Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 9th February 2010, CERN
Chair: A.Tsaregorodtsev
Minutes: S.Paterson
CERN: J.Closier, Z.Mathe, R.Nandakumar, S.Paterson, M.Rashid, V.Romanovski, R.Santinelli, A.C.Smith, F.Stagni, A.Tsaregorodtsev
Phone: E.Lanciotti, R.Nandakumar, M.Sapunov
Apologies: P.Charpentier, R.Graciani, A.Casajus
0. Ongoing Action List
#10916 - Release procedure for web pages - pending an update from Adri
#10919 - Setting up the new Production Setup on volhcb17-26 - configuration of the servers is ongoing, the disk partitioning is including /tmp which should be changed
#12526 - New LogSE server - Joel mentioned that the LogSE machine is available but is not partitioned correctly, this is being investigate
#13348 - Update downtime calendar to use GOCDB interface - Vladimir will investigate the current usage of the calendar and reassess the situation
#13402 - Data access tests at DPM sites - Having demonstrated DPM access works at several sites we close this action. When sites agree to the conditions of the document they can be included.
#13404 - FTS+WMS client middleware in LCG-AA - Ongoing
#13405 - Keeping Real Data on disk at CERN - After some discussion no immediate conclusions were drawn, technically the host of the file must change but it is unclear when this can occur. Ongoing.
#13406 - Testing xrootd at IN2P3 - It was agreed to formulate a testing module to make this kind of operation easier, this will be pursued. Ongoing.
#13407 - LHCb platform detection and capability, tests are ongoing after a problem was found with install_project - Vladimir confirmed that bug fixes are ongoing by Hubert
#13596 - Detection of sgm pool and static accounts - Joel commented this is in the testing phase and has been used to successfully identify problematic sites
1. Production Status Report
- Current productions
The recent ramp up of MC highlighted problems with the CERN SRM instance. Nothing had changed on our side but it
was still unclear what happened at 3am to bring the system to its knees. We will carefully watch things the next time
we start to ramp up and saturate the system. A few small reconstruction productions (5 data taking conditions but
less than 20 jobs) have been running smoothly. Vladimir mentioned the recent point that jobs at NIKHEF entered a
short queue and the benchmarking had returned an incorrect value, some production jobs subsequently stalled and
this will be investigated. A 0.5% failure rate for MC productions is observed with the watchdog stalling projects at
various stages, an action will be added to production operations.
After some discussion about CPU efficiencies reported by Elisa it was agreed to expose more CPU / WallClock
information about jobs as they proceed. During the reported period it was predominantly user jobs running at PIC
and therefore the lower efficiency is as expected. The inclusion of the batch monitoring utility by U.Schwickerath will
also be investigated at PIC and RAL in order to provide more fine grained information.
2. DIRAC Release Update
- DIRAC Certification pre-release
The certification pre-release is ready and testing will continue. The problem observed with Mike uploading job outputs
was resolved. We are almost ready to freeze the release candidate in certification.
- Preparing the new production system
The release candidate will be installed as a new setup initially on the new hosts. The configuration parameters for the
new setup will be tuned such that a simple renaming of "NewProduction" to "Production" can be performed during the
major intervention. The production database schema during the intervention will be updated and the LogSE contents
can be archived. In the days leading up to the intervention no new productions will be created. By avoiding having
active productions in the system we greatly simplify the migration procedure. Andrei will formulate the migration
procedure and circulate it to the developers list.
3. DIRAC Tasks
- SVN migration
o release tagging, building and installation scripts
The automatic update of CMT requirements will be added to the DIRAC tagging script.
- Infrastructure monitoring (SAM,SLS,Dashboard)
All non-DIRAC SAM tests were recently consolidated and can now be trusted to spot real problems. Several Savannah
bugs were opened against the LHCb Dashboard related to performance issues. From the questionnaire sent to the site
admins there is a twiki page describing the feedback and a document is being prepared. A prototype for the
visualisation of the available information would be very nice to have in time for the NIKHEF 'jamboree'.
- Framework status
Andrei mentioned that the system administrator service is being prepared to allow remote service control.
- Bookkeeping status
Zoltan is in the process of adding BK bookmarks, online event statistics and so on.
- Web Portal
Light httpd may be used instead of Apache, Joel will investigate with IT whether this is an acceptable solution. Andrei
commented that we can run the web portal fully in user space if this is OK. After the DIRAC SVN release the reshuffling of the top level menus will be discussed.
4. AOB
Joel sent a mail regarding the end of life for SLC4, Andrei confirmed this is no problem for DIRAC.
There are minutes attached to this event.
Show them.