Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 23rd February 2010, CERN
Chair: A.Tsaregorodtsev
Minutes: S.Paterson
CERN: P.Charpentier, J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovski, R.Santinelli, A.C.Smith, F.Stagni
Phone: R.Graciani, E.Lanciotti, R.Nandakumar, A.Tsaregorodtsev
Apologies: M.Sapunov
0. Ongoing Action List
#11163 - Site summary pages - Federico commented that work has started after the circulation of the site summary proposal. Hopefully something should be available next week.
#12165 - Hardware metrics - the sensors from last October will be resurrected in the Quattor template, volhcb16 will be used for testing.
#12382 - Site status view in the Web Portal - merged with #11163
#13029 - Develop periodic testing infrastructure in SAMSystem - ongoing.
#13347 - Follow updates to CORAL LFC interface - the fix is being tested by Marco prior to release.
#13348 - Update downtime calendar to use GOCDB interface - ongoing.
#13403 - DIRAC Web Portal at PIC - Matvey had a suggestion to use a Nginx server to dispatch the queries for better load balancing between PIC and CERN.
#13404 - FTS+WMS client middleware in LCG-AA - closed.
#14096 - Date in RawIntegrityMonitor - Fixed, action closed.
#13405 - Keeping Real Data on disk at CERN - It was agreed to pursue adding a new workflow module to re-register data at CERN after processing.
#13406 - Testing data access with xrootd protocol - Andrew is starting to think about the testing infrastructure for data access protocols.
1. Production Status Report
- Current productions
MC has ramped up with requests for ~18M events (~40k jobs) all running
smoothly. The physics stripping of the last L0HLT1 pass is proceeding
for the minimum bias but even for 1 file per job there are many stalled
jobs (58 stalled / 102 done and 342 still running). The b and c inclusive samples may have to be rerun to ensure small enough file sizes.
2. DIRAC Release Update
- DIRAC v5r0 and patch releases
Since the v5r0 release there have been a couple of patch releases for
minor problems in several subsystems. Zoltan confirmed that the recent
fixes to the BK were working in production. Andrei suggested that
everyone should look at the tagging procedure for the next releases, in
addition to a meaningful tag name the versions.cfg file should be updated
and release notes be added. The next release (v5r1) will include a new
SystemAdministrator interface and service to better manage the DIRAC
components.
- New production system migration post-mortem
Andrei reported that the migration was fairly smooth apart from a few
minor fixes as needed during the morning. By noon Wednesday all the
services were up and the configuration master was switched over. By 5pm
on Wednesday the DIRAC v5r0 migration was ready. After the Ganga patch
release there were no problems for users. Since the MC ramped up
recently to 10k jobs the load on the central machines was observed to be
very reasonable. The memory consumption of the Service and Agent hosts
was ~ 4.8 GB which is well below the limit of 16GB. To conclude the new
DIRAC setup has a much larger capacity for load.
- Next steps
Ricardo summarised the progress on database mirroring to ensure a
periodic backup is available in case of failures. The initial mirror can
be installed on volhcb25. In the future the procedure for backups and
the possibility for hot-swap machines will be investigated. Joel asked
whether some of the old machines can be retired, particularly volhcb06,
the LogSE. It was agreed to copy over everything from the old LogSE and
then the cleaning can start on the new LogSE. The final list of hosts to
be retired are: volhb04, 06, 08, 09 and 10. volhcb12 is used for
development and volhcb11 is an old web portal that will move to
volhcb17. volhcb13 can be migrated to SLC5 and be used for hosting the
CERN web portal.
3. DIRAC Tasks
- Infrastructure monitoring (SAM,SLS,Dashboard)
Roberto updated the root macro test to better track down
incompatibilities. The periodic tests have been consolidated to work with
the latest DIRAC version and all seem reliable.
- ResourceStatus System status
Discussed above.
- Framework status
All the agents have been migrated to the new framework. Andrei mentioned
that many commits were made recently using the eclipse debugging tool,
these should be carefully checked. The SystemAdministrator service
allows to restart services and install software.
- Bookkeeping status
No updates from Zoltan.
- WMS status
Ricardo commented that a new reworking of the WMS is being planned. The
CNAF CE direct submission will be pursued as discussed in the production
operations meeting, Roberto will also look into an update of the site
occupancy factor of the ranking expression.
- DMS
A plugin to allow merging by file types will make the streaming for the
stripping much easier.
- Production Management status
There was a small update to the production status machine in order to
treat associate transformations in addition to MC and merging.
- VO-boxes
An extra service to allow to run the VO-boxes themselves will be deployed
and tested.
- Documentation
The API documentation (Epydoc) was updated and is available as PDF / HTML
from the DIRAC menu.
4. AOB
Roberto mentioned that the discussion following the recent Castor
problems has highlighted a need for us to get in touch with sites outside
of working hours (without using ALARM tickets). The official routes to
report problems i.e. GGUS team tickets, don't trigger the same effect as
a mail to Castor support.
Joel just circulated an announcement of an intervention on Castor LHCb at
CERN. Raja mentioned a similar event at RAL.
There are minutes attached to this event.
Show them.