Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 16th June 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: P.Charpentier, J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovsky, A.C.Smith
Phone: R.Graciani, R.Santinelli, A.Tsaregorodtsev
Apologies: M.Sapunov
0. Ongoing Action List
(94) - 23/09/08 - Andrew,Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(111) - 14/09/08 - Alexei,Andrei - Production request page, file merging and associated data distribution implementation
(116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(133) - 03/03/09 - Matvey - Provide request DB interface page
(139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release
(140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging...
(146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification. Should be complemented with the installed SW monitoring tools.
(151) - 26/05/09 - Andrew,Andrei - Update central components with optimized BK methods
(152) - 26/05/09 - Stuart - Investigate the crashes when using the copy local option for SQL DDDB files
(153) - 26/05/09 - Ricardo,Joel - Investigate optimal hardware configurations for DIRAC servers
(156) - 02/06/09 - Ricardo,Joel - Refine VO-box setup to minimize impact of updates and maintenance procedures
(157) - 16/06/09 - Andrei,Ricardo - Prepare plan with timestamps for obtaining new hardware for production system
(158) - 16/06/09 - Andrei - Correct the stalled job reporting on the production monitoring page
(159) - 16/06/09 - Philippe - Investigate the issue of persisting production log files with the PPG
Completed Actions:
Done - (143) - 14/04/09 - Andrei - Update the links in the job monitoring info sent to SiteMonitoring
Done - (148) - 21/04/09 - Joel - Setup DIRAC SVN/Trac project portal and configure access to the DIRAC developer team.
Done - (154) - 02/06/09 - Stuart - Define production parameter set to be persisted outside of workflow object
Done - (155) - 02/06/09 - Roberto,Stuart - Open bug against gLExec for environment issue
1. Action Updates
- Action(94) - SystemLoggingDB example
Ongoing.
- Action(111) - File merging productions
Alexey reported progress on the production request page via mail, Andrei agreed to test this tool.
- Action(116) - CPU studies
Ricardo attached some images to the agenda page. The HepSpec results show that for our internal normalization we
must continue to rely on the published site normalization values since not enough data is available for the range of
CPUs on the Grid. To define the scaling factor Ricardo proposed 444.namd as the correlation between our applications
is strong. The filling mode was enabled last week in certification with up to two jobs per pilot allowed. There was also
an update on the sandbox service testing, the procedure discussed last week was reiterated and agreed.
- Action(122) - Develop specialized SAM Agent to track and report errors with regard to pilots
Ongoing.
- Action(133) - RequestDB monitoring
Not yet ready after comments from Andrew.
- Action(139) - gLexec testing after deployment at new sites
Procedure for preserving the environment is established but is being verified by OSG. Andrei suggested attending the
gLExec working group meeting to convey our tests are on hold until a final decision is taken regarding the
configuration such that the same procedure can be used at all sites.
- Action(140) - Banning faulty storage elements
Ongoing.
- Action(143) - Update links in the job monitoring info
Fixed.
- Action(145) - Sensor for SAM production vs. sgm mappings at sites
Procedure is understood, action can be closed.
- Action(146) - SAM software distribution synchronization with the CS
Joel started to look at this, the cleaning of the shared area orphan packages was implemented.
- Action(148) - Setup DIRAC SVN/Trac project portal and configure access to the DIRAC developer team
The developers are starting to play with SVN, action closed.
- Action(151) - Update of central BK components with optimized methods
Ongoing.
- Action(152) - Investigate crashes for local DB copying
Retaining the action to test with latest LHCb version.
- Action(153) - Investigate optimal hardware configurations for DIRAC servers
After fruitful discussions last week the decision was taken to retain MySQL and request two powerful file servers. Joel
mentioned that the warranty on some existing machines is due to expire. After a long discussion on how to proceed
with a testing and production setup it was agreed to prepare a plan with timestamps. ( ACTION: Andrei,Ricardo )
- Action(154) - Define production parameter set to be persisted outside of workflow object
Work has started on this in parallel with the recent production activities, tests are successful, action closed.
- Action(155) - Open bug against gLExec for environment issue
Done.
- Action(156) - Refine VO-box setup to minimize impact of updates and maintenance procedures
Ricardo will confirm the procedure to follow in the event of updates / maintenance of the VO-boxes after discussions
with Raja and Joel.
2. Production Status Report
- Report from the Production meeting
Happily running > 9k concurrent jobs at the moment. Stuart reported that the 10^9 min bias production is ongoing
with a low priority, this is almost 20% complete today. The 19 signal channel samples were created last night with
higher priority and all are currently running. Merging will hopefully start by the end of today. Stalled job reporting
problems mentioned last week persist, the numbers on the production monitoring reflect stalled jobs as a counter not
a real set of jobs in this status. ( ACTION: Andrei ) After the release of AppConfig v2r7p2, WorkflowLib v9r22 and a
new LHCbSystem utility FEST was restarted and jobs are running fairly smoothly, no problems yet discovered relating
to Oracle CondDB access (~500 running jobs at the Tier-1s currently). Sporadic failures observed at GridKa and RAL.
-LogSE
Since Grid storage elements are not capable of storing logs at the level we require Stuart asked whether the log files
should really be persisted along with the data. Philippe agreed to pursue this with the PPG. ( ACTION: Philippe ).
- volhcb01
Andrei suggested migrating services from volhcb01 -> volhcb04 (development BK). This was agreed.
- Reboot of the production hosts
It was agreed to stagger the reboot most of the volhcb machines this afternoon and volhcb09 tomorrow morning.
- volhcb13 migration
Migration of the current WMS was agreed for next Monday. A test will first be performed using the development
system.
- Weekly reports
To be discussed on Thursday at the operations meeting. Stuart suggested that the GEOC prepares this, Vladimir
agreed to try this for this week.
3. DIRAC Tasks
All components discussed above.
- Web Portal
Andrei commented that there is a 'Help' button available at all web pages. All developers are encouraged to
contribute to the twiki pages linked from there.
4. AOB
None
There are minutes attached to this event.
Show them.