Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 7th July 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: A.Tsaregorodtsev
CERN: B.Panzer-Steindel,Z.Mathe,F.Stagni,J.Closier,A.C.Smith,R.Mamunur,Ph.Charpentier,N.Brook,R.Santinelli
Phone: A.Zhelezov, A.Tsaregorodtsev,R.Graciani,M.Sapunov,R.Nandakumar
0. Ongoing Action List
(94) - 23/09/08 - Andrew,Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(111) - 14/09/08 - Alexei,Andrei - Production request page, file merging and associated data distribution implementation
(116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(133) - 03/03/09 - Matvey - Provide request DB interface page
(139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release
(140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging...
(146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification.
Should be complemented with the installed SW monitoring tools.
(151) - 26/05/09 - Andrew - Update central components with optimized BK methods using file time stamps
(156) - 02/06/09 - Ricardo,Joel - Refine VO-box setup to minimize impact of updates and maintenance procedures
(160) - 07/07/09 - Ricardo - update the DIRAC hardware configuration paper with the current DIRAC server load plots
(161) - 07/07/09 - Alexei - define Production Request for ongoing production to allow its monitoring
(162) - 07/07/09 - Ricardo,Andrei - document the TQ priorities algorithm
(163) - 07/07/09 - Joel - investigate possible use of CERN backup tools for log files archiving.
Actions closed:
(152) - 26/05/09 - Stuart - Investigate the crashes when using the copy local option for SQL DDDB files
Reported as fixed
(156) - 02/06/09 - Ricardo,Joel - Refine VO-box setup to minimize impact of updates and maintenance procedures
Closed. Solved by the updated DNS alias changing procedure and by the new hardware proposal
(157) - 16/06/09 - Andrei,Ricardo - Prepare plan with timestamps for obtaining new hardware for production system
Document prepared, action closed. Hardware procurement and depoyment becomes a permanent task.
{159) - 16/06/09 - Philippe - Investigate the issue of persisting production log files with the PPG
Closed
1. DIRAC hardware configuration
Ricardo has presented the LHCb proosal for the DIRAC System hardware configuration ( the document is attached to the
agenda ). In the subsequent discussion the following points were raised.
Bernd: The document will be examined by the FIO expertes (Olof Barring and Tim Bell) to see which options are
feasible and which are not. Other suggestion may come up as well. In general there is no problem with the requested
hardware with the following remarks:
- Having two network cards for redundancy looks to be an overkill. These components are (almost) never failing.
However, configuring network settings with multiple network interfaces is complicated. This is also not a standard
configuration. However, the network redundancy can be achieved by splitting the DIRAC servers into two sets each
sitting in a separate local network. In case one network is failed, the other one continues although with less
capacity;
- Having NAS storage rather than standard file servers can be considered but can easily take 6 months to purchase
and deploy, this is also not a standard solution for IT;
- Basic servers are now 8 cores, so only those should be considered;
- RAID6 lareg disk arrays are providing redundancy but in case of a disk failure can take a very long time to rebuild.
The performance during this time is degraded. However, in our case the performance for the LogSE and Sandbox is not
an issue, we need rather the increased capacity.
- The services are consuming a relatively small amount of CPU. In this case the use of virtual machines can be considered.
However, we should be careful since the memory consumption is quite important. The opportunities opened by the use
of virtual machines should be investigated.
- If no non-standard hardware is requested (double NIS cards, NAS storage), the requested hardware can be acquired in
1 month + 1 month for proper configuration and testing.
The paper will now be sent to Olof Barring and Tim Bell complemented with few plots illustrating the current loads
on the DIRAC production machines to justify better our projected load estimates (Action: Ricardo)
1. Action Updates
- Action(94) - Logging system usage examples
Ongoing.
- Action(111) - Production definition tools
Alexei reported that the ProductionRequest page is now able to produce production generation scripts. This
was tested with the MC production scripts to obtain the same workflow and parameters. The ProductionRequest
tool documentation is being prepared and will be released tomorrow.
It was requested to provide a link between a production request and a real production to test the production
request monitoring. Alexei will create a production request corresponding to the current MC'09 production
and will create a link to between the request and the production.
Andrew would like to try out the Production Request tools to generate the new FEST productions. However,
this requires the possibility to define input data BK query in the Production Request. This extra input
form will be provided shortly (Action: Alexei);
The ProductionRequest page will be deployed now in Production (Action: Andrei, Alexei) to start using it
by the relevant people ( PPG coordinators, BK experts, production managers ).
- Action(116) - CPU normalization studies, Filling mode
It was decided to move to the use of HEP CPU units quickly (Action: Ricardo). The site CPU capacity will
be either taken from the sites that already report it in HEP units or by applying a factor to the old
fashioned 500 SI2K units. Once this is done, a memo with instructions to users on how the CPU time limits
are treated will be compiled together with suggestions on how best decompose the user payload. In particular,
The use os a large number of small jobs is not favored in the grid unlike the LSF batch system.
- Action(122) - Analysis of the SAM pilot submission errors.
Ongoing.
- Action(133) - RequestDB monitoring
Ongoing.
- Action(139) - glExec deployment and tests on sites
Roberto reported that basic glexec tests were performed at GridKa and Lancaster sites including the tests
of the wrapper scripts that allow to pass the pilot environment to the payload environment. The experience
showed that the glexec configuration on sites stays rather complicated and does not work out of the box.
However, the sites passing basic tests should now undergo more profound tests with real payload. Ricardo
accepted to update the glexecComputingElement to take into account the glexec wrapper scripts and proceed
with real job tests
It was pointed out that all the sites accepting Pilot role jobs are obliged (strongly advised) to deploy
glExec in the ID changing mode. However, for many sites glexec in logging mode would be enough, but this
glexec mode was never tested. Roberto will raise the point of testing glexec in logging mode in the coming GDB
meeting and in the glexec working group (Action: Roberto).
- Action(140) - Banning faulty storage elements
Andrew reported that technically the SEs can be banned now. This should be properly handled by various components,
like for example, Input Data Optimizers. There was a discussion on how to treat jobs temporarily not having any
eligible site to run. It was suggested to introduce a new job state "OnHold", which will be periodically checked
by a dedicated Optimizer in view of releasing hold jobs. Other technical possibilities to realize this functionality
were decided to discuss off-line.
- Action(146) - SW installation framework using CS data
Ongoing.
- Action(147) - new DIRAC service host
The host OS should be updated to SLC4. The machine is expected this week.
- Action(148) - SVN/Trac server
Ongoing. SW shared area celaning procedure is done.
- Action(151) - Update central components with optimized BK methods
In production for the WMS part. Still to be deployed for the DMS part - action on Andrew.
- Action(152) - Investigate the crashes when using the copy local option for SQL DDDB files
Closed, reported as fixed "long ago"
- Action(156) - Refine VO-box setup to minimize impact of updates and maintenance procedures
Closed
- Action(157) - Plan for new hardware for production system
The document is ready and discussed in the PASTE meeting. More plots are to be added about the
current system loads on teh DIRAC servers. The action is closed.
- Action(159) - Investigate the issue of persisting production log files with the PPG
Philippe reported that there are the following purposes of the log files:
- Check for the reasons of failing jobs. These logs are needed for several days (1 week);
- Evaluation of statistics on cross-sections and other MC parameters which are determined
by passing the application log files. These logs are necessary only for several (dozens)
of jobs per production;
- Histograms for DQ group to evaluate the data quality. The histograms will be all stored
in a dedicated disk storage with File Catalog registration by standard tools for the
reconstruction type productions
The action is closed. See below the discussion on the LogSE operation issues.
2. Production issues
MC'09 production was restarted yesterday after a discussion on how to handle log files without
overloading the LogSE. This discussion is summarized in the eLog entry:
http://lblogbook.cern.ch/Operations/1785
Joel has started to archive the subdirectories for production 4837. Vladimir is preparing a script
to compress job directories which are at least 1 week old. This should allow to use the current 1TB
LogSE with the current production pace keeping logs up to one month easily available through the
Web portal.
In the discussion there were suggestions to use "standard" CERN backup tools to archive the log
directories using journaling. Joel will investigate what is available (Action). Ricardo, suggested
to investigate the possibility to configure the Apache server to serve files from tar archives.
This deserves further investigation. Another Ricardo's proposal of storing log files in directories
by time stamp needs special coding in the LogSE service and was put on hold for the moment.
The current system load is slightly below 2K jobs per hour. the load on volhcb04 and volhcb09
is reasonable. volhcb13 is starting to develop ~10% I/O wait which is a sign of it being close
to a limit. volhcb10 runs at ~70% of CPU load. The Matching rate is exceeding the production job
submission rate making the system starve. It was decided to start a second ProductionJob agent
to speed up submissions. But the real solution to this problem is bulk job submission with
sharing sandboxes. This will be addressed in the near future development.
Ricardo mentioned that few days ago volhcb04 was rebooted as it has generated kernel panic.
This can be a sign of some hardware failure. It was decided to continue using volhcb04 as it is.
However, volhcb05 is freed from the running services and can serve as its eventual replacement.
The v4r16 release was compiled and is now under the test by GANGA experts and in the development.
It fixes the problem of picking up several times the sam production job and is deployed already
on volhcb09 to avoid it. The new TQ priority corrections which were supposed to go into the release
are not yet ready and need more testing (see below).
3. DIRAC Tasks
WMS tasks
There was a discussion on the procedure of handling Job priorities. Some details on the procedure
were explained. The part which is applying static priorities ( job shares ) is now implemented and
is working. Its description can be found here: https://twiki.cern.ch/twiki/bin/view/LHCb/DIRACWMSPrioritySchema
However, because of a different nature of user ( short ) and production (long) jobs
sharing by a number of jobs is not satisfactory. What is expected is sharing the amount of CPU
by different groups according to predefined settings. This is addressed in the updated procedure
where the TQ priorities are dynamically updated taking into account the history of the job execution.
This procedure is coded and is under the test now. The exact algorithm will be documented in the
TWiKi shortly (Action: Ricardo, Andrei)
4. AOB
There are minutes attached to this event.
Show them.