Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 21th April 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: A.Tsaregorodtsev
CERN: Z.Mathe,R.Santinelli,F.Stagni,J.Closier,A.C.Smith,Ph.Charpentier
Phone: N.Brook, M.Sapunov, A.Tsaregorodtsev
0. Ongoing Action List
(94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(111) - 14/09/08 - Alexei,Andrei - Productions with file merging and associated data distribution implementation
(116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side and protect data operations
(126) - 20/01/09 - Andrei - Document the design of the workflow / production management components
(132) - 03/03/09 - Stuart - Update LFN construction utility to store production output directories as a parameter
(133) - 03/03/09 - Matvey - Provide request DB interface page
(136) - 03/03/09 - Vladimir,Roberto,Joel - Integration of existing SAM infrastructure tests into DIRAC SAM framework.
Check the Nagios based SAM framework deployment time scale.
(139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release
(140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging...
(142) - 14/04/09 - Adri - verify that the data transfer monitoring info sent to SiteMonitoring is functioning
(143) - 14/04/09 - Andrei - update the links in the job monitoring info sent to SiteMonitoring
(144) - 21/04/09 - Zoltan - propose and implement solutions to reduce the Bookkeeping service load for production input data queries
(145) - 21/04/09 - Vladimir,Roberto,Joel - check why SAM application test jobs are running by sgm user on some sites
(146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification
(147) - 21/04/09 - Joel - renew the request to IT for deployment of the new LHCb/DIRAC central host machine
(148) - 21/04/09 - Joel - setup DIRAC SVN/Trac project portal and configure access to the DIRAC developer team.
Actions closed:
(109) - 14/09/08 - Matvey - Expose DataLogging interface on the web page (pending lhcbweb deployment)
(134) - 03/03/09 - Matvey,Andrew - Provide RunDB monitoring page
(141) - 07/04/09 - Vladimir,Roberto - Investigate reason for downtime calendar not being updated
1. Action Updates
- Action(94) -
Ongoing.
- Action(95) -
New test is developed to check site VOMS configuration, certificates. Parsing of the voms-proxy-info command is
needed to reveal problems, can not rely on the exit code of the command.
SRM spcae tokens unit test is updated to provide more verbosity in case of problems to help the work of the
site managers to fix them.
action point dropped and moved to regular ongoing activity report
- Action(96) -
Pending Pablo Saiz to publish the LHCb information in the dedicated Web Page.
action point dropped and moved to regular ongoing activity report
- Action(109) - done and closed
- Action(111) -
Ongoing. The file merging production is defined and tested. Still the input data definition for the merging production
is to be improved as it can put a high load on the Bookkeeing and ProductionDB services. BookkeepingWatchAgent was updated
to update ProductionDB only in case of the number of returned files in the BK query is changed. Zoltan is asked to investigate
which means can help lowering the load on the Bookkeeping service. Suggested possible solutions include interface method
to return just a number of files for a given BK query and returning files only registered in the Bookkeeping DB after
a given date (ACTION).
- Action(116) - CPU studies
Ongoing. Ricardo reported (by e-mail) that the CPUTimeLeft utility is updated to return the results in the same units
as used in the pilot JDL for the MaxCPU time specification ( 500 SI2K CPU equivalent ). Since the way how sites are
publishing the length of the queues is rather confusing, Ricardo is asked to explain how the queue length is evaluated
for the matching purposes ( ACTION ). It was also suggested that all the CPU time related measurements are now converted
to the new common HEP units. The action point is updated.
- Action(122) -
Ongoing.
- Action(124) - Apply dataset limit check
Ongoing with low priority.
- Action(126) -
Ongoing. Will be updated once the MC'09 procedures are all sorted out
- Action(132) -
Ongoing with lower priority
- Action(133) - RequestDB monitoring
Single request monitoring is available. Summary page is ongoing
- Action(134) - done and closed
- Action(136)
Roberto is asked to verify the time scale for the new Nagios based SAM infrastructure to be deployed.
Depending on this time scale and the estimated time to adapt the LHCb SAM tests to the new infrastructure,
the decision on continuing the development on putting the SAM infrastructure tests into the DIRAC SAM
framework will be taken.
- Action(139) -
Ongoing, pending more sites deploying glexec.
- Action(140) - Banning faulty storage elements
Ongoing.
- Action(141) - done and closed
- Action(142) - data transfer monitoring info sent to SiteMonitoring
Ongoing.
- Action(143) - links in the job monitoring for SiteMonitoring
Ongoing.
2. Production Status Report
- Production status
Last week production activities were concentrating around the FEST production. These tests revealed
problems with the data access at IN2P3 and NIKHEF. The updated version of Brunel necessitated also
update of the output log file analyses. There are still fixes needed in the way how the raw data files
parameters are reported to the Bookeeping DB by the RunDB tools.
MC'09 preparation
Small productions with final application versions are defined and are running OK with short jobs.
Later today Gauss options for the final production will be provided by the physics groups. Then
larger jobs will be created in order to evaluate the CPU requirements and output file sizes.
Tools for the data distribution of RDST data to CERN+localT1+T1 are ready but should be configured
for this case.
Other production matters discussed:
- There are still problems with the user identity mapping on some sites resulting in the SAM application
test jobs running as sgm user. This can be a faulty site configuration problem. Vladimir and Roberto
will check that, possibly providing a correponding SAM sensor (ACTION).
- Accessing SQLite database in the shared area can cause locking problems especially at the sites
with NFS mounted shared areas. An ultimate solution would be to always copy the SQLite files local
to the running jobs. Marco will be contacted to see if this can be done on the application level.
3. DIRAC Tasks
- Framework status
There was a quick discussion of the possibility to have a monitoring of the software installed at
the sites. It was agreed that such tool can be very useful providing for each site info about the
installed applications, versions and platforms. To realize that there should be a tool to inspect
the shared area to get this information. It was pointed out that SetupProject can be used but it
does not provide platform information. Further investigation of the subject is necessary.
Related to the software installation, it was suggested that the logic of the SAM SW installation
jobs should be based on comparison of the software available at a site and the list of the
mandatory software in the CS. Missing application-version-platforms combinations should be
installed. The same jobs in the end can report the site software installation status to the
software monitoring service (Action)
- Bookkeeping status
Few bug fixes to go into the new release - v4r12.
- WMS status
There is a problem of slow job entry-point verification when there are many jobs with large input
data samples. The problem is due to slowness in the LFC interrogation to get file metadata. Andrew,
Adri and Ricardo are looking at how this can lightened by, in particular, deferring the file metadata
extraction from LFC at the Worker Nodes rather than in the central services to maximize parallelization.
The amount of metedata to be extracted should be also reevaluated.
- DMS
Updated RunDB service interface to suit the needs of the Web Monitoring.
- Production Management status
Ongoing tests of the new Stuart's Production API for the needs of the upcoming MC'09 production
- VO-boxes
No update.
- Web Portal
RunDB and Data Logging monitoring pages are now available. Integrity DB monitoring is coming soon.
- Services deployment ( hardware, configuration )
Joel will renew the request to CERN IT for the new production service host machine.
- Documentation
Joel will set up and test the SVN/Trac software project repository. This will be used to evaluate
the capacity of the Trac portal which can eventually replace the Savannah/TWiKi for DIRAC depending
on the evaluation results.
4. AOB
There are minutes attached to this event.
Show them.