Production and PASTE Meeting, 26th January 2010, CERN
Chair: A.Tsaregorodtsev
Minutes: S.Paterson
CERN: P.Charpentier, J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovski, R.Santinelli, A.C.Smith, F.Stagni
Phone: V.Fernandez, E.Lanciotti, R.Nandakumar, M.Sapunov, A.Tsaregorodtsev
Apologies: R.Graciani, A.Casajus
0. Ongoing Action List
#10916 - Release procedure for web pages - pending an update from Adri
#11223 - CREAM CE direct submission was pending a GridFTP server that allows to save pilot standard output
#12526 - Joel mentioned that the LogSE machine is available but is not partitioned correctly, this is being investigated
#13402 - Data access tests at DPM sites - Philippe commented that making use of Tier-2 sites that provide data storage requires an addendum to the computing model and agreement on the level of service a site should provide in this case. A proposal has been circulated for the approval of the OPG.
#13403 - Elisa mentioned that the lhcbweb -> CERN connection could be enhanced by moving through the OPN but not by much (34->31ms). It was agreed to retain both instances of the web portal and investigate a load balanced alias.
#13406 - Testing xrootd at IN2P3, it was agreed to formulate a testing module to make this kind of operation easier, this will be pursued.
#13407 - LHCb platform detection and capability, tests are ongoing after a problem was found with install_project
1. Production Status Report
- Current productions
The minimum bias L0HLT1 Moore stripping and c inclusive stripping's are complete, b inclusive is pending 5 jobs (250
/ 3k files). There were two passes of the V0.DST stripping productions that went smoothly from our side. A few small
simulation requests for 2009 settings are pending, these initially failed due to a backwards incompatible change in the
RICH detector description requiring a new Boole release. Andrew kindly agreed to take over in the absence of Stuart
during the next week.
2. DIRAC Release Update
- DIRAC Certification pre-release
Andrei mentioned that the certification setup is complete, production tests have been running but we must start to
define procedures for performing certification.
- Certification tests
o Bookkeeping usage for tests
With one instance of the BK shared by development and certification it was agreed to use the namespaces
/lhcb/certification and /lhcb/development to separate the activities. After a certification release is made we can
trigger a cleanup of this namespace and for the next release candidate we reinstall the databases.
o Failover usage for tests
For testing request propagation it was agreed to install a request forwarding agent on volhcb14.
o Integrated tests
The whole chain of production is being tested in certification. Others are mimicking typical user activity.
o Subsystem tests
Certification tests by subsystem should be put in place in a common framework such that testing can be decoupled
from the system responsibles.
o Web portal tests
A mechanism for testing the web portal such that it can be installed easily in certification should be defined.
- Client tests (Ganga)
For client tests using Ganga the client installation will be performed in $LHCBDEV on lxplus and configured to point to
LHCb-Certification. For major releases the installation will be on hold pending the Ganga testing. For minor or patch
releases the Ganga testing is only mandatory if the client is directly affected.
- Preparing the new production system
After some discussion it was agreed that the new major version of DIRAC should tentatively be made in mid-February
(taking 1 day). This has to be coordinated very closely with the Ganga team. The current pre-release should be
stable for the API so can be tested after installation in $LHCBDEV, hopefully today. In order not to complicate the
new release with also coordinating a Ganga release it was proposed to add the old import structure temporarily to the
SVN repository.
Hot swapping of the production machine database is being tested in Barcelona but may not be ready for the first v5
release. DNS load balancing is also being investigated.
3. DIRAC Tasks
- SVN migration
o release tagging, building and installation scripts
Joel asked whether the versions.cfg file will keep all previous tags. After a certification release moves into productions
the intermediate tags will be purged.
- Infrastructure monitoring (SAM,SLS,Dashboard)
Roberto will try to use the updated unit test code from Andrew.
- ResourceStatus System status
Federico now has a common interface for his clients. There were several bug fixes for RSS components. The new
notification system alarms were attempted to be used but do not appear to be deployed in development. The
Notification DB should be updated simply by dropping the existing development DB. For keeping track of the current
shifters and GEOCs the developer of the ShiftDB was contacted. This requires an Oracle connection to the database
so Zoltan's wrapper classes can be utilised. Andrei mentioned that some of the RSS modules were in conflict in
development, Federico will check.
- Bookkeeping status
No updates, there were a few bug fixes recently.
- Production Management status
Any LHCb specific code was removed from the Transformation system. There are now two places to retain production
code including the Transformation system and LHCb Production Management system.
4. AOB
Next week is a software week, there is a slot on Tuesday morning to discuss experiences with real data and the current status of the sub-projects. Andrei agreed to present the DIRAC v5 status. Philippe will coordinate the session
and contact the prospective participants.
Next week there will be no PASTE meeting due to the Software Week.
There are minutes attached to this event.
Show them.