Production and PASTE Meeting, 1st October 2007, CERN Chair: A.Tsaregorodtsev Minutes: S. Paterson CERN: J.P.Baud, G.Castellani, P.Charpentier, J.Closier, G.Kuznetszov, S.Paterson, R.Santinelli, M.Sapunov, A.Tsaregorodtsev Phone: R.Bernet, N.Brook Apologies: R.Nandakumar, M.A.S.Miguelez 1. Production Status Report Minutes from the production meeting are available here. Philippe suggested a check for site targeted productions to ensure consistency between LFC and SEs. Stuart suggested checking the metadata of files in the Processing DB before job creation. After some discussion it was agreed that three types of integrity checking should occur: 1. Background checking at low level; 2. Checking before usage in the ProcDB; 3. For DIRAC3, checking after usage problems are reported e.g. stager, jobwrapper errors. Jean-Philippe mentioned that an extra metadata field exists for marking the 'checked' (lfcsetrstatus) status and the background activity can report there. This should be implemented asynchronously in the ProcDB and the stager agent can be reused for this ( ACTION Gennady, Stuart ). 2. Dressed Rehearsal Discussion Andrei summarised the discussions so far: - 20K files to be migrated from pit to Castor, files are moved to disk cache initially and then Agent checks whether files are migrated to tape or to another Tier-1 site - If backup replica exists with correct checksum, the Online machinery is notified - Distribution to the Tier-1's should be performed according to the site capabilities (see Nick's table attached to agenda page) Nick raised the issue of rfcp data transfers from the pit to Castor, how is the checksum calculated? Roberto mentioned rfcp from ALICE uses checksum for transfers to disk cache. This could also be adopted by LHCb. Nick also queried when a file can be marked as 'good'? Andrei clarified that an agent can check the status of files and reports back that the migration was successful. - After replica registration in the LFC, reconstruction jobs can be submitted Andrei mentioned that the main issue is that CERN + Tier-1's must receive a specific proportion of reconstruction workload. How can we ensure that the distribution of the number of jobs reflects the fraction of data at the Tier-1's (CERN has 100% of the data)? Also how can we dynamically assign jobs to sites maintaining the requested proportions? Everything is in place except the ProcDB fractions of job submission and this presents a fairshare problem. For the dressed-rehearsal it was decided to proceed with the 'by hand' approach and consider more complicated solutions for a later date. - Once rDSTs are produced, the stripping jobs can be launched. - The outcome of stripping jobs should then be replicated to all Tier-1 sites via asynchronous operation with FTS - The stripping phase is over when replicas exist at all the Tier-1's - Finally, analysis jobs can be sent to all Tier-1 sites. Nick suggested 25 concurrent jobs at CERN and 75 at the remaining Tier-1 sites The question of whether these tests should be performed with gLexec and generic pilot agents was raised? It was agreed that the Computing Model should be based on generic pilots for the dressed-rehearsal. Philippe mentioned that other experiments are placing effort on the use of lxbatch at CERN for dedicated analysis activities. It was suggested to resurrect DIRAC.CERN.ch to submit to lxbatch. The LHCb shares should be setup correctly and we probably need one long queue for simulation jobs. This should be pursued with a fairly high priority. ( ACTION: Andrei, Joel, Stuart ) 3. Production Operations Issues SLC3 support at CERN is set to finish at the end of this month. Joel indicated that we can ask for new machines or upgrade existing machines to SLC4. volhcb03 - already SLC4 lxgate03 - ask for new machine (volhcb04) and migrate test system to there After successful migration of the Test system, the user and production systems (volhcb01, volhcb02) can be upgraded. Due to the non-trivial migration of the Bookkeeping it was decided to keep volhcb01 in the short-term and ask for new machines for lxgate03 + lxgate05 for the production and user systems. volhcb02 can be migrated to SLC4 and lxgate34 can be replaced by volhcb01 once the bookkeeping is migrated. In order to support DIRAC.CERN.ch it was agreed to install LSF clients on all the WMS machines ( ACTION: Joel) Andrei agreed to prepare a schematic of machine requirements and requested upgrades ( ACTION: Andrei ). Andrei raised the issue of WMS mirroring and maintaining a 'hot swap' of the JobDB in case of failures. A mirror of the JobDB should run on a separate machine that can be permanently kept up to date. This dedicated machine should be setup with passive services waiting to be turned on in case of problems. ( ACTION: Andrei, Joel ) The LogSE is currentlly a classic GridFTP SE running on 1TB local disk storage on volhcb01. Jean-Philippe clarified that GridFTP servers are OK for transfers outside of FTS. Andrei mentioned that sandboxes, logs and the software tarballs are each typically ~100GB in size which is well within the 1GB / box limit. Jean-Philippe also conveyed that SLC4 has GridFTP2 that has many improvements on GridFTP. Since Castor performs less well than a classic SE for this use-case it was agreed to keep this outside of Castor. A spare 5TB at CERN can now be added to the lhcbdata pool. Disk storage at NIKHEF / SARA was mentioned by Philippe. Should the dCache or DPM be pursued? A single storage technology at the site would be advantageous for LHCb and since no support at NIKHEF exists for LHCb DPM usage so it was agreed not to push for DPM storage at this time. Joel mentioned that the Castor durable storage is ready to be added next week. There are two options, - Can have new pool to replace lhcbdata and migrate all existing data - Alternatively can change the configuration of the pool & retain the existing data Up to 70 TB can be added for LHCb in both cases. Either one or two pools are available and the advantages / disadvantages are to be discussed with Miguel. The preferred option is to avoid any serious data migration. ( ACTION: Joel ) 4. DIRAC3 Tasks DIRAC3 developers session commences after today's PASTE meeting. Adri and Andrew should arrive tomorrow. 5. AOB Roberto asked whether large Tier-2's should start to be used for analysis. After some discussion, Nick confirmed that the official policy should be to run at the Tier-1's only for now. Philippe mentioned that the AFS UI installation (currently gLite 3.0) can be updated to 3.1. Stuart suggested that an lhcb-proxy-init (based on dirac-proxy-init) would remove the UI dependence on lxplus and resolve any problems with proxies for LHCb users (proxies can be extended to VOMS inside the WMS). The API version of pyOpenSSL and the Condor ClassAd should be recompiled for SLC4 to be released in parallel with v2r16. ( ACTION: Andrei, Adri ) Roberto reported that after investigations around 20% of sites do not provide correct VOMS schema, this number increases when further site updates are required. The ATLAS and CMS schema (default mapping) works everywhere. Out of the Tier-1 sites, CERN and IN2P3 don't offer correct VOMS mapping. It was agreed to adopt the default mapping everywhere and maintain roles in the DIRAC CS.