WLCG/EGEE OPs meeting 27th November 2006. OSG GOC Rob, Leigh EGEE ROCs --------- Asia-Pacific Min Central Europe Marcin CERN DECH Sven France Rolf, Gilles, Osman Italy no Northern Europe Ron Russia Lev South East Europe no South West Europe Gonzalo UK/I Philippa WLCG Tier 1 sites ----------------- ASGC Min BNL Dantong Yu CERN Maite FNAL Jo Kaiser GridKA Sven IN2P3 Rolf INFN no NDGF Michael PIC Gonzalo RAL Derek, Matt SARA /NIKHEF Ron TRIUMF Reda GGUS Torsten VOs --- Alice no ATLAS Gilbert BioMed no CMS Ian Fisk LHCb Roberto OSCT no + Any comments on previous meeting. None + CIC On Duty Handover. From CERN to South East. Comment that PPS SAM are not running at the moment: Is expected to be the same problem as hit the production one at the weekend with a hanging voms-proxy-init. + Can SFT framework be relied on? in particular the SE region is using this. The recomendation is very much to switch for SAM, good documentation now exists since it has been deployed in various regions. + VOMS Error returned by mkgridmap cron. Will investigate with VOMS group. It errored every day untill Saturday but they have now gone. Possibly the VOMS crashed at the weekend and that may have fixed it, just a guess it must be followed up with VOMS people. + Maite and Steve commented about what was going to happen at the next 1-day operations workshop, during the WLCG collaboration week. Steve requested feedback on what tutorials are needed for the sysadmins to increase reliability. Please contact him with requests of offers of talks. Links to the preliminary agendas: Thursday afternoon: http://indico.cern.ch/sessionDisplay.py?sessionId=21&slotId=0&confId=3738#2007-01-25 Friday morning: http://indico.cern.ch/sessionDisplay.py?sessionId=22&slotId=0&confId=3738#2007-01-26 Please, remember to register so we know how many people is expected to come (the registration is common to the WLCG Collaboration Workshop). + Update 3.0.9 to gLite is now available for the production systems. In particular includes many fixes to the WMS system. + WLCG o Alice suspeded for today for ALIEN upgrade. o CMS and ATLAS events still processing for the time being. o Question to IN2P3 about dcache, instabilities following upgrade but IN2P3 reports things should be okay now. IN2P3 are running gPlazma with dCache for authorisation. Action Items + Request for LFC certificate reload. Is it answered by the wiki page? https://uimon.cern.ch/twiki/bin/view/LCG/TheLCGTroubleshootingGuide#How_to_replace_host_certificates No comment received to this, we assume it is clear and close the action. + Maintainence of External Packages. Policy is currently that external packages are not maintained but will make best effort to maintain a decent set of packages. This is being looked at now within the SA3 all hands meeting. Questions to be asked, how many externals are there and which are way out of date and so have lost upstream support. Leave action open since it is being discused right now. + AOB UK/I submited things that do not appear to have made it. This was later explained by the fact that ROC admins must copy and paste things the sites want raised. In the past we had a situation where the ROCs had things posted they were unhappy or at least unclear about. This is why it is how it is. Request for an better itemised work list for SAM. To give people requesting updates to SAM idea as how progress is being made and what is being worked on right now. It is suggested that SAM produce some kind of plan to produce a report much as GGUS and GOCDB does. A programmed of work (POW). This is agreed. We will request the SAM team to make th plan for the next year available and to present it to teh ROC managers meeting (action for the ROC managers) Comments from LHCb. LHCb want to schedule a time with PIC to test how CE load is a function of number of resource brokers. PIC is happy to arrange a time to do the controled DOS of PIC. TIER-1 REPORTS ----------------- * ASCG Taiwan: no report * BNL/USATLAS site report: BNL dCache had very high load and many USATLAS production transfer from Tier 2 sites to BNL were stuck. We found that the transfer logs residing on PostGres Database at dCache SRM door nodes were over 30G Bytes, and slowed down the disk I/Os, which caused time out on most SRM requests. The data transfer backlog reached several thousand files. We manually cleaned up the transfer log and the production went back on track. The backlog files were transfered from Tier 2 to BNL. The planned dCache upgrade to 1.7.0 will be the first week of December. * CERN: Nothing to report * FNAL: no report * IN2P3: T1/IN2P3-CC comments: - GGUS ticket #15120 reopened on Nov. 20th about VOMS errors returns by mkgridmap cron. voms.cern.ch and lcg-voms.cern.ch are involved. The errors are : internal server and unknown data format errors. The ticket is still not answered, and errors still occur. - problems with SRM SE after the Dcache update. After solving some misconfiguration problem, we noticed unstability of GPLAZMA service (this is a new authentication service). After a while, this new service hangs up and becomes unavailable for SRM services. Dcache developers has been informed of our problem. We study the possibility to come back to the previous DCACHE authentication system to temporary solve the problem. - Even though SRM SE is still quite unstable at the time being, we are going to start the intensive tranfer tests scheduled with the 3 LCG VOs (Alice, Atlas, Cms). * Tier1 GridKa (FZK): - Storage: sometimes the SFT rm tests fail. Until the next upgrade of dcache 1.7, planned early December, there will be no activity to track this issue - MW: 20/11 a minor glitch in the operation of the lfc host was detected and fixed. A leftover of the powerdown of last week. 23/11 a flood of jobs by CMS seriously lamed the batch system. After consulting CMS representatives it was decided to remove the jobs. During deletion of 25k jobs, submission of new jobs stalled * INFN: - ALICE: production and FTS tests stopped at all sites for the deployment of the AliEn release v2-12. Operations likely to restart around tomorrow first on test sites (Torino in Italy) and subsequently after deployment in all others. * NDGF: No SAM tests for ARC hence the errors... * PIC: lcg-CE service: - Approx. 24h problems from 15-Nov noon till 16-Nov noon: due to problems with the quattor torque configuration of the WNs (epilogue/prologue files created with wrong permissions). - Approx. 12h problems on the 17-Nov morning: Configuration problem when introducing a new bunch of WNs. The CE did not have the proper ssh keys. SRM-tape service: -Castor name server hanged on 18-Nov evening. Service recovered on monday 20th Nov in the morning. SRM-disk service: no problems. * RAL: It was discovered that heavy load on an RB was caused by some gahp_server processes which were not owned by the condor scheduler process, killing them dramatically reduced the load on the RB ralsrma.rl.ac.uk - Castor SRM at RAL in maintenance this week. This was only in production for CMS, with whom the downtime was agreed upon in advance. Apparently some, possibly all, of RAL-LCG2''s LCG CE''s queues are not (yet) accessible from gLite RBs - https://savannah.cern.ch/bugs/?21122 How important is this to the experiements?