UKI Monthly Operations Meeting (TB-SUPPORT)

Thursday 25 Oct 2007, 10:30 → 11:30 Europe/Zurich

EVO - GridPP Deployment team meeting (under GridPP community)

Monthly review and discussion meeting for those involved with UKI EGEE/WCG deployment and operations.

UKI minutes Attendees ========= Brian Davies Duncan Rand Matthew Doidge Mona Aggarawal Andrew Elwell Phil Roffe Jeremy Coles Stephen Burke Derek Ross Yves Coppens Giuseppe Mazza Peter Love Geoff Quigley Chris Brew Winnie Lacesso Mingchao Ma Site Status review and issues ============================= Accounting ---------- Some sites still having publishing delays LESC - Old version of java needing upgraded UCL-HEP - site is down OX-HEP - ? csTCDie, giHECie, mpUCDie - thought to be a transient problem, but will look again Is there a missing edg-gatekeeper rpm in the EGEE repository [Action : Jeremy to follow up] Monitoring status ----------------- Some sites contacted to ask how they will meet 90% target in October GridPP monitoring page http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/ukgrid.html Publishing may need to be updated if a site they have multiple CEs, receipe on LCG-Rollout YAIM will be improved to cope with this Current experiments issues ========================= Pilot jobs - Site Managers will be asked for their position on glExec CCRC - in February and May 2008 - overlapping activites for all activities T1 only for now. Atlas ----- M4 now large complete 70-80% of subscriptions met M5 being planned, plans to improve Data Acquisition and data quality monitoring and get higher rates Performing functional tests of data distribution - looking to get Glasgow involved Issues with missing resource management tools for VOs CMS --- Memory requirements are growing, going above 1GB for 32bit systems Didn't manage to meet transfer targets Did manage to get all T1 transfers to succeed Now commission links before use, once link is commissioned , transferred from Debug to Production Achieved 20K jobs/day (target 25K) better than CSA06 LHCb ---- Software distribution via SAM Reliability of shared software areas increasing concern (NFS limits) Pool accounts a nightmare up to 10,000 jobs running simultaneously Various file access problems Want gLite WMS, full VOMS support, generic pilot jobs Alice ----- Peak of 6400 running, equal split T1 and T2 Alice central services >95% availability Now use xrootd (DPM first choice for T2 SE due to integrated xrootd) Upward trend of users but slow and sporadic Updates/Changes =============== May need more top-level BDIIs - on a T2 level. Suggestion is that any site with >1000 CPU should have one lcg-CE for SL4, now in PPS but needs new YAIM, some packaging problems, something available to Production in 4 weeks possibly Site downtimes now announced automatically via GOCDB - no need for seperate Broadcast Downtimes get sent to VO Managers not Users is this correct [Action : Jeremy to check] SAM admin tool has some small improvements VOView consistency ------------------ OXford and Birmingham: probably okay - shared queues Liverpool - fixed but CE overloaded LESC - numbers should be correct now RALPP - should be fixed, but can be unfixed by subsequent YAIM reruns IC - fix underway Security policies: Grid Security Policy approved, VO operations policy - final call for comments, Pilot Jobs Policy in progress http://www.gridpp.ac.uk/deployment/security/policies/index.html YAIM 4 is coming Talk at EGEE 07 about CREAM CE VO specific SAM tests: https://twiki.cern.ch/twiki/bin/view/LCG/SAMVOSpecificTests Steve Lloyd working on hardware allocations using formula based on SAM Test Efficiency X (CPU Delivered + Disk available) GridPP oversight committee - need to develop disaster recovery scenarios and planning Discussion on revised MOUs for GridPP3 coming Response to recent security incident ==================================== A head node at CNAF containing 1100 CDF proxies valid for a week plus kerberos keytab file which is allowed to create new proxies was compromised. Attacker got user's passwords and logged in to a service account on the CDF head node. JObs can be used to submit jobs and modify/delete files belonging to CDF Vos, and could possibly get logins to other sites Need to check logs for proxies of the form /DC=fnal/O=Fermilab/OU=Robots/CN=cdf/ and for connections from 207.36.118.14, bastion.cnaf.infn.it and cdfhead.cnaf.infn.it. CDF use pilot jobs therefore only "/DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/UID=benjamin" and "/DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin are valid proxies Contact security-officer@gridpp.ac.uk if anything is noticed. AOB === Please register by end of tomorrow for SRM conference at Edinburgh in November, DPM, dCache and Storm developers will be attending. HEPSYSMAN next Wednesday, talks on MonAMI and Nagios. Want to share experiences with 64bit and virtualisation - will set up wiki areas

10:30 → 10:50 Site status review and issues

- Accounting -- There has been a problem with the APEL archiver processing new data. This has led to delays in publishing. Most sites now look up-to-date. Those with a recent gap: csTCDie; giHECie; mpUCDie; UKI-LT2-IC-LeSC; UKI-LT2-UCL-HEP; UKI-SOUTHGRID-OX-HEP - Monitoring status -- The summary table for GridPP sites http://www.gridpp.ac.uk/wiki/SAM_availability:_Monthly_summary_table was reviewed by the PMB on Monday. Some sites have been contacted to understand how the target of 90% can be met for October and November. -- Steve Lloyd has developed a combined view page that will be of use to quickly check the status of a GridPP site or group of sites: http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/ukgrid.html. - Site concerns -- Security - will come on to in item later

10:50 → 10:55 Current experiment/VO activities and issues

- ATLAS ATLAS throughput tests have finished and the M5 detector cosmics run has started. This will run untill 5th November. There may be T0-T1 transfers shortly but don't expect too much T2 activity. - CMS CSA07 now to continue till mid-November. There have been problems maintaining rates. - LHCb Currently working on event stripping. No immediate plans to utilise T2. - Regional VOs - including camont and totalep camont has received a new award and is funded for a couple of more years. There has been no activity from totalep. gridpp now has many members. We need to renew focus on getting the Tier-2 (DNS-style named) VOs enabled. - Other such as biomed Other countries have recently noted that biomed jobs are even less efficient now than previously. Some sites have seen a surge in calice jobs.

10:55 → 11:05 Recent GDB/MB/DTEAM discussions

- GDB/MB -- Tier-1 accounting figures are now being recorded. UK as a whole did not reach the committed levels for Q3 larelgy due to under use. Site availability is generally improving. -- Use of pilot jobs and glexec is back on the agenda and rising in priority. Site managers will shortly be asked their position by WLCG. -- The experiments are now planning a common computing readiness challenge for Feb and May 2008. -- ATLAS summary --- M4 (23/8-objectives (SFOs->analyse data @ T2s) largely met --- O(70-80%) subscriptions met --- M5 - improved TDAQ & data quality monitoring. T0->T1 export to be improved. --- Functional tests of data distribution (time & eficiency) to select sites now "regular" --- Still have data management stability issues (stagein & stageout). Still see many proxies expire. --- Big issue with missing resource management tools for VOs missing (job priorities, shares, ACLs, quotas) CMS: --- Memory requirements seem to be growing! --- Not all sites had enough data subscriptions to meet transfer target for T0 --- Most T1 transfers eventually succeed but overall quality to be improved --- Commission links and monitor (PhEDEx runs in production or debug mode) --- All 7 T1 now managing to archive to tape --- 20K jobs per day achieved (target 25K) = better than CS06 --- Analysis rate is much lower than target - more T2s being included --- Skimming at T1s started and data being sent to T2s. LHCb: --- Software distribution now via SAM (see Joel Closier's CHEP poster) --- Reliability of shared areas increasing concern (limits of NFS). Access permissions has been a problem. Moving to pool accounts a nightmare --- Up to 10,000 jobs running simultaneously. SE unavailability for data output upload major issue. Also full disks awkward. --- Various file access problems for reconstruction phase - inconsistencies between SRM TURLs and root access; ROOT finding HOME; rfio reliability (now fixed with ROOT); not enough disk guranteed; files not available (want protocol for site to warn via LFC flag that replica is unavailable) --- Wants: gLite WMS; full VOMS support; support for generic pilot jobs --- Main problem is D1TX. Too many SE instabilities and now migration issues. --- SL4: inconsisencies between MW and OS - m/w 32-bit. WN should expose 32-bit arch but even CERN has puthon = 64-bit ALICE: --- Peak of 6400 jobs running: 50% T1 and 50% T2 --- ALICE central services >95% availability last 4 month --- Now use xrootd (DPM now first choice for T2s due to integrated xrootd) --- Upward trend of more users but slow and sporadic --- Main concern is stability of storage --- ALICE still has a shortfall in pledged resources DTEAM:

11:05 → 11:15 Updates/Changes

- ROCs requested to have sites "with more than, say 1000 CPUs set up a local top-level BDII for local usage". We may do this on a T2 basis. - lcg-CE for SL4. It is now on the PPS. Requires the new YAIM (the one with the configurator options). - new tool in the CIC Portal for announcing site and service downtimes. Most of you will have seen the announcements. Notification seems better for me. Documentation is http://cic.gridops.org/index.php?section=home&page=SDprocedure. The ops manual will be updated. - Some small improvements with the SAM tool for sysadmins: https://cic.gridops.org/index.php?section=roc&page=samadmin - Problem of the VOView consistency. Signaled 3 weeks ago, still 90 queues have problems. The new list under the usual http://voatlas01.cern.ch/atlas/data/VOViewProblem.log -- CE:epgce2.ph.bham.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:2 TOTVOrun:0 TOTwait:0 TOTVOwait:0 -- CE:hep-ce.cx1.hpc.ic.ac.uk:2119/jobmanager-pbs-heplt2 TOTrun:336 TOTVOrun:6384 TOTwait:1465 TOTVOwait:27835 -- CE:hepgrid2.ph.liv.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:0 TOTVOrun:0 TOTwait:4444 TOTVOwait:13332 -- CE:heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:3 TOTVOrun:0 TOTwait:0 TOTVOwait:0 -- CE:heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:7 TOTVOrun:0 TOTwait:9 TOTVOwait:0 CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-10min TOTrun:1 TOTVOrun:0 TOTwait:0 TOTVOwait:0 (most LeSC queues have problem) -- CE:t2ce03.physics.ox.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:9 TOTVOrun:0 TOTwait:0 TOTVOwait:0 =TBC: pgce2.ph.bham.ac.uk:2119/jobmanager-lcgpbs-short, t2ce03.physics.ox.ac.uk:2119/jobmanager-lcgpbs-short are probably OK - these are shared queues after all, so VOjobs < QueueJobs is normal. Liverpool are "fixed", but their BDII suffers from CE load (hence the 4444). =LeSC have deployed a new version of the SGE plugin (as have IC-HEP), so their numbers are probably correct (VOJobs < QueueJobs). =RALPP fixed, but it's somehow been unfixed? =The IC hep-ce.cx1.hpc.ic.ac.uk queue is broken - Security policies: Grid Security Policy (v5.7) approved. Grid site operations policy (v1.4) approved. Still need covering documents for specific grids. VO operations policy (v1.4) - final call for comments. Pilot Jobs Policy (v0.3) in progress. - YAIM 4 is coming. Discussed in contect of support (http://indico.cern.ch/sessionDisplay.py?sessionId=13&slotId=0&confId=18714#2007-10-02) - VO specific SAM tests: https://twiki.cern.ch/twiki/bin/view/LCG/SAMVOSpecificTests - The overall GridPP deployment links page has been updated: http://www.gridpp.ac.uk/deployment/links.html. New links to some of the experiment monitoring - we need to learn how to best make use of it! - GridPP T2 Q3 quarterly reports are now done. Steve Lloyd is working on the GridPP3 hardware allocations using a formula based on: SAM Test Efficiency x (CPU Delivered + Disk Available). - There was a GridPP Oversight Committee meeting on 10th October. The main item of concern to operations is the development of disaster recovery scenarios and planning. - From the start of the OC T2 document: "By the end of Q207, the UK Tier-2s had made 8588 KSI2K of CPU and 743 TB of disk available to GridPP. This represents 136% of the CPU and 84% of the disk agreed in the GridPP2 Tier-2 MoU signed in 2004. This MoU covered the period to September 2007 and is in the process of being extended unchanged to cover the GridPP2+ period to April 2007." - gstat publishing (ref: LCG-ROLLOUT)

11:15 → 11:25 Response to recent security incident

- Update and discussion on CNAF head node incident. Largest impact on CDF supporting sites. - sites please check information in GOCDB, especially the CSIRT email and telephone information.

11:25 → 11:30 AOB

- Please support and register for the HEPSYSMAN meeting on 31st October. It is in London and will focus on improving monitoring at sites. Based on progress/feedback from this session we will organise more indepth training if required: http://hepwww.rl.ac.uk/SYSMAN/oct2007/main.html - Also, remember there is going to be an SRM2.2 workshop with a Tier-2 focus at NeSC (Edinburgh) 13th-14th November: http://indico.cern.ch/conferenceTimeTable.py?confId=21405. You need to register for this event! - Sharing experiences with 64-bit and virtualisation