Operations team & Sites

Tuesday 29 Nov 2011, 11:00 → 12:16 Europe/London

EVO - GridPP Operations team meeting

- This is the biweekly ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 115728 with code: 4880. Apologies: David C; Raja

Minutes Opsteamminutes.pdf

11:00 → 11:20 Meetings & updates

- ROD team update - EGI ops SLURM survey: http://www.zoomerang.com/Survey/WEB22DWEANYTKU - Tier-1 update - Security update -- T2 issues geant4 GGUS 'on solution' default -- General notes. JG message wish to stop sites pointing to CERN's top level BDII because of the load it imposes on them. UK features strongly in list. UKI-LT2-Brunel UKI-LT2-RHUL UKI-LT2-UCL-HEP UKI-SCOTGRID-ECDF UKI-SCOTGRID-GLASGOW UKI-SOUTHGRID-BRIS-HEP UKI-SOUTHGRID-OX-HEP UKI-SOUTHGRID-RALPP - TEG Take a look at the summary talks from yesterday: https://indico.cern.ch/conferenceDisplay.py?confId=161830. WG1. Monitoring. https://docs.google.com/a/pic.es/document/d/1o5uesrQz3AZMhzmve4wTGhc15QKionaSLE9Vvd5n5Jw/edit?pli=1. Examples: "Many monitoring systems exist, but they are not exposed outside the site or outside the VO, or they are of difficult accessibility." "very few services come with a native monitoring." "In particular, Nagios (and Nagios probes) is considered a suitable fabric monitoring solution, but many sites addressed several shortcomings, such as the impossibility to monitor services rather than nodes (problematic in case of load balanced front-ends), the limitations in alarms masking, the counter intuitive interface (especially for historical views). At the same time, other sites indicated Nagios as a very complete solution. In general, it is difficult (and probably not beneficial from the site perspective) to identify a common fabric monitoring system. Therefore, developers of middleware provided sensors should avoid extremely tight integration and coupling with any particular monitoring system and be as generic as possible with respect to the framework from which the sensor is launched." WG2. Tools & Services (fuller example) Areas of improvement – Stability: service info is prone to disappear, bad because use cases shifted towards needing more stability – Information validity: info provider info very fragile, configuration very error prone – Better policies for resource publication – Lower latency for dynamic information Areas of potential efficiency gains – Better validation tools – Accurate storage information would make storage accounting a lot easier – Provide more powerful and user-friendly client tools Largest use of operational effort – Configuration and validation of information – Debugging IS problems for users and sites Missing areas – A continuous certification and auditing of the BDII information by WLCG ' WG3:Software config & management. Main issue seems to be Re-deployment of (almost) identical common software packages for the experiments'. WG4: Operational requirements on middleware: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGTegOperationsWG4WG5#WG4_Operational_Requirements_on. Main areas error logging; robustness; failure investigation; deployment complications and strategic direction. WG5: Middleware (config; deployment; distribution) Dependencies. Repositories. Reactive development. Validation methods. Tools for deployment. - Nagios status Plans for backup server at Lancaster. - Tickets General: A few "dteam" tickets at sites instigated by Chris' proxy renewal tests. It seems a few of us have let our dteam settings go stale! Also a general reminder goes out to those who still have cvmfs tickets. Observation: Whilst checking on https://ggus.eu/ws/ticket_info.php?ticket=76750 I noticed that t2k.org wasn't on the list of VOs in the "Change concerned VO" drop down box. t2k have had to make do with "Other". Glasgow: https://ggus.eu/ws/ticket_info.php?ticket=76661 Still having problems openmpi but Dave is working on it. If there's time it might be interesting to hear about it. Birmingham: https://ggus.eu/ws/ticket_info.php?ticket=76624 Load/Network problems caused temporary(?) SE troubles for atlas. Did things hold up over the weekend? Can this ticket be closed? Brunel: https://ggus.eu/ws/ticket_info.php?ticket=76608 Biomed nagios tests. Bounced back to biomed as their nagios tests don't seem to be dynamically reading bdii information, should be in "Waiting for reply". EFDA-JET: https://ggus.eu/ws/ticket_info.php?ticket=76588 Fusion jobs were failing due to symbolic links in the Input files. Looks like it could be set to solved or at least waiting for reply. Tier-1: https://ggus.eu/ws/ticket_info.php?ticket=76564 geant4 jobs were having trouble (lcgadmin voms role didn't have permission to use one of the queues). Asked to retest on the 23rd, been waiting for reply since then. https://ggus.eu/ws/ticket_info.php?ticket=76521 snoplus support enabled, ticket can be closed. Oxford: https://ggus.eu/ws/ticket_info.php?ticket=76556 hone having trouble submitting to some of Oxford's queues. Not a "problem" ticket, but an interesting one. Heavy involvement from Daniela to rule out the WMS being an issue. Durham: https://ggus.eu/ws/ticket_info.php?ticket=76288 atlas job failures, had a bunch of bad nodes that looked like they were killing and eating jobs. Did removing them from the queues sooth things? https://ggus.eu/ws/ticket_info.php?ticket=75488 compchem still having trouble - looks like authentication issues. Manchester: https://ggus.eu/ws/ticket_info.php?ticket=76190 lhcb - Some WNs failed to install cvmfs properly, causing problems. Presume fixed now? https://ggus.eu/ws/ticket_info.php?ticket=76119 biomed - complaining about not having much space. Alessandra freed up enough space to give them a small but positive amount. Not much more can be done. RalPP: https://ggus.eu/ws/ticket_info.php?ticket=75960 biomed complaining about duplicate SE entries at RalPP when issueing a lcg-infosites. Stephen B pointed out that there are two volumes for biomed, a share in the group volume and a space token of their own. Lancaster: https://ggus.eu/ws/ticket_info.php?ticket=75798 lhcb having trouble with one of the Lancaster CEs. Although all the original problems appear to be fixed poor performance is keeping us banned. https://ggus.eu/ws/ticket_info.php?ticket=75393 Lancaster was being rubbish for t2k.org, waiting for word back from them about testing our clusters. Cambridge: https://ggus.eu/ws/ticket_info.php?ticket=75538 Atlas space token resizing. How's it going?

11:20 → 11:40 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb For LHCb, no significant issues to report. We have finished our re-processing and are waiting for the Monte Carlo MC11 to be validated before submitting significant amounts of jobs again - CMS - ATLAS - Other Proxy renewal testing - current status - Experiment known events affecting job slot requirements - Site performance/accounting issues - Metrics review

11:40 → 11:55 Hardware discussion

- Pledges/expectations - Site timetables - Hardware directions Please record latest thinking/purchases here: https://www.gridpp.ac.uk/wiki/Guidance_and_recent_purchases. It is useful to share HEPSPEC06 updates: https://www.gridpp.ac.uk/wiki/HEPSPEC06 Also useful to track site information if maintained here: https://www.gridpp.ac.uk/wiki/Site_information.

11:55 → 12:05 November GDB

- Meeting agenda: https://indico.cern.ch/conferenceDisplay.py?confId=106650 - Summary: https://www.gridpp.ac.uk/wiki/GDB_9th_November

12:05 → 12:10 Actions

- https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items

12:10 → 12:11 AOB