Deployment team

→ Europe/London
EVO - GridPP Deployment team meeting

EVO - GridPP Deployment team meeting

Pete Gronbech
Description
- This is the biweekly DTEAM meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 64688 with code: 4880.
Minutes ======= Atendees ------- Alessandra Forti Andrews McNab Brian Davies Daniela Bauer Dave Colling Derek Ross Duncan Rand Jeremy Coles Mohammed Kashif Pete Gronbech Richard Hellier Sam Skipsey Wahid Bimji Gareth Smith Mingchao Ma Stuart Purdie 2: Experiment Reports ------------------ CMS No problems! Would like to move date of castor upgrade, in discussion with castor people on that. Reprocessing campain due at the end of this week; hence want to avoud the 3 day downtime in the middle of that. LHCb Upload problems still at Brunel - looks like it's only one of thier clusters, took them out to Atlas Looks like many files at RHUL that's Dark Data - so might be the case at other Tier-2's too; RHUL expected to be worst, however. Atlas subscriptions: request to go through the Atlas UK operations team; not something for the site admins to handle directly. Hammercloud analysis script: Sam has an updated script; not offically published, and definatly not supported! ROD Team updates ---------------- BHAM - sounds serious; looks like passing again. Glasgow - Firmware issues (not hardware, thankfully), mostly resolved, and operating at reduced capacity today. QMUL - Cream CE in commising, downtime juggling is awkward. There's a status for CREAM, but not others. Tier 1 update ------------- LHCb SRM endpoint. Outage on tape system - tape robot power supply. Quattorising MyProxy. Rolling update to gLite 3.2 BDII. Atlas srm issue? Batch system issues yesterday - batch server failing on DNS lookups. Castor issue: Hit by bug in SRM for how VOs' check to see if files brought online was causing : Crash in SRM bring online request ( ATLAS ES issue.) have now updated the srm, hope this is going to solve issue. GDB --- Pete going as Tier 2 rep. Tier 2 ------ Publishing status: Sept/Oct: Some problems at Sheffiled - 2 weeks of outage. Allesandra has noted that to the site a week ago. Glasgow a few days (put down to power cut). Birmingham (put down to filesystem issues). Pete to note to them once they come back up. Meetings -------- Hepix is on, has Evo link. Storage today, security and networking tomorrow. Tickets ------- Usual group; No major change on them. There are some ticket that there arn't known procedures for handling. (e.g. user trying to set up a UI on their laptop). Need to work out how to handle that. Ticket metrics: Lots got BHAM, probably down to the filesystem issue being duplicated (per CE). Some of the metrics don't look right. BDII Freshness -------------- This will become a critical test, and we should make sure we're in good shape before that happens. There was a difference between gridpp-nagios and gstat; and no clear answer on that. Now looks all ok; Jeremey would like to know if there are differences between the two. SE for Nagios testing. Some discussion as to the merits of particualar SE's; end conclusion was to use Tier-1, with Oxford as a (manual) failover plan. (As oxford would be co-located with the nagios server). Procedures ---------- Missing instructions: Introducing a new site node VO creation. New GOC DB ---------- Ducan entered downtime; to clear the downtime he hit 'delete', which didn't do what he expected. Noted that it should be 'edit'. Some discussion about not being able to extend downtimes. Managing downtimes with each service independantly. Guidance on security issues --------------------------- Some sites were looking for guidance on what to do under security advisories. E.g. in the event of no public exploit released for things. There was some inconsistancy with the Grid policy being forcing in some areas defering to site policy in others. This is down to the level of risk depends on the site usage and Actions review -------------- Web pages and wiki: getting more urget. Resiliancy: Nothing after Scotgrid; presumable sites are doing things, perhaps return to it at HEPSysMan. Closed. DN list: done, but not used. Logical ang physical: Keep open, sites to check. Migration strategy: Still working on it. Current setup appears to be fine. Depends on the NGS situation; which is less certain with funding things. Hepsysman talk: done. Closed. Upcoming events: Done. Not similar for CMS and LHCB. Closed. NDGF: Done. Closed. WMS monitoring: Not at IC, as WMS not used enough. GStat numbers: Unclear. GStat pledge: Jeremy happy it these match spreadsheet. Closed. PMB 400.2: Sam to followup with more grumpiness. AOB --- HepSysMan: Agenda just updated. Space for a couple more talks, if interested. Maybe one on 'new kit', to inform furture purchases - tada! a Procudement panel now added. Brian might talk about the Glasgow link, as it's larger than the average. Availabilty and reliablity results are good. UCL and RALPP have yet to update the name of the Tier-1 to RAL-LCG. RALPP have agreed to do this. Evo on windows 7? No one else appears to have tried Chat Window ----------- [10:52:33] Jeremy Coles Andrew - let me know if you need to test your audio. Thanks. [10:53:18] Andrew McNab hi - I don't have an external microphone so I'm just going to listen in this time or phone in if necessary. I'll sort one out for next week [10:53:50] Andrew McNab I've done some tests with the EVO test page and verified I can make feedback! [10:56:20] Jeremy Coles Ok. Let us know if you need help. [11:08:58] Brian Davies Crash in SRM bring online request ( ATLAS ES issue.) have now updated the srm, hope this is going to solve issue. [11:10:45] Brian Davies video crashes on NT as well [11:11:25] Pete Gronbech new version on 14th Oct version 2.6 which I'm blaming [11:12:58] Mohammad kashif https://gridppnagios.physics.ox.ac.uk/nagios/cgi-bin/status.cgi?servicegroup=org.apel.CE&style=detail [11:33:09] Mingchao Ma back in 10 minutes [11:36:22] Duncan Rand i vote for ral tier-1 [11:37:28] Alessandra Forti me too [11:42:03] Mingchao Ma back [12:03:35] Alessandra Forti GOCDB ticket in case someone is interested to add [12:04:06] Alessandra Forti it has been closed and the unsolved things have been moved to savannah but still this is the pointer [12:04:10] Alessandra Forti https://gus.fzk.de/ws/ticket_info.php?ticket=63245
There are minutes attached to this event. Show them.
    • 11:00 → 11:20
      Experiment problems/issues 20m
      Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS -- Script for hammercloud plots. - Other - Experiment blacklisted sites - Experiment known events affecting job slot requirements - Site performance issues
    • 11:20 → 11:35
      Meetings & updates 15m
      - ROD status -- Two items from the handover report: BHAM - cluster wide filesystem corruption. Two tickets, one expires Tuesday - other nagios alarms almost certainly due to same issue. GLASGOW - Dual power spike bork'd everything. Hardware damage, extent as yet unknown; in downtime. - Tier-1 update - Next Grid Deployment Board is on 10th November: http://indico.cern.ch/conferenceDisplay.py?confId=83604. Pete is T2 rep this month (http://www.gridpp.ac.uk/wiki/GDB_reports). - T2 issues -- Accounting:http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php (August-October 2010 - UKI level - show data for site as a function of date. ... Problems at Sheffield; Glasgow? & Birmingham? Also csTCDie. - HEPiX. - http://indico.cern.ch/conferenceTimeTable.py?confId=92498#20101101 - Escalated tickets https://gus.fzk.de/download/escalationreports/roc/html/20101101_EscalationReport_ROCs.html 3 tickets on hold... 56316 - NGS-RAL not in BDII 58733 - RAL-PP biomed. dcache issue. 61228 - ALICE RAL T1. - Tickets that have no specific responsible agent: Have we seen others like this one? https://gus.fzk.de/ws/ticket_info.php?ticket=63391 - Ticket metric spreadsheet attached for reference.
      Ticket-metrics
    • 11:35 → 11:45
      BDII "freshness" test and gstat 10m
      - Nagios https://gridppnagios.physics.ox.ac.uk/nagios/cgi-bin/status.cgi?servicegroup=BDII&style=detail - gstat http://gstat-prod.cern.ch/gstat/site/UKI-LT2-Brunel/treeview/bdii_site/dgc-grid-35.brunel.ac.uk/ And semi-related to testing... From yesterday a change of the default SE used for SAM replica management tests was required by each NGI/Grid. The CERN endpoint (samdpm002.cern.ch) is running out of free space. In the UK a temporary CERN node is now being used but shortly we need to decide on a national endpoint. Which SE should be used?
    • 11:45 → 11:55
      Procedures 10m
      - Links and missing instructions -- Introducing a new site node -- VO creation - Guidance around security incidents/vulnerabilities
    • 11:55 → 12:00
      Actions 5m
      - http://www.gridpp.ac.uk/wiki/Deployment_Team_Action_items 400.2 JC to confirm that priorities have been documented for the major experiments for recovering files from disk servers.
    • 12:00 → 12:01
      AOB 1m
      - HEPSYSMAN 22nd November: http://hepwww.rl.ac.uk/sysman/Nov2010/agenda.html - Many GridPP sites have very high availability/reliability for October. Well done to everyone. - Site BDII changes. Sites still using UK-T1-RAL dn: GlueSiteUniqueID=UKI-LT2-UCL-HEP dn: GlueSiteUniqueID=UKI-SOUTHGRID-RALPP