Operations team & Sites

Tuesday 13 Dec 2011, 11:00 → 12:16 Europe/London

EVO - GridPP Operations team meeting

- This is the biweekly ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 115728 with code: 4880. Apologies: Rob H; Matt D

Minutes GridPPOpsMinutes13-12-11.txt

11:00 → 11:20 Meetings & updates

- ROD team update - EGI ops https://wiki.egi.eu/wiki/Agenda-13-12-2011 - Nagios status - Tier-1 update - Security update -- T2 issues IRC -- General notes. Issues with new CA certs: https://ggus.eu/ws/ticket_info.php?ticket=77179. "Maarten: so far there does not appear to be a way to handle this easily on the VOMRS side; we may need to handle each case manually; proposal: do the current case and record the necessary steps in the ticket - then we may judge if the necessary effort can be tolerated for this exceptional case; the total number of UK CA users in WLCG currently is 425 (ALICE: 9; ATLAS: 264; CMS: 40; LHCb: 80; DTEAM: 31; OPS: 1) and their certificate renewals probably will be spread over many months to come, so on average there would be at most a few per day to deal with, depending on the affected VO" - TEG - Tickets Of interest" tickets are: 77350 (Durham) - Do they actually support multicore jobs? 77302 (Manchester) - There was a mistake made during the assignment process, just checking that Manchester know about it as they're usually much faster at replying. 77064 (Glasgow) - As of the time of writing there been no addition to this ticket since its assignment a week ago. 76996 (Birmingham) - A problem that looks solved reemerged (or another problem took its place), however the assigned to field of the ticket was stripped. Is Mark still getting updates? I also noticed that Chris Brew set all his tickets "On Hold" whilst he was afk for a few days, and this seemed like a very good working practice. Only 35 tickets open this week. Good work! DURHAM https://ggus.eu/ws/ticket_info.php?ticket=77350 Multi-CPU job problem sent do multiple sites. Does Durham support multicore jobs? https://ggus.eu/ws/ticket_info.php?ticket=77028 atlas problems, looks like the root cause is the same as 76487 (overly full software area). MANCHESTER https://ggus.eu/ws/ticket_info.php?ticket=77302 biomed ticket, it went around the houses before finally get assigned, just checking that they know about it. QMUL https://ggus.eu/ws/ticket_info.php?ticket=77110 Having deletion errors for atlas, Chris and Elena called in the Storm developers. Seemed okay as of the 8/12. Are things settled? GLASGOW https://ggus.eu/ws/ticket_info.php?ticket=77064 Possible WMS problem for ops jobs going through WMS. No visible progress on the ticket till it's inception on the 5/12 Tier-1 https://ggus.eu/ws/ticket_info.php?ticket=77026 Chris W spotted a problem with the output of multiple BDII queries- about 50% failed. The problem appeared to be one of the UK top BDII aliases at RAL, but noone is really sure. Kudos to Chris for spotting this wierd behaviour, but what does it mean? Birmingham https://ggus.eu/ws/ticket_info.php?ticket=76996 This one cropped up last week as missing libraries troubles killing ATLAS jonbs. Since then the problems have faded away and then came back so it might have changed in nature. I notice the "assigned to" field has been wiped, so Mark might not be getting notifications anymore. RALPP Chris put all his tickets on hold whilst he was away for a few days, this seems like a very sensible thing to do. Cambridge: https://ggus.eu/ws/ticket_info.php?ticket=57746 This ancient ticket is sadly being set to "unsolved". General: https://ggus.eu/ws/ticket_info.php?ticket=74353 The pheno my-proxy trouble ticket. Just a quick note to say that Chris W put it to "In Progress" from "Unsolved" and some excellent work has been done.

11:20 → 11:40 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS - Other - Experiment blacklisted sites - Experiment known events affecting job slot requirements - Site performance/accounting issues - Metrics review

11:40 → 11:50 Proxy renewal issues

https://ggus.eu/ws/ticket_info.php?ticket=74353 - pheno - T2K - snoplus

11:50 → 11:55 Networks update

MM reports from recent meetings OR if Mark does not join a look at the GDB agenda for tomorrow: http://indico.cern.ch/conferenceDisplay.py?confId=106651

11:55 → 12:00 Hardware latest

64 core and all that jazz. Martin's comments: "Our recent stuff is all rather conservative - 12 cores per box in two chips. We shied away from the Interlagos and Sandy Bridge due them not being released or guaranteed delivery for our orders. The ITTs specified the usual HEPSEPC total, RAM and disk constraints, and this year for the first time some I/O bandwidth per core (not large - 80Mb/s (bits/s)). For more than 12 cores in a box, this implies bonded pairs or triplets of 1Gb/s NICs, or 10Gb/s NICs."

12:00 → 12:01 AOB