Choose timezone
Your profile timezone:
Operations team (& sites)
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the biweekly OPS team meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area.
- The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 78425 with code: 4880.
Apologies: Mark S
GridPP Operations team (& sites) meeting 31/05/11
In attendance: Jeremy Coles (Chair), Brian Davies, Stuart Purdie, Raul Lopes, Matthew Doidge, Raja Nandakumar, Andrew McNab, Rob Harper, David Crooks, Catalin Condurache (minutes), Mark Mitchell, Gareth Smith, Mohammad Kashif, Stephen Jones, Stuart Wakefield, Ewan Mac Mahon, Alessandra Forti, Pete Gronbech, Santanu Das, Duncan Rand, Govind Songara, Mingchao Ma, Wahid Bhimji, Rob Fay
11:00
Meetings & updates (20')
- ROD team update
Andrew: nothing to report
- Nagios status
Kashif: nothing to report
- Tier-1 update
Gareth: fairly quiet week at RAL T1; some load issues on LHCb small service classes; plan to rollout new version of Castor client (2.1.10) on all worker nodes.
- GDB "timetable"
-- http://www.gridpp.ac.uk/wiki/GDB_reports
Jeremy: put names against various GDB meetings for this year; people asked to check their availability
- CA Technical Advisory Group update (see text attached)
Jeremy: the UK eScience CA certificate expires 2012. A new CA cert will be in operation by Sep 2011 (IGTF release). Sites will need to install two CA certificates for the CAs. More details will follow.
- Security update
--SSC5
--SSC4
Mingchao: SSC4 finished from site point of view; sites need to send the final report; any feedback from participants sites?
About SSC5: details not yet known; will be targeting sites that support ATLAS pilot analysis jobs.
- EGI
-- Still many UK sites with the EGEE tag. http://gstat.egi.eu/gstat/summary/GRID/EGEE/ . Sites on this list on Wednesday will receive a ticket.
Jeremy: re-tagging exercise: sites to change EGEE tag to EGI. Proposal to update by Thursday, followed up by tickets to each 'faulty' site.
-- Some EMI-1 components will reach staged rollout next week.
-- glite 3.2: glite-SGE_utils and CREAM 1.6.6: problem with glic update now understood by developers.
-- Site admin input requested for batch system survey: http://www.zoomerang.com/Survey/WEB22CE9DXKZDT/.
Stuart: sites asked to fill in the survey. Following the SA1 meeting (30 May), invitation have been sent to submit requirements re: middleware
-- gstat availability figure issues?
There are sites with 0% and -100% availability. Possible problems with site-BDII. Sites asked to report such anomaly.
- Open tickets:
http://tinyurl.com/3ulldrr
57746 - Camont issue with Cam-HEP. Condor related? More job outputs given but it needs a WMS expert to help.
Catalin and/or Daniela to have a look and try to help.
64995 - RAL T1 No GlueSACapability defined for WLCG Storage Areas. On hold but now progressing. Change ticket status? (still on hold).
https://ggus.org/ws/ticket_info.php?ticket=66306 - Creation of Irish NGI. Still "almost". Now problem with transmission of alarms to dashboards for ops.
https://ggus.org/ws/ticket_info.php?ticket=70093. hone issue. Needs wider input? (Any Tier-1 comment on this ticket?).
Catalin to update this ticket, although there is no clear what is the problem there.
Several other tickets on hold.
https://gus.fzk.de/ws/ticket_info.php?ticket=68853
Retirenment of SL4 and 32bit DPM Head nodes and Servers
Awaits end of SL4 gLite 3.1 T2 machines.
https://gus.fzk.de/ws/ticket_info.php?ticket=68858
Retirenment of SL4 and 32bit DPM Head nodes and Servers - Glasgow sub-ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=68859
Durham sub-ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=68865
UCL-HEP sub-ticket.
11:20
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
Raja: nothing to report
- CMS
Stuart: operationally OK. CMS wants to do some Whole Node scheduling testing at Tier1 (next 6 months). Also very limited testing at Tier2 sites.
Catalin: Will check with Derek about the Whole Node scheduling documentation
Pete: Also WN scheduling supported at Oxford for Atlas
- ATLAS
Alessandra: quiet week; 3 sites with transfer issues: QMUL - timeout due to checksum (maybe upgrade gridftp?), Manchester - one data server down, Birmingham - one VO overloading the Storage Element; 4 sites not on line: IC in test mode, UCL, LANC, CAM in automatic broker-off; 2 tickets for QMUL and MANC
Brian: similar checksum related at other sites, possibly gridftp overloaded; also transfer failures because of an US DN (it contains a "'" character within DN)
- Other
- Experiment blacklisted sites
- Experiment known events affecting job slot requirements
- Site performance/accounting issues
- Metrics review
Alessandra: What is meaning of the 'Points' columns in http://pprc.qmul.ac.uk/~lloyd/gridpp/metrics.html? How is it calculated (any formula)?
Jeremy: Will ask SteveL to update the documentation
11:40
Site issues/discussion (15')
- Current status across the sites.
Reminder of the glexec deployment wiki page https://twiki.cern.ch/twiki/bin/view/LCG/GlexecDeployment.
Matt - LANC - some pbs with ATLAS
Brian - looking into different failures modes for FTS
Rob - RALPP - nothing to report
David - Glasgow - nothing to report, keep an eye on the network traffic
Kashif - work on SAM Nagios 2nd level support
Ewan + Pete - Oxford - re-install a CREAM-CE; nothing else to report, running OK
Santanu - Cambridge - some issues with LHCb production and pilot on CREAM-CE
Stephen - implementing an alternate CREAM-CE
Chris - few more WNs to be put on-line
Wahid - power outage yesterday; couple of CREAM-CE showing some oddities
Alessandra - dealing with SONAR tests
Andrew - job submissions are fine; configuring ARGUS and glexec
Govind - glexec and Argus installed and in production;
11:55
Actions (05')
- http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
O-110516-01 - on hold
O-110516-02 - pending on Ireland process to finalise
O-110516-03 - done - results in the Storage wiki (http://www.gridpp.ac.uk/wiki/Suggestions_for_suitable_hardware_to_run_a_Grid_SE#Network_capabilities)
O-110516-04 - on going
O-110524-01 - to be updated by Jeremy
O-110524-03 - done
O-110524-05 - to be updated and closed by Alessandra
O-110524-06 - problem not resolved in production yet
O-110524-07 - no updates yet
12:00
AOB (01')
- WLCG workshop in July https://indico.desy.de/conferenceTimeTable.py?confId=4019#all. Let Jeremy know if you expected to go but have not heard back.
- HEPSYSMAN (30th June - 1st July) http://hepwww.rl.ac.uk/sysman/June2011/main.html
Gareth: Registration needed especially if accommodation needed
Pete: Topics for discussions and speakers are required
Jeremy: Also for GridPP in September
- Lustre workshop QMUL.
Stuart: Mon 4 - Tue 5 July
Pete: The Viglen Storage servers firmware issue. Following a recent firmware upgrade problems disappeared. What are other sites experiencing? Are they waiting for an official Adaptec firmware release?
Alessandra: Installed 10 days ago a server without problems. Will provide the version number on the mailing list.
Chat transcript:
[10:54:01] Brian Davies joined
[10:55:56] Stuart Purdie joined
[10:57:02] raul lopes joined
[10:57:46] Matthew Doidge joined
[10:58:27] Jeremy Coles joined
[10:58:44] Raja Nandakumar joined
[11:00:35] Andrew McNab joined
[11:00:36] Rob Harper joined
[11:00:53] David Crooks joined
[11:01:31] Jeremy Coles Catalin will take minutes today.
[11:01:41] Mark Mitchell joined
[11:01:41] Gareth Smith joined
[11:01:56] Mohammad kashif joined
[11:01:57] RECORDING Catalin joined
[11:02:18] Catalin Condurache i muted myself
[11:02:22] RECORDING Matthew joined
[11:02:54] Stephen Jones joined
[11:03:10] Stuart Wakefield joined
[11:03:26] Ewan Mac Mahon joined
[11:03:40] Alessandra Forti joined
[11:03:42] Pete Gronbech joined
[11:04:02] Stephen Jones left
[11:04:02] Santanu Das joined
[11:04:16] Stephen Jones joined
[11:04:24] Duncan Rand joined
[11:04:58] Stephen Jones my evo keeps crashing
[11:05:07] Queen Mary, U London London, U.K. joined
[11:07:02] Govind Songara joined
[11:08:05] Stephen Jones I'm fine with Nov. I'll let Rob know.
[11:08:21] Stephen Jones Cheers!
[11:13:04] Pete Gronbech will do
[11:14:44] Alessandra Forti SITE_OTHER_GRID="EGI|WLCG|GRIDPP|NORTHGRID"
[11:14:49] Alessandra Forti in site-info.def
[11:15:13] Alessandra Forti clearly replace northgrid with you region
[11:16:00] Pete Gronbech SITE_OTHER_EGEE_ROC="UK/I"
SITE_OTHER_EGEE_SERVICE="prod"
[11:16:13] Pete Gronbech what about these other entires
[11:16:27] Govind Songara i also want to know that
[11:16:32] Pete Gronbech Should they now be otherEGI_ROC
[11:18:27] Mingchao Ma joined
[11:20:09] Wahid Bhimji joined
[11:24:14] Rob Fay joined
[11:25:12] Jeremy Coles https://ggus.org/ws/ticket_info.php?ticket=70093
[11:26:15] Catalin Condurache ok
[11:27:55] Catalin Condurache i'll check with derek
[11:29:32] Catalin Condurache nope
[11:29:48] Alessandra Forti left
[11:29:53] Alessandra Forti joined
[11:39:15] Brian Davies ATLAS sites with failure to complete prepareto put were qmul,manchester,ucl,rhul
[11:40:01] Queen Mary, U London London, U.K. I wonder if they are having the same problem - but at a lower level.
[11:41:24] Jeremy Coles http://pprc.qmul.ac.uk/~lloyd/gridpp/metrics.html
[11:45:02] Ewan Mac Mahon In the fullness of time it would be good to actually put the money on the metrics page.
[11:49:31] Ewan Mac Mahon Rod is at RAL(PPD)
[11:49:37] Ewan Mac Mahon Er. Rob.
[11:49:39] Jeremy Coles Rob.
[11:49:43] Rob Harper
[11:49:44] Jeremy Coles I was asking Raul.
[11:49:52] Jeremy Coles Sorry Rob!
[11:49:53] Rob Harper Ah, sorry, misheard
[11:50:13] Ewan Mac Mahon Ah; Raul =/= RAL
[11:55:13] Govind Songara left
[11:55:31] Govind Songara joined
[12:04:03] Jeremy Coles http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
[12:04:41] Brian Davies http://www.gridpp.ac.uk/wiki/Suggestions_for_suitable_hardware_to_run_a_Grid_SE#Network_capabilities.
[12:09:31] Stephen Jones left
[12:09:38] Stephen Jones joined
[12:11:31] Stephen Jones left
[12:12:20] Raja Nandakumar Apologies - have to go.
[12:12:33] Raja Nandakumar left
[12:14:34] Stephen Jones joined
[12:15:42] Mohammad kashif left
[12:15:42] Brian Davies left
[12:15:43] David Crooks left
[12:15:44] Andrew McNab left
[12:15:44] Rob Fay left
[12:15:46] Govind Songara left
[12:15:46] Mark Mitchell left
[12:15:47] Mingchao Ma left
[12:15:48] Gareth Smith left
[12:15:48] Alessandra Forti left
[12:15:49] Stephen Jones left
[12:15:52] Wahid Bhimji left
[12:15:53] raul lopes left
[12:15:54] Ewan Mac Mahon left
[12:15:55] Rob Harper left
[12:16:00] Matthew Doidge left
[12:16:04] Queen Mary, U London London, U.K. left
[12:16:05] Santanu Das left
[12:16:34] Duncan Rand left
[12:16:39] Pete Gronbech left
[12:16:45] Stuart Purdie left
In attendance: Jeremy Coles (Chair), Brian Davies, Stuart Purdie, Raul Lopes, Matthew Doidge, Raja Nandakumar, Andrew McNab, Rob Harper, David Crooks, Catalin Condurache (minutes), Mark Mitchell, Gareth Smith, Mohammad Kashif, Stephen Jones, Stuart Wakefield, Ewan Mac Mahon, Alessandra Forti, Pete Gronbech, Santanu Das, Duncan Rand, Govind Songara, Mingchao Ma, Wahid Bhimji, Rob Fay
11:00
Meetings & updates (20')
- ROD team update
Andrew: nothing to report
- Nagios status
Kashif: nothing to report
- Tier-1 update
Gareth: fairly quiet week at RAL T1; some load issues on LHCb small service classes; plan to rollout new version of Castor client (2.1.10) on all worker nodes.
- GDB "timetable"
-- http://www.gridpp.ac.uk/wiki/GDB_reports
Jeremy: put names against various GDB meetings for this year; people asked to check their availability
- CA Technical Advisory Group update (see text attached)
Jeremy: the UK eScience CA certificate expires 2012. A new CA cert will be in operation by Sep 2011 (IGTF release). Sites will need to install two CA certificates for the CAs. More details will follow.
- Security update
--SSC5
--SSC4
Mingchao: SSC4 finished from site point of view; sites need to send the final report; any feedback from participants sites?
About SSC5: details not yet known; will be targeting sites that support ATLAS pilot analysis jobs.
- EGI
-- Still many UK sites with the EGEE tag. http://gstat.egi.eu/gstat/summary/GRID/EGEE/ . Sites on this list on Wednesday will receive a ticket.
Jeremy: re-tagging exercise: sites to change EGEE tag to EGI. Proposal to update by Thursday, followed up by tickets to each 'faulty' site.
-- Some EMI-1 components will reach staged rollout next week.
-- glite 3.2: glite-SGE_utils and CREAM 1.6.6: problem with glic update now understood by developers.
-- Site admin input requested for batch system survey: http://www.zoomerang.com/Survey/WEB22CE9DXKZDT/.
Stuart: sites asked to fill in the survey. Following the SA1 meeting (30 May), invitation have been sent to submit requirements re: middleware
-- gstat availability figure issues?
There are sites with 0% and -100% availability. Possible problems with site-BDII. Sites asked to report such anomaly.
- Open tickets:
http://tinyurl.com/3ulldrr
57746 - Camont issue with Cam-HEP. Condor related? More job outputs given but it needs a WMS expert to help.
Catalin and/or Daniela to have a look and try to help.
64995 - RAL T1 No GlueSACapability defined for WLCG Storage Areas. On hold but now progressing. Change ticket status? (still on hold).
https://ggus.org/ws/ticket_info.php?ticket=66306 - Creation of Irish NGI. Still "almost". Now problem with transmission of alarms to dashboards for ops.
https://ggus.org/ws/ticket_info.php?ticket=70093. hone issue. Needs wider input? (Any Tier-1 comment on this ticket?).
Catalin to update this ticket, although there is no clear what is the problem there.
Several other tickets on hold.
https://gus.fzk.de/ws/ticket_info.php?ticket=68853
Retirenment of SL4 and 32bit DPM Head nodes and Servers
Awaits end of SL4 gLite 3.1 T2 machines.
https://gus.fzk.de/ws/ticket_info.php?ticket=68858
Retirenment of SL4 and 32bit DPM Head nodes and Servers - Glasgow sub-ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=68859
Durham sub-ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=68865
UCL-HEP sub-ticket.
11:20
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
Raja: nothing to report
- CMS
Stuart: operationally OK. CMS wants to do some Whole Node scheduling testing at Tier1 (next 6 months). Also very limited testing at Tier2 sites.
Catalin: Will check with Derek about the Whole Node scheduling documentation
Pete: Also WN scheduling supported at Oxford for Atlas
- ATLAS
Alessandra: quiet week; 3 sites with transfer issues: QMUL - timeout due to checksum (maybe upgrade gridftp?), Manchester - one data server down, Birmingham - one VO overloading the Storage Element; 4 sites not on line: IC in test mode, UCL, LANC, CAM in automatic broker-off; 2 tickets for QMUL and MANC
Brian: similar checksum related at other sites, possibly gridftp overloaded; also transfer failures because of an US DN (it contains a "'" character within DN)
- Other
- Experiment blacklisted sites
- Experiment known events affecting job slot requirements
- Site performance/accounting issues
- Metrics review
Alessandra: What is meaning of the 'Points' columns in http://pprc.qmul.ac.uk/~lloyd/gridpp/metrics.html? How is it calculated (any formula)?
Jeremy: Will ask SteveL to update the documentation
11:40
Site issues/discussion (15')
- Current status across the sites.
Reminder of the glexec deployment wiki page https://twiki.cern.ch/twiki/bin/view/LCG/GlexecDeployment.
Matt - LANC - some pbs with ATLAS
Brian - looking into different failures modes for FTS
Rob - RALPP - nothing to report
David - Glasgow - nothing to report, keep an eye on the network traffic
Kashif - work on SAM Nagios 2nd level support
Ewan + Pete - Oxford - re-install a CREAM-CE; nothing else to report, running OK
Santanu - Cambridge - some issues with LHCb production and pilot on CREAM-CE
Stephen - implementing an alternate CREAM-CE
Chris - few more WNs to be put on-line
Wahid - power outage yesterday; couple of CREAM-CE showing some oddities
Alessandra - dealing with SONAR tests
Andrew - job submissions are fine; configuring ARGUS and glexec
Govind - glexec and Argus installed and in production;
11:55
Actions (05')
- http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
O-110516-01 - on hold
O-110516-02 - pending on Ireland process to finalise
O-110516-03 - done - results in the Storage wiki (http://www.gridpp.ac.uk/wiki/Suggestions_for_suitable_hardware_to_run_a_Grid_SE#Network_capabilities)
O-110516-04 - on going
O-110524-01 - to be updated by Jeremy
O-110524-03 - done
O-110524-05 - to be updated and closed by Alessandra
O-110524-06 - problem not resolved in production yet
O-110524-07 - no updates yet
12:00
AOB (01')
- WLCG workshop in July https://indico.desy.de/conferenceTimeTable.py?confId=4019#all. Let Jeremy know if you expected to go but have not heard back.
- HEPSYSMAN (30th June - 1st July) http://hepwww.rl.ac.uk/sysman/June2011/main.html
Gareth: Registration needed especially if accommodation needed
Pete: Topics for discussions and speakers are required
Jeremy: Also for GridPP in September
- Lustre workshop QMUL.
Stuart: Mon 4 - Tue 5 July
Pete: The Viglen Storage servers firmware issue. Following a recent firmware upgrade problems disappeared. What are other sites experiencing? Are they waiting for an official Adaptec firmware release?
Alessandra: Installed 10 days ago a server without problems. Will provide the version number on the mailing list.
Chat transcript:
[10:54:01] Brian Davies joined
[10:55:56] Stuart Purdie joined
[10:57:02] raul lopes joined
[10:57:46] Matthew Doidge joined
[10:58:27] Jeremy Coles joined
[10:58:44] Raja Nandakumar joined
[11:00:35] Andrew McNab joined
[11:00:36] Rob Harper joined
[11:00:53] David Crooks joined
[11:01:31] Jeremy Coles Catalin will take minutes today.
[11:01:41] Mark Mitchell joined
[11:01:41] Gareth Smith joined
[11:01:56] Mohammad kashif joined
[11:01:57] RECORDING Catalin joined
[11:02:18] Catalin Condurache i muted myself
[11:02:22] RECORDING Matthew joined
[11:02:54] Stephen Jones joined
[11:03:10] Stuart Wakefield joined
[11:03:26] Ewan Mac Mahon joined
[11:03:40] Alessandra Forti joined
[11:03:42] Pete Gronbech joined
[11:04:02] Stephen Jones left
[11:04:02] Santanu Das joined
[11:04:16] Stephen Jones joined
[11:04:24] Duncan Rand joined
[11:04:58] Stephen Jones my evo keeps crashing
[11:05:07] Queen Mary, U London London, U.K. joined
[11:07:02] Govind Songara joined
[11:08:05] Stephen Jones I'm fine with Nov. I'll let Rob know.
[11:08:21] Stephen Jones Cheers!
[11:13:04] Pete Gronbech will do
[11:14:44] Alessandra Forti SITE_OTHER_GRID="EGI|WLCG|GRIDPP|NORTHGRID"
[11:14:49] Alessandra Forti in site-info.def
[11:15:13] Alessandra Forti clearly replace northgrid with you region
[11:16:00] Pete Gronbech SITE_OTHER_EGEE_ROC="UK/I"
SITE_OTHER_EGEE_SERVICE="prod"
[11:16:13] Pete Gronbech what about these other entires
[11:16:27] Govind Songara i also want to know that
[11:16:32] Pete Gronbech Should they now be otherEGI_ROC
[11:18:27] Mingchao Ma joined
[11:20:09] Wahid Bhimji joined
[11:24:14] Rob Fay joined
[11:25:12] Jeremy Coles https://ggus.org/ws/ticket_info.php?ticket=70093
[11:26:15] Catalin Condurache ok
[11:27:55] Catalin Condurache i'll check with derek
[11:29:32] Catalin Condurache nope
[11:29:48] Alessandra Forti left
[11:29:53] Alessandra Forti joined
[11:39:15] Brian Davies ATLAS sites with failure to complete prepareto put were qmul,manchester,ucl,rhul
[11:40:01] Queen Mary, U London London, U.K. I wonder if they are having the same problem - but at a lower level.
[11:41:24] Jeremy Coles http://pprc.qmul.ac.uk/~lloyd/gridpp/metrics.html
[11:45:02] Ewan Mac Mahon In the fullness of time it would be good to actually put the money on the metrics page.
[11:49:31] Ewan Mac Mahon Rod is at RAL(PPD)
[11:49:37] Ewan Mac Mahon Er. Rob.
[11:49:39] Jeremy Coles Rob.
[11:49:43] Rob Harper
[11:49:44] Jeremy Coles I was asking Raul.
[11:49:52] Jeremy Coles Sorry Rob!
[11:49:53] Rob Harper Ah, sorry, misheard
[11:50:13] Ewan Mac Mahon Ah; Raul =/= RAL
[11:55:13] Govind Songara left
[11:55:31] Govind Songara joined
[12:04:03] Jeremy Coles http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
[12:04:41] Brian Davies http://www.gridpp.ac.uk/wiki/Suggestions_for_suitable_hardware_to_run_a_Grid_SE#Network_capabilities.
[12:09:31] Stephen Jones left
[12:09:38] Stephen Jones joined
[12:11:31] Stephen Jones left
[12:12:20] Raja Nandakumar Apologies - have to go.
[12:12:33] Raja Nandakumar left
[12:14:34] Stephen Jones joined
[12:15:42] Mohammad kashif left
[12:15:42] Brian Davies left
[12:15:43] David Crooks left
[12:15:44] Andrew McNab left
[12:15:44] Rob Fay left
[12:15:46] Govind Songara left
[12:15:46] Mark Mitchell left
[12:15:47] Mingchao Ma left
[12:15:48] Gareth Smith left
[12:15:48] Alessandra Forti left
[12:15:49] Stephen Jones left
[12:15:52] Wahid Bhimji left
[12:15:53] raul lopes left
[12:15:54] Ewan Mac Mahon left
[12:15:55] Rob Harper left
[12:16:00] Matthew Doidge left
[12:16:04] Queen Mary, U London London, U.K. left
[12:16:05] Santanu Das left
[12:16:34] Duncan Rand left
[12:16:39] Pete Gronbech left
[12:16:45] Stuart Purdie left
There are minutes attached to this event.
Show them.