Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly ops & sites meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area.
- The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 108203 with code: 4880.
Apologies: Jeremy
GridPP Operations team (& sites) meeting 11/10/11
In attendance: Pete Gronbech (Chair), Mark Mitchell, Mohammad Kashif, Mark Slater, Daniela Bauer, Andrew Washbrook, Sam Skipsey, Rob Fay, Elena Korolkova, David Crooks, Wahid Bhimji, Andrew McNab, Mingchao Ma, Brian Davies, Rob Harper, Mark Mitchell, Raja Nandakumar, Catalin Condurache (minutes), Chris Brew, Gareth Smith, Raul Lopes, Govind Songara, Santanu Das, Alessandra Forti, Stephen Jones, Duncan Rand
Apologies: Jeremy Coles, Matt Doidge
11:00
Meetings & updates (20')
- ROD team update
- Nagios status
Kashif: no issues
- Tier-1 update
Gareth:
problems on ATLAS Castor instance: file duplication - resolved this morning;
lcgce08 - certificate related problems last week - resolved on Friday;
network problems last week - resolved by flashing out the cache on few switches
FTS reconfiguration this morning;
- Security update
Mingchao:
incident with proxy user certificate made public (2 weeks ago); recommendation of 24 hrs lifetime for proxy and VOMS proxy certificates (http://osct.web.cern.ch/osct/op-notices/proxy-lifetime-02-11-2007.txt)
EGI CSIRT has released the security dashboard: https://operations-portal.egi.eu/csiDashboard; still in test phase
-- T2 issues
-- General notes.
There is a GDB tomorrow: https://indico.cern.ch/conferenceDisplay.py?confId=106649. Matt D is the Tier-2 rep for the UK so please let him know any points you want raised.
Checking Red/Amber tickets for NGI_UK:http://tinyurl.com/3endfg7
Or go to https://ggus.eu/ws/ticket_search.php and select Support Unit:NGI_UK and Creation date: Any and Status: open states - then click Go.
Checking Red/Amber tickets for ROC UKI: http://tinyurl.com/3ctjrna
GGUS#75172 - enable UK regional VOs in RAL LFC
need to understand the difference between localLfc and centralLfc
'scotgrid' and 'southgrid' want to be registered as 'centralLfc' in the RAL LFC
GGUS#74098 - CREAM Nagios tests failing (alice)
on hold - intermitent failures; need to check with Alice if can be closed
GGUS#74353 - Proxy not renewing properly from WMS vo = pheno
still in progress
11:20
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
Raja:
Extensive use of CPU, extending to all available T2s. Primary work is reprocessing. Also reconstruction. To be finished by XMas. Also small Monte Carlo simulation.
Request to extend LHCb queues to 2 days.
- CMS - n/a
- ATLAS
Alessandra: CVMFS business. Some T2s which do mot want to move yet to CVMFS should put tickets on hold. Needs to be done by end of 2011 though.
- Other
- Experiment blacklisted sites
- Experiment known events affecting job slot requirements
- Site performance/accounting issues
- Metrics review
11:40
Site roundtable (20')
- Quick check of what everyone is doing and what problems they are facing... starting with whoever is first in EVO!
Sam: LHCb queues extended to 2 days at Glasgow
Brian: need a list of all sites that have more than 1GB network link to superJanet.
12:00
AOB (1')
- Reminder that quarterly reports are now due!
Pete: Hepsysman - 10 Nov 2011 at QMUL
Gareth: registration page is available (http://hepwww.rl.ac.uk/sysman/Nov2011/main.html)
Update from Lancaster (Matt Doidge) sent by email:
=-=-=-=-=-=-=-
Quick Lancaster highlights:
We've deployed cvmfs without trouble, and we've asked atlas to "make the switch".
New CREAM CE up and running, we seem to have ironed out the last gotchas. No lcg-CE at the site anymore.
September availability numbers seem to be right (a little on the lowside due to a weekend power outage.
=-=-=-=-=-=-=-
Chat transcript:
[11:00:36] Mark Mitchell back in 5
[11:02:01] Mohammad kashif any one speaking ?
[11:03:17] Pete Gronbech alessandra are you able to take minutes?
[11:03:17] Mark Slater I hope not because if they are I can't hear them
[11:03:30] Daniela Bauer joined
[11:03:30] Pete Gronbech joined
[11:03:30] Andrew Washbrook joined
[11:03:31] Mark Slater joined
[11:03:31] Mohammad kashif joined
[11:03:32] Sam Skipsey joined
[11:03:32] Rob Fay joined
[11:03:32] Elena Korolkova joined
[11:03:33] David Crooks joined
[11:03:33] Wahid Bhimji Yes
[11:03:34] Wahid Bhimji joined
[11:03:35] Andrew McNab joined
[11:03:35] Mingchao Ma yes
[11:03:36] Brian Davies joined
[11:03:36] Rob Harper joined
[11:03:36] Mingchao Ma joined
[11:03:37] Mark Mitchell joined
[11:03:38] Raja Nandakumar joined
[11:03:48] Catalin Condurache yes
[11:04:00] Catalin Condurache ok
[11:04:10] Chris Brew joined
[11:04:16] RECORDING Catalin joined
[11:05:09] Elena Korolkova Can someone post a link to the agenda, please
[11:05:19] Gareth Smith joined
[11:05:55] Andrew McNab It's Stuart this week according to the wiki
[11:05:56] Rob Harper http://indico.cern.ch/conferenceDisplay.py?confId=153031
[11:06:09] Elena Korolkova thanks, Rob
[11:06:24] raul lopes joined
[11:09:04] Govind Songara joined
[11:09:32] Santanu Das joined
[11:16:41] Phone Bridge joined
[11:17:06] Mingchao Ma join via telephone, sorry about that
[11:19:58] Elena Korolkova I can't understand why we are forced to install cvmfs now.
[11:20:18] Elena Korolkova There atre still problem with cvmfs
[11:20:36] Elena Korolkova and there is no atlas official deadline for this
[11:20:43] Alessandra Forti you have to do it sooner or later. We decided to move within the end of the year. You can do it in december if you want
[11:21:32] Elena Korolkova Yes, I don't want to do this now because I don't have problem with atlas sw
[11:22:27] Alessandra Forti then reply to the ticket and put it on hold.
[11:22:33] Elena Korolkova I would prefer to wait till the beginning on November when more problem with cvmfs will be fixed
[11:23:18] Elena Korolkova I can install it at one wn, check and do not switch to it.
[11:24:43] Rob Fay sorry, I missed something there, done what?
[11:24:49] Alessandra Forti me too
[11:26:00] Elena Korolkova I've changed this for Sheffield
[11:31:23] Stephen Jones joined
[11:43:14] Stephen Jones Should we write up the use cases where local LHCs are used, and uses cases where cenral LFCs are used?
[11:43:55] Catalin Condurache at RAL: local "biomed zeus hone cdf dzero babar pheno t2k.org esr ilc magic fusion geant4 cedar gridpp camont na62.vo.gridpp.ac.uk cernatschool.org snoplus.snolab.ca dteam ops"
[11:44:06] Catalin Condurache central: "minos.vo.gridpp.ac.uk ralpp mice ngs.ac.uk neiss.org.uk"
[11:53:29] Duncan Rand brunel
[11:53:34] Duncan Rand 2 Gbps
[11:54:07] Alessandra Forti manchester has 10Gbs redundant from the site to jnet. 4gbps is from the nodes
[11:54:24] Wahid Bhimji ecdf has 2 I think - at weakest point
[11:54:41] Wahid Bhimji 2 Gbs I mean is the narrowest pipe
[11:54:47] Rob Fay Liverpool has 2x1Gbps right now
[11:55:17] Mark Mitchell Glasgow limited to 6 Gig max Realistic 4.5 Gig
[11:56:09] Mingchao Ma http://osct.web.cern.ch/osct/op-notices/proxy-lifetime-02-11-2007.txt
[11:57:59] Mingchao Ma https://operations-portal.egi.eu/csiDashboard
[11:58:12] Duncan Rand left
[11:59:34] David Crooks I get the same thing
[12:00:12] Govind Songara I can see critical nagios alert for both CE
[12:05:14] Mohammad kashif left
[12:05:58] Mark Slater left
[12:05:59] Raja Nandakumar left
[12:05:59] Elena Korolkova left
[12:05:59] David Crooks left
[12:06:01] Govind Songara left
[12:06:02] Andrew McNab left
[12:06:02] raul lopes left
[12:06:03] Rob Harper left
[12:06:03] Rob Fay left
[12:06:04] Phone Bridge left
[12:06:04] Mingchao Ma left
[12:06:05] Gareth Smith left
[12:06:06] Daniela Bauer left
[12:06:09] Wahid Bhimji left
[12:06:11] Chris Brew left
[12:06:12] Andrew Washbrook left
[12:06:19] Brian Davies left
[12:06:20] Stephen Jones left
In attendance: Pete Gronbech (Chair), Mark Mitchell, Mohammad Kashif, Mark Slater, Daniela Bauer, Andrew Washbrook, Sam Skipsey, Rob Fay, Elena Korolkova, David Crooks, Wahid Bhimji, Andrew McNab, Mingchao Ma, Brian Davies, Rob Harper, Mark Mitchell, Raja Nandakumar, Catalin Condurache (minutes), Chris Brew, Gareth Smith, Raul Lopes, Govind Songara, Santanu Das, Alessandra Forti, Stephen Jones, Duncan Rand
Apologies: Jeremy Coles, Matt Doidge
11:00
Meetings & updates (20')
- ROD team update
- Nagios status
Kashif: no issues
- Tier-1 update
Gareth:
problems on ATLAS Castor instance: file duplication - resolved this morning;
lcgce08 - certificate related problems last week - resolved on Friday;
network problems last week - resolved by flashing out the cache on few switches
FTS reconfiguration this morning;
- Security update
Mingchao:
incident with proxy user certificate made public (2 weeks ago); recommendation of 24 hrs lifetime for proxy and VOMS proxy certificates (http://osct.web.cern.ch/osct/op-notices/proxy-lifetime-02-11-2007.txt)
EGI CSIRT has released the security dashboard: https://operations-portal.egi.eu/csiDashboard; still in test phase
-- T2 issues
-- General notes.
There is a GDB tomorrow: https://indico.cern.ch/conferenceDisplay.py?confId=106649. Matt D is the Tier-2 rep for the UK so please let him know any points you want raised.
Checking Red/Amber tickets for NGI_UK:http://tinyurl.com/3endfg7
Or go to https://ggus.eu/ws/ticket_search.php and select Support Unit:NGI_UK and Creation date: Any and Status: open states - then click Go.
Checking Red/Amber tickets for ROC UKI: http://tinyurl.com/3ctjrna
GGUS#75172 - enable UK regional VOs in RAL LFC
need to understand the difference between localLfc and centralLfc
'scotgrid' and 'southgrid' want to be registered as 'centralLfc' in the RAL LFC
GGUS#74098 - CREAM Nagios tests failing (alice)
on hold - intermitent failures; need to check with Alice if can be closed
GGUS#74353 - Proxy not renewing properly from WMS vo = pheno
still in progress
11:20
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
Raja:
Extensive use of CPU, extending to all available T2s. Primary work is reprocessing. Also reconstruction. To be finished by XMas. Also small Monte Carlo simulation.
Request to extend LHCb queues to 2 days.
- CMS - n/a
- ATLAS
Alessandra: CVMFS business. Some T2s which do mot want to move yet to CVMFS should put tickets on hold. Needs to be done by end of 2011 though.
- Other
- Experiment blacklisted sites
- Experiment known events affecting job slot requirements
- Site performance/accounting issues
- Metrics review
11:40
Site roundtable (20')
- Quick check of what everyone is doing and what problems they are facing... starting with whoever is first in EVO!
Sam: LHCb queues extended to 2 days at Glasgow
Brian: need a list of all sites that have more than 1GB network link to superJanet.
12:00
AOB (1')
- Reminder that quarterly reports are now due!
Pete: Hepsysman - 10 Nov 2011 at QMUL
Gareth: registration page is available (http://hepwww.rl.ac.uk/sysman/Nov2011/main.html)
Update from Lancaster (Matt Doidge) sent by email:
=-=-=-=-=-=-=-
Quick Lancaster highlights:
We've deployed cvmfs without trouble, and we've asked atlas to "make the switch".
New CREAM CE up and running, we seem to have ironed out the last gotchas. No lcg-CE at the site anymore.
September availability numbers seem to be right (a little on the lowside due to a weekend power outage.
=-=-=-=-=-=-=-
Chat transcript:
[11:00:36] Mark Mitchell back in 5
[11:02:01] Mohammad kashif any one speaking ?
[11:03:17] Pete Gronbech alessandra are you able to take minutes?
[11:03:17] Mark Slater I hope not because if they are I can't hear them
[11:03:30] Daniela Bauer joined
[11:03:30] Pete Gronbech joined
[11:03:30] Andrew Washbrook joined
[11:03:31] Mark Slater joined
[11:03:31] Mohammad kashif joined
[11:03:32] Sam Skipsey joined
[11:03:32] Rob Fay joined
[11:03:32] Elena Korolkova joined
[11:03:33] David Crooks joined
[11:03:33] Wahid Bhimji Yes
[11:03:34] Wahid Bhimji joined
[11:03:35] Andrew McNab joined
[11:03:35] Mingchao Ma yes
[11:03:36] Brian Davies joined
[11:03:36] Rob Harper joined
[11:03:36] Mingchao Ma joined
[11:03:37] Mark Mitchell joined
[11:03:38] Raja Nandakumar joined
[11:03:48] Catalin Condurache yes
[11:04:00] Catalin Condurache ok
[11:04:10] Chris Brew joined
[11:04:16] RECORDING Catalin joined
[11:05:09] Elena Korolkova Can someone post a link to the agenda, please
[11:05:19] Gareth Smith joined
[11:05:55] Andrew McNab It's Stuart this week according to the wiki
[11:05:56] Rob Harper http://indico.cern.ch/conferenceDisplay.py?confId=153031
[11:06:09] Elena Korolkova thanks, Rob
[11:06:24] raul lopes joined
[11:09:04] Govind Songara joined
[11:09:32] Santanu Das joined
[11:16:41] Phone Bridge joined
[11:17:06] Mingchao Ma join via telephone, sorry about that
[11:19:58] Elena Korolkova I can't understand why we are forced to install cvmfs now.
[11:20:18] Elena Korolkova There atre still problem with cvmfs
[11:20:36] Elena Korolkova and there is no atlas official deadline for this
[11:20:43] Alessandra Forti you have to do it sooner or later. We decided to move within the end of the year. You can do it in december if you want
[11:21:32] Elena Korolkova Yes, I don't want to do this now because I don't have problem with atlas sw
[11:22:27] Alessandra Forti then reply to the ticket and put it on hold.
[11:22:33] Elena Korolkova I would prefer to wait till the beginning on November when more problem with cvmfs will be fixed
[11:23:18] Elena Korolkova I can install it at one wn, check and do not switch to it.
[11:24:43] Rob Fay sorry, I missed something there, done what?
[11:24:49] Alessandra Forti me too
[11:26:00] Elena Korolkova I've changed this for Sheffield
[11:31:23] Stephen Jones joined
[11:43:14] Stephen Jones Should we write up the use cases where local LHCs are used, and uses cases where cenral LFCs are used?
[11:43:55] Catalin Condurache at RAL: local "biomed zeus hone cdf dzero babar pheno t2k.org esr ilc magic fusion geant4 cedar gridpp camont na62.vo.gridpp.ac.uk cernatschool.org snoplus.snolab.ca dteam ops"
[11:44:06] Catalin Condurache central: "minos.vo.gridpp.ac.uk ralpp mice ngs.ac.uk neiss.org.uk"
[11:53:29] Duncan Rand brunel
[11:53:34] Duncan Rand 2 Gbps
[11:54:07] Alessandra Forti manchester has 10Gbs redundant from the site to jnet. 4gbps is from the nodes
[11:54:24] Wahid Bhimji ecdf has 2 I think - at weakest point
[11:54:41] Wahid Bhimji 2 Gbs I mean is the narrowest pipe
[11:54:47] Rob Fay Liverpool has 2x1Gbps right now
[11:55:17] Mark Mitchell Glasgow limited to 6 Gig max Realistic 4.5 Gig
[11:56:09] Mingchao Ma http://osct.web.cern.ch/osct/op-notices/proxy-lifetime-02-11-2007.txt
[11:57:59] Mingchao Ma https://operations-portal.egi.eu/csiDashboard
[11:58:12] Duncan Rand left
[11:59:34] David Crooks I get the same thing
[12:00:12] Govind Songara I can see critical nagios alert for both CE
[12:05:14] Mohammad kashif left
[12:05:58] Mark Slater left
[12:05:59] Raja Nandakumar left
[12:05:59] Elena Korolkova left
[12:05:59] David Crooks left
[12:06:01] Govind Songara left
[12:06:02] Andrew McNab left
[12:06:02] raul lopes left
[12:06:03] Rob Harper left
[12:06:03] Rob Fay left
[12:06:04] Phone Bridge left
[12:06:04] Mingchao Ma left
[12:06:05] Gareth Smith left
[12:06:06] Daniela Bauer left
[12:06:09] Wahid Bhimji left
[12:06:11] Chris Brew left
[12:06:12] Andrew Washbrook left
[12:06:19] Brian Davies left
[12:06:20] Stephen Jones left
There are minutes attached to this event.
Show them.