Operations team & Sites

Tuesday 29 May 2012, 11:00 → 12:16 Europe/London

EVO - GridPP Operations team meeting

- This is the biweekly ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 126540 with code: 4880. Apologies: Duncan,

text Atlas_report_29May2012.txt

GridPP Operations team (& sites) meeting 29/05/12

In attendance: Jeremy Coles (Chair), Catalin Condurache, Elena Korolkova, Daniela Bauer, Gareth Roy, Brian Davies, Wahid Bhimji, Stuart Purdie, Emyr James, John Bland, Stephen Jones, David Crooks, Rob Harper, Mark Mitchell, Matthew Doidge, Raja Nandakumar, Sam Skipsey, Mohammad kashif, Mark Norman, Gareth Smith, Mark Slater, Chris Brew, Andrew McNab, Ewan Mac Mahon, John Kelly, Duncan Rand, raul lopes, Mingchao Ma, Pete Gronbech, Ian Collier, Govind Songara, John Gordon

11:00         
Experiment problems/issues (20')     
Review of weekly issues by experiment/VO

- LHCb
Raja: Nothing to report. Data taking and processing in UK

- CMS - nothing to report

- ATLAS
Elena: couple of problems at Glasgow, both ticketed, resolved on Friday and Monday

- Other

11:20         
Meetings & updates (20')     
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest

- GOCDB problems over weekend - databases issues, probably sorted out

- Tier-1 status
  . siteBDII issue last Saturday, host removed from alias

- Accounting
- Documentation
  . StephenJ: documentation re: GridPP approved Vos updated

- Interoperation
  . Stuart: no updates; meeting on Fri 1 June; probably EMI-2 discussions

- Monitoring
  . Sussex being certified

- On-duty
- Rollout
- Security
  . Mingchao: security update re: CERN incident to come. GOCDB was down last Fii - Mon

- Services
  . MarkM: perfsonar - deployment - deadline July; more checks for next week
  . Brian: only Oxford and QMUL are publishing

- Tickets
- Tools
- VOs
  . Sussex trying to be certified

- Site updates

 11:40         
CHEP/Workshop feedback (20')     
- From those who attended
ChrisW: "decommissioning" of WMS and LFC - end of 2012
JohnG: it means LHC VOs won't depend on WMS; small VOs will still use it
SamS: glexec future - it should continue to be installed

12:00         
VOMS and VO membership suspensions (5')     

JohnG (via email): "The helpdesk people are working on adding the relevant email addresses for sites to GOCDB. If this results in any duplicate GGUs update mails on existing tickets, we apologise. I'll alert you when this is ready and we are ready to adopt the new workflow discussed a couple of weeks ago."

- Discussion about the causes
Jeremy: the incident - users required to renew membership agreement at short notice (only 24 hrs); probably an earlier message has been missed (or it didn't go through channels)

- Ongoing problems to be resolved
Ewan: still problems with SouthGrid; what is the grace period for signing AUP (should be increased)
Jeremy: will look into the still possible problems

12:05         
AOB (1')
- No meeting next week - Tue 5 June


EVO Chat window
=============
[11:01:02] Elena Korolkova joined
[11:01:02] Daniela Bauer joined
[11:01:02] Gareth Roy joined
[11:01:03] Brian Davies joined
[11:01:03] Jeremy Coles joined
[11:01:04] Wahid Bhimji joined
[11:01:04] Stuart Purdie joined
[11:01:05] Emyr James joined
[11:01:06] John Bland joined
[11:01:06] Stephen Jones joined
[11:01:07] David Crooks joined
[11:01:08] Rob Harper joined
[11:01:09] Mark Mitchell joined
[11:01:09] Matthew Doidge joined
[11:01:09] Raja Nandakumar joined
[11:01:10] Sam Skipsey joined
[11:01:11] Mohammad kashif joined
[11:01:11] Mark Norman joined
[11:01:47] Elena Korolkova left
[11:01:58] RECORDING Catalin joined
[11:02:01] Gareth Smith joined
[11:02:07] Daniela Bauer CMS: Stuart is on holiday and Dave is sick.
[11:02:39] Mark Slater joined
[11:02:54] Chris Brew joined
[11:02:55] Elena Korolkova joined
[11:03:04] Jeremy Coles Catalin is taking minutes. We'll start with general updates.
[11:03:08] Andrew McNab joined
[11:04:18] Ewan Mac Mahon joined
[11:05:24] John Kelly joined
[11:05:28] Jeremy Coles https://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest#
[11:05:39] Duncan Rand joined
[11:06:23] raul lopes joined
[11:07:38] Mingchao Ma joined
[11:08:58] Pete Gronbech joined
[11:13:15] Stephen Jones All: Here are the updated VOMS records, taken direct from CIC Portal:
[11:13:16] Stephen Jones https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs
[11:14:06] Ian Collier joined
[11:14:51] Govind Songara joined
[11:16:43] Daniela Bauer I can't hear you!!!
[11:22:13] Ewan Mac Mahon It's not just you.
[11:22:59] Elena Korolkova ucUCL was not the only site with this problem but there were other sites.
[11:23:10] Jeremy Coles Which others?
[11:23:38] Elena Korolkova These jobs are jobs subitted from the test factory and atlas experts investigating the problem now
[11:24:16] Elena Korolkova IMO. ticket should not be keept open as theseb are test jobes
[11:25:31] Elena Korolkova It was also Liv and couple of other sites. For some reason I haven't seen Sheffield at this factory
[11:25:58] Elena Korolkova Matt is very quiet
[11:27:50] Elena Korolkova https://ggus.eu/ws/ticket_info.php?ticket=82296 is not ATLAS TEAM ticket and so it is not followe by atlas teAM
[11:29:09] Stephen Jones Elena: At liverpool, the "glideins" had been submitted to a non-production CE, hepgrid3. It has been out of use in GOCDB for years.
[11:29:26] Elena Korolkova In ECDF there were oinly specific WNs
[11:29:36] Elena Korolkova not all WNs were affected
[11:30:22] Wahid Bhimji Ineed as I say we have some WNs that are hyperthreaded but don't have a lot of local disk space - it wasn't previously a problem. I can reduce the slot count but seems a shame as its normally fine.
[11:30:28] Elena Korolkova Steve, I'll check this
[11:30:38] John Gordon II joined
[11:30:49] Wahid Bhimji so was first investigating whether there was stale stuff not cleaned up after job completes or a different reason
[11:30:50] Ewan Mac Mahon Wahid - could you make those nodes production jobs only?
[11:31:01] Wahid Bhimji It is production jobs that cause the problem!
[11:31:11] Ewan Mac Mahon Oh! I was assuming analysis.
[11:31:45] Elena Korolkova prod jobs have big input files now
[11:32:09] Christopher Walker joined
[11:32:30] Wahid Bhimji yes only certain prod jobs though - but there is no way to ensure one mixes the different types (as they come to the same pilots)
[11:32:39] Ewan Mac Mahon Make them analysis only then  
[11:33:03] Wahid Bhimji hah indeed that is a choice ...
[11:34:11] Elena Korolkova perhaps we can ask about possibility to flag reco+digi jobs which require more space on WNs and more memory
[11:34:45] Elena Korolkova (separate pilot?)
[11:34:57] Wahid Bhimji that would be useful - the memory part has the same issue - there are sites that overcommit much more than ECDF
[11:35:45] Elena Korolkova I'll write to atlas experts
[11:37:53] Chris Brew I'm not aware of any
[11:38:12] Chris Brew other than they messed up their CVMFS area for a few days
[11:38:35] Jeremy Coles Do we know how that happened?
[11:38:40] John Kelly left
[11:41:24] Emyr James left
[11:41:31] Pete Gronbech sound is braking up aain
[11:42:07] Catalin Condurache much better
[11:43:00] Wahid Bhimji not for CVMFS?
[11:43:11] Wahid Bhimji I'm confused now too.....
[11:43:20] Ewan Mac Mahon I think the default size of the cvmfs local cache is 20GB
[11:43:22] Elena Korolkova Atlas is asking for 20 GB
[11:44:12] Elena Korolkova this is split as 15 for input (parameter in panda) and output 5 GB
[11:44:22] Ewan Mac Mahon Well, mine's set to 20 per repo, and I don't remember picking that myself; I copied it from somewhere. Quite happy to take your word for it that it's OTT, but I got it from someone at some point.
[11:46:07] Sam Skipsey https://ggus.eu/ws/ticket_info.php?ticket=81498
[11:46:13] Jeremy Coles Thanks
[11:46:26] Elena Korolkova Some jobs require input set to 20 GB. So if site has >25 GB we can change parameter for input files and the site can run these jobs
[11:46:48] Ewan Mac Mahon Elena - but that's scratch space, not cvmfs cache, surely?
[11:47:02] Ewan Mac Mahon And the more space spent on cvmfs caches, the less is left for scratch.
[11:47:26] Elena Korolkova ATM such sites are RAL, Manchester, QM, PALPP, Oxford,
[11:47:44] Elena Korolkova this is scratch/tmp used by job
[11:47:54] Jeremy Coles https://www.gridpp.ac.uk/deployment/support/index.html
[11:47:58] Sam Skipsey Elena - there's some confusion here, as Ewan notes. There *is* an increasing requirement for scratch space per job, but there's *also* a 20-25GB "requirement" for space in the CVMFS repo for atlas
[11:48:09] Ian Collier Elena, What do you mean by 'such sites'?
[11:48:12] Sam Skipsey (I use quotes because I really dont believe this)
[11:49:21] Ewan Mac Mahon Ian: ATLAS have an internal config that says how much scratch space they expect to be able to use per job at a site; that seems to have been set to 14GB for virtually everyone for ages, and it's now being upped to 20GB once people confirm that it's OK.
[11:49:46] Ewan Mac Mahon Since many of us seem to have been buying to a spec of 50GB per slot for years, it seems to be mostly OK.
[11:50:04] Ian Collier Ah, so 'such sites' are teh ones that have said they have more scratch space available.
[11:50:16] Ewan Mac Mahon That's my understanding, yes.
[11:50:24] Elena Korolkova Ewan is right, 14 GB is for majority of T2s
[11:50:25] Sam Skipsey Yes, this is the case.
[11:50:36] Stephen Jones Who do we ask for one set of unambiguous ATLAS Requirements? Who collects their requirements?
[11:50:43] Ewan Mac Mahon There were some emails on tb-support last week or so, sith various people OKing the change.
[11:50:43] Sam Skipsey But basically everyone has always assumed that they need to provide 25GB+ per job.
[11:50:49] Sam Skipsey Since forever.
[11:51:02] Sam Skipsey (I was surprised that ATLAS still used the 14GB figure)
[11:51:20] Elena Korolkova If you think you can provide 25 GB, I'll change panda config
[11:52:01] Elena Korolkova If you can't, atlas is not requiring 25 GB
[11:52:13] Elena Korolkova ATLAS asks for 20 GB
[11:52:35] Sam Skipsey Right, my point was that, as Ewan noted, we've all been assuming you wanted more than that for age.
[11:52:38] Sam Skipsey +s
[11:53:30] Elena Korolkova no. If glashow has 25 GB per job slot I'll change maxinputsize for Glasgow queues
[11:54:45] Elena Korolkova Sam, pls let me know
[11:55:32] Sam Skipsey Yes, we can easily manage 25GB/slot
[11:55:46] Sam Skipsey I would be surprised if a site couldn't.
[11:56:17] Elena Korolkova ok. I'll change this for Glasgow
[11:58:44] Ewan Mac Mahon Elena: You should be able to set us to 25GB too, in fact I think 30GB would be a reasonable limit for us, if the extra would be useful.
[11:58:53] Brian Davies Just to clarify, my understanding that the responses to the TB-support mail thread "disk space on WN fo ratlas jobs" was regarding scratch space not cvmfs space. If people responded differently please e-mail me.
[12:00:31] Wahid Bhimji I am writing a blog on the storage list (finally) - so that will contain my summary of interesting things at chep
[12:01:16] Wahid Bhimji standards .... and xrootd   
[12:02:16] Elena Korolkova yes, it was disk space for atlas jobs
[12:02:16] Jeremy Coles Workshop https://indico.cern.ch/conferenceDisplay.py?confId=146547
[12:05:58] Christopher Walker EVO looks like it will have gone by the end of the year
[12:07:28] Wahid Bhimji I did clairfy with Romain on the security TEG glexec position. The requirement is for sites to provide fine grained traceability not necessarily to install glexec though the group did not know of any other current way to satisfy the requirement.
[12:08:02] Wahid Bhimji There was also some discussion on whether the requirement amounted to requiring identity switching
[12:09:01] Wahid Bhimji Probably a bit premature to comment

11:00 → 11:20 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS - Other

11:20 → 11:40 Meetings & updates

With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest - Tier-1 status - Accounting - Documentation - Interoperation - Monitoring - On-duty - Rollout - Security - Services - Tickets - Tools - VOs - SIte updates

11:40 → 12:00 CHEP/Workshop feedback

- From those who attended

12:00 → 12:05 VOMS and VO membership suspensions

- Discussion about the causes - Ongoing problems to be resolved

12:05 → 12:06 AOB