Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234.
To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +442030510622
-- The meeting extension is 9308582.
Apologies: Ian Neilson.
Operations Minutes 3/11/2015
============================
Present:
A. Forti
A. Washbrook
A. McNab
B. Davies
C. Brew
D. Traynor
D. Bauer
D. Crooks
D. Rand
E. Korolkova
E. MacMahon
F. Melaccio
G. Roy
G. Smith
G. Stewart
G. Songara
I. Loader
J. Coles
J. Bland
J. Hill
K. Mohammad
M. Doidge
O. Smith
P. Lacesso
P. Gronbech
R. Nandakumar
R. Lopez
R. Frank
S. Skipsey
S. Jones
T. Whyntie
1) Experiments
==============
LHCb - DURHAM had some failed pilots, which has been fixed
- Some data that was stuck at Oxford and could be deleted.
- An LHCb user had submitted a multithreaded job which had been causing some issues.
Atlas - ASAP metrics for October (portal down so from observation)
- RALPP low (<80%)for this month (likely due to SE issues reported below with CMS jobs).
- PRODDISK has been decommissioned all files left are dark and can be deleted.
- Deleting data can be time consuming due to a number of small files.
- QMUL had questioned why there is no production work, this was due to a spacetoken being blacklisted
CMS - Upcoming reset of Data transfer password
- RALPP reported as underpar, Chris states CMS are submitting a high rate of jobs (mini AOD processing, with merge)
which is killing the SE. It was affecting other VOs so they have rate limited CMS jobs to try to mitigate.
Other VO
========
DEAP3600 (pend) - New VO: Dark Matter studies
- Would like to mirror DB from Canada (CouchDB)
- Discussion about service levels
- Researchers present at RHUL, Sussex and RAL
HPC DiRAC - Started bulk copying
- Looking at methods for providing a file-list with checksums
- PG asks if they were trying to add Leicester as an additional transfer site (no GridPP assistance at site)
- Lydia is handing over data movement to Durham to another member of staff to assist Leceister
LIGO - no report
LOFAR - no report
LSST - no report
LZ - Everything is running fine, a big production run of MC had gone well at Imperial.
- Using Disk storage at Imperial, with good instructions provided by the team there.
- Production files also copied to US sites as well
UKQCD - Craig taking over ownership of the UKQCD VO
UCLAN - “No update from GalDyn - still writing up”
PRaVDA - no report
Ewan comments about OPS action regarding new VOs being moved to the PMB. Jeremy states he needs to create a proposal to present to the PMB to move things forward.
DIRAC (WMS) - Things are working as before.
- Duncan looked at sites missing from the reporting (Brunel). There seems to be an internal issue where pilots
not being sent.
- VAC sites need and updated version of the pilot to work with the new version of DIRAC
- Duncan asks for sites to check if there are stuck GridPP pilots and if so remove them.
Meetings and Updates
====================
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
* Upcoming GDB Agenda
* Brian - Jens had a question about having multiple hosts behind an CNAME style alias, requiring a different certificate. Brian asks if this is an issue for other sites.
- Robert comments that VOMS has this issue
* Nagios GLEXEC timeouts, Kashif is looking into the issue and subscribed to the ticket. Ewan asks if this is an actual issue or a test artefact, Kashif states only QMUL is timing out so the problem may be something else. Daniel sees other VOs succeeds so wonders if that is the case. Kashif states more investigation is needed.
* HW spend almost finalised, money to be spent this FY
* Upcoming DPM workshop. Sam states he will be presenting the UK status, and will be asking for input from the community.
* Tier 2 A/R October figures out this month, Jeremy asks if contacted sites could send responses.
* VM only sites A/R need to be implemented.
* RCUK Cloud Workshop 1st December - Brian notes that Tier-1 staff found the WorkShop was already full.
* MB update - 2GB/core baseline but 4GB/core is useful for some Workflows
- Working groups for technology evolution, workshops taking place in February in Lisbon (https://indico.cern.ch/event/433164/)
- Pre-commercial procurement and HNSciCloud
WLCG Ops Co-Ordination
=======================
- No updates
Tier-1
======
- Issues: - some high load in the Atlas castor tape system
- more ongoing work still required to remove old core switch, this will mean a scheduled downtime
- Gareth states that the interruptions should be very short.
Storage & Data Management
=========================
- No updates
Tier-2 Evolution
================
- Adding GOCDB entries for resource BDII so sites can run without a site BDII (in particular VAC and VCYCLE sites)
- T2C tests, Ewan states they have been developing plans attempting to run CPU jobs at Oxford and pulling data from elsewhere. The plan is to use Atlas workloads. Sam states there are a few wrinkles in configuring a site that doesn’t care where it gets it’s data from. Proposal is for Atlas to create a pseudo site to allow tests to happen.
- Jeremy states there is a number of tickets open in JIRA without many completed. Andrew states these are being used to map out strategy and plans
- Raja makes a comment about the T2C plans and that LHCb has not tested streaming data into a site, which has not worked well in the past. Sam states that the Atlas testing will copy data rather than streaming. Although Sam would like to try streaming to see what the issues would be. Ewan states streaming is interesting as third parties will not have the ability to stage data (HPC with diskless CPU resources). Raja states this would be down the list of priorities from LHCb. Discussion ensues about Tier model and serving data to T2C style sites.
Accounting
==========
- Some Delays in APEL accounting.
Documentation
=============
- Jeremy states query is failing in bringing up the keydocs page.
Interoperation
==============
- Next meeting Monday 9th.
Monitoring
==========
- No update
On-Duty
=======
- UCL availability alarm, VM sites not well tested.
- Tickets:
- Five for availability / reliability: Sussex, Sheffield, Liverpool, Lancaster and UCL.
- Two for GLUE2 validation: Liverpool and QMUL. One for the CEs at QMU
Rollout
=======
- No update
Security
========
- Java vulnerability, no specific cause for concern but some advice to update Java
Services
========
- No updates
Tickets
=======
SUSSEX
======
116915 (14/10) - no update
116865 (12/10) - no update
*Aside* Raja comments, problems with central banning that caused issues on the CERN batch farm and CE systems. Question was that every site should have seen the issue but wasn’t. Problem occurred 28th October.
RALPP
=====
117261 (28/10) - waiting for reply, Elena will look more carefully, Analysis queue is still in test mode. Chris thinks it’s correlated to CMS SE issues.
BRISTOL
=======
116775 (6/10) - no update
117303 (30/10) - no update
ECDF
====
95303 (Long long ago) - tarball glexec not available.
SHEFFIELD
=========
114460 (18/6) - Elena looking in to it.
116560 (30/9) - Waiting for reply, currently jobs don’t look to be failing.
116967 (17/10) - Elena had question about the need for a rolling 30 day figure. On Hold
LANCASTER
=========
116478 (28/9) - no update
116882 (13/10) - no update
95299 (Far far away) - rpath being a last hurdle to overcome
QMUL
====
117151(23/10) - waiting for response from network team
117011 (19/10) - can be closed
116689 (6/10) - Nagios glexec ticket (see discussion above)
BRUNEL
======
117324 (2/11) - no update
100IT
=====
116358 (22/9) - no update
THE TIER 1
==========
116866 (12/10) - Ticket being pushed by Gareth
116864 (12/10) - Ticket being pushed by Gareth
117171 (24/10) - Could be closed
117277 (30/10) - Ticket being pushed by Gareth
117248 (28/10) - Discussed Earlier
Discussion
==========
Site efficiency
===============
* Difference between Wallclock and Wallclock*number of CPUs. Alessandra states this has come up a number of times in the past 2 years, and John Gordon had promised that the development portal would go into production “soon”. See Johns slides here https://indico.cern.ch/event/393617/.
HEPSYSMAN
=========
* Discussion about timing, location and topics.
* Hosted in Manchester with likely dates 14th-15th January.
* Suggested talks and topics via email.
Actions
=======
* GridPP Document storage and categorisation - no update
* HTTP TF SAM Test organising for volunteers - in progress update after task force meeting next week
* Get SAM Tests working on new Dirac Server - progress ongoing see above discussion
* Put Northgrid VO and other regionals in VO Nagios project - Kashif states no progress, asks whether it is useful. Alessandra comments that she believes it would be useful as it can be used for testing.
Chat Logs
=========
Daniela Bauer: (03/11/2015 11:03)
I can't hear anyone.
I'll try reconnecting
Chris Brew: (11:05 AM)
It it correllated with that
Samuel Cadellin Skipsey: (11:07 AM)
Yeah, the problem is that there's a lot of small files and other metadata stuff.
John Bland: (11:08 AM)
Ours has been going for over about 2 days so far and is probably half way through
Alessandra Forti: (11:08 AM)
remember to delete the space token too
Daniela Bauer: (11:08 AM)
My VIdyo keeps dying :-(
Chris Brew: (11:10 AM)
I'm taking the opportunity to test find <dir> -type d -empty -delete to remove empty directory trees.
Alessandra Forti: (11:13 AM)
on dcache?
Jeremy Coles: (11:14 AM)
Gareth is taking minutes today. Thanks!
Matt Doidge: (11:15 AM)
checking on my atlas prod disk deletion we stalled - it's trying to delete a file that was on a retired disk server, and the recursive rfrm stops.
Chris Brew: (11:15 AM)
Yes, on the NFS mount. I'm not sure I'll every be brave enough to run it on a area not scheduled for deletion.
Tom Whyntie: (11:17 AM)
No update from GalDyn - still writing up
Ewan Mac Mahon: (11:17 AM)
Is moving to Leicester maybe a good time to get Dirac to start using tarballs?
Alessandra Forti: (11:20 AM)
@Matt yes, if you have the DB-disks not syncronised or some overloaded data servers it can happen that rfrm needs to be restarted.
Matt Doidge: (11:23 AM)
Thought so, is there a good way of removing the "speed bump" file? I'm thinking of a dpns-rm -a on it?
Ewan Mac Mahon: (11:24 AM)
Not sure, but there's always database hackery.
Matt Doidge: (11:24 AM)
The dpns-rm -a seems to have worked.
Ewan Mac Mahon: (11:24 AM)
You could also try mv-ing it to another location as a bit of a temporary hack - it'd still exist, but out of the way of your recursive rm.
But never mind.
Matt Doidge: (11:26 AM)
Disappointingly easy fixes are disappointing.
Samuel Cadellin Skipsey: (11:26 AM)
(and yes, the dpns-rm -a was the "correct" approach, officially ;) )
Robert Wolfgang Frank: (11:27 AM)
yes we do
voms being one
Alessandra Forti: (11:33 AM)
yes dpns-rm -a is what I use when I ync db and disks
sync
https://indico.cern.ch/event/433164/
Ewan Mac Mahon: (11:43 AM)
Though I'm not completely sure whether LHCb do this for all their workload, or just all the work that they currently run at Oxford.
Arguably it doesn't matter though - they have a way to use T2C type resources, even if they also need some 'normal' sites as well.
Whereas for ATLAS and CMS it's a bit less certain how well they can use a CPU only resource without a lot of special-casing.
Samuel Cadellin Skipsey: (11:45 AM)
Indeed, we've essentially been mostly used as T2C resources for LHCb for the history of GridPP.
Chris Brew: (11:47 AM)
my unmute button has dissapeared
Streaming might work for "federated" sites
i.e. from RALPP to Oxford
Samuel Cadellin Skipsey: (11:48 AM)
Indeed, I was wondering if the trade-off between "not copying all the file for sparse access" actually would look "okay" for close enough sites.
Chris Brew: (11:48 AM)
If we had some way of designating "parent" sites
Samuel Cadellin Skipsey: (11:48 AM)
HTTP DynaFed can do that, but noone uses it :)
Chris Brew: (11:49 AM)
I can ask for it :)
Matt Doidge: (11:50 AM)
I thought I fixed our accounting yesterday but no joy
Ewan Mac Mahon: (11:50 AM)
You might have done - IME it takes a couple of days or so after fixing something at a site for the results to trickle through to the point where they're visible.
Matt Doidge: (11:51 AM)
True that, ta.
Federico Melaccio: (11:54 AM)
no
Matt Doidge: (11:55 AM)
Sorry, working from home again and for some reason vidyo doesn't like my laptop Mic
Although my biological audio systems aren't functioning very well either!
John Hill: (11:57 AM)
We didn't see the problem because a re-yaim had removed the central banning configuration!
It's back in now
Matt Doidge: (12:00 PM)
Bristol keep getting tickets for the same issue
I've been talking a lot to the glexec devs
It's been a bit of a rabbit hole to jump down, having library problems at the moment that need fixing.
That one wsa closed this morning
On top of my technical issues the bulletin wouldn't let me edit it.
Gareth Smith: (12:08 PM)
Apologies. I have to leave now.
Ewan Mac Mahon: (12:09 PM)
Enablement is so a word, btw: http://www.oed.com/view/Entry/61509
Matt Doidge: (12:09 PM)
My spellchecker lied to me!
Alessandra Forti: (12:11 PM)
it's me
Daniel Peter Traynor: (12:12 PM)
I guess this is important fo the new metrics? http://pprc.qmul.ac.uk/~lloyd/gridpp/metrics2.html
Ewan Mac Mahon: (12:14 PM)
From where is the 'number of CPUS on which it ran' figure got? Given that things don't always run on the number of CPUs that they ask for.
Jeremy Coles: (12:17 PM)
https://indico.cern.ch/event/393617/
Paige Winslowe Lacesso: (12:18 PM)
Sorry, must - bye for now!
Ewan Mac Mahon: (12:18 PM)
I'm not seeing that there's a great deal of complexity here. The bottom line seems to be that the figures in the production portal are known to be bad, so no-one should use them.
Alessandra Forti: (12:19 PM)
https://twiki.cern.ch/twiki/bin/view/LCG/WLCGOpsMinutes151001
Ewan Mac Mahon: (12:19 PM)
At some point in the future there will be a new portal that fixes things, but in the meantime the people that consume the figures need to just live without.
Daniel Peter Traynor: (12:22 PM)
the day again?
stft computing insight is the 8 and 9th in coventry
Samuel Cadellin Skipsey: (12:23 PM)
and the DPM Workshop is 8,9th too
Ewan Mac Mahon: (12:24 PM)
The non-MEW is the 8th-9th
Federico Melaccio: (12:24 PM)
I think it is useful
Ewan Mac Mahon: (12:24 PM)
Oh, What Dan said.
It's definitely useful in principle.
And the next HPC-SIG seems to be in February.
So I'm not sure what I was thinking of, but it wasn't that.
Daniel Peter Traynor: (12:25 PM)
it was QM
Ewan Mac Mahon: (12:26 PM)
Somewhere more northerly would be good.
Federico Melaccio: (12:26 PM)
agree with Ewan
Ewan Mac Mahon: (12:26 PM)
Durham.
And also maybe push it to January?
December tends to get a bit busy with stuff.
It's useful - seems like a no-brainer to me.
Even if the VOs aren't being used much right now, it's worth running the tests so that they work /before/ we point new users at them.
Andrew McNab: (12:32 PM)
got to go too, bye
raul: (12:33 PM)
Sorry, got to leave now
Daniela Bauer: (12:33 PM)
bye
Govind: (12:33 PM)
sorry go to leave now
bye
Federico Melaccio: (12:35 PM)
bye
Tom Whyntie: (12:35 PM)
Thanks, bye
There are minutes attached to this event.
Show them.