Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234.
To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +442030510622
-- The meeting extension is 9308582.
Apologies:
GridPP Operations Meeting
Tuesday, 6 May 2014
Attendees:
Alessandra Forti, Andrew Lahiff, Andrew McNab, Chris Brew, Chris Walker, Dan Traynor, Daniela Bauer, David Crooks (minutes), Ewan MacMahon, Gang Qin, Govind Songara, Janusz Martyniak, Jeremy Coles (chair), John Bland, John Hill, Kashif Mohammad, Elena Korolkova, Matt Doidge, Matt Williams, Pete Gronbech, Raul Lopes, Robert Frank, Rob Harper, Robert Fay, Sam Skipsey, Steve Jones, Tom Whyntie
Agenda:
11:00 - 11:20 Experiment problems/issues 20'
Review of weekly issues by experiment/VO
- LHCb
- CMS
DB: Some unfavourable mention of RALPP in CMS computing meeting yesterday related to A/R [correction later in meeting].
[inserted from later]
Jeremy: Are you aware of any issues?
Chris Brew: Network problems and ARGUS server problem for A/R.
- ATLAS
- Other
Chris Walker: Nothing to add
11:20 - 11:40 Meetings & updates 20'
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
General bulletin updates:
Jeremy: WLCG workshop arrangements. Next step for those going is to submit GridPP visit notice as per usual. Dave Kelsey has list of applicants to cross-check. Please don't leave it too late.
: WLCG A/R reports circulated this morning and summary sent to TB-SUPPORT. Some sites to follow up. Please get in touch if you're below 90% to keep a record.
: RAL outage last week, BDII affected Nagios. Some post-mortem in progress to look at handover and monitoring of core services/propagation of changes. In Services section of the meeting will cross-check announcements against things which might affect these services.
Chris Walker: BDII configured to failover; if RAL is replying with wrong answer, supersedes other ones. Filed GGUS ticket.
Jeremy: Looking at how RAL BDII monitoring goes down and comes back up in a staged way.
Jeremy: Needs to make sure that it is authoritative when it comes back.
Gareth Smith: BDII never went down, just unavailable. Could have shut it down beforehand and bring it up later - as it was possibly ended up with incomplete data as the network came back up.
Ewan MacMahon: [see also transcript] There were 3 IPs behind the alias. One of them was producing as much info as any other, one with partial, one was blank. All three were claiming to be authoritative.
Gareth Smith: Aught to have have shut them down, brought them up when stable network.
Jeremy: Things to follow up on.
Ewan MacMahon,Steve Jones,Chris Walker: Discussion around possible solutions.
Jeremy: Actions?
: Gareth Smith: Some form of tracking would be useful. ACTION
Jeremy: Mention link to EGI Statistics for April.
WLCG Operations Coordination:
Jeremy: Meeting this week, 8th May, standard agenda, see pre-meeting agenda just before meeting. Jeremy can't make it.
Alessandra: Perfsonar TF has finished and will be replaced with a data transfer one. Has been created, no structure yet, will be discussed on Thursday.
Jeremy: Just monitoring or transfers in general?
Alessandra: Transfers in general and how monitoring can be used in applications.
Jeremy: Tuning and federation?
Alessandra: Yeah, federation and those kind of things.
- Tier-1 status
Gareth Smith: Covered the main item, the work last week. The intervention was successful; no immediate in increase in bandwidth, but be able to route to OPN and CERN and T2s via main RAL network. Then be able to drop light router out, remove 10G bottleneck to T2s. Relatively small change, no date yet.
Storage:
DPM collab meeting.
Sam: Jeremy and I were there. Double length meeting. collaboration + extended dev meeting. Summary is good. UK nice that collab generally quite happy with our contribution, testing and support. Did sign up for quite a lot of things. Feedback on docs. If anyone has any comments on what is good that would be useful.
Jeremy: Clarification: we weren't signing up, but expressing interest.
Sam: Yes - we were being enthusiastic, maybe more so that other countries.
- Accounting
Jeremy: Discussion in EGI context, our inter EGI contributions, look rather low compared to others? Full of WLCG work. Use of stats to comment on international collaboration and research. Jeremy will look at this.
- Documentation
Agreed at last core ops to review KeyDocs to make sure they are focused.
- Interoperation
David C: See bulletin update: UMD-3 EA programme and EMI-2 decommissioning.
- Monitoring
Next meeting Friday
Jeremy: What does MVC stand for?
David C: Model View Controller: find out more on Friday.
- On-duty
Jeremy: Notes from bulletin.
- Rollout
Jeremy: Should update list wrt EMI-3
- Security
No update
- Services
Jeremy: Recording outages in coming weeks or things that will be impacted by work elsewhere. Updates:
Gareth Smith: Nothing that I'm aware of.
Jeremy: Following up on status of DIRAC. Heard back from Mark Slater, not much progress since HEPSYSMAN. Still intending to do this.
MS: Working with Januscz with cern@school and IC DIRAC instance, which works. Jobs running at Glasgow and QMUL; QMUL . na62 working as well.
[inserted from later in meeting]
Janusz: NA62 success last week. Managed to replace the WMS from the portal with DIRAC. Replaced glite-job-submit with dirac-job-submit. Jobs running, some not-DIRAC related problems.
One major thing is to replace the LFC with DIRAC file catalogue, next thing to do.
C@S: Got jobs running, a couple of transcription errors, VO not known to box you have to add it... Fixed on Friday, managed to run jobs.
Other VOs - one of biggest implementation was for T2K, who became silent. Not sure why - I can ask. User had quite complicated JDL, storage elements.
Chris Walker: Does that make it easier to add that support for other VOs?
Janusz: VOs have to make request. It's not a 5 minute job to do this.
Jeremy: Need some commitment from VO
Chris Walker&Janusz: Discussion around how easy it is to add things to DIRAC.
Janusz: There's a workshop at the end of May, would be useful to get an idea of what the VOs want so I can prepare beforehand. Depends what kind of requirements the jobs have.
Note: DFC = DIRAC file catalogue.
Jeremy: Enabling sysadmin through gridpp vo to understand DIRAC. Suspect not many people have done this because of other things.
Tom: Looking at metadata and DIRAC.
Jeremy: EGI will run this as a service through a French site. Should we run one of our own? Come back to: also look at interaction with dev community.
Janusz: Invested a lot of time into DIRAC.
Jeremy: Useful to feed back, look at overlapping issues.
Janusz: Might be a dissemination issue.
- Tickets
Matt: Bank holiday got in the way of a full update.
SNOPLUS+ ticket : last update, Catalin looking for updates.
Tom: Hold up is getting VM to test the deployment of SNOPLUS software. Do have a guide to do this.
Matt: EMI3 tickets, down to Edinburgh, Sussex, UCL and Bristol. ECDF/Sussex progressing nicely. Bristol/UCL updated tickets, adamant that things will be done. Short on time and manpower. Edinburgh triggered an alert to ROD, button wasn't clicked at right time, tickets progressing.
CVMFS for ILC ticket: last we left it Glasgow had updated, waiting for it to percolate. Oxford waiting on return of Kashif.
Kashif: Do it today or tomorrow. Waiting to check that T2K updates work. Working for them, now moving to ILC.
Gareth Roy: Glasgow should be done.
Chris Walker: You need to individually update per VO for puppet? Catalin was asking for help, assistance would be appreciated.
Kashif: I use hiera, changes passed through OK. Changes done per VO.
Chris Walker: Would be useful to change once only, not per VO.
ACTION: Kashif to look into this.
Matt: Just Bristol left, might put in separate ticket.
Matt: 103028: covered in ATLAS meeting.
ATLAS jobs using more RAM than asked for then being killed by batch system.
In essence, if you want to dynamically assign things, can't rely on PANDA alone.
Chris Walker: Ball's in my court for that one, need to send email to cloud support. Fundamentally a difficult problem to solve.
- Tools
Jeremy: Probably covered this, any updates? In next couple of weeks have update from technical meeting.
- VOs
Jeremy: Already covered. Asked about FTS3 frontend, haven't heard anything.
- Site updates
11:40 - 11:50 LHCONE/LHCOPN update 10'
Chris Walker: Went over the notes here: https://indico.cern.ch/event/317792/contribution/2/material/0/0.txt
Ewan MacMahon: Why is Tony Cass not worried about us not being part of LHCONE?
Chris Walker: Would be a problem if people assumed that we were on it and we weren't. LHCONE driven by a couple of things. Part of it is to split off traffic. People want these links to bypass site firewalls.
Jeremy, Chris, Ewan, Pete Gronbech: Discussion. How much would it cost to join? Problem might come if overloaded links are fixed by using LHCONE. Does the community join or individual sites? Has there been any performance benefit shown? Has problems potentially for non-LHC user communities. Not clear if this is a big resurgence - Mark was going but obviously isn't now, this was Chris' first meeting. Issues around negotiating between working across different links.
11:50 - 12:05 Batch systems - plans 15'
Jeremy: Switch this conversation to next time and also follow up on multicore.
12:05 - 12:06 AOB 1'
- Quarterly reports!
Pete Gronbech: Request for topics and talks for HEPSYSMAN.
Jeremy: Monitoring?
Pete: One of the options, but hot topics like multicore should be on there.
Chris Walker: Lightning talks that people can decide on nearer the time would be a good idea.
Transcript:
Jeremy Coles: (06/05/2014 10:59)
David is taking minutes today.
We will start in a couple of minutes given current low numbers....
we'll wait until 11:05.
Ewan Mac Mahon: (11:11 AM)
I think it might have been me that was supposed to file that ticket; I haven't.
Matt Doidge: (11:11 AM)
No ticket was submitted by anyone else that I can see.
Ewan Mac Mahon: (11:14 AM)
The three nodes behind the alias didn't all behave the same way.
Or at least, didn't all come back at the same time.
Daniela Bauer: (11:18 AM)
From teh CMS report: RALPP didn't get pulled up for storage, but for low site readiness.
Christopher John Walker: (11:23 AM)
It's all gone quiet for me...
Alessandra Forti: (11:23 AM)
the mandate hasn't been decided yet
Jeremy Coles: (11:23 AM)
We are generally fine.
Tom Whyntie: (11:36 AM)
I have something to add on DiRAC
I can go into more detail at tomorrow's storage meeting
Samuel Cadellin Skipsey: (11:39 AM)
(Certainly, Dan seems happy with DiRAC for NA62 from our conversations with him here at Glasgow)
Tom Whyntie: (11:39 AM)
https://www.gridpp.ac.uk/wiki/A_quick_guide_to_CVMFS
Christopher J. Walker: (11:40 AM)
Matt needs to talk to Alex Owen actually.
Or Cozmin, but that's a QMUL internal problem.
Catalin was asking about updating the puppet module.
Ewan Mac Mahon: (11:43 AM)
We* should blog this.
*Kashif should blog this.
Jeremy Coles: (11:46 AM)
Kashif is this something we can create an action for you please?
Daniela Bauer: (11:47 AM)
Janusz can't get his Vidyo to work
Janusz is here now :-)
Jeremy Coles: (11:51 AM)
DiRAC enbled: NA62,
vo.landslides.mossaic.org
t2k.org
snoplus
gridpp
northgrid
cernatschool.org
Tom Whyntie: (11:52 AM)
Huge thanks!
The guide is very good
https://www.gridpp.ac.uk/wiki/Quick_Guide_to_Dirac
Christopher J. Walker: (11:58 AM)
I have interest, but little time...
Tom Whyntie: (12:02 PM)
Thanks Janusz
Jeremy Coles: (12:03 PM)
Chris's notes: https://indico.cern.ch/event/317792/contribution/2/material/0/0.txt
Due to the DiRAC extended discussion today I think I will move the batch system discussion to next week and also cover then updates on multicore experiences at GridPP sites.
Any objections?
Ewan Mac Mahon: (12:11 PM)
Sounds reasonable. Batch systems and multicore rather go together anyway.
I think we also want our services accessible to non-LHC users and sites.
It's not all T2-t2 traffic, some of its T2->random desktop traffic.
I also think it's striking that the Italian T2s had a 10Gbit link between them.
I think we can safely say that our way is doing rather better than that.
Indeed, and I'm sure Mark will have put that pint strongly. My point is that it should be Dave B putting that point.
And in management meetings, not networking ones.
Andrew McNab: (12:29 PM)
This is a bit like the financial transaction tax. How to not just opt out but prevent being shut out.
Jeremy Coles: (12:29 PM)
Pint?
Samuel Cadellin Skipsey: (12:30 PM)
Sure, I understand that point. Just noting that it's not like we've not been providing critique.
Ewan Mac Mahon: (12:30 PM)
Er. Point.
Jeremy Coles: (12:30 PM)
The UK view has been given at the WLCG MB. I'll check on the latest updates in that area.
Ewan Mac Mahon: (12:30 PM)
And indeed, I'm not suggesting radical change, just an increase in emphasis.
@Jeremy was the UK view "we're not interested" or "please stop, you're breaking things"?
Because I think it's mostly been the former, and I think we should up the ante to something closer to the latter.
Daniela Bauer: (12:33 PM)
Bye
Tom Whyntie: (12:33 PM)
Thanks
There are minutes attached to this event.
Show them.