Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234.
To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +442030510622
-- The meeting extension is 9308582.
Apologies:
Participants: Andy Washbrook, Andrew Lahiff, Andrew McNab, Brian Davies, Chris Brew, Dan Traynor, Daniela Bauer, David Crooks (minutes), Duncan Rand, Elena Korolkova, Ewan MacMahon, Gang Qin, Gareth Roy, Gareth Smith, Govind Songara, Ian Loader, Janusz Martyniak, Jeremy Coles (chair), John Bland, John Hill, Kashif Mohammad, Mark Slater, Matt Raso-Barnett, Matt Williams, Raja Nandakumar, Raul Lopez, Robert Fay, Robert Frank, Sam Skipsey, Steve Jones, Tom Whyntie, Wahid Bhimji
11:00 - 11:20 Experiment problems/issues 20'
Review of weekly issues by experiment/VO
- LHCb
Raja: A little work, primarily MC and user, otherwise nothing to report
Steve Jones asked for multicore jobs to be sent to new arc/condor at Liverpool after some requested changes were made. Jeremy clarified whether this was multicore specifically or arc/condor after Raja noted that not much multicore work was being done (it was arc/condor). Raja will look into this.
- CMS
Daniela: not much news. Bristol email to say they looked at Phedex from last week. Think it had more to do with central instance, looks like it's getting better.
- ATLAS
Elena: Beginning of September; time for ATLAS efficiency to define ABCD status for data distribution, coming from HC tests. (transcript for link). Only RHUL is C status, because there are some struggles with new disk servers. Sheffield and Liv are still T2 not T2D. Sent email to ADC experts to find out what's going on. Set up multicore queues for IC. Not enough software releases validated, 13 out of 120. Put queues online to speed up procedure, if software not validated no jobs will run. Will wait to see what happens. Birmingham had problem, a ticket with storage now closed, still in test. One worker node which gives problem.
Mark Slater: we'll offline the node and investigate
Elena: Ticket for ECDF, for instabilities in webdav. Update for DPM because problem with xrootd; needs update.
Sam: Update on storage problem. There was a bug in the ATLAS N2N thing which caused "malformed URLs" which caused xrootd to fall over. In combination with ATLAS user with malformed URLs, lots of xrootd's falling over. Emergency release, problem fixed.
Wahid: everyone has done it.
Sam: Info flow interesting. Outside UK, several threads started up saying "my xrootd is broken" after the release issued.
Jeremy: Maybe raise at Ops
Wahid: On Webdav. Was opened for ECDF, but more general point to DPM team. Fixed at ECDF but not sure whether to close ticket. Probably should open ticket against someone else. Someone should take a look at the whole of the UK and encourage webdav.
Elena: Not a team ticket.
Wahid: Maybe shouldn't have been assigned to ECDF. Not really a site issue. Another opened against DPM team, closed as it was a site issue. Within ATLAS opened as a JIRA task to ensure deployment? Go to Thursday meeting. ATLAS position is xrootd is stable first, then webdav. Chasing up instabilities does increase load. It is an ATLAS request that webdav is enabled.
Jeremy: Sounds like an issue for Thursday, but track it here.
- Other
- DIRAC status
-- discussion at GridPP33 about encouraging sites to support the GridPP DIRAC service via VMs either on IaaS Cloud or Vac so more sites get experience with the new technologies. Andrew M proposed to start running SAM tests against such endpoints. We'll review progress in this slot.
Andy: 2 parts. 1) Want jobs running regularly on gridpp service at sites 2) Like as many sites able to run jobs in VMs. Have SAM testing of gridpp DIRAC infrastructure to make sure jobs run at sites. Can be done by deploying VM solution, Vac/Openstack, or enabling gridpp dirac VO on conventional WLCG site. Idea is to mon page on gridpp site and to report it in this slot at each meeting.
Had a look at statistics on DIRAC portal. If you have an account as user, you can make plots etc.
[refer also to transcript]
Clarification: In WLCG terms its the gridpp VO. You also need to be registered in the DIRAC central instance at Imperial so that the site director can send out the pilot jobs. You have to tell them that you want to receive jobs.
Ewan: gridpp VO probably basically supported at lots of sites. If it still works is another question; hasn't been used in production for a while.
Jeremy: Does VO Nagios support gridpp VO?
Kashif: [See link: https://vo-nagios.physics.ox.ac.uk/nagios/]
Andrew: Issue of VO setup, but also needs to be activated at central portal the two parts. If you go down the VM route, the gridpp VO in WLCG terms doesn't come into it. This is the beauty of doing some SAM tests is that we can see what works end to end.
Looked over accounting since start of the year, only run 870 jobs through infrastructure, mostly cern@school and things, separate subgroups within the gridpp dirac entity for different projects and experiments. Can also use default gridpp. The jobs have only run at 8 sites. 2 of which are convential/Vac sites at same place. Would like to get this rolled out to more sites and particularly to test the VM stuff in production; once it works for gridpp it's easy to extend to other experiments. Idea to report this here each week.
Janusz: Do you require any configurations on DIRAC side?
Andrew: Don't think so.
Janusz: I was adding sites defensively. Clearly this list is open - if someone needs a site let me know. The mask is another thing. I have to mask them specifically. Adding more sites is not a problem, clearly should any VO need more than that... sites and also storage element - ready to do this if there's a request.
Jeremy: Looking at quick guide to DIRAC, covers client setup but not contacting Janusz. Maybe we could add something?
Janusz: Usually people notice quickly if they try to set the VO on the web portal and not get it and then contact me.
Tom: In terms of support, is direct email the best way? GGUS?
Janusz: GGUS ticket is probably better.
Tom: What's the best way of flagging the DIRAC?
Jeremy: Do we need to set up a support unit?
Duncan: Probably best to use Imperial.
Ewan: Notified site or specific email address field?
Jeremy: Notified site, probably.
Duncan: And cc Janusz. We should add him.
Jeremy: Assign to Imperial - if it doesn't work, we look at it again. The hope is that every site will get a chance to start trying some of this VM stuff. What do we want to do for the next week?
Andrew: Principally the 8 sites include WLCG sites. The thing that sites can do straight away is to go out and try Vac or cloud stuff. Link for Vac config for gridpp dirac page in the wiki. I'm going to put SAM tests together. [in chat]
Jeremy: Any questions for those not at GridPP? Currently gaining experience with this approach. Sites should read documentation and we'll come back to it next week. As Andrew noted, there's a pre-GDB on clouds next week.
Ewan: Authentication to Vac side of things is with a hostcert in the name of your whole VAC site and put the certificate in the Vac instances. (Documented). Minor setup phase; worth applying for your hostcert in advance of setting things up.
Jeremy: Should be in beginning of documentation.
Ewan: It is, just thought I would flag it.
11:20 - 11:40 Meetings & updates 20'
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
- General updates
A/R, pretty good. SUSX generally down to updates and DT. EGI Ops for July, UK 96% for Ops VO.
CA TAG meeting. Might be moving to commercial CA with JANET? At the moment a bit unclear. Any other CA comments or issues let Jeremy know.
WN lockups:
Steve Jones: Running SuperMicro motherboards, getting period lockups. Quite a long story, a lot of work, think it's nailed now. New kernel, problem gone away. (see transcript). Might want to look at it if you've had this problem.
Gareth R: 2 things, SuperMicro machines locked up as Steve said but also DELL Interlagos (so completely different systems) had major load spikes. Wouldn't fall over but would have serious issues. New kernel has so far improved stability greatly.
VOMS checks: Bristol, RHUL, UCL, no ticket; want to check on situation.
Andy Washbrook: bouncycastle-mail? Is that in tarball?
Steve Jones: Think it would be better done as root, might be possible in tarball. SHA-2 certificates.
Gareth Roy: It doesn't work unless you update bouncycastle-mail 1.46-2 . Otherwise will not understand SHA-2 (assuming VOMS extensions). Assumed it was for glexec for WN.
Andy Washbrook: Will follow up.
- WLCG ops coordination
We can put in our own Tier-1/2 feedback. Might mention ATLAS issue, webdav, maybe. Covers Baselines, middleware issues, change rollout ,feedback to expts, IPv6.
- Tier-1 status
Gareth Smith: On Saturday, network switch failed. Odd effects, not yet understood on virtual infrastructure. Much wider that one might have expected. In case people saw issue, just to flag it up. FTS2 goes away today, NFS mounted software server previously used by small VOs goes away as well.
- Storage and data management
Reminder for FAX sites to update RPM.
Wahid/Sam: Everyone has done it, got the RPM (this was the issue we discussed earlier).
- Accounting
Oversight documents for STFC: please check REBUS capacities and report back to Jeremy, as some discrepancies had been noted.
- Documentation
Need to kickstart KeyDocs. Discussion over options to revamp the website, Drupal and Wordpress being considered.
Duncan: Wordpress appears unhappy with IPv6; Blogger seems OK.
Tom/Andrew: Discussions about to get underway.
- Interoperation
David: No meetings over the summer.
- Monitoring
David: 2 updates. Monitoring consolidation meeting on SAM3 progress and validation. Some discrepancies noted for 4 ATLAS UK sites; noted in wiki pages and explained. David will follow up with a request for people to check their sites in the 2 versions and feedback any discrepancies they see.
CVMFS/squid monitoring TF meeting the previous week; outcome was that new CMVFS monitoring from ALICE could be added in to WLCG squidmon, with SNMP port access being the default requirement. Additional monalisa host sensor would be noted as an optional extra. Any discussion of upcoming WLCG notifications should wait for the official announcement, but note that one will be coming soon. Possible clarification on Thursday in WLCG Ops.
- On-duty
See bulletin
- Rollout
- Security
Ewan: No new news.
- Services
Some RIPE probes, if you have questions please get back to me.
JANET to deploy a PerfSonar at an exchange point in London. Part of a strategy being adopted.
Ewan: Target it with tests from our one?
Jeremy: They want to get familiarity with it. They haven't asked for us to test against it. If I have one I'll pass that on. I have suggested that we're interested in being involved; not overlapping. Some coordination required.
- Tickets
See bulletin: tickets linked below where there was discussion or comment.
https://ggus.eu/index.php?mode=ticket_info&ticket_id=107880
Gareth Smith: Some update on this during meeting
https://ggus.eu/index.php?mode=ticket_info&ticket_id=107911
https://ggus.eu/index.php?mode=ticket_info&ticket_id=107910
Ewan: In a sense not a site issue. Really a more general issue, what's the solution? We've been mostly assuming that software tags go away in the landscape of CVMFS. They're using software tags as a blacklist. Long term/short term option might be to stop using WMS, switch to DIRAC. Not sure there's a better answer than "that's not possible". Open to suggestions, will affect anyone trying to do this kind of thing.
Sam: I generally agree. For software tags, blacklisting outside of their scope. A deeper question is why do they need to blacklist sites?
Jeremy: 2 tickets should become 1 ticket with general discussion?
Sam: Using it in this way because there isn't a general blacklisting method. Others might like to use it but haven't thought of using it? Bigger experiments do blacklist but have the people to develop their own systems.
Ewan: Worth flagging WMS -> DIRAC as a long term solution, but might not be useful for the short term.
- Tools
- VOs
- Site updates
11:40 - 11:45 Multi-core status and actions 5'
Elena: Haven't finished yet, work in progress. Set up test condor server but in parallel want to allow multicore on Torque/Maui
11:45 - 12:00 Feedback for operational effort optimisation 15'
- A follow on from the previous discussion
- wiki: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGOperationCosts
Jeremy is writing up notes from this section shortly; please refer to them for a record of this discussion.
- Previously:
* AF: Feedback is important. We are given a voice for once.
* EM suggested some kind of "European Middleware Initiative" to coordinate middleware production and maintenence. Also suggested for people stop knocking holes in our infrastructure.
* SS: Middleware under resourced though (e.g. Argus lacking support).
* glexec mentioned as a possible source of wasted effort.
* SS: Reducing layers seems to be the way to go (parameter passing to batch systems. glexec again brought up as an example of an unneeded layer.
* JC: Cloud impacts?
* SS: For better or worse, losing what batch systems do for you.
* GS: It has been brought up that mature sites in same countries could be grouped together as a single point of contact/submission for VOs. Ticket
review/ROD mentioned as points of effort. Not sure if helps UK.
* AF: the history of the hammercloud tests, and how they've been useful for atlas and adopted by cms.
12:00 - 12:01 AOB 1'
- Actions https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
Transcript:
Daniela Bauer: (02/09/2014 11:00)
I'm trying to join from home which might or moght not work.
Jeremy Coles: (11:00 AM)
It worked so far...
Starting in 1 minute
Daniela Bauer: (11:01 AM)
Yes, I can hear you :-)
Steve Jones: (11:02 AM)
hepgrid2.ph.liv.ac.uk
ARC/Condor
Duncan Rand: (11:03 AM)
s
Steve Jones: (11:04 AM)
Thanks
Jeremy Coles: (11:04 AM)
s?
Duncan Rand: (11:05 AM)
That was just a test, sorry.
Elena Korolkova: (11:06 AM)
https://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistory?columnid=10102#time=720&start_date=&end_date=&values=false&spline=false&debug=false&resample=false&sites=T2&clouds=UK
epgg03
Wahid: (11:12 AM)
Everyone did it http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteview#currentView=FAX+endpoints&highlight=false
1234: (11:12 AM)
Hello - sorry I'm late
Ewan Mac Mahon: (11:12 AM)
We're just awesome.
David Crooks: (11:16 AM)
Sorry, could I check who 1234 is for the minutes?
Elena Korolkova: (11:18 AM)
I think all dpm sites have webDav, stability is a different issue
Ewan Mac Mahon: (11:19 AM)
Just in case the person we see as '1234' doesn't see themselves as that (it's apparantly possible to get an inconsistency between the local Vidyo view and what everyone else sees), they're the person that said 'Hello - sorry I'm late' at 11:12.
1234: (11:19 AM)
WHOOPS
Samuel Cadellin Skipsey: (11:20 AM)
...WHOOPS isn't an identification?
Ewan Mac Mahon: (11:20 AM)
I think they're going out and coming back in again.
Samuel Cadellin Skipsey: (11:20 AM)
:D
Andrew McNab: (11:20 AM)
https://www.gridpp.ac.uk/wiki/Quick_Guide_to_Dirac https://www.gridpp.ac.uk/wiki/Vac_configuration_for_GridPP_DIRAC
Duncan Rand: (11:22 AM)
which VO?
Ewan Mac Mahon: (11:22 AM)
It's the 'gridpp' vo though, right?
Not a new one just for the dirac, it's just we haven't used it for a whole lot in the past.
But a lot of CEs will already support it, because it's been used for site security challenges and the like.
Though whether it works is a whole other thing.
Kashif Mohammad: (11:24 AM)
VO nagios supports gridpp vo
Jeremy Coles: (11:24 AM)
Thanks. So we can check the status on sites.
Kashif Mohammad: (11:25 AM)
https://vo-nagios.physics.ox.ac.uk/nagios/
Tom Whyntie: (11:26 AM)
Hello!
That would be my question too - the Vac stuff is transparent to the end user, right?
Andrew McNab: (11:29 AM)
yes, Tom
Samuel Cadellin Skipsey: (11:29 AM)
They're "just" "bottled pilots" from the perspective of DIRAC, right?
Tom Whyntie: (11:31 AM)
Great, thanks
Andrew McNab: (11:32 AM)
the payload is running in a pilot, from its point of view, yes
Samuel Cadellin Skipsey: (11:33 AM)
(the heavy quoting there is to indicate that I know that VAC/OpenStack/other VM hosting tools are doing more complex stuff behind the scenes, but what DIRAC sees is nice and simple)
Steve Jones: (11:35 AM)
OK, got it.
Ewan Mac Mahon: (11:37 AM)
In a nutshell the VM has an init script that runs the pilot agent. It runs it as a non-root account too, so the environment the job sees should be very normal SL6+cvmfs.
Jeremy Coles: (11:37 AM)
https://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
Samuel Cadellin Skipsey: (11:37 AM)
Oh, indeed, I read all of the VAC source code last week, Ewan ;)
Andrew McNab: (11:39 AM)
I like systems where it's practical for users to sit down and read all the source code!
Ewan Mac Mahon: (11:39 AM)
I think one of the things that's particularly nice about vac is that it's just so small.
Coupled with the cernvm 3 images, which are just downright clever, it's a very nice system.
Samuel Cadellin Skipsey: (11:40 AM)
Jeremy - is the CA TAG open, or close invites? It would be nice to know a bit more about it.
Jeremy Coles: (11:41 AM)
Closed at the moment.
Steve Jones: (11:41 AM)
kernel-2.6.32-431.23.3
Samuel Cadellin Skipsey: (11:41 AM)
Righto, I sort of expected that.
Steve Jones: (11:42 AM)
Cheers Gareth.
Govind : (11:44 AM)
Sorry Jermey. what issue you we are talking about RHUL ?
Jeremy Coles: (11:48 AM)
Hi Gvoind. I was looking at the pre-prod results and noted RHUL was failing - as a CMS T3 actually.
Govind : (11:52 AM)
VOMS are updated yesterday for cms,.. I drilled down the test it looks like failing glexec test
Duncan Rand: (11:53 AM)
There is also a plea to make the SAM dirac monitoring IPv6 compatible.
David Crooks: (11:54 AM)
https://twiki.cern.ch/twiki/bin/view/LCG/ValidationStatus
Jeremy Coles: (11:57 AM)
Thanks Govind.
Duncan Rand: (12:06 PM)
They should get it added here http://ps-dashboard.es.net/index.cgi?dashboard=6%3A%20ESnet%20to%20Europe
Note how poor Lancs perfsonar is.
Ewan Mac Mahon: (12:12 PM)
We should probably encourage them to set up their own mesh config file sooner rather than later, even if they only have one box initially pulling from it. It's clearly the best way to configure a proper network of perfsonars.
Kashif Mohammad: (12:13 PM)
no
Duncan Rand: (12:21 PM)
As you suggested earlier it depends who they want to ineract with. We at GridPP will no doubt be happy, but they might only be interested in other NRENs.
Tom Whyntie: (12:22 PM)
DIRAC has solved a huge number of problems for us (CERN@school) so far as a small VO, and support via email and GGUS has been great.
Andrew McNab: (12:24 PM)
that's a great idea. would make it easier if we could assum everyone had one.
Tom Whyntie: (12:24 PM)
The CERN VM service!
Ewan Mac Mahon: (12:25 PM)
And that.
Lots of stuff.
Indeed.
proper cern accounts with the little plastic cards that open the doors and get you through the front gate too. Proper cern accounts.
Or, put differently, WLCG being on an equal footing with an LHC experiment.
Samuel Cadellin Skipsey: (12:27 PM)
(you get a little plastic card? I should ask for my little plastic card!)
Daniela Bauer: (12:32 PM)
soory, I have to leave
Ewan Mac Mahon: (12:32 PM)
To be clear on the VAC thing, it's not necessarily something I'd advocate actually doing right now, but I tihnk it's worth submitting as an example of GridPP thinking big and doing good stuff.
It's worth mentioning, partly for the sake of mentioning it.
Tom Whyntie: (12:33 PM)
Thanks, bye
Ewan Mac Mahon: (12:33 PM)
Though that said, given that WLCG does de facto have central pilot frameworks, moving lock stock and barrel to VAC would make the whole stack, and sites in particular, a lot simpler.
And it would save us from OpenStack, which would be quite a win.
Samuel Cadellin Skipsey: (12:34 PM)
Well, quite, although starting VMs is also easy in HTCondor, so it's not clear that OpenStack is the only opposing solution (or that VMs, strictly are the only implementation of such a solution)
Ewan Mac Mahon: (12:37 PM)
Speaking of our ops meeting - I do think it's worth shouting about the things the UK does well already in addition to asking for new things we want.
As I said at the top of the meeting, we're just awesome.
Andrew Lahiff: (12:41 PM)
Following on from Sam's comment - I've recently been dabbling with starting CernVMs in HTCondor, and may try to reproduce VAC type funtionality.
Jeremy Coles: (12:43 PM)
A washbrook; A Lahiff; A McNab; Brian; Chris B; Dan T; David C; Elena; Ewan; Gang; Gareth R; Gareth S; Govind; Janusz; Jeremy; John B; John H; Kashif; Mark S; Matt RB; Matt W; Raja; Raul; Rob F; Robert; Sam; Steve J; Tom; Wahid.
Samuel Cadellin Skipsey: (12:43 PM)
Andrew: I think the main issue is how you do the bit that VAC calls "contextualisation" - you need to get appropriate certs etc into each VM to make it appropriately flavoured. But I don't think that's a problem with a bit of scripting in Condor's hooks. (I've also been dabbling with Condor's VM stuff)
Andrew McNab: (12:43 PM)
I wonder if something like Vcycle would help with that Andrew? You're going to need some agent poping up the VMs centrally on the Condor WNs?
Ewan Mac Mahon: (12:44 PM)
Having an architecture document assumes that we're a city.
We're not, we're a favella.
Jeremy Coles: (12:44 PM)
But you can still come up wit a general description.
Samuel Cadellin Skipsey: (12:44 PM)
Ewan: you might say we're a jungle.
(On that note, I recently discovered some papers on "Jungle computing", which appears to be an attempt to bring back means of managing heterogeneity in distributed resources, which Cloud does not that well.)
Andrew Lahiff: (12:52 PM)
Andrew: Vcycle is something I'd like to look into at some point. I also want to try getting Condor to pop up the VMs itself.
There are minutes attached to this event.
Show them.