Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234.
To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +44 (0)161 306 6802. Phone bridge ID 1001002
-- The meeting extension is 109308582. PIN 1234
Chair: Jeremy
Minutes: Andy Washbrook
Apologies:
Present: Adam Boucher, Alessandra Forti, Andrew Washbrook, Andrew McNab, Chris Brew, Daniela Bauer, David Crooks, Elena Korolkova, Gareth Roy, Gordon Stewart, Govind Songara, Ian Loader, Ian Neilson, Jeremy Coles, John Bland, John Hill, Kashif Mohammad, Lukasz Kreczko, Marcus Ebert, Matt Doidge, Paige Lacesso, Raul Lopes, Robert Currie, Steve Jones, Terence Froy, Vip (?)
Ops Meeting minutes
- Widened volunteer pool to take minutes
- Suggestion to create a template for the minutes. There was one available a long time ago but this has since been lost
Experiment problems/issues
LHCb (update given by Andrew McNab on behalf for Raja)
- Nothing much to report
- Disk server at Tier 1 taken out of service but now back again
CMS (No report)
ATLAS (update given by Elena)
- Three tickets for Glasgow last week but problems all hopefully solved
- One ticket for Sheffield, and one for Birmingham regarding xrootd/webdav service
- Request for ATLAS UK cloud support mailing list (atlas-support-cloud-uk@cern.ch) to be used in case of issues rather than contact individuals
- Someone on team will deal with issue even if they are small problems
- Arose due to specific request from Steve Jones determining reasons for drying up of multi-core ATLAS jobs at Liverpool
- Link to brokerage page in broken. Alessandra opened ticket to fix
CMS (no report)
Other
- Daniela Bauer: NA62 are currently undergoing a ramp up in activity which is in hand
GridPP DIRAC (Andrew McNab)
- Added Datacentred back in use for SKA telescope tests.
- Gave Openstack details, created a login and added details into Vcycle configuration. Now starting running VMs
Meetings and Updates
General updates
- GridPP Oversight Committee meeting this week
- ATLAS - APEL accounting records comparison. Bug found in scripts for ARC and Torque.
- Marcus: xrdcp test for sites with DPM
- Bug in the ATLAS pilot code leads to xrdcp checksum job failures
- Only seen at ECDF so far but only because other sites using a different tool to stageout files
- Followed up in Storage group
- 2GB limit effecting xrdcp transfers from bristol using DPM with HDFS
- Solved the problem followed up in storage list
- The EGI Security Policy Group has produced a new revised and updated Tthe op-Level Security Policy document
- Applies to all sites, new draft available for comment
- Respond by next Thursday, suggest security team have a look
- HEPSYSMAN 2017 dates to be confirmed soon
- Confirm dates next week, avoid clash with the UK based WLCG workshop
- A new release of the Accounting Portal is ready for testing
- WLCG meeting (pre-GDB) on networking: 10 Jan 2017 at CERN. It will be the opportunity for LHC experiments, Network operators, WLCG service and site managers to meet and discuss the network requirements for the 3rd run of the LHC, planned for 2021-2023.
- The ARC brainstorming camp takes place 28th November - 2nd December. Reminder of Andy W's request on: Call for UK feedback to the ARC brainstorming camp
- Suggest sites provide feedback either through general call from Andrej or Andy as long as we have good UK representation - though if they go through me I can send this on to Andrej and also talk directly to developers about any UK site issues.
WLCG ops coordination
- No update
Tier-1 status
- No update
Storage and data management
- Some curious operational issues with failed xroot transfers - need more testing (covered by Marcus Ebert above)
- Duncan provided feedback from the JISC data transfer/campus networking workshop
Tier-2 Evolution (Andrew McNab)
- Next generation LHCb pilots / VMs deployed at all VM-based sites. Includes multiple parallel payloads support for multiprocessor VMs and new WLCG SAM probes framework
- Vac 2.0 mixed-sized VMs support in testing, using Super Slots mechanism to create multiple single-processor VMs in groups to avoid breaking up N processor slots for multiprocessor VMs.
- In production next month after testing for a couple of weeks
- Cambridge HPC demo
- Still discussing mechanics of joining
- Need get certificates and run vcycle or get access to their setup
- Dune VO
- Call for sites to add VO to allowed list
- They are using DIRAC
- (Elena) No storage requirements, only CPU is needed
Accounting
- No update
Documentation (Steven Jones)
- Accounting scripts documentation to get current APEL values from ARC, Torque and VAC to allow sites to drill discrepancies to see where these things match up
- Next work is to break this down by VO
- Other people can add other batch systems to scripts via github
- Docs put in Github FAQ for now but agreed with Alessandra to share this link around
Interoperation
- No update
Monitoring
- No update
On-duty
- No update
Security (Ian Nielson)
- We are still seeing sites pop-up in the dashboard in relation to EGI-SVG-2016-11476.
- CVE-2016-7117 still waiting for distro. kernel updates
- New kernel update yesterday but does not include patch
Services
- No update
Tickets (Matt Doidge)
- 21 open
- AFS tickets
- Suggest either put on hold or perfectly acceptable to close with response along the lines of "poiltely forward on your message to your Tier-3 rep / local system administrator"
- Tier 1 open tickets
- Andrew Lahiff is on leave and Gareth Smith is now part time
- Needs someone to respond to open tickets
- (Daniela) 2 tickets against the storage which affects all small VO (GGUS 125126)
- Birmingham ticket
- Mark Slater unavailable for comment (usually teaches on Tuesday)
- LSST tickets
- Side note: LSST are positive about experience so far and are talking to associated communities interested in using Grid resources
Tools (Kashif Mohammad)
- No plan for ARGO to release a VO-based Nagios service
- Could provide service for VO but NGI has to talk to them first
VOs
- No update
Site updates
- No update
Discussion: Site roundtable
Adam Boucher, Oliver Smith (Durham)
- Moving middleware services (ARC, BDII, ARGUS) to SL7/CentOS7
Alessandra Forti, Andrew McNab (Manchester)
- Marshalling new round of LSST jobs
- Investigating ARC and HTcondor services and installed a testbed
Andrew Washbrook, Marcus Ebert, Rob Currie (Edinburgh)
- Assessing requirements for scaling up Openstack use on Eddie3 cluster
- Can provide notes or documentation of our work, but may be highly site-specific
- Moving DPM (production and test) headnodes to VMs
- Continuing the multi-VO Xrootd network studies that has been discussed in more detail in storage meeting
Chris Brew (RAL PPD)
- Queried status of ARC under SL7 / CentOS 7
- Detailed discussion of issue in chat log
- SL7 build APEL SSM storage accounting testing for John Gordon
- Looking at dashboarding software as follow up from HSM meeting
- Working on running CMS jobs from Oxford but reading from RAL
Daniela Bauer (IC)
- Mostly DIRAC work and keeping stuff running
- Also working on CHEP proceedings
- Seems to be no way (or a hidden way) for submitting proceedings
David Crooks, Gareth Roy (Glasgow)
- VM hosting of services near to medium future
- More flexibility in providing services in a reduced hardware estate and to minimise time to migrate to new data centre
- Bill system and management tools background project
- Ownership and migration new computing facility Sep 2017 (was 2016)
- Modular data centre situated on campus
Elena Korolkova (Sheffield)
- Drain all storage nodes and convert them to SL6
- Last one has no free disk space on storage
- This week plan to add memory to disk pool
Govind Songara (RHUL)
- Last week moved DB on DPM headnode to another machine
- Move headnode from SL5 to SL6 or maybe wait for Centos7
- IPv6 discussion with central IT start looking get some IP from Janet make some progress
Ian Neilson (RAL / Sec)
- Still intending to run communication challenge in near future
John Bland, Steven Jones (Liverpool)
- IPv6 testing and negotiation with university firewalls to get higher bandwidth
- CHEP proceedings on VAC testing underway
- Evaluate new new features in Vac 2.0
- Approved VO rpms and meta rpm is work in progress
- Hardware purchase pending
- Add Dune VO
- Input to EGI accounting TF
- SL7 migration plans in future
John Hill (Cambridge)
- Standardise middleware on UMD4
- Work with Duncan on two Dirac site perfsonar hosts
Kashif Mohammad (Oxford)
- ARC CE problem due to globus update, upgrade to 5.2 seems to work
- Looking in InfuxDB, Grafana and telegraf
- IPv6: central IT accidentally disabled but now back
- Lustre FS testing for local storage
Lukasz Kreczko, Paige Lacesso (Bristol)
- Working with netcomms team on new JANET link, Dedicated 10G link for Grid site bypassing uni firewall
- new network tagged VLAN
- Deploy on all WN
- Jeremy suggested sharing relevant plots to help with network forward look presentations
- Query from Winnie on Southgrid BDII glue2 validate error
Matt Doidge (Lancaster)
- CentOS7 migration will have to happen sooner rather than later,
- Small test cluster 4 machines by Chrstmans
- Working on UMD based tarball
Raul Lopes (Brunel)
- Moving middleware services to CentOS7
- ARGUS IPv6 long GGUS ticket interacting with dev team
- ARC CE on CentOS7
- Move DB for storage for CentOS7
- Head node on CC7 pretty soon when DPM is ready
- IPv4 20 G down and reverted to 6G link after problem with link outside campus, but now back on
AOB
- Monitoring: David Crooks reported on traffic light queues for Grafana instance and suggested we reuse the GridPP monitoring blog for new developments across sites
Chat Window
raul: (22/11/2016 11:08)
Email for Atlas support?
Jeremy Coles: (11:09 AM)
atlas-support-cloud-uk@cern.ch
Daniela Bauer: (11:10 AM)
Apologies. Vidyo trouble
Elena Korolkova: (11:14 AM)
Could sites support dune vo, please
All jobs are coming to sheffield ATM
Govind: (11:27 AM)
How much storage they need ?
Matt Doidge: (11:28 AM)
Dune access should be available at Lancaster - I'll check why it isn't.
Elena Korolkova: (11:28 AM)
Thanks
Alessandra Forti: (11:29 AM)
if we have them somewhere public we can add them to the FAQ
Steve Jones: (11:31 AM)
https://github.com/gridpp/audit/wiki
Thats the wiki to the gridpp/audit project.
Chris Brew: (11:36 AM)
I spoke to Brian about it last week
Daniela Bauer: (11:39 AM)
https://ggus.eu/?mode=ticket_info&ticket_id=125126
raul: (11:49 AM)
I do run on Centos 7
Latest Arc is working with globus. On CentOS7
I am testing it for MW readiness
I am supposed to write a report for UMD-4. Next week...
Chris Brew: (11:53 AM)
I'm running nordugrid-arc-5.1.3-1.el7.centos.x86_64
raul: (11:54 AM)
The one I am testing is arc-5.2
Chris Brew: (11:54 AM)
rpm -qa | grep globus globus-xio-5.12-1.el7.x86_64 globus-rsl-10.10-1.el7.x86_64 globus-gram-client-13.13-1.el7.x86_64 globus-gsi-callback-5.8-1.el7.x86_64 globus-gssapi-error-5.4-2.el7.x86_64 nordugrid-arc-plugins-globus-5.1.3-1.el7.centos.x86_64 globus-xio-gsi-driver-3.10-1.el7.x86_64 globus-gsi-sysconfig-6.9-1.el7.x86_64 globus-gsi-cert-utils-9.12-1.el7.x86_64 globus-gssapi-gsi-12.1-1.el7.x86_64 globus-gss-assist-10.15-1.el7.x86_64 globus-ftp-control-7.2-1.el7.x86_64 globus-gsi-openssl-error-3.5-2.el7.x86_64 globus-openssl-module-4.6-2.el7.x86_64 globus-gsi-proxy-core-7.9-1.el7.x86_64 globus-callout-3.14-1.el7.x86_64 globus-gram-protocol-12.12-3.el7.x86_64 globus-gass-transfer-8.9-1.el7.x86_64 globus-common-16.4-1.el7.x86_64 globus-gsi-credential-7.9-1.el7.x86_64 globus-io-11.5-1.el7.x86_64 globus-gsi-proxy-ssl-5.8-1.el7.x86_64 globus-xio-popen-driver-3.5-2.el7.x86_64 globus-ftp-client-8.29-1.el7.x86_64
Ah OK, I'll look for that.
raul: (11:55 AM)
I'll send you the packages I've got installed
Chris Brew: (11:57 AM)
Hmmm 5.2 is avaiable from UMD-4-untested and the lcmaps stuff I had to install from SL6 is now in testing
raul: (12:00 PM)
It's been in epel-testing for a couple of weeks. I am testing it for UMD-4.
Chris Brew: (12:01 PM)
Must have hit it just after we installed our latest batch
raul: (12:02 PM)
The globus is
globus-gssapi-gsi-12.9-1.el7.x86_64
Chris Brew: (12:03 PM)
Yes, I'm pulling in the full set of updates on one of our new ArcCEs
Alessandra Forti: (12:07 PM)
is none of you using the repositories from nordugrid?
and do you know if there are any differences?
Chris Brew: (12:08 PM)
Oh, yes. I've got ipv6 at home! No sign of it on the general network at RAL yet though :(
Alessandra Forti: (12:09 PM)
I installed the ARC/HTC CE without UMD but I'm worried this is going to bite me when I'll get on to the WNs
raul: (12:09 PM)
I do install from nordugrid, but UMD asked me to test arc 5.2 and pointed to epel-testing.
Alessandra Forti: (12:09 PM)
UMD and nordugrid are incompatible because of latest globus
Chris Brew: (12:10 PM)
Alessandra: No, our setup relies on other things from UMD (like lcmaps) so I've always thought it safest to go for a suposedly integrated release
John Bland: (12:10 PM)
who's paying for the dedicated link?
Alessandra Forti: (12:10 PM)
we removed UMD completely from the ARC-CE installation
right
raul: (12:11 PM)
I keep UMD disabled in everythiing. I enable selectively
Alessandra Forti: (12:11 PM)
it was too good to be true
raul: (12:12 PM)
There seems to be a problem with Staged rollout comingalive from time to time and MW readiness, and nordugrid all in parallel
Lukasz Kreczko: (12:13 PM)
globus-gssapi-gsi.x86_64 12.9-1.el6 @epel on our ARC CE
but also nordugrid-arc.x86_64 5.2.0-1.el6 @epel
nothing from UMD
Elena Korolkova: (12:14 PM)
Thanks, Matt :)
raul: (12:15 PM)
arc 5.2 wors fine with globus 12.9-1 both on el6 and el7
it should be in UMD soon
Daniela Bauer: (12:16 PM)
@Matt: dune has two fnal servers (voms1 and voms2) and I think you are missing one
Your proxy is valid until Wed Nov 23 12:15:50 GMT 2016 2016-11-22 12:15:51,680 FATAL - Authorization failure: Authorization error: Failed to get the local user id via glexec: glexec error: [gLExec]: LCAS failed. The reason can be found in the syslog. VO dune: LSC validation failed: LSC file matching VOMS attributes not found in store. VO dune: AC signature verification failure: no valid VOMS server credential found.
Chris Brew: (12:18 PM)
Raul: It's in untested, I've just installed from there on one node
Matt Doidge: (12:18 PM)
Thanks Daniela, you just saved me an hour or two of poking - I have out of date voms settings for dune.
Daniela Bauer: (12:18 PM)
Is the the CA ?
/DC=org/DC=cilogon/C=US/O=CILogon/CN=CILogon OSG CA 1
Chris Brew: (12:19 PM)
Got to go!
Daniela Bauer: (12:19 PM)
Because I think you have the same problem for lsst
Lukasz Kreczko: (12:19 PM)
So, do we have any grid services other than FTS that works in docker? Stuff like that would make OS upgrades easier, no?
Matt Doidge: (12:20 PM)
It's the DN that's wrong - it still has "opensciencegrid" for Dune.
Gareth Douglas Roy: (12:21 PM)
@Lukasz, I think most services need certs which means you need to run your container using the host network namespace.... which makes it difficult to run anything else on the machine
Grid services don't co-host well as far as I can tell
Matt Doidge: (12:21 PM)
The CA settings for LSST look okay, but I'll double check, they could be wrong elsewhere.
Lukasz Kreczko: (12:21 PM)
I see
Gareth Douglas Roy: (12:21 PM)
But yes if it worked it would make life much easier :D
Lukasz Kreczko: (12:22 PM)
but only the "outwards" facing part would need to run on host network namespace, right?
Robert Andrew Currie: (12:22 PM)
I'm in Edinburgh is the question was to me for Imperial
Matt Doidge: (12:22 PM)
@Daniela - aha, one of the fnal vomsdirs does have the wrong CA, thanks!
Gareth Douglas Roy: (12:22 PM)
Yes but sharing ports etc can make running multiple components on the same machine and if your only running a single container then I'm not sure what you gain
Lukasz Kreczko: (12:23 PM)
yeah, one would need multiple containers per host/VM
Alessandra Forti: (12:23 PM)
@Lukas: RAL is going to move to CentOS7 using containers
Gareth Douglas Roy: (12:23 PM)
It's actually a more compelling story for VM's as you do actually want to give each sub-item it's own external IP
Alessandra Forti: (12:24 PM)
https://indico.cern.ch/event/505613/contributions/2227447/attachments/1347485/2041461/Oral-531.pdf
I think the work is based on something Brian did at Nebraska Uni.
Lukasz Kreczko: (12:25 PM)
@Alessandra: can we copy that for smaller sites?
Gareth Douglas Roy: (12:25 PM)
@Alessandra aI'd seen that but only WN as far as I could tell they didn't mention services?
Lukasz Kreczko: (12:25 PM)
RAl is runnign some services in docker, right? This whole self-healing infrastructure sutff
Gareth Douglas Roy: (12:26 PM)
I knew they were using Mesos and I'd see lots of components but I hadn't seen how they did "grid" style services
I was looking to see if Andrew had come up with a good idea
Alessandra Forti: (12:26 PM)
I don't know. The presentation looked really interesting whether it is doable for small sites is hard to say.
Lukasz Kreczko: (12:26 PM)
BDII is the only one that I remember
Mesos is not a bad idea ;)
There are minutes attached to this event.
Show them.