Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234.
To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +44 (0)161 306 6802. Phone bridge ID 1001002
-- The meeting extension is 109308582. PIN 1234
Chair: Pete
Minutes:
Reserve:
Apologies:
Minutes for GridPP ops & sites meeting 14/02/2017:
Chair: Peter Gronbech
Minutes: Oliver Smith
Participants:
Adam Boutcher, Alessandra Forti, Andrew Washbrook, Andrew McNab, Brian Davies, Catalin Condurache, Chris Brew, Daniela Bauer, David Crooks, Elena Korolkova, Gareth Roy, Gordon Stewart, Ian Loader, Ian Neilson, John Bland, John Hill, John Kelly, Kashif Mohammed, Lukasz Kreczko, Mark Slater, Matt Doidge, Oliver Smith, Paige Winslowe Lacesso, Peter Gronbech, Raul Lopez, Rob Currie, Sam Skipsey, Steve Jones, Terence Froy, Vip Davida
Experiment problems/issues
LHCb: AMN: No issues
CMS: DB:
- Bristol out of waiting room. Small issue left to resolve
ATLAS: EK:
Tickets:
- ECDF: Deletion failures but now looks fine
- Oxford: 125822 – Dark data
- QMUL 125627 – Transfer problems, waiting on submitters
- QMUL Squid issue – on hold
Other VO:
- HPC Dirac: HPC Cambridge, EPCC Edinburgh, Leicester – test transfers done.
- LIGO: SS to have a chat with
- LZ: EK to update page with public info.
- DUNE: EK to update page
- PRAVDA: Contact left PRAVDA, no updates for the moment. MS to get update.
- SKA: AMN: Data loss faced due to Dirac file loss. AMN to add update to wiki.
- SNO+: Do we keep on Incubator page? To be discussed. Starting to run out of space at sites, discussion to add space tokens for SNO+.
Dirac: All fine. IC Cloud to be removed.
Meetings & updates
General updates:
- The GDB for February took place last Wednesday. Summary will appear here.
- Notes from today's WLCG ops meeting are available. No specific UK issues but useful experiment summaries.
- Next HEPSYSMAN meeting 13th June to 15th at RAL.
- Request for topics for main meeting
- A reminder to register for the next GridPP collaboration meeting please.
WLCG ops coordination:
- No report
Tier 1 status:
- As reported last week the Castor 2.1.15 upgrade has been completed. However, there have been problems with CMS xroot redirection tests following this. The next change for Castor is to to update the SRMs.
- The migration of LHCb data to the T10KD tape media has been completed. All Tier1 data is now on this media.
- CC: Looking to retire WMS services at T1. Looking for feedback on this. DB: MICE depending on this at the moment?
- CC: Moving LFC from Oracle to MySQL, looking to remove LFCs that no longer use this. Will contact VOs directly.
Storage management:
- SS: Focus on lightweight sites at the minute.
Tier2 Evolution:
- Vac installed at Birmingham. Running LHCb production jobs. Preparing to run ALICE VMs.
- MS: Single node install at present, will start moving more nodes over. Aiming to add more VOs and move whole site over
- ALICE VMs switched to using GSI proxies internally rather than host cert/keys.
- AMN: Proxy generated per VM from host certificate and inserted into VM rather than full host certificate. Proxy lifetime is matched to VM lifetime.
Accounting:
- No report
Documentation:
- Review of keydocs, reminder functionality broken at present. Either update docs or request deletion of pages deemed unnecessary
Interoperations:
- DC: EGI Ops meeting
- UMD4.4: Frontier Squid 3, configuration changes required if updating from v2.
- IPv6 status: April 2018 dual stack target, EGI ops request for NGIs to begin discussion with sites. “NGIs/ROCs please start discussing with sites and provide suggestions for the overall plan”
- Decommissioning dCache 2.10
- Can’t do direct upgrade from 2.10 – 2.16, suggested to upgrade 2.10>2.13>2.16
- CB: 2.13 at the minute, planning upgrade to 2.16 in future
- DB: Will look at, likely recent version already
- Storage accounting, looking for test sites. UK looking well
- Proposal to modify the declaration of scheduled interventions
- Additional tier added, looking for feedback
- ARGO: proposal to use GOCDB as only source of topology information
- VAPOR: Mentioned on agenda but not discussed
- SL5 UK updates:
- GS SL5 DPM to SL7 next week
- T1 SL5 following Castor upgrade March
- ARGO: Stale ARC probes – known problem, being looked into. Two endpoints, results might be different. Recommendation to check both endpoints.
- VO Nagios – no update at present
- Next meeting 13th March
- PG: CentOS7 vs SL7 vs CC7 – thoughts?
- AF: SL7 looks to be the final version of SL, suggestion to use CentOS7.
- PG: CERN version of CentOS7 has CERN specific additions, recommendation not to use it.
- Majority seem to be moving to CentOS7, OX using SL7
- JK: T1 test node looks to be using SL7, not sure if this is a final decision
- EGI bringing up Docker hub service – looking at providing services in docker containers
Monitoring:
- AF: CERN moving to Kibana, dashboards/SAM tests changing. Looking for feedback. Will send links once available.
On-duty:
- Gordon on duty this week
Rollout:
- Please can we hear something from Raul and Sam (or others contributing to the middleware readiness work) about news in this area at the ops meeting next week? Thank you.
- Does anyone want specific questions in this area tackled in a roundtable discussion?
Security:
- EGI-SVG-2017-12391 - Just a check (JC). Attacks on Hadoop installations - sites should check they are securely configured. [1]
- Security team meeting this afternoon.
Services:
- The development dashboard is now operational again. However the production dashboard is not currently displaying latency results (OSG ticket submitted).
- Birmingham pS host had lost its entire network connection and both Lancaster's hosts their IPv6 connection. Durham results are incomplete, GGUS ticket submitted (#126587).
Tickets:
25 Open UK Tickets this Week
ATLAS want your INPUT 126184
Atlas request for input on sites monitoring.
TOKEN AFFECTION
126554 (10/2)
Sno+ jobs failed at Liverpool, and once again John B had to educate a user group that space tokens are a thing (thanks John!).
SS: Campaign to run without SRMs, this will remove space tokens. DPM plans to move away from space tokens to quotas.
BD: Only other site with space tokens for SNO+ is Alberta, suggestion to ask SNO+ what tokens they need.
UNRELIABLE AVAILABILITY
126349 - ECDF
125743- RALPP
Both of these availability tickets are confusing the sites and myself (although the latter is still quite easy to do). ECDF are getting negative results again (and a lot of unknowns) and RALPP seem to be not updating results very often at all, suffering a several day lag by the looks of it.
DC: RALPP - ARC Stale probe problem? Suggestion to submit ticket for this. Kashif to send existing ticket number
Tools
Actions
- Enquire if Docker is run in privileged mode at IISAS-GPUCloud.
- Create wiki table tracking recommendations for sites from Jan-17 ATLAS jamboree
- SS: To do storage section
- AF: Will review next week
GDB
https://indico.cern.ch/event/614010/contributions/2475517/attachments/1412039/2160229/GDB-Feb-17.pdf
- April pre-GDB – collaborating with other communities
- AMN: Looking for people with links to other communities, how we can work on things together.
- WISE 27th -29th March; LHCOPN-LHCONE 4-5 April; 24th -28th April HEPiX Budapest
- CERN AFS phaseout
- Phaseout process in link above
- Pre-GDB on Benchmarking
- LHCb: DB12 and MJF
- Atlas: KV
- No recommendation yet, CMS/ATLAS don’t yet have data. Deferred until April GDB.
- EOS Workshop
- CERNs replacement filesystem
- SS: Not suggested to install EOS at T2 sites.
- Cloud Services for Synchronisation and Sharing
- CERN Tape Archive
- CASTOR Tape replacement, working with EOS. Not for use outside of CERN.
- Baseline for WLCG Stratum 1 Operation
- CVMFS discussion on scale due to widespread use
Vidyo Chat window log:
kashif: (14/02/2017 11:13:07)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=126473
For Elena
Elena Korolkova: (11:13 AM)
thanks, Kashif
Peter Gronbech: (11:26 AM)
https://www.gridpp.ac.uk/gridpp-dirac-sam
Andrew McNab: (11:33 AM)
I've removed the IC CLOUD site so that DIRAC sam page is all successes from today now. Well done everyone!
Peter Gronbech: (11:49 AM)
https://www.gridpp.ac.uk/keydocs?sort=area
David Crooks: (11:49 AM)
https://wiki.egi.eu/wiki/Agenda-13-02-2017
Daniela Bauer: (11:57 AM)
dcache-2.13.25-1.noarch
Paige Winslowe Lacesso: (11:58 AM)
SORRY have to leave now
Brian Davies @RAL-LCG2: (12:01 PM)
http://operations-portal.egi.eu/vapor/resources/GL2ResSummary?NGI=NGI_UK&filter=site
Gareth Douglas Roy: (12:04 PM)
Sorry, another meeting across campus so need to leave
Gordon Stewart: (12:04 PM)
Also leaving for another meeting. I'm ROD this week, but have nothing to report.
raul: (12:07 PM)
brunel on CentOS for several years
John Kelly (RAL): (12:07 PM)
I have just checked one of Andrew's test workers - is is running sl7
Lukasz Kreczko: (12:09 PM)
7.2 or 7.3?
Mark Slater: (12:10 PM)
for I'm afraid I have to go - apologies!
John Kelly (RAL): (12:10 PM)
[root@lcg1628 jpk]# cat /etc/redhat-release
Scientific Linux release 7.2 (Nitrogen)
[root@lcg1628 jpk]#
Chris Brew: (12:10 PM)
Rolling! If Redhat doesn't bother about minor versions I'm not going to!
Lukasz Kreczko: (12:11 PM)
:D
I hope that goes all well
not that suddenly you have incompatible binaries or something ;)
yup
not accessible from the outside
but we will update anyway to be safe
raul: (12:16 PM)
Sorry, I've got to leave
John Bland: (12:19 PM)
they'll take space tokens from my cold, dead hands (or when they replace them with quota-tokens, whichever's easier)
Andrew John Washbrook: (12:20 PM)
I am not even sure what negative availability means ;)
anti-available
Terence Stephen Froy: (12:24 PM)
Nothing to report from QM except Dan is republishing our accounting data from November onwards (shame really as >100% availability was quite a nice statistic to boast!)
kashif: (12:25 PM)
https://ggus.eu/?mode=ticket_info&ticket_id=125804
The ticket has been closed now but as David mentioned, we still see stale results in Argo
Daniela Bauer: (12:29 PM)
Sorry, I will have to leave now.
Peter Gronbech: (12:36 PM)
https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
There are minutes attached to this event.
Show them.