Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234. To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +44 (0)161 306 6802. Phone bridge ID 1001002
-- The meeting extension is 109308582. PIN 1234
Chair: Jeremy C
Minutes: Duncan R
Apologies: Daniela B, Andrew M
Present: David C, Duncan R, Elena K, Gordon S, Ian L, Jeremy C (Chair/notes), John H, Kashif M, Linda C, Lukasz K, Mark S, Matt D, Winnie L, Raja N, Sam S, Steve J, Tony (RHUL)
Apologies: Andrew M, Daniela B
Experiment problems & issues
LHCb – Sheffield. Problem being followed up. Otherwise okay.
1PB from CASTOR to ECHO. Waiting on green light for more transfer. Doing about 100TB/week.
DUNE – RAL up and running. Storage is working. Able to direct copies from FermiLab with minor catch – working around at Fermi needed. Trying to to 3rd party copy from CERN to ECHO. FTS – need same protocol on both ends. (EOS-ECHO check).
CMS (from Daniela by email) - There seems to be no news from CMS, except the T1 is looking a bit dejected on the availability plots (seems to be getting better though). The T2s look fine.
ATLAS – Yesterday WLCG meeting. ATLAS has some problems at T0. Rucio readers are overloaded. Also issues with GOCDB link not working for some.
Short UK meeting last time. No further news other than several tickets being open. Sheffield – transfer operation timeout. Elena does not see the errors in logs though. Will check Sheffield FTS configuration. ECDF ticket – jobs failing with lcgcp command not found. The other issue is ECDF deletion errors. One ticket on Manchester to do with transfers and a data server that recently came online. Matt said there was an issue with multi-thread user jobs and the user was contacted.
For other VOs JC mentioned LZ moving on to production jobs and the 8GB memory requirement leading the jobs mainly to RALPP and IC. Also LSST have had some issues with Ganga - they should perhaps use GridPP DIRAC tooling rather than Ganga given their needs?
Looking at the GridPP DIRAC SAM results - Sheffield was noted as not having jobs flowing recently. Elenena noted that a new sysadmin at the site had made changes that she needs to correct with with the configuration.
https://www.gridpp.ac.uk/gridpp-dirac-sam
The meeting looked at the bulletin updates. https://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
Mainly information updates (see bulletin directly)
Vip: CVMFS – Nagios test starts failing. Can we run script to automatically reload CVMFS. Will it affect any running job? Oxford has issue with caching. Nagios checks for mismatch. If warning offline that node for that VO. Wait for job to finish and reload automatically or wait? Nobody had an answer so JC suggested Oxford try it and report the results.
Quiet meeting storage and data management.
On-duty - JC will update Rota for the medium term.
Interoperation area – Kashif mentioned that there was a cream update umd-4.7 that people should be aware of now.
There was a question about whether anybody was interested in MPI for CentOS7? CREAM-CEs could do it but it was not much used. Sites on the call asked for more information about which VOs wanted to use MPI.
Security: Not much operationally.
Linda has a talk at DI4R.
Matt reviewed the tickets - at this point it was clear that Matt's VidyoConnect client did not allow his audio to be heard by JC's client! Everybody else could hear everything... anyway no minutes from this part of the meeting.
Under discussion topics:
1. PMB interested to know if there is enough commonality of site hardware (CPU, storage, configuration...) to explore a DELL framework agreement as happened some years ago. PB and AM actioned to follow up by the PMB.
2. JC reviewed the background papers being prepared ahead of GridPP41. Basically covering areas: storage, deployment, Tier-1, experiment support etc. The idea is to capture the options and discuss in dedicated sessions at Ambleside.
3. If anyone wants to volunteer a talk for GridPP41 please let Pete or Jeremy know. Also be reminded that in parallel with the PMB on the Tuesday is a hack event.
4. JC gave a summary of the July GDB and talked to the slides available at https://indico.cern.ch/event/651355/
There are minutes attached to this event.
Show them.