Operations team & Sites

EVO - GridPP Operations team meeting

EVO - GridPP Operations team meeting


- This is the weekly GridPP ops & sites meeting

- The intention is to run the meeting in VidyoConnect: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6

-- The PIN is 1234. To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone.

-- The London (UK) service is on +442030510622.

-- The meeting extension is 109308582. PIN 1234

Chair:  Matt



Videoconference Rooms
- This is the weekly GridPP ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the Janet(UK) Community area. Direct link http://evo.caltech.edu/evoNext/koala.jnlp?meeting=MDMaM82v2nD2Du999sD99D - The phone bridge number is +44 (0)161 306 6802. The phone bridge ID is 1001002 with code: 4880. Apologies:
Alessandra Forti
Auto-join URL
Useful links
Phone numbers

Minutes 30/4/2019

* Alessandra Forti
* Andrew McNab
* Brian Davies
* Chris Brew
* Daniel Traynor
* Daniela Bauer
* Darren Moore
* David Crooks
* Elena Korolkova
* Emanuele Simili
* Gareth Roy
* Gordon Stewart
* Ian Loader
* John Hill
* Kashif Mohammad
* Linda Cornwall
* Matt Doidge
* Paige Lacesso
* Pete Clarke
* Pete Gronbech
* Raja Nandakumar
* Robert Frank
* Sam Skipsey
* Steve Jones
* Teng

LHCB Report
* Problems with aborted Pilots at ECDF
* Another storage pool moved from CASTOR to ECHO at RAL, this leaves only the USER space to move

* Brunel has data transfer problems, nothing else to report for CMS>

* Brunel having problem with file deletions for ATLAS. BD says this is an issue with WebDAV being unstable.
* RHUL has an issue with Squid servers being shown as red when on of the HA servers is available

VO Incubator
* Discussion of DUNE queues at Lancaster, jobs being submitted to old SL6 queue. MD to email AMcN to have queues moved over.
* LFC to DFC migration for T2K and Mice ongoing, progress being made by DB and SF

Meetings and Update

General Updates
* Discussion of GridPP42 and associated presentations.
* Consultation on GridPP43 potential dates and locations (nominally located at Ambleside 20-22nd August).
    - Problems were raised in respect to the timing being close to the end of the summer holidays.
    - PC encouraged emails to DB to raise any questions/concerns.

* Technical meeting regarding DPM and future within the UK.
    - AF comments that many sites are planning to move from DPM (whether as smaller sites or moves to different storage solutions).
    - KM comments that Oxfords plan is to update to DOME.

    - Registration now appears to be open for HEPSYSMAN.
    - DC asks if anyone has security topics to cover as part of the training if the could let him know.

* TB-SUPPORT discussion about SW areas and whether or not they are needed. It appears that now all VOs are either using CMVFS or containers so it is unlikely that a SW area is needed.

Tier-1 Status
* High CMS failures seem but this appears to have improved.

* no report

Tier-2 Evo
* no report

* no report

* SJ will check Fermilab VO information

* no report

* no report

On Duty
* DB on duty, nothing to report

* Nothing ongoing that sites need to be concerned about.
* Dockerhub breach for sites that may have been using this for any reason.
* Trust anchors have been updated, sites should install as soon as they are able.

* no report

* 131608 - Needs an update as being escalated to VO Manager
* 139101 - no news
* 140679 - ongoing

Discussion & AOB
* Site Round Table

Manchester     - going into downtime to upgrade DPM headnode
RALPP        - nothing to report
QMUL        - nothing to report
IC        - preparing for the move to Slough and IRIS cloud
Sheffield    - nothing to report
Glasgow        - CEPH and HTCondor-CE
Cambridge    - nothing to report
Oxford        - SL6 CE now in downtime to be retired (CentOS7 only)
Lancaster    - Robin leaving Lancaster
Bristol        - Condor and OS upgrades.
Liverpool    - Attending EGI to talk about CREAM-CE migration
Edinburgh    - working on a second CE as current is having problems

* HEPSYSMAN - registration open please register if you'd like to attend.

* GridPP6 - PC updates on current status, panel to take place next week.
* PC congratulates GridPP on it's ability to aid LSST and getting it's payloads running via GridPPs infrastructure.
* Indigo IAM vs EGI Check-In as a SSO solution, both AF and DC felt that IAM was a better solution then the EGI Check-In system.



Matt Doidge: (30/04/2019 11:03)

Gareth is kindly taking minutes

Elena Korolkova: (11:06 AM)



Matt Doidge: (11:12 AM)


David Crooks: (11:31 AM)


Raja: (11:33 AM)

Apologies - got to go now

Paige Winslowe Lacesso: (11:33 AM)

apologies back soon

John Hill: (11:43 AM)


David Crooks: (11:43 AM)

I was just about to post that as well :)

Alessandra: (11:51 AM)


There are minutes attached to this event. Show them.
    • 11:00 11:01
      Ops meeting minutes 1m
      • This is a reminder that this is an important task. The minute taker gives access to the discussions for those not present and provides a reference for others to refer back to afterwards.

      • The team composition has been changing. If everybody contributes then the task comes around less often.

      • Please extract actions from the meeting and add them to our table here: https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items#Action_list.

      • Recent allocations: See above link. The page should be updated each week by the minute taker (if they don't the task will keep coming to them!).

      • Upcoming allocations:

    • 11:01 11:20
      Experiment problems/issues 19m

      Review of weekly issues by experiment/VO

      • LHCb

      • CMS
        T1: https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL
        T2: https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T2_UK_London_Brunel

      • ATLAS

      • Other: Updates should be recorded in https://www.gridpp.ac.uk/wiki/GridPP_VO_Incubator.

      Also from Daniela:
      *T2K (LFC to DFC): We really really need QMUL which is a major T2K site to deal with their storage. Details in:
      We haven't quite got round to the three small sites (LIV, OX, SHEF) still missing (because I spend all my time setting up an IRIS cloud), but we haven't forgotten.

      *MICE (LFC to DFC): This is going much better (less sites, less data).

      *LZ changed one of their voms servers. The Operations Portal has updated now. If you support LZ, please check if:
      [root@gfe02 ~]# cat /etc/grid-security/vomsdir/lz/voms.hep.wisc.edu.lsc
      /DC=org/DC=incommon/C=US/ST=WI/L=Madison/O=University of Wisconsin-Madison/OU=OCIS/CN=voms.hep.wisc.edu
      /C=US/O=Internet2/OU=InCommon/CN=InCommon IGTF Server CA
      is up to date.

      • GridPP DIRAC status [Andrew McNab]
        -- https://www.gridpp.ac.uk/gridpp-dirac-sam
    • 11:20 11:40
      Meetings & updates 20m

      With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest

      • General updates
      • WLCG ops coordination
      • Tier-1 status
      • Storage and data management
      • Tier-2 Evolution
      • Accounting
      • Documentation
      • Interoperation
      • Monitoring
      • On-duty
      • Security
      • Services
      • Tickets
      • Tools
      • VOs
      • Site updates
    • 11:40 12:20
      Discussion topics 40m
      • February GDB: https://indico.cern.ch/event/739875/
      • Site roundtable.
    • 12:20 12:25
      Actions & AOB 5m