ATLAS UK Cloud Support

Name: ATLAS UK Cloud Support
Start: 2020-02-20T10:00:00+00:00
End: 2020-02-20T11:00:00+00:00
Location: Vidyo

Thursday 20 Feb 2020, 10:00 → 11:00 Europe/London

Vidyo

Tim Adye (Science and Technology Facilities Council STFC (GB)), Stewart Martin-Haugh (Science and Technology Facilities Council STFC (GB))

Hide

● Outstanding tickets

GGUS #145543 James will declare the files lost.
GGUS #145610 the disk is in test, so the users shouldn't need any files there.
GGUS #145510 Stage-in issues alleviated by restarting WN xrootd gateways. Stage-out problems look like something odd in the pilot logfile. James contacted DDM Support.
GGUS #144759 not quite sure what happened, perhaps local network issues. Seems to be better now.

● CPU

QMUL dropped off over last day.
Durham outage resulted in DPM fully DOMEed. Would like to discuss removing SRM.
Glasgow has CPU sitting idle, waiting for the Ceph storage.
The LT2 pledges seem low on ATLAS chart. REBUS has 39871, but plot has 32.6k.
Alessandra: annoying that pledge line is often obscured by the graphs.
Stewart and Alessandra discussed whether http://tinyurl.com/ATLAS-Cloud-Status-UK is still valid.
Probably better to use: http://atlas-agis.cern.ch/agis/pandablacklisting/list/
Elena: ANALY_SHEF_SL7 is brokeroff there. Alessandra will put it back.

● CentOS7 - Sussex

Matt reported from the Tuesday meeting: Sussex are sorting out their worker nodes.

● Storageless sites

Sheffield is now storageless. Removed from GOCDB. Alessandra will close the JIRA ticket.

● Glasgow Ceph storage

Will move all systems to Nautilus, following new plans at RAL.
Currently jobs are failing writing back to Ceph. Sam understands the issue and will fix it after the move to Nautilus.

● Grand Unified queues

Alessandra: Grand Unification has been stopped for the UK. Now doing other countries.

Alessandra: Manchester has lots of accesses from workers from outside, eg. from Glasgow.
Gareth: Think this could be local users at Glasgow.
Alessandra has a fix for DPM/xrootd storage memory use: https://its.cern.ch/jira/browse/LCGDM-2903 . The memory use can be seen in a screenshot sent to atlas-support-cloud-uk@cern.ch. This happened when the fix was revoked by Puppet, and refixed by Alessandra.

● News round-table

Alessandra: nothing else to report
Elena: request from ATLAS for all sites to check HTCondor proxy lifetimes configuration. Will send to TB-SUPPORT@JISCMAIL.AC.UK.
Gareth: in limbo during data centre move. Will need a week downtime.
James Walder is the new ATLAS - RAL Tier1 Liaison, taking over from Stewart and Tim.
Matt: will send an email about a few lost files.
Sam: NETR
Stewart: was at the Workshop on Efficient Computing for High Energy Physics in Edinburgh. Some interesting talks, with slides here: https://indico.ph.ed.ac.uk/event/66/timetable/?view=standard
Tim: NETR

There are minutes attached to this event. Show them.

- 10:00 → 10:20
  Status 20m
  - Outstanding tickets 10m
    
    Open ATLAS UK GGUS tickets
    
    GGUS #145543 James will declare the files lost.
    GGUS #145610 the disk is in test, so the users shouldn't need any files there.
    GGUS #145510 Stage-in issues alleviated by restarting WN xrootd gateways. Stage-out problems look like something odd in the pilot logfile. James contacted DDM Support.
    GGUS #144759 not quite sure what happened, perhaps local network issues. Seems to be better now.
  - CPU 5m
    
    UK Cloud jobs over last week
    
    QMUL dropped off over last day.
    
    Durham outage resulted in DPM fully DOMEed. Would like to discuss removing SRM.
    
    Glasgow has CPU sitting idle, waiting for the Ceph storage.
    
    The LT2 pledges seem low on ATLAS chart. REBUS has 39871, but plot has 32.6k.
    
    Alessandra: annoying that pledge line is often obscured by the graphs.
    
    Stewart and Alessandra discussed whether http://tinyurl.com/ATLAS-Cloud-Status-UK is still valid.
    
    Probably better to use: http://atlas-agis.cern.ch/agis/pandablacklisting/list/
    
    Elena: ANALY_SHEF_SL7 is brokeroff there. Alessandra will put it back.
  - Other new issues 5m
- 10:20 → 10:40
  Ongoing issues 20m
  - CentOS7 - Sussex 5m
    
    Centos 7 deployment Twiki
    
    Matt reported from the Tuesday meeting: Sussex are sorting out their worker nodes.
  - Storageless sites 5m
    
    ADCINFR-147: Sheffield
    
    Sheffield is now storageless. Removed from GOCDB. Alessandra will close the JIRA ticket.
  - Glasgow Ceph storage 5m
    
    ADCINFR-152: Glasgow Ceph
    
    Will move all systems to Nautilus, following new plans at RAL.
    Currently jobs are failing writing back to Ceph. Sam understands the issue and will fix it after the move to Nautilus.
  - Grand Unified queues 5m
    
    ADCDPA-235
    
    Alessandra: Grand Unification has been stopped for the UK. Now doing other countries.
    
    Alessandra: Manchester has lots of accesses from workers from outside, eg. from Glasgow.
    Gareth: Think this could be local users at Glasgow.
    Alessandra has a fix for DPM/xrootd storage memory use: https://its.cern.ch/jira/browse/LCGDM-2903 . The memory use can be seen in a screenshot sent to atlas-support-cloud-uk@cern.ch. This happened when the fix was revoked by Puppet, and refixed by Alessandra.
- 10:40 → 10:50
  News round-table 10m
  Alessandra: nothing else to report
  
  Elena: request from ATLAS for all sites to check HTCondor proxy lifetimes configuration. Will send to TB-SUPPORT@JISCMAIL.AC.UK.
  
  Gareth: in limbo during data centre move. Will need a week downtime.
  
  James Walder is the new ATLAS - RAL Tier1 Liaison, taking over from Stewart and Tim.
  
  Matt: will send an email about a few lost files.
  
  Sam: NETR
  
  Stewart: was at the Workshop on Efficient Computing for High Energy Physics in Edinburgh. Some interesting talks, with slides here: https://indico.ph.ed.ac.uk/event/66/timetable/?view=standard
  
  Tim: NETR
- 10:50 → 11:00
  
  AOB 10m