ATLAS UK Cloud Support

Name: ATLAS UK Cloud Support
Start: 2019-11-21T10:00:00+00:00
End: 2019-11-21T11:00:00+00:00
Location: Vidyo

Thursday 21 Nov 2019, 10:00 → 11:00 Europe/London

Vidyo

Tim Adye (Science and Technology Facilities Council STFC (GB)), Stewart Martin-Haugh (Science and Technology Facilities Council STFC (GB))

Hide

● Outstanding tickets

https://ggus.eu/?mode=ticket_info&ticket_id=144103 a file was lost from Glagow. Sam generated a fresh dump, which should mark the files as "suspicious".
https://ggus.eu/?mode=ticket_info&ticket_id=144155 one disk server had 2 failures on old disks, meaning that 28 TB of data was lost. Sam will send a list of files to be marked lost in Rucio. Will be reported on atlas-support-cloud-uk@cern.ch, and Tim will do Rucio.

https://ggus.eu/?mode=ticket_info&ticket_id=144034 Birmingham was using the CERN Frontier proxy. This was due to incorrect settings in a a local user's jobs. Should now be fixed, though Tim remarked that the new settings shouldn't refer to the RAL Frontier service (which is being decommissioned). He'll comment on the ticket.

https://ggus.eu/?mode=ticket_info&ticket_id=143901 Sheffield: Elena made fixes. Will see how it goes.

https://ggus.eu/?mode=ticket_info&ticket_id=142329 Patrick now has the ARC-CE set up at Sussex. Configured lc maps. Next is ATLAS setup. Will send name of CE (grid-arc-01.hpc.susx.ac.uk) to Cloud support. Elena will help.

● Other new issues

Oxford Datacentre downtime for A/C upgrade and power supply upgrades. Also retired some old VO nodes. Should be back Friday or Monday.

● Storageless sites

Elena: once other problems resolved, can try switching directio->staged for analysis queues.
Will continue to clean DATADISK once a Rucio bug is fixed.

● Glasgow Ceph storage

Storage off for couple of days. Now back. Will test this week. Then test internal data flows before testing with ATLAS.
Tim will open a JIRA ticket to track ATLAS testing.

● News round-table

Dan:
Lots more info about arc6 can be found here https://indico.cern.ch/event/739898/. Looks like there was a pre GDB dedicated to arc6.
QMUL is red on Alessandra's heatmap (despite green other UK sites). A quick test seemed to work, so will investigate further.
Elena: NTR
Matt: is concerned that the Quarterly Reports have been taking a long time. Might help if we keep track in the minutes of problems, even if they are fixed.
Patrick: NTR
Sam: NTR
Vip: Oxford have a new Sysadmin. Mike Leech arrives 2nd December. Vip will introduce him to this group.
Tim: Testing Rucio copytool (instead of xrdcp) for accessing Echo from jobs in test queue. Following a fix in Rucio, stage-in works, but stage-out fails. Investigations continue. https://its.cern.ch/jira/browse/ATLDDMOPS-5503

There are minutes attached to this event. Show them.

- 10:00 → 10:10
  
  Outstanding tickets 10m
  
  Open ATLAS UK GGUS tickets
  
  UK Cloud jobs over last week
  
  https://ggus.eu/?mode=ticket_info&ticket_id=144103 a file was lost from Glagow. Sam generated a fresh dump, which should mark the files as "suspicious".
  https://ggus.eu/?mode=ticket_info&ticket_id=144155 one disk server had 2 failures on old disks, meaning that 28 TB of data was lost. Sam will send a list of files to be marked lost in Rucio. Will be reported on atlas-support-cloud-uk@cern.ch, and Tim will do Rucio.
  
  https://ggus.eu/?mode=ticket_info&ticket_id=144034 Birmingham was using the CERN Frontier proxy. This was due to incorrect settings in a a local user's jobs. Should now be fixed, though Tim remarked that the new settings shouldn't refer to the RAL Frontier service (which is being decommissioned). He'll comment on the ticket.
  
  https://ggus.eu/?mode=ticket_info&ticket_id=143901 Sheffield: Elena made fixes. Will see how it goes.
  
  https://ggus.eu/?mode=ticket_info&ticket_id=142329 Patrick now has the ARC-CE set up at Sussex. Configured lc maps. Next is ATLAS setup. Will send name of CE (grid-arc-01.hpc.susx.ac.uk) to Cloud support. Elena will help.
- 10:10 → 10:20
  
  Other new issues 10m
  
  Oxford Datacentre downtime for A/C upgrade and power supply upgrades. Also retired some old VO nodes. Should be back Friday or Monday.
- 10:20 → 10:40
  Ongoing issues 20m
  - Centos 7 migration 5m
    
    Centos 7 deployment Twiki
  - Storageless sites 5m
    
    Sheffield
    
    Elena: once other problems resolved, can try switching directio->staged for analysis queues.
    Will continue to clean DATADISK once a Rucio bug is fixed.
  - Glasgow Ceph storage 5m
    
    Storage off for couple of days. Now back. Will test this week. Then test internal data flows before testing with ATLAS.
    Tim will open a JIRA ticket to track ATLAS testing.
- 10:40 → 10:50
  
  News round-table 10m
  
  Dan:
  Lots more info about arc6 can be found here https://indico.cern.ch/event/739898/. Looks like there was a pre GDB dedicated to arc6.
  QMUL is red on Alessandra's heatmap (despite green other UK sites). A quick test seemed to work, so will investigate further.
  Elena: NTR
  Matt: is concerned that the Quarterly Reports have been taking a long time. Might help if we keep track in the minutes of problems, even if they are fixed.
  Patrick: NTR
  Sam: NTR
  Vip: Oxford have a new Sysadmin. Mike Leech arrives 2nd December. Vip will introduce him to this group.
  Tim: Testing Rucio copytool (instead of xrdcp) for accessing Echo from jobs in test queue. Following a fix in Rucio, stage-in works, but stage-out fails. Investigations continue. https://its.cern.ch/jira/browse/ATLDDMOPS-5503
- 10:50 → 11:00
  
  AOB 10m