ATLAS UK Cloud Support

Name: ATLAS UK Cloud Support
Start: 2020-04-02T10:00:00+01:00
End: 2020-04-02T11:00:00+01:00
Location: Vidyo

Thursday 2 Apr 2020, 10:00 → 11:00 Europe/London

Vidyo

Tim Adye (Science and Technology Facilities Council STFC (GB)), James William Walder (Science and Technology Facilities Council STFC (GB))

Hide

● Outstanding tickets

GGUS #146374 Elena reported issues in job configs. "We have a problem with job submission to our cluster. We are trying to fix arc ce and to install a new arc ce. We are working on this. Unfortunately we can’t work ops this full time."
GGUS #146355 timeout on stage-out. Could be same issue RAL sees with Rucio. (Alessandra says Cisco box is now 15 years old.)
GGUS #146280 Matt reported that as they get more data off the bad node, they are seeing fewer failures. Will take another ~week for draining. Will keep ticket open in the meantime.
GGUS #146159 issues consolidated. Dimitrios has sorted out the assignments. Glasgow Network busy, but continuing. Matt can get any files requested, but can't make them available for transfers.
GGUS #145688 Alessandra said this is not a priority, so will put ticket on hold.
GGUS #145510 James said this issue is on hold while waiting for new WN with SSDs. In the meantime, he is studying transfer rates.
GGUS #144759 Sam said this is low priority, and would require access to datacentre to fix. Currently this is difficult. Gareth is ill.

● CPU

All pledges increased on 1st April.
Sam will follow up on Durham patchy production.
ScotGrid pledge down a bit. Sam doesn't know why, but can check individual pledges from sites. REBUS information provided only at federation level (for pledges). LHCb is almost halved in pledge.

● Other new issues

James: A few blacklisting events, which recovered quite quickly. Could be due to stage-outs or disk issues.

● CentOS7 - Sussex

Peter updated the GGUS ticket. We can update AGIS settings when know memory limit from Patrick.

● Glasgow Ceph storage

Ceph migration going slowly due to a weird problem. Waiting for help from the Ceph community.

● Grand Unified queues

All Grand Unified queues now created. Many analysis and production queues now draining in brokeroff.

● News round-table

Alessandra: NTR. James asked about unpacked containers in cvmfs. Alessandra says that the check of the sites is postponed due to Covid-19 work.
Dan: NTR
James: NTR
Matt: Elena is no longer in ATLAS. Need replacement for her ATLAS reports in the GridPP Ops meeting. Alessandra can feedback tickets and news from the ATLAS Cloud meeting, but won't be able to attend WLGC Ops meeting to pass on news from there.
Peter: NTR
Sam: NTR
Tim: Wants RAL to be added to stress tests for the DOMA TPC group. Alessandra said we are waiting for several sites to be added. She will poke.

● AOB

Vidyo or Zoom in future? Stick with Vidyo for the moment. Review in a month's time.
Best wishes to Gareth.

There are minutes attached to this event. Show them.

- 10:00 → 10:20
  Status 20m
  - Outstanding tickets 10m
    
    Open ATLAS UK GGUS tickets
    
    GGUS #146374 Elena reported issues in job configs. "We have a problem with job submission to our cluster. We are trying to fix arc ce and to install a new arc ce. We are working on this. Unfortunately we can’t work ops this full time."
    
    GGUS #146355 timeout on stage-out. Could be same issue RAL sees with Rucio. (Alessandra says Cisco box is now 15 years old.)
    
    GGUS #146280 Matt reported that as they get more data off the bad node, they are seeing fewer failures. Will take another ~week for draining. Will keep ticket open in the meantime.
    
    GGUS #146159 issues consolidated. Dimitrios has sorted out the assignments. Glasgow Network busy, but continuing. Matt can get any files requested, but can't make them available for transfers.
    
    GGUS #145688 Alessandra said this is not a priority, so will put ticket on hold.
    
    GGUS #145510 James said this issue is on hold while waiting for new WN with SSDs. In the meantime, he is studying transfer rates.
    
    GGUS #144759 Sam said this is low priority, and would require access to datacentre to fix. Currently this is difficult. Gareth is ill.
  - CPU 5m
    
    UK Cloud jobs over last week
    
    All pledges increased on 1st April.
    
    Sam will follow up on Durham patchy production.
    
    ScotGrid pledge down a bit. Sam doesn't know why, but can check individual pledges from sites. REBUS information provided only at federation level (for pledges). LHCb is almost halved in pledge.
  - Other new issues 5m
    
    US transfers rate to Glasgow
    the rate from BNL and Chicago to Glasgow is poor - less than 1MB/s. Files bigger than a certain size timeout
    
    https://lcgfts05.gridpp.rl.ac.uk:8449/var/log/fts3/2020-04-02/dcsrm.usatlas.bnl.gov__svr018.gla.scotgrid.ac.uk/2020-04-02-0336__dcsrm.usatlas.bnl.gov__svr018.gla.scotgrid.ac.uk__1921237810__b7410d4c-06b3-54c0-a4ea-c53397a55cb5
    https://lcgfts05.gridpp.rl.ac.uk:8449/var/log/fts3/2020-03-31/uct2-dc1.uchicago.edu__svr018.gla.scotgrid.ac.uk/2020-03-31-2030__uct2-dc1.uchicago.edu__svr018.gla.scotgrid.ac.uk__1919147842__3f985a4b-401a-52c0-b2c3-7153c0b9dba7
    
    Hammercloud blacklisting events:
    http://hammercloud.cern.ch/hc/app/atlas/robot/blacklisting/?cloud=UK&action=blacklisting&hours=168
    
    James: A few blacklisting events, which recovered quite quickly. Could be due to stage-outs or disk issues.
- 10:20 → 10:40
  Ongoing issues 20m
  - CentOS7 - Sussex 5m
    
    Centos 7 deployment Twiki
    
    Peter updated the GGUS ticket. We can update AGIS settings when know memory limit from Patrick.
  - Glasgow Ceph storage 5m
    
    ADCINFR-152: Glasgow Ceph
    
    Ceph migration going slowly due to a weird problem. Waiting for help from the Ceph community.
  - Grand Unified queues 5m
    
    ADCDPA-235
    
    Migration plans
    
    All Grand Unified queues now created. Many analysis and production queues now draining in brokeroff.
- 10:40 → 10:50
  News round-table 10m
  Alessandra: NTR. James asked about unpacked containers in cvmfs. Alessandra says that the check of the sites is postponed due to Covid-19 work.
  
  Dan: NTR
  
  James: NTR
  
  Matt: Elena is no longer in ATLAS. Need replacement for her ATLAS reports in the GridPP Ops meeting. Alessandra can feedback tickets and news from the ATLAS Cloud meeting, but won't be able to attend WLGC Ops meeting to pass on news from there.
  
  Peter: NTR
  
  Sam: NTR
  
  Tim: Wants RAL to be added to stress tests for the DOMA TPC group. Alessandra said we are waiting for several sites to be added. She will poke.
- 10:50 → 11:00
  AOB 10m
  Vidyo or Zoom in future? Stick with Vidyo for the moment. Review in a month's time.
  
  Best wishes to Gareth.