ATLAS UK Cloud Support

Name: ATLAS UK Cloud Support
Start: 2020-08-27T10:00:00+01:00
End: 2020-08-27T11:00:00+01:00
Location: Vidyo

Thursday 27 Aug 2020, 10:00 → 11:00 Europe/London

Vidyo

Tim Adye (Science and Technology Facilities Council STFC (GB)), James William Walder (Science and Technology Facilities Council STFC (GB))

Hide

Outstanding tickets

148342 UKI-SCOTGRID-GLASGOW less urgent assigned 2020-08-24 11:56:00 UKI-SCOTGRID-GLASGOW with transfer efficiency degraded and many failures
- One (DPM) disk server with a possible issue, trying to rebuild raidset.
- Finish current set of drains before tackling above issue
- JW to send current set of transfer failures to SS.
148339 UKI-NORTHGRID-LIV-HEP less urgent in progress 2020-08-24 08:41:00 0 transfer efficiency to UKI-NORTHGRID-LIV-HEP_LOCALGROUPDISK endpoint
- No more failures; unclear why endpoint was available given endpoint full.
- JW to prioritise UK localgroupdisk cleanup.
147979 UKI-NORTHGRID-MAN-HEP less urgent in progress 2020-08-27 05:55:00 UKI-NORTHGRID-MAN-HEP timeout transfer errros and also deletion errors
- JW updated ticket asking for status report.
146771 UKI-SCOTGRID-ECDF less urgent in progress 2020-08-20 14:44:00 UKI-SCOTGRID-ECDF deletion failures with “The requested service is not available at the moment.”
- No further changes applied sofar
146651 RAL-LCG2 urgent on hold 2020-08-10 10:59:00 singularity and user NS setup at RAL
- on hold
146374 UKI-NORTHGRID-SHEF-HEP urgent in progress 2020-07-22 14:53:00 ATLAS pilot jobs idle on UKI-NORTHGRID-SHEF-HEP CE
- no update
144759 UKI-SCOTGRID-GLASGOW less urgent on hold 2020-08-10 09:54:00 High traffic from UKI-SCOTGRID-GLASGOW on RAL CVMFS Stratum1
- on hold
142329 UKI-SOUTHGRID-SUSX top priority on hold 2020-06-04 14:05:00 CentOS7 migration UKI-SOUTHGRID-SUSX
- on hold

CPU

RAL
- Difference in cpu to pledge. updating corepower (set to 10), to 11.7 (approx current value) would give consistent values with internal monitoring.
Northgrid
- General comments on the nature and use of current storage model.
  - Agressive use of storage by the sites; how to move forward, while keeping legacy ‘frameworks’ supported
- LANCS downtime from disk related issues.
London
SouthGrid
- OX back running after Power failure; not yet at full capacity.
Scotgrid

Other new issues

RAL: Storage downtime; 10hr; batch farm will also be offline.
Alessendra email: TPC, DPM, Macaroons
- ECDF, OX aware of news.

Ongoing issues

CentOS7 - Sussex
Grand Unified queues

News round-table

Dan
- Migration to new lustre; Atlas scratch disk migration can cause problems (highly volatile).
  - Set up a scratchdisk on another site temporarily ?
  - Could new disk be installed first, and Old data still readable? Probably not.
  - JW to create Jira - ticket, for DDM to advise on the migration.
Peter
- NTR
Sam
- Current storage set at 500TB for CEPH; site to update atlas srr manually to reasonable value.
Gareth
- ntr
Tim
- NTR
JW
- test version of xrootd-ceph plugin with fix to xrdcp -f issue avaialble.

AOB

There are minutes attached to this event. Show them.

- 10:00 → 10:20
  Status 20m
  - Outstanding tickets 10m
    
    Open ATLAS UK GGUS tickets
    
    Outstanding tickets
    
    148342 UKI-SCOTGRID-GLASGOW less urgent assigned 2020-08-24 11:56:00 UKI-SCOTGRID-GLASGOW with transfer efficiency degraded and many failures
    
    One (DPM) disk server with a possible issue, trying to rebuild raidset.
    
    Finish current set of drains before tackling above issue
    
    JW to send current set of transfer failures to SS.
    
    148339 UKI-NORTHGRID-LIV-HEP less urgent in progress 2020-08-24 08:41:00 0 transfer efficiency to UKI-NORTHGRID-LIV-HEP_LOCALGROUPDISK endpoint
    
    No more failures; unclear why endpoint was available given endpoint full.
    
    JW to prioritise UK localgroupdisk cleanup.
    
    147979 UKI-NORTHGRID-MAN-HEP less urgent in progress 2020-08-27 05:55:00 UKI-NORTHGRID-MAN-HEP timeout transfer errros and also deletion errors
    
    JW updated ticket asking for status report.
    
    146771 UKI-SCOTGRID-ECDF less urgent in progress 2020-08-20 14:44:00 UKI-SCOTGRID-ECDF deletion failures with “The requested service is not available at the moment.”
    
    No further changes applied sofar
    
    146651 RAL-LCG2 urgent on hold 2020-08-10 10:59:00 singularity and user NS setup at RAL
    
    on hold
    
    146374 UKI-NORTHGRID-SHEF-HEP urgent in progress 2020-07-22 14:53:00 ATLAS pilot jobs idle on UKI-NORTHGRID-SHEF-HEP CE
    
    no update
    
    144759 UKI-SCOTGRID-GLASGOW less urgent on hold 2020-08-10 09:54:00 High traffic from UKI-SCOTGRID-GLASGOW on RAL CVMFS Stratum1
    
    on hold
    
    142329 UKI-SOUTHGRID-SUSX top priority on hold 2020-06-04 14:05:00 CentOS7 migration UKI-SOUTHGRID-SUSX
    
    on hold
  - CPU 5m
    
    Monit: Site-oriented dashboard
    
    UK Cloud jobs over last week
    
    RAL
    
    Difference in cpu to pledge. updating corepower (set to 10), to 11.7 (approx current value) would give consistent values with internal monitoring.
    
    Northgrid
    
    General comments on the nature and use of current storage model.
    
    Agressive use of storage by the sites; how to move forward, while keeping legacy ‘frameworks’ supported
    
    LANCS downtime from disk related issues.
    
    London
    
    SouthGrid
    
    OX back running after Power failure; not yet at full capacity.
    
    Scotgrid
  - Other new issues 5m
    
    RAL Storage downtime 8th September
    
    ** htpp-tpc and macaroons
    (Email from Alessandra)
    As you might know from the updates we have given at the TCB in July and August with gsiftp and srm going to be done with http instead. At the moment we are enabling only dcache and DPM sites.
    
    dcache works well enough with the supported versions with preference with >5.2.18 and 6.2.x previous releases do have minor bugs that require restarting http.
    DPM 1.14 was released yesterday in EPEL this is a required upgrade to enable HTTP-TPC
    
    Sites will also have to enable macaroons at the moment not all sites have this enabled. Here is an initial list of sites that do not have it, but you can check also Petr's monitoring for the various storage settings. If macaroons is not enabled we will haveto disable TPC for these sites.
    
    Australia-ATLAS
    FMPhi-UNIBA
    GRIF-LAL
    IFIC-LCG2
    IN2P3-LPC
    IN2P3-LPSC
    NCG-INGRID-PT
    RO-07-NIPNE
    TR-10ULAKBIM
    UKI-SCOTGRID-ECDF
    UKI-SOUTHGRID-OX-HEP
    UNIBE
    
    Further updates on this ticket ADCINFR-166
    
    So here are two requirements:
    
    1) DPM sites need to upgrade to 1.14
    2) dcache sites are ok better is with releases >5.2.18
    3) Make sure macaroons are enabled
    
    RAL: Storage downtime; 10hr; batch farm will also be offline.
    
    Alessendra email: TPC, DPM, Macaroons
    
    ECDF, OX aware of news.
- 10:20 → 10:40
  Ongoing issues 20m
  - CentOS7 - Sussex 5m
    
    Centos 7 deployment Twiki
  - Grand Unified queues 5m
    
    ADCDPA-235
    
    Migration plans
- 10:40 → 10:50
  News round-table 10m
  Dan
  
  Migration planned to new lustre; Atlas scratch disk migration can cause problems (highly volatile).
  
  Set up a scratchdisk on another site temporarily ?
  
  Could new disk be installed first, and Old data still readable? Probably not.
  
  JW to create Jira - ticket, for DDM to advise on the migration.
  
  Peter
  
  NTR
  
  Sam
  
  Current storage set at 500TB for CEPH; site to update atlas srr manually to reasonable value.
  
  Gareth
  
  ntr
  
  Tim
  
  NTR
  
  JW
  
  test version of xrootd-ceph plugin with fix to xrdcp -f issue avaialble.
- 10:50 → 11:00
  
  AOB 10m