ATLAS UK Cloud Support

Thursday 19 Nov 2020, 10:00 → 11:00 Europe/London

Zoom

Tim Adye (Science and Technology Facilities Council STFC (GB)), James William Walder (Science and Technology Facilities Council STFC (GB))

Meeting to be held via Zoom (https://ukri.zoom.us/j/97404730356)
Password protected (same as OPs Mtg)

Outstanding tickets

• 149362 UKI-SOUTHGRID-RALPP urgent in progress 2020-11-13 08:45:00 ATLAS CE failures on UKI-SOUTHGRID-RALPP-heplnx207
  • Issue in adding new CE with AGIS/CRIC (see below)
  • Site to take ce into downtime on Monday for general cleanup
• 148968 UKI-NORTHGRID-LANCS-HEP less urgent in progress 2020-11-19 06:53:00 UKI-NORTHGRID-LANCS-HEP: deletion and transfer failures
  • gridFTP restarted; looking better, but will keep an eye
  • Other non-Lancs issues with Italy sites, adds a bit of confusion
    • Napoli issue with https available only on LHCONE (via certain IPvX?) whereas,
    • gridFTP available on non LHCONE
• 148342 UKI-SCOTGRID-GLASGOW less urgent in progress 2020-11-12 17:24:00 UKI-SCOTGRID-GLASGOW with transfer efficiency degraded and many failures
  • Sam to take a look at problem files
• 146651 RAL-LCG2 urgent on hold 2020-10-16 11:56:00 singularity and user NS setup at RAL
  • no update
• 142329 UKI-SOUTHGRID-SUSX top priority on hold 2020-11-05 10:52:00 CentOS7 migration UKI-SOUTHGRID-SUSX
  • no update

CPU

• RAL

• Northgrid

• London

• Server with occasional bad memory issue

  • To discuss with manufactuer to attempt a proper fix (no just bios update)

• SouthGrid

• Scotgrid

  • Durham Priorty user that takes the all the priorty, causing loss of ATLAS jobs.
    • Some job loss also from HC test failures from missing files
  • GLA: CMVFS update for CMS, some unindented consequences caused problems
  • GLA: Bringing online additional capacity slowly; aim before Christmas full capactiy.
    • Probable identification of high iops in ceph cluster from offsite xroot direct access reads
      • user from cern, accessing scratchdisk
      • Swtich off to see if this solves the issue.

Other new issues

• Recent Switcher problem with AGAS/CRIC sending many emails

  • Concerns raised on approprate use of mailing lists.
  • ATLAS uk has cloud-support, uk comp operations, and uk comp users.
    • The comp users list has been unused for 5 years, and it was agreed to be removed
    • For cloud support, this remains the most active discussion list, and will be unchanged.
    • The comp operations contains the daily summary and Switcher notifications. Non automated traffic is on the order of 1 email per year; which may have been unintentionally intended for cloud support.
      • It was decided to keep the Swticher and Daily summary in this list. A simple filter can remove any unwanted emails.

• Queues:

  • Long-term queues that are not disabled, but not running production:
    • UK ANALY_MANC_TEST_SL7: Still needed
    • UK ANALY_QMUL_GPU_TEST: -> could be renamed to non test
    • RAL-LCG2_TEST: -> not actively used (see comment from Peter)
    • RAL-LCG2_UCORE: Can be disabled
    • UKI-NORTHGRID-LANCS-HEP_TEST (see comment from Peter)
    • UKI-NORTHGRID-MAN-HEP_TEST; testbed -> keep
    • UKI-SCOTGRID-GLASGOW_CEPH_TEST: keep
    • UKI-SOUTHGRID-OX-HEP_TEST: (see comment from Peter)
    • UKI-SOUTHGRID-SUSX_UCORE: not test, should become production, might want remaining
    •  
    • Peter uses TEST queuse for dev test work monitoring
    • QM test queue might be useful

Ongoing issues

• CentOS7 - Sussex

  • no update

• Datadisk; watermark reduced.

• LOCALGROUP disk

  • New pool to be created shortly

• TPC:

  • Naples -> moved to DPM 1.14.2, networking blocked 443 ipv6, ipv4 open on general network

  • Affects whole of UK (e.g. Lancs, MAN)

  • Retry transfer failures

  • Vunerability from DPM, and dCache

    • Beleive all UK DPM sites up to date (or not affected)
    • dCache issue announced in appropriate channels

News round-table

• Vip
  • Had to leave before end; NTR
• Dan
  • NTR
• Matt
  • NTR; away for next week’s meeting
• Peter
  • NTR
  • http://atlas-cric.cern.ch/api/atlas/pandaqueue/query/?json for Cric status of queues
• Alessandra
  • NTR
• Gareth
  • The two CE’s recently added to Glasgow will stay in downtime for time being.
  • JW to check they are included correctly in CRIC / AGIS.
• JW
  • NTR
• Sam;
  • Final talk available for Workshop.
  • Positive comments on updates to talk draft
    • Tables are now much better

AOB

NTR

10:00 → 10:20 Status

10:20 → 10:40 Ongoing Items

• CentOS7 - Sussex

  • no update

• Datadisk; watermark reduced.

• LOCALGROUP disk

  • New pool to be created shortly

• TPC:

  • Naples -> moved to DPM 1.14.2, networking blocked 443 ipv6, ipv4 open on general network

  • Affects whole of UK (e.g. Lancs, MAN)

  • Retry transfer failures

  • Vunerability from DPM, and dCache

    • Beleive all UK DPM sites up to date (or not affected)
    • dCache issue announced in appropriate channels

10:40 → 10:50 News round-table

News round-table

• Vip
  • Had to leave before end; NTR
• Dan
  • NTR
• Matt
  • NTR; away for next week’s meeting
• Peter
  • NTR
  • http://atlas-cric.cern.ch/api/atlas/pandaqueue/query/?json for Cric status of queues
• Alessandra
  • NTR
• Gareth
  • The two CE’s recently added to Glasgow will stay in downtime for time being.
  • JW to check they are included correctly in CRIC / AGIS.
• JW
  • NTR
• Sam;
  • Final talk available for Workshop.
  • Positive comments on updates to talk draft
    • Tables are now much better

10:50 → 11:00 AOB

NTR