ATLAS-T0 AFS test followup

Friday 8 May 2015, 11:30 → 12:30 Europe/Zurich

R2 (CERN)

ATLAS T0 job vs AFS 2015-05-8, follow-up from load test

present: Luc, Armin, Jan

ATLAS T0 ops have performed a T0 load "scale" test on Wed May 6th:

Goal eventually is to have 100k jobs/day, i.e. a submission rate of ~60/min. The test job takes ~1200 seconds. At 20/min submission rate the expected equilibrium of ~400 concurrent jobs was reached. At higher rates (even as little as 25/min) the jobs slowed down causing them to pile-up in the batch system. On Wednesday the jobs piled-up at some point to 1800 concurrent jobs, while at the same time execution times rose to on average 12000 seconds.

The AFS volume p.atlas.pro.tzsoft was visibly saturated at 3k accesses/sec. However, no further request pileup occurred (no warning mail sent, not seen when looking into logs), so the test did not even trigger the "usual" overload conditions.

A cross-check with an invalid patch directory (still on AFS, but not existing) was running at expected speed.

Problem might have been compounded via access from Wigner batch nodes (all AFS servers are in Meyrin), might be worth investigating/correlating.

Actions:

• p.atlas.pro.tzsoft will be replicated. Ideally will be split into sub-volumes (e.g. per-release, per-package-and-release?) - see http://information-technology.web.cern.ch/book/afs-administrator-guide
  • Armin (or service account) needs to become an AFS admin for ATLAS, can be done by ATLAS : "afs_admin set-acl atlas ACCOUNT"
  • optional: create sub-volumes: "afs-admin create  ..."
  • then "afs_admin create-replica ...." to make readonly, repeat in order to create more replicas (suggest 3)
  • after changes got written into read-write path (/afs/.cern.ch/project/atlas/...): "afs_adminin vos_release ...."
• Alternatives / further options that can be explored in case the 3 read-only replicas are not sufficient:
  • IT can set higher thread quotas per volume (needs looking at, used to work but unused). Current is 10 threads (out of 255)
  • IT could add (a small number of) more servers (unused disk is wasted = cost) to host more replicas. Will be easier once on "virtual" AFS servers with CEPH backend
    • "reserving" a server for this volume is not useful, won't stay special, will create even more single-point-of-failure
  • ATLAS will see whether
    • the patch area can be removed altogether (only use "real" releases for T0 activity, and/or
    • patch are be moved to CVMFS (dedicated fast-sync service?)
    • LD_LIBRARY_PATH lookup logic can be pushed to job submission engine (lookup once, only add entries that actually contain a patched library into job)
• AFS Console has 20-40min latency (and only 3 data points/hour), frustrating when trying to watch behaviour after changes
  • workaround: directly look at access counters in "vos exa VOLUMENAME", derive rate
• IT will check whether overload mails are working Ok (but these did come as usual on Tue, no recent changes..)