ATLAS-T0 AFS meeting

Thursday 19 Mar 2015, 09:30 → 11:00 Europe/Zurich

Rest1 (CERN)

AFS vs ATLAS-T0 meeting 2015-03-19

present: Luc, Armin, Jan

• RQF0437013 : ATLAS T0 AFS setup for run-2? Current situation
• "prompt reprocessing":
  • runs quickly (<2 days) after data is taken
  • uses AFS directly, including non-"released" (user) software on non-replicated AFS volumes (and direct ORACLE connections)
• "bulk processing":
  • runs later
  • uses software on CVMFS
  • uses cached DB data via Frontier
• T0 jobs will
  • run from a job definition script (on AFS) that gets created by the T0 framework
    • expect several layers of AFS dependencies inside
  • write actual job logs into EOSATLAS (copy via xrdcp at end of job)
  • try to write the output from job wrapper scripts into one of 2 AFS directories (load-balanced, with failover to the other dir on errors)
    • AFS failures tend to get the client stuck (in kernel space) until the target server is back, so failover needs to be implemented on top of that - might not actually work?
    • size: o(100k) directories, each with o(10) small text files, auto-cleanup. Access is one immediate read, then occasional access via HTTP (failed job debugging)
  • original purpose of the meeting was to make sure that these directories are hosted on different machines, and not co-hosted with a "heavy" user
    • Series of bottlenecks:
      • each volume can use 20 threads on the fileserver, then gets throttled
        • one observed overload scenario is running out of fileserver threads under massively parallel accesses
      • each volume is on RAID-1 partition (I/O ops limit,  disk bandwidth limit) - this is shared with other volumes on the same partition
      • all volumes on the same machine share the same 256 thread pool
      • all volumes on the same machine share bus and network bandwidth (usually not an issue)
    • CERN AFS project space is split by experiment. ATLAS project volumes are only on 3 machines
      • ATLAS-induced overload will hurt ATLAS, not other experiments
      • we regularly see individual overloaded volumes also for ATLAS. Will eventually implement a notification mechanism
      • heaviest user is probably the ATLAS software release process (compiles on AFS?), as well as concurrent access to libraries on AFS from batch jobs
    • CERN AFS volume placement is dynamic (scripts move these around, no easy way to "avoid" colocation), so cannot durably separate
    • AFS support will help in case of observed issues (identify causes of overload, move volumes etc). No formal AFS piquet exists, but IT CC operator can call out to AFS support out of working hours for catastrophic failures.

Discussion/"sermon":

FUD: Expect OpenAFS to gently decay as companies/institutes pull out, support money dries up. Might be accelerated by the US gov push for IPv6 (specified in government contracts), as well as the lingering DES. CERN has no single drop-in replacement that covers all use cases. Experiments should review AFS use and try to minimize it, some "easy" cases will already be better suited on other systems.

• ATLAS T0 will see whether the job wrapper logs can also go into EOSATLAS.
• Batch job input: LSF can copy scripts and input files from the submission machine (i.e don't need to use AFS for these), but deeper dependencies into AFS (libraries etc) will not be copied. As far as possible these should come from CVMFS. Will need to see for highly dynamic code (that is not going the full release cycle).