ATLAST0 - AFS meet

Thursday 30 Apr 2015, 11:00 → 12:00 Europe/Zurich

R2 (CERN)

ATLAS T0 job vs AFS, 2015-04-30

present: Luc, Jan

Meeting got triggered by the persistent overloads of the AFS volume "p.atlas.pro.tzsoft", plus the fact that ATLAST0 plans to increase the number of cores (and hence jobs accessing this volume) by factor of 5 over the next few weeks..

Discussion

• discussion on the monitoring (AFS console, LSF monitoring, ad-hoc plot of number of "rescheduling" messages)
  • AFS should get hit mostly at job start
  • observe "echo" pattern (concurrently started jobs exit together, next round of jobs starts together - smears out over time)
  • job submission rate is adapted to not have PEND jobs, cannot look at PEND decrease rate
• structure:
  • full release is on CVMFS, ~10GB
  • AFS has same per-project directory structure, and is preferred via LD_LIBRARY_PATH ("many" entries - but unclear whether one/project?)
    • reducing the number of entries will make a significant difference on any filesystem, glibc actually looks into several subdirectories for each entry..
  • intention is to quickly deploy new versions of shared libraries, without doing a full release. Typically have ~> 10 patched libraries at any given time
• AFS cache might no be working as expected, most likely too small to hold working set
  • [after meeting: example machine had 512MB memory cache - which is the default for VMs. But ATLAST0 are "fat" exclusive VMs and would not suffer from disk I/O contention on the hypervisors "real" disk in the case of a disk-based AFS cache]
  • AFS cache monitoring - see Jarka's dashboard at https://meter.cern.ch/public/_plugin/kibana/#/dashboard/temp/ATLAS-dev::HostMetricsWithAFS
  • note: 99% cache hit rate would not disprove this - AFS is block-based and therefore does prefetching, the prefetch might cover repeated access to the same block (good!) and nevertheles get evicted before the next job runs (and acceses the data again). Would need a dedicated test:
    • drop AFS cache ("fs flushall", or disruptive "service afs stop && rm -rf /usr/vice/cache/*; service afs start")
    • run job interactively
    • run job interactively
    • check whether any AFS server access during second run
• Why was this not a issue before?
  • "overload" mails are recent - issue most likely present before
  • ATLAS might not have noticed a bad efficiency at job start (i.e. several minutes to start for a multi-hour job would still be only a few percent loss)
• How to check whether 5x increase in T0 jobs would work?
  • scale number of jobs up by making each job do less (reduce events/job). AFS library access is at start of job.
  • should watch for side effects on other ATLAS AFS activity (in particular nightly builds)

ToDo:

• arrange scale test (many shorter jobs)
• compare AFS cache to working set/patch area
  • done: 512MB cache size vs 92..170MB ("du -s-h -L", depends on release)
• investigate behaviour on "real" jobs

  • can use  "strace -T -e trace=file -o sometempfile -f /the/path/to/the/job arguments_for_job" and then grep '/afs' sometempfile