PH-SFT & AFS

Tuesday 5 May 2015, 09:30 → 10:30 Europe/Zurich

R2 (CERN)

PH-SFT and AFS

present: Patricia, Jan

Background: IT is worried about long-term health of OpenAFS project (several indicators); trying to get in touch with major CERN stakeholders, list use cases/workflows and see which options exist for selective replacements of AFS.

General discussion: home directories ~> local disk + CERNBOX lazy sync, dedicated LXPLUS-like VM and authenticated batch access, GIT for code collaborations..

 

AFS use by SFT:  AFS is used to distribute add-on compilers, external software releases and "nightly" builds for continuous integration. Clients are the LHC experiments. AFS also used inside SFT.

• SFT compilations are done into local filesystems, using compilers+cmake from AFS and linking against libraries on AFS. Results are copied back into AFS (and in some cases into CVMFS, if asked for by clients. Example: COOL, CORAL)
• AFS provides  a convenient way to have several compiler versions on the same OS.
  • Discussion: installing several compilers locally via RPM should be possible (at least for gcc, keeps versions in different directories), not sure whether wanted.
• SFT will use CVMFS more widely (aim for full synchronisation incl nightly builds), timescale: next few months
  • might need changes to CVMFS - e.g shorter sync period
• SFT is looking at Docker slices (base, compiler, other dependencies), initially internally but might be interesting for the experiments (at least for nightly/continuous integration builds). Very early stage.
• SFT would also consider building against EOS (once mounted as a filesystem). IT does not see EOS as primary software distribution method, but would probably work

Summary: SFT will start to look at alternatives to AFS, will need more discussion (internal and with IT).

 

AFS "overload" mails: recently introduced by IT. Underlying issues have been there before, only the notification is new.. SFT receives them regularly.

• IT willing to investigate (can only trace ongoing activity; not enough logging to go back in time)
  • AFS console might help to identify overload periods (and correlate with build activity), "unknown" access are worth investigating (might come from experiments)
• using many small volumes (one per package+version) spreads load and helps to identify affected package
  • Suggestion for IT: keep mapping from AFS volume -> mountpoint (hook into afs_admin command; even if this would be unreliable due to unprivileged mount changes later)
• suggest to aggressively mark volumes as "readonly" and create several replicas (spreads load, easier caching)
  • "readonly" volumes can still be changed later, just need "vos release"
• review LD_LIBRARY_PATH, PATH - keep as short as possible (SLC6:multiplication factor 8 on LD_LIBRARY_PATH; uncached negative stat()s)