CCC & AFS

Wednesday 1 Apr 2015, 10:00 → 13:00 Europe/Zurich

CERN

AFS use in the CCC, 2015-04-01 (no joke)

present: Alastair, Stephen, Massimo, Jan

Current state (provided by Alastair before the meeting):

• AFS switched off on most TN servers, on for GPN servers, off for RHEL5 servers, off for Redhat MRG systems
• AFS on on consoles
• AFS tolerated for CTF3 Operation (typically for Matlab)
• AFS frowned on in CCC, sometimes used for Machine Development storage or more recently Matlab execution
• All the attempts to transfer our long term data storage currently on CS-CCR-NFS99 to AFS on hold due to the limits of number of files per directory needing to be at least 100,000
• The last TN Disco test of 8/9/2014 had very serious blockages on the consoles due to AFS and/or Kerberos and/or Active Directory. This included not being able to login as root. There is hope to debug this using the “micro TN disco” method Luigi Gallerani developed with David Rueda for understanding the licensing problems for Windows Terminal servers on the TN. You are welcome to volunteer!

AFS update: IT worried about long-term health of the project, going round to discover major AFS use cases, in particular those that would be painful (or impossible) to migrate. Also start to discourage major AFS uptake for new projects. Timeframe is month..years. No single drop-in replacement exists, but expect many use cases to be ok with EOS (directly or via FUSE mount).

Discussion:

• Most of CCC (actually BE+EN+TE) infrastructure is on NFS (own servers, ~10).
  • home directories (~1200 users, incl ~100 operators; ~400 users have 'root')
  • non-ORACLE data (multidimensional/"blob")
    • incl archival storage
  • boot / root images for "diskless" machines (~1000)
• CCC consoles have local directory for login (with crucial files overwritten regularly), but have AFS mounted (users can "kinit" and copy things to/from AFS manually)
• CCC would like to have alternatives for (some) of the current NFS use cases (including the 20-year "radioactive materials" logdata archive, 35TB, 130k files/dir); system is complex and has too many root users on untrusted machines (ongoing refactoring). Would look for
  • "home directory"
  • "secrets storage" (DB passwords etc)
  • massive/fast data writes from Java programs (direct Java Stream API would be a plus for any future storage system)
• a "lazy sync" approach for user directories might work, as long as local cache files (java webstart, java cache, browser cache, ..) are not synced. Bad experience with the initial Windows "briefcase" and "roaming profiles" lazy sync.
  • "rsync" server on EOS? won't work, EOS has block-level checksums but incompatible algorithm and not exposed. IT prefers full-file sync via CERNBOX 

Other users/use cases:

• BE-ABP (e.g. H.Burkhardt) runs simulations, uses BATCH - will have different access patterns, might use AFS