CTA deployment meeting

→ Europe/Zurich
31/1-012 (CERN)

31/1-012

CERN

6
Show room on map
Hide

FTS:

  • polling of GC'd files: needs to be validated with modified GC (Steve).
  • checking for tape residency ("m" bit): This functionality was requested by CMS and will increase reliability on single-copy SSD buffers. Before implementing this check, the newly discovered DB performance issues in FTS need to be solved. See also the following presentation by Andrea: link to WLCG ops. Implementing tape residency check implies a new FTS state and generates additional load by having to poll EOS over longer periods (as the lifecycle now would include the time to migrate to tape).

CTA:

  • backpressure locally tested by Eric, waiting for Julien to come back from holidays for extended testing on ctapps/ctaatlas. The code is still on a separate branch which will be merged by Eric/Julien.
  • cancel: Coding and unit testing completed. Eric is adding python code to the system tests to ease its testing with evict (cf last meeting). WIP but should be completed in one week's time.
  • Repack:
    • Repacking of missing copies implemented. Repacking a broken tape with multi-copy files will reinstate the correct number of copies by reading problematic files from other tapes containing another copy of the file. 
    • In the case of changes to the number of copies in a storage class, repack will create additional copies. Two flags will be available to repack: -m (move tape contents) and -a (add copy); by default both flags are active.
    • Waiting for Julien to run a large test with multiple drives.

Migration tools:

  • Adding migration tools to CI, few bugs/fixed identified such as an EOS mishandling of flags/mode bits.
  • Better reporting of gRPC errors now available in EOS (4.5.6)
  • Idempotence: Import scripts won't stop on errors but will store them on a special table that will then be processed on a second pass (similarly to what the CASTOR->CTA DB import scripts do).
  • Deltas (directories/files created after DB snapshotting) can be identified by CTA DB import script and used for the EOS import process. No support for directory renames but these cases will be detected and can be addressed otherwise.
  • Next tests involve a complete re-import of ATLAS into CTAPPS, followed by a full import of CASTOR afterwards.

Meeting with CMS - presentations to be prepared:

  • Julien: Proposed EOS/CTA production setup: Hardware, capacity, configuration, ATLAS experiences
  • Michael: Current CASTOR use cases and their translation into CTA. Which ones will be taken over by Rucio and which ones will require direct EOS/CTA access?
    • pit->T0 transfers
    • QA checks
    • local users with CASTOR home directories and EOS->CASTOR archiving
  • Eric / Steve: CASTOR vs EOS/CTA configuration
    • CMS service classes, file classes, tape pools in CASTOR; their equivalents in CTA and how to select/set them
    • CMS CASTOR migration and recall policies and their equivalents in CTA
    • GC / DB
  • Eddy (with Andrea): FTS specifics for CTA (disk buffer eviction, request canceling, handling GC, checking for tape residency)
  • Michael: Migration of CASTOR metadata
    • Analysis of CASTOR/CMS namespace (directories, files, users, groups, etc)
    • handling special cases (symlinks, POSIX permissions/ACL's, zero-length files, empty directories)
    • Read-only pre-import onto EOSCTA for validation
    • Migration milestones
There are minutes attached to this event. Show them.