CTA deployment meeting

Europe/Zurich
31/S-023 (CERN)

31/S-023

CERN

22
Show room on map

FTS:

  • Eddy representing FTS today. Eddy to be added to castor-tape-dev.
  • Explicit stager_rm from FTS: Full description in ticketJulien will follow up with Andrea.
  • Storage collocation hints:
    • ATLAS proposes a pair of collocation hints to be transferred from Rucio to FTS.  (Example: ['data15_13TeV.00284473.physics_Main.merge.AOD.f643_m1518','Main_AOD'], [<md5sum1>,'Main_AOD']).
      • It is left up to each storage system to decide if and how to use them for data placement in order to optimise future reading. Initially, CTA will just receive the hints and log them.
      • FTS does the translation into each back-end storage system (CTA, dCache, etc), similarly to what has been sketched for storage classes and hints.
      • We propose a maximum string size of 64 chars per hint (so 128 chars in total)
    • Andrea is in contact with ATLAS for agreeing on the metadata that Rucio will use for this purpose.
    • Eric will propose and discuss with FTS team the format how to receive the metadata from FTS.

ATLAS Data Carousel recall exercise:

  • Julien will present to ATLAS the outcome of the recall exercise (done after meeting, see link)
  • GC problem (wrong space estimation causing too high cleanup) analysed and understood, fix being worked on
  • Performance degraded also as a consequence of RAO only being executed on the first retrieve batch within a mount, which brings down retrieve performance during the rest of the mount (cf gitlab)
  • When retrying failed file retrievals, these will be issuing a sparse tape access (few files remaining to be retrieved on each tape) which will result in lower performance.

EOS branching, tagging and merging

  • Steve's proposal can be found attached to the agenda.
  • EOS commits related to CTA go into a dedicated CTA branch based on a specific EOS (stable) tag. Untagged EOS commits are therefore left out. This requires EOS tags to be sufficiently stable. 

Status of migration tools

  • Tool for injecting fileID's to populate EOS has been developed by Andreas.
  • File ID's will be the CASTOR nsfileids in both EOS and CTA catalogues.
  • Michael's injection tools are still to be modified.
  • What is the right hardware to run migration? Julien to check with Michael/Giuseppe.

Actions follow-up:

  • Storage classes defined by ATLAS: atlas_raw, atlas_derived, atlas_default
  • FIFO queueing is default
  • Activities??
Actions
who what by when
Eric Agree with ATLAS on list of "activities" and configure via cta-admin. Deploy "activities" on ATLAS 27/5
Michael Complete (with Cédric S.) namespace split-up 30/5
Cédric Implement repacking disabled tapes once disabling is honored by CTA 30/5
Julien Ensure CTA team is copied in exchanges with ATLAS and other experiments. 24/5
Julien talk to procurement and network people (to ensure all network infrastructure is in place when nodes arrive) 30/5
Michael Ensure that Georgios gets in touch with Luc to advance discussions on modelling collocation hints and assessing their usefulness. 30/5
Julien/Andrea Explict stager_rm follow-up 13/6
Andrea Agree Rucio->FTS metadata format for collocation hints and storage classes  13/6
Eric propose and discuss with FTS team format how to receive collocation hints (in addition to storage classes and activities) from FTS. 13/6
Julien Identify what is the right hardware to run migration 13/6

 

 

There are minutes attached to this event. Show them.
    • 10:35 10:55
      FTS status and timelines 20m
      • storage classes and writing/grouping hints (size limitations?)
      • explicit stager_rm from FTS
    • 11:15 11:35
      EOS branching, tagging and merging for CTA 20m
    • 11:35 11:55
      Status of migration tools (Michael/Giuseppe) 20m
    • 11:55 12:15
      Review of actions, AOB, items for next meeting 20m