CTA deployment meeting, 3/5/2019

• Feedback from Rucio meeting (Julien)
  • Discussed T1 export with both ATLAS and CMS. 
    • Convinced them that it is more convenient to do export via “big” EOS instance as it provides more space and multi-protocol support (eg GridFTP).
    • This however requires experiments to manage file caching. A solution is to implement FTS job dependencies for the EOSCTA->EOS->T1 transfer chaining: Two FTS prepare jobs are launched (by Rucio). The first one stages out from EOSCTA and transfers to EOS; the second job waits for the first prepare to complete (and therefore for the file(s) of the first transfer to appear on EOS) and then transfers them to T1 sites. Rucio does the cleanup of the temporary EOS files.
    • A few clarifications are needed (Andrea):
      • This requires changes over the classic BringOnline as the 2nd job will issue a prepare request for a file that (potentially) doesn’t exist in the namespace of the SE.
      • How will this work for multi-file jobs?
      • How will this work for T1 file imports (that are less frequent but do happen)?
• CTA production setup and web pages (Julien)
  • Agreed with Vlado the transfer of tape drives to CTA (2 x 3 enterprise drives now, 3 LTO drives later). Julien to go ahead and reinstall them for CTA ASAP
  • Web pages: site created (http://cern.ch/eoscta), basic structure (cf. eg. cern.ch/eos) to be populated:
    • Introduction (main page)
    • Running CERN instances (description and monitoring links)
    • Resources (documentation, links to Git code) 
    • Contact (GitLab tracker, support flow, email)
• Tape pool and storage classes setups for ATLAS (via Giuseppe)
  • ATLAS (Alessandro) agrees to the principle to start with O(10-20) storage classes. We can map these into tape pools as required, can initially start with a reduced number (5-10). Exact storage class names to be agreed with ATLAS.
• Review of action points:
  • Status of migration (Michael)
    • see input below
    • Michael(+Giuseppe) to send experiment/user namespace split-up proposal to ATLAS
    • Q: When will we be able to perform a test import? A: requires some DB changes to be completed and deployed first
  • FTS status update (Andrea)
    • Propagate “activity” to EOS/CTA: Completed
  • Repack status (Eric, Cédric):
    • Repack can be used in a crude way in production (cleanup necessary, including of buffered files).
    • Auto-deletion of  buffered files and other cleanup operations being developed.
    • -> Cédric to discuss with Vlado for setting up and executing field testing
  • Support for fair sharing via activities (Eric):
    • Activities is underway.
• Others:
  • Julien to ask CMS to delete/re-create pre-offset files (in order to avoid nsfileid clashes during import)

• Videoconf with dCache

• • CERN position: 
    • CTA currently focusing on T0 deployment
    • T1 support will follow (~mid next year)
    • CTA code will be released as open source. 
    • Support will be available in priority for WLCG related sites (T1’s in particular), best-effort for other sites.

• • dCache team: Interested in CTA as back-end to dCache as alternative to commercial (TSM, HPSS, ..) and open-source (ENSTORE) solutions. In particular, looking at XFEL sites that want to deploy tape.
  • Currently, two integration options (see link):
    • “script”: simple get/put/list interface using scripts. Neither complex nor efficient by itself (eg. doesn’t know about tape<->file mapping but this could be handled by EOSCTA)
    • “driver”: newer, API-based interfacing with more in-depth interfacing required. Currently, only NDGF providing an implementation for TSM.
  • “driver” integration would likely require stable interfaces directly with CTA back-end. From our perspective, integration with dCache would be in principle at the level of EOSCTA rather than CTA as this is where we feature (mostly) stable and public interfaces
  • We can let the dCache team experiment with EOSCTA. Can we provide Paul Millar with a shrink-wrapped EOSCTA container set so that they can have a look? (Julien)