ESCAPE Task 2.1 DIOS (Datalake Infrastructure)

Wednesday 11 Dec 2019, 10:00 → 11:35 Europe/Rome

Presents: Aleem, Andrew Pickford, Bastien Gounon, Bernardino, Frederic Gillardo, Frederique Chollet, Gonzalo Merino, Guido Aben, James Collinson, Kilian, Mario Lassnig, Martin Barisits, Nadine Neyroud, Patrick Fuhrmann, Paul Millar, Paul Musset, Riccardo Di Maria, Stephane Jezequel, Xavier Espinal, Ron Trompert, Rizart Dona, Ghita, Tommaso Boccali

Minutes were taken by Riccardo Di Maria

Partners round table

• LAPP:
  • dCache installed with WebDav
  • to do: activate open id token auth
  • CTA use case in coordination with other people within the CTA experiment - define interface between DIRAC and data lake  - possible usage of DESY endpoint
  • RUCIO client installed
  • meeting planned on CTA archive implementation
  • federated DPM storage for ESCAPE in south France
  • 100 TB shared among 4 sites
  • to do: integration of ESCAPE VO

• PIC/IFAE:
  • dCache endpoint up and running
  • to do: look at open id token auth
  • one node of distributed CERN EOS @PIC
  • to do: deploy XCache for VIRGO and CMS - overlapping with ESCAPE plans
  • Task2.3 to do: deploy and test data transfer use case for gamma rays with RUCIO - DESY storage dCache instance available (dCache-demo)
  • hiring is in  Task2.3 only
  • to do: follow-up with Frederic Gillardo

• IN2P3:
  • testbed setup
  • server less, FAAS - PoC successful - will be presented to wp2
  • RUCIO instance for Nessie (data lake) testbedLSST workflows
  • to do: dCache access tests
  • to do: install XCache - tests with remote sites (Asia, Japan)
  • to do: auth, bearer tokens

• GSI:
  • hiring almost completed
  • machines up and running
  • xrootd endpoint ready
  • QoS: posix, disk, no replicas
  • certificate access not working
  • to do: xrootd resources in data lake
  • FAIR: first prototype of data lake  up and running - can be potentially connected to ESCAPE data lake
    • to do: add more sites (dCache @DESY)
  • to do: improve nginx configuration

• DESY:
  • dCache-support model: running exemplary instance at DESY and disseminate knowledge to other dCache sites
  • dCache-support chat group
  • ESCAPE wiki page
  • Prometheus (unstable - will stay up) and dCache-demo (stable and reliable - suitable for ESCAPE), both supports tokens
  • perfSONAR up and running
  • alternative caching model to XCache using dCache (in production - AGLT2)
  • on-going:  further optimisation of such deployments

• SURF/SARA:
  • dCache - macaroons and scitokens
  • datatransfer tests (AARnet, NDGF)
  • to do: test against other dCache (DESY)
  • to do: transfer tests with South Africa
  • integration CS3MESH4EOSC and ESCAPE - data management system integrate with an analysis platform for astro community

• INFN:
  • primary testbed site: CNAF (with Storm) in production
  • XrootD + WebDav working
  • ESCAPE VO enabled
  • integrated with Indigo IAM
  • to do: integrate with tape (via srm)
  • CEPH under test @CNAF (to be discussed a possible integration for ESCAPE)
  • Interest from 2 DPM Italian sites (both ATLAS :”driven”)
  • to do: multiple tests possible - 2 separated sites (preferred at least initially) and 2 sites beyond the same endpoint

• RUG:
  • handing over the project to a new person
  • hiring soon
  • more news next year

• AARnet:
  • Need help to bring TCP for data transfers
  • On board for LOFAR's, SA and AUS data transfers

• CERN:
  • RUCIO instance is ready (puppet, k8s)
  • RSE per storage endpoint
  • integration with FTS
  • monitoring in place for RUCIO and FTS
  • CRIC being deployed
  • minimal continuous data lake testing
  • ESCAPE wiki contains a table summarising status
  • to do: consolidate monitoring
  • to do: develop machinery for full mesh data transfer tests
  • to do: TCP/HTTP and token based auth on EOS
  • caching layer ready and deployed
  • HammerCloud instance ready
  • to do: update and clarify wiki

• Strategy and plans for 2020
  •  Dec 2020: Functional Data Transfer Tests Machinery in place
    • To demonstrate stable and sizeable data movement across sites in the datalake
    • Performance monitoring in place (e.g. transfer matrix)
    •  Targeting Feb-2021 Workshop (M2.3) and pilot phase testing completion (report needed)
  • Preparations for WP2-M2.4 (Mar 2021)
    •  Targeting Apr-2021 WP2 D2.2 assessment and analysis of the performance of the pilot data lake
    •  Targeting M2.4: Expanded prototype to 3rd party, RI workloads accessing data from compute resources including commercial clouds

• AOB: support for data lake through RocketChat and mail - https://chat.escape2020.de/channel/datalake_status_and_support