DOMA LHCC Review Prep - Storage

Tuesday 8 Jun 2021, 16:00 → 18:00 Europe/Zurich

Second of two sessions inviting feedback on the DOMA LHCC review doc.

This session focuses on the storage section, including CTA, dCache, Echo, EOS, StoRM and XRootD.

Please use the Indico integrated Zoom (not the one distributed previously by email).

Further information - https://twiki.cern.ch/twiki/bin/view/LCG/LHCC2021

1 Presentation of review doc

Speakers: Mario Lassnig (CERN), Oliver Keeble (CERN)

Session2.pdf

2 Atlas requirements and priorities

Speaker: David Cameron (University of Oslo (NO))

ATLAS input to storage.pdf

3 CMS requirements and priorities

Speakers: Danilo Piparo (CERN), James Robert Letts (Univ. of California San Diego (US))

210608_DOMA_MiniWorkShop_2.pdf

4 Other input - Alice, LHCb, facilities - tbc

Speaker: Reda Tafirout (TRIUMF (CA))

DOMA-part2-TRIUMF.pdf

NDGF-T1

From INFN-T1:

Minor comment: in XrootD description (4.2.1) it worth to mention also GPFS as a "kind of storage" and GEMSS as "tape access": Numerous production quality plug-ins have been developed over the years to address community needs such as xroot[s] and http[s] protocols, popular authentication and authorisation mechanisms (Macaroons, SciTokens, x509), interfaces to various kinds of storage (Ceph, DPM, EOS, GPFS, HDFS, Lustre, Unix), commonly used check summing, tape access (CTA, GEMSS and HPSS), Cheers, Vladimir

 

 

From Frank:

Both UCSD and Caltech are in the middle of their respective switch from HDFS to Ceph. Caltech is probably more advanced than UCSD.
But both will be Ceph sites by the end of the summer.

At UCSD, we are also changing the overall structure from disks in worker nodes to large fileserver with 102 HDDs per server.
So the fundamentals change in many ways.

The change is to benefit from erasure encoding, and because I’m sick and tired of corruption due to the large entropy in our HDFS setup.
A single 2GB file is spread across too many physical devices, and as a result, 2 systems going down results in thousands of corrupted files.

We never had the to fix this architecture in HDFS. So we are changing it now, as we move to Ceph.
 

5 Discussion