DOMA / TPC Meeting

Wednesday 27 Jan 2021, 17:30 → 18:35 Europe/Zurich

Topic: WLCG DOMA TPC Meeting

Join Zoom Meeting
https://cern.zoom.us/j/99836057922?pwd=ZFhWN3NpYi9oZmwvM3pIRE9zdzFnZz09

Meeting ID: 998 3605 7922
Passcode: 733660
One tap mobile
+41315280988,,99836057922# Switzerland
+41432107042,,99836057922# Switzerland

Dial by your location
        +41 31 528 09 88 Switzerland
        +41 43 210 70 42 Switzerland
        +41 43 210 71 08 Switzerland
        +33 1 7037 2246 France
        +33 1 7037 9729 France
        +33 1 8699 5831 France
Meeting ID: 998 3605 7922
Find your local number: https://cern.zoom.us/u/aeB4ArMgmT

News:

* Reorganisation of the agenda to add TAPE REST API slot

  * Related to this the next GDB is on TAPE evolution https://indico.cern.ch/event/876801/ sites with tape systems are encouraged to attend

* Agreed a new slot, every other week we start at 4 pm

SRM+HTTP

* Code is installed on FTS-devel-next it needs to be installed on FTS-devel before we can setup a testbed. Mihai saidhe cando it in a couple of weeks. The next step is to patch the DOMA rucio instance which is a bit old (Martinor Thomas will do  that) and add 3 T1 one per storage implementation (Petr can do that). We are still using dteam so we will ask the sites to enable it on storage end points.

There was a question about using smoke tests for this but we agreed they are not the right level of testing because they don't use neither FTS nor GFAL.

Experiments 

ATLAS

* CRLs are not respected by HTTP-TPC, some discussion on whether we should implement them at all. We agree that ATM CRLs are part of our infrastructure and we should implement them where missing.

* Discussion on the UNIBE case with the swiss CA updated the intermediate CA and there is a problem with the validity of the site certificate. UNIBE transfers fail mostly from large dcache sites. Paul will look at the DESY failures, but the general suggestion is that the site might want to ask for new certificates.

* Disccussion about INFN-T1 transfer rates

CMS

has enabled all T1 bar RAL they will open a ticket for them. they also are alligning their timeline with ATLAS in order to move the disks endpoints to use HTTP. Alessandra will send the list of minimum versions of storage ATLAS requires.

LHCb

LHCb has also stated to open tickets for sites to enable HTTP-TPC they are also asking for xrootd as a back up but is less important.

Protocols news

Wei reports there were two bugs in xrootd http and they were fixed.

 

1 SRM+HTTP tape access

Speaker: Mihai Patrascoiu (CERN)

2 Future uniform tape access

Speakers: Oliver Keeble (CERN), Paul Millar

3 Experiments production

Speakers: Alessandra Forti (University of Manchester (GB)), Diego Davila Foyo (Univ. of California San Diego (US)), Petr Vokac (Czech Technical University (CZ))

Missing CRL verification while establishing HTTP connections

• CRL validation is mandatory in the EGI/WLCG infrastructure
• Only dCache and StoRM correctly deals with CRLs for HTTP-TPC
• Davix / gfal2 no CRL support for HTTP DMC-1235
• DPM HTTP-TPC LCGDM-2959
• Dynafed rely on gfal2 with no CRL support
• XRootD HTTP-TPC xroot#1383 (also used by EOS)
• FTS no CRL support(?)
  • started with XrdSecGSICRLCHECK=0 environment
  • rely on gfal2 with no CRL support for HTTP
• To be checked - Echo
• Brian mentioned CRLs are obsolete and only grid still rely on this revocation mechanism
  • OCSP Stapling - necessary to modify clients
  • Create additional tickets and require OCSP Stapling support?
    • No, GDB is right forum to discuss CRL -> OCSP Stapling

New issues

• XRootD & EOS on CentOS7 don't terminate stuck HTTP-TPC transfers root#1303
  • CentOS7 provide too old libcurl 7.29 (speed limit option)
  • Terminated only by FTS after reaching total transfer timeout (can be hours)
  • FTS has limit for active transfers (per site and per link)
  • GridFTP in gfal2/FTS can be configured with GRIDFTP PLUGIN:PERF_MARKER_TIMEOUT
  • We need something similar for in gfal for HTTP-TPC DMC-1236
• BNL dCache HTTP-TPC push sometimes fails with
  • Failed to select pool: java.lang.IllegalStateException: Replica exists with state: REMOVED
  • Failed to select pool: java.lang.IllegalStateException: Replica exists with state: CACHED
• DPM use client IP in security token while redirecting from headnode to disknode LCGDM-2961
  • Transfers fails if connection to headnode is IPv6 (IPv4) and IPv4 (IPv6) after redirection
  • Client usually use same IP version, but curl supports happy-eyeball since 7.59.0 available in CentOS8
• HTTP-TPC transfers from UNIBE-LHEP to some dCache sites fails GGUS:150314
  • dCache 5.x - failure: java.lang.NullPointerException
  • dCache 6.x - failure: javax.net.ssl.SSLException
  • It is difficult to understand what is going on just from exception class name
  • Occurs probably since Swiss intermediate CA update(?)
    • Is server certificate considered OK if it is valid before (intermediate) CA validity, e.g.
      • QuoVadis Grid ICA G2 - valid Sep 22 2020 ... May 23 2026
      • dpmdisk01.cscs.ch - valid Aug 27 2020 ... Aug 27 2021
• HTTP-TPC transfer from UNIBE-LHEP to one INFN-T1 StoRM endpoint fails
  • ds-102-10-20.cr.cnaf.infn.it - transfer succeeds
  • ds-102-11-20.cr.cnaf.infn.it - transfer fails
    • failure: SSLHandshakeException while fetching https://dpm.lhep.unibe.ch:443/dpm/lhep.unibe.ch/home/atlas/atlasscratchdisk/SAM/test_unibe_dcache.src: java.lang.NullPointerException

Old issues

• INFN-T1 StoRM upload performance GGUS:150181
  • 1GB files with 250 connections transfer test from EOS to INFN-T1 with SRM ~ 62Gb/s
  • 1GB files with 250 connections transfer test from EOS to INFN-T1 with WebDAV ~ 12Gb/s
  • Transfer time plot for SRM+gsiftp vs. WebDAV

4 Token Authorization testbed

Speakers: Andrea Ceccanti (Universita e INFN, Bologna (IT)), Andrea Ceccanti (Unknown)

5 http and xrootd protocol news

Speakers: Brian Paul Bockelman (University of Nebraska Lincoln (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

6 AOB