DOMA / TPC Meeting

Wednesday 11 Mar 2020, 17:30 → 18:55 Europe/Zurich

17:30 → 17:50 ATLAS TPC tests

Speaker: Petr Vokac (Czech Technical University (CZ))

ATLAS non-GridFTP test with production infrastructure
* https://its.cern.ch/jira/browse/ADCINFR-166
* S&C notes https://indico.cern.ch/event/881125/contributions/3723661/attachments/1987192/3318955/atlas_tpc.pdf
  * PRAGUELCG2 (DPM devel), UKI-NORTHGRID-LANCS-HEP (DPM 1.13.2), UKI-NORTHGRID-MAN-HEP (1.13.2)
* number of issues with WebDAV TPC with tokens hidden by preferred X.509 delegation
  * generally WebDAV has a lot of options
    * push, pull, stream
    * gridsite (X.509 delegation), tokens (macaroons, oidc)
  * not all sites supports everything

Timeouts (speed limits)
* DPM 1.13.2 WebDAV TPC speed limit set to 1MiB/s (fixed)
* XRootD 4.11.2 WebDAV TPC same speed limit as DPM, but applicable only for CentOS8 (fixed)
* dCache wait 2 minutes for HEAD (used also for checksum calculation)
  https://github.com/dCache/dcache/issues/5353

FTS -> gfal2 fails WebDAV TPC with tokens within same site
* token cached per hostname, but macaroons issued for specific file
* ATLAS use multiple RSE within one storage => WFMS can create TPC within one storage
* we can't really continue with WebDAV tests without fixing this issue
  https://its.cern.ch/jira/browse/FTS-1546
  https://its.cern.ch/jira/browse/FTS-1528
  https://its.cern.ch/jira/browse/FTS-1520

WLCG dCache upgrade task force - partial success for TPC
* XRootD TPC not enable by default
  * discussion about request signing - our preference / requirements still not clear
* AGLT2 and ifae already enabled "pool.mover.xrootd.tpc-authn-plugins=gsi"

dCache issue with macaroons
* Internal error: KeeperErrorCode = Session expired for /dcache/macaroons/secrets/2020-03-11T09:42:13.21
  https://github.com/dCache/dcache/issues/5253

dCache / DPM issue validating certificates
* some dCache servers are not able to use WebDAV TPC with tokens and DPM destination
* failure: Remote copy failed with status code 0: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
  https://its.cern.ch/jira/browse/ADCINFR-166?focusedCommentId=3119772&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-3119772

StoRM - XRootD TPC never tested
* no storage endpoint configured with XRootD TPC

XRootD
* we would like to see SLACXRD(?) in our ATLAS XRootD TPC tests
* add in AGIS also WebDAV for TPC(?)

EOS
* it is up to ATLAS to ask EOS Ops to upgrade to version that supports XRootD and WebDAV TPC

DPM/dmlite 1.13.2 (latest official release)
* mutiple issues with ACL permissions (not really TPC problem, but makes debugging issues difficult)
* first TPC HTTP transfer with proxy delegation always fail
  https://its.cern.ch/jira/browse/LCGDM-2898
* gridsite delegation preferred even in presence of tokens (fixed)
  * DPM doesn't use "Creadentials: something" header but still rely on non-standard X-No-Delegate
    https://its.cern.ch/jira/browse/LCGDM-2909
    * unfortunatelly X-No-Delegate was added in gfal2 sources last week
  * partially fixed by in February but update broke TPC when source did not support tokens
    https://gitlab.cern.ch/lcgdm/dmlite/commit/a501779e130f5e92d2d45d7ac82a7cec17f45f96
  * now hopefully fixed https://its.cern.ch/jira/browse/LCGDM-2908
* default configuration with redirection to HTTP can expose tokens (fixed)
  * davix (gfal -> fts) by default set "Secure-Redirection: 1"
  * don't rely on client to set non-standard header to secure transfers
  https://its.cern.ch/jira/browse/LCGDM-2910
* macaroon without before: caveat (fixed)
  https://its.cern.ch/jira/browse/LCGDM-2906
* memory issues with xrootd - switched to jemalloc (fixed)
  https://its.cern.ch/jira/browse/LCGDM-2903
* poor checksum performance due to small read buffer (fixed)
  https://its.cern.ch/jira/browse/LCGDM-2902

Update list of supported TLS protocols and ciphers
* for security reasons we should move to TLS1.2 and higher
* for performance reasons we should limit advertised ciphers accelerated in hardware
  https://its.cern.ch/jira/browse/LCGDM-2911

DPM OIDC / FTS XDC
* unable to submit transfer manually (failing)
  curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credentials: oidc' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: bearer $TSRC" -H "Source: $SRC" -H 'OIDC_CLAIM_sub: 58280cfd-ed7f-4954-90c7-cfde610cb963' -H 'OIDC_CLAIM_iss: https://wlcg.cloud.cnaf.infn.it/' -H 'OIDC_CLAIM_aud: https://wlcg.cern.ch/jwt/v1/any' -H 'OIDC_CLAIM_wlcg.groups: wlcg,wlcg/xfer' "$DST"
* fts3-xdc.cern.ch returns internal error 500
  ./fts-rest-transfer-submit -s https://fts3-xdc.cern.ch:8446 davs://golias100.farm.particle.cz/dpm/farm.particle.cz/home/wlcg/1M davs://golias100.farm.particle.cz/dpm/farm.particle.cz/home/wlcg/x

17:50 → 18:05 Token Authorization testbed

Speaker: Andrea Ceccanti (Universita e INFN, Bologna (IT))

18:05 → 18:20 Xrootd Protocol Update

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

need to add 

root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test/ to xrootd TPC stress tests

https://griddev03.slac.stanford.edu:1094/xrootd/atlas/tpctest to http TPC stress tests 

both should support dteam VO.

18:20 → 18:35 HTTP Protocol Update

Speaker: Brian Paul Bockelman (University of Nebraska Lincoln (US))

18:35 → 18:55 Discussion