DOMA / TPC Meeting

Wednesday 5 Aug 2020, 17:30 → 18:50 Europe/Zurich

We will use Zoom
https://cern.zoom.us/j/98672039467?pwd=dXgzVW9BalhjS3lKdEVTV1F0NTFvZz09

Meeting ID: 215 543 044

We will use Zoom
https://cern.zoom.us/j/98672039467?pwd=dXgzVW9BalhjS3lKdEVTV1F0NTFvZz09

Meeting ID: 215 543 044

Token Authorization testbed

* doodle poll for the hackaton has been out for a while and is starting to be populated

* rucio new version allows better monitoring of transfers donewith OIDC AAI

Experiments tests

* ATLAS 2 sites in production AGLT2 (dcache) and prague (DPM). Added 3 more dcache sites FZK, pic and DESY-HH. The change is done on a destination basis so all the transfers willbe initiated with the http protocol if it is available also at the source. This has caused some problems with CERN. Until only prague and AGLT2 were enabled transfers with CERN were ok. But with the 3 extra sites transfers from CERN were degraded. The main problem seems to be that nginx cannot take the load and to become really production EOSATLAS should start using Xrdhttp. The code exists and has already been committed by Elvin but it needs to be backported from xrootd 5 and properly released. The best estimate we can have for this to be put in production is in 4 weeks i.e. at the beginning of September. StoRM hasn't presented any problem with this 5 sites but the previously reported instability means sites have to apply a workaround or upgrade to 1.11.18 which hasn't been released yet. We should start looking at for the tests is INFN-T1. Petr asked also to add a purely xrootd site. Wei thinks it is better to wait for release 5 in particular 5.0.1 which has become RC today because the libraries underneath are quite different from release 4 and the change will create some disruption so it is better to avoid this in production. Brief talks of the http errors. Need to talk to dcache about more optimal configuration.

* CMS Diego reports everything is working fine for CMS they are adding two extra sites one in India (TIFR) and the other in switzerland(CSCS) which are now working on their macaroons configuration. At Florida there is a problem with the checksums request, the site has xrootd with lustre. KIT was working with Nebraska for a while and was added recently for transfers to UCSD and seems to be working fine

Http protocol

* RAL-ECHO update, the problem ECHO was having was with the checks gfal does around the copy, in particular mkdir and stat checks. It is possible to trigger the FTS transfer with --strict-copy which does only the transfer and skips the tests. In the standard ATLAS environment this would be configured in AGIS but with the DOMA rucio this needs to be configured by hand. Petr has access and will look into it. Alessandra is worried that not doing these checks might be dangerous, but apparently RAL is already doing production transfers with gsiftp and --strict-copy. Same warning as above about change of xrootd release, but ECHO instance for DOMA is a testbed and it ok to break it.
 

17:30 → 17:50 Token Authorization testbed

Speakers: Andrea Ceccanti (Universita e INFN, Bologna (IT)), Andrea Ceccanti (Unknown), Jaroslav Guenther (CERN)

17:50 → 18:10 Experiments tests

Speaker: Petr Vokac (Czech Technical University (CZ))

HTTP-TPC sites last 2 days

I think we should make progress with our infrastructure in two weeks
(once DPM 1.14 is out) -> drop HTTP endpoints that doesn't support
macaroons/ask DPM sites to configure them (15 sites), because only than
we can include e.g. StoRM (INFN-T1 is candidate for active TPC) and
we'll have to wait for native XRootD endpoint, because as Wei said they
it is better to wait for 5.x with HTTP-TPC ... ideally we could gather
test data not just for DPM+dCache for September GDB.

* dCache SRR and empty storage section with no SRM (e.g. IN2P3-CC)
  * https://rt.dcache.org/Ticket/Display.html?id=9825
  * no ticket yet(?)

* is there optimal dCache configuration for HTTP-TPC
  * I think I saw it is possible not to redirect transfer from head node
to disk but use doors
  * also some sites use multiple DNS A/AAAA records for "headnode" while
others just one (may be load-balancer)
  * we can't easily say / detect that site has optimal dCache configuration

* StoRM < 1.11.18 default configuration for WebDAV performance issues
observed in the past at INFN-T1
  * details in https://issues.infn.it/jira/browse/STOR-1200
  * should we ask all sites to follow
https://issues.infn.it/jira/browse/STOR-1207

* new dCache sites with preferred HTTP-TPC (PIC, FZK, DESY-HH)
  * IN2P3-CC asked if they are fine to be moved
  * should we add also MWT2(?)

* EOS old nginx WebDAV interface disabled for TPC
  * XrdHttp will be used in future
  * no point to test something that is not going to be used
  * scalability issues / horrible transfer performance while increasing
number of ATLAS HTTP-TPC sites

* still unable to include storages except dCache (DPM 1.14 with release
data August 15)
  * macaroons not supported by all source sites => we need storage that
supports gridsite delegation
    * add macaroon support to all sites or drop their HTTP-TPC from
allowed TPC protocols
    * once we have DPM 1.14 and once EOS works with macaroons
(preferably till the end of August)
    * than we'll include INFN-T1 in HTTP-TPC + some XRootD site
  * 15 DPM sites without enabled macaroons
    Australia-ATLAS FMPhI-UNIBA GRIF-IRFU GRIF-LAL IN2P3-CPPM IN2P3-LPC
IN2P3-LPSC INFN-COSENZA RO-07-NIPNE TR-10-ULAKBIM UKI-SCOTGRID-DURHAM
UKI-SCOTGRID-ECDF UKI-SOUTHGRID-OX-HEP UNIBE-LHEP ZA-WITS-CORE
  * EOS ATLAS with no macaroon support (last attempt with 4.8.6 ->
crashing XRootD -> fixed(?), can we try again(?))
  * we would like to see also "normal" XRootD in our HTTP-TPC (SWT2_CPB,
NET2 or OU_OSCER)
  * add back HTTP-TPC to INFN-T1 (stability issues solved 2 months ago)

18:10 → 18:15 HTTP Protocol Update

Speaker: Brian Paul Bockelman (University of Nebraska Lincoln (US))

18:15 → 18:20 Xrootd Protocol Update

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

18:20 → 18:35 Discussion