WLCG DOMA BDT Meeting

Wednesday 20 Jul 2022, 16:30 → 17:30 Europe/Zurich

Brian Paul Bockelman (University of Wisconsin Madison (US)), Maria Arsuaga Rios (CERN), Petr Vokac (Czech Technical University in Prague (CZ))

Topic: WLCG DOMA BDT Meeting (twiki)

16:30 → 16:35 News

16:35 → 16:50 Transfers with tokens

Speaker: Francesco Giacomini (INFN CNAF)

Token protection (enforced TLS)

• by design for HTTPS protocol
  • never redirect to HTTP with WLCG JWT token
  • probably fine for dCache that use internal token (do I remember this correctly?)?
• xroot client behavior
  • token is opaque cgi data for client and it can be sent on unsecured channel
  • everybody should always use xroots:// when they use tokens
    • basically exactly same requirement as in https:// case

=>

• we have to move to xroots (TLS) everywhere before we start to use token
  • in future (timeline not yet defined) we must organize campaign where we'll ask sites to provide additional xroots:// protocol
• minimum xrootd client / libraries version 5.x
  • older experiment software releases may be built with 4.x
  • impossible to access storage directly from job payload that use older releases
• experiments should think (now) how to support older workflows once we move to tokens => xroots

Status

• ATLAS
  • use directIO only for specific jobs
  • jobs can be forced to use copy2scartch for older releases
    • technically we are "ready" for xroots:// except transferring whole files can dramatically increase network bandwidth (+ storage utilization)
    • should be OK by the time we will be ready to move to tokens
      • new software releases built with XRootD 5.x
• ALICE
  • everything (read/write) is directIO
  • old releases probably still use even XRootD 3.x libraries
  • it may be very difficult to move legacy software to use 5.x
• CMS
  • reading is always directIO
    • hope that use of legacy releases will go down (to zero) over time
  • writing done outside main job
    • easy to replace xroot version
• LHCb
  • ???

Token configuration for WLCG sites

• prepare configuration for each experiment and each storage implementation
  • may not always cover very complicated sites
  • these sites have sufficient local expertise to understand requirements
• prepare testsuite to validate site storage configuration
  • not just operation that should succeed but also operation that should fail
  • test regularly
    • CMS use SAM tests for protocol compliance
    • standard way to communicate issues with sites
• organization
  • start with friendly sites to come with reference configuration for each experiment and each implementation
  • be ready before winter 2022/23 shutdown
    • good opportunity for wider deployment of "token compliant software and configuration"

WebDAV with tokens compliance testbed

• documentation update (dCache, XRootD -> results)
• does dCache configuration for prometheus supports also X.509(?)
• recent tests for EOS and Echo
  • most probably just incorrect mapping for "protected" area
• xrootd storage.create
  • open email thread with developers about XRootD 5.5 release plans
  • necessary for compliance testbed
    • people plan to configure their XRootD flavor storage properly only once they have XRootD release that have a chance to pass tests

Audiences vs. different token issuers

• do we expect all issuer agree on the same "aud" format?
  • Resource indicators described RFC8707
    • WLCG JWT profile recommends to use schema://fqdn:port
    • Even our own software doesn't strictly follow this recommendation (e.g. HTCondor)
    • Some VO can rely on generic https://wlcg.cern.ch/jwt/v1/any while others may forbid such wide "aud" claims
  • Should our (storage) implementations provide per issuer configurable audiences?
    • currently not supported by dCache scitoken plugin and scitoken-cpp library
    • we decided that it is sufficient to have support for multiple audiences
      • already tested with XRootD and dCache

16:50 → 17:05 Tape REST access

Speaker: Cedric Caffy (CERN)

Mihai:

- Gfal2 v2.21.0 released (8th of July 2022) -- bring full support for Tape REST API
- FTS v3.12.0 released (14th of July 2022) -- brings support for /Archiveinfo and /Release operations in FTS

- The FTS team has the following tape-rest-api tests: fts-scripts/tape-rest-api
- The testsuite requires python3-gfal2 v1.12.0 + gfal2 v2.21.0 (found in DMC production repos)
- Running the testsuite:

$ export TAPE_ENDPOINT=<url-to-test-endpoint-directory> # example: https://ctadevfts.cern.ch:8444/eos/ctaeos/preprod/gfal2_tests/
$ export TEST_FILE=<url-to-test-file> # example: file:///tmp/file.1mb
$ python3 -m pytest test_*.py

- Running against the CTA endpoint passed all tests
- Individual tape commands have been run against the dCache endpoint. At the time, the endpoint was using real tape, so there was no point in running the test suite. The report has been written here: CodiMD

P.V. notes

• dCache file release and staging operations should now work (bad request routing configuration in the past)
• regular tests can't be run with real tape backend because of latency for operations with real tape

17:05 → 17:20 Packet marking

Speakers: Marian Babik (CERN), Shawn Mc Kee (University of Michigan (US))

17:20 → 17:30 AOB