WLCG DOMA BDT Meeting

Wednesday 21 Jun 2023, 16:30 → 17:35 Europe/Zurich

Brian Paul Bockelman (University of Wisconsin Madison (US)), Maria Arsuaga Rios (CERN), Petr Vokac (Czech Technical University in Prague (CZ))

Topic: WLCG DOMA BDT Meeting (twiki)

16:30 → 16:35 News

Actions:
* 3 questions/numbers to be answered by the experiments (ATLAS, CMS, LHCb):
- Token rates
- Unique token per file transfer or one unique token per storage endpoint per hour
- Lifetime token

• Official statement that there will be no reliance on the revocation of access tokens.

16:35 → 16:45 IAM tests and updates

Speaker: Berk Balci (Istanbul Technical University (TR))

WLCG IAM Performance Testing and Improvement

• Request rate is not the only important performance parameter - also latency is important
  • "reasonable" latency depends on use-case
  • for interactive user interaction it must be short (definitely not seconds)
  • increasing response time for busy IAM
• For Rucio / FTS workflow we would like to understand also refresh token request rate and token exchange.

16:45 → 17:25 Rucio and FTS transfers with tokens

Presentation and discussion about Rucio&FTS design document for transfers with tokens.

Speakers: Dimitrios Christidis (CERN), Mihai Patrascoiu (CERN)

2023.06.21 Rucio and FTS Transfers with Tokens.pdf

FTS_Revised_Token_Workflow_2023.pdf

Rucio

• all tokens always audience restricted to reduce impact of compromised AT to one storage
• Q: why not to rely fully on FTS to be able to obtain transfer tokens?
  • this would require full transfer privileges built into FTS
  • may not be always acceptable for multi-VO FTS instances
  • FTS use token exchange to get refresh token with VO defined granularity and limited lifetime
• Q: described Rucio path prefix on slide 4
  • this is just about token granularity and not about storage configuration for particular issuer
  • simplification - RSE & file level $PATH granularity
    • different experiment use different storage path construction (example is ATLAS specific)
    • it'll be possible to define also granularity between RSE <-> file level
• Q: expected granularity for storage.create
  • for majority of experiments RSE level should be fine
    • still very limited damage
  • this tokens can't be used to destroy stored files
    • with stolen token attacker could rename all files in storage.create:$PATH
    • completely safe for file level granularity, but attacker could cause quite a lot of mess in the namespace with e.g. RSE level granularity
      • it may be useful to have more precise storage.create definition in the profile
      • e.g. rename only files to a different name within same directory
        • test behavior of SE implementation
• Q: is it necessary to decrease the data loss risk from considerable (with x.509) to practically zero during the transition to tokens? I would like to avoid over-engineering
• Q: does FTS request the tokens itself using client credentials?
  • FTS uses the initial AT to obtain a refresh token. Further on, when needed, it will obtain new ATs.
  • And yes, FTS does have a client ID + secret

FTS

• "Identity token" mentioned in the slides really means normal access token that can be used to talk with FTS (REST) endpoints
  • don't confuse it with OIDC Identity Tokens
  • that's why called "Identity token" in quotation marks
    • technically it is used to associate client with internal FTS identity
• plan to provide more details about FTS client interaction with tokens in more detailed documents
• Q: what are the future plans with described OAuth flow after implementing "V. Extended callback support after DC24"?
  • this will be always supported
  • callback URL approach allows to offload all communication with IAM to the experimental framework (e.g. Rucio, Dirac, ...)
  • even more limited privileges given to FTS (e.g. no token exchange done by FTS)
• Q: how FTS store / cache token passed from client for source & destination storage - should their storage be considered as security risk
  • they are stored in the FTS database
    • delegated X.509 also stored in database (allow to distribute transfer load to multiple FTS instances)
  • developers considering encryption of these data
• Q: are you planning to support (test) everything also with different identity providers
  • CILogon is used e.g. by DUNE (with WLCG JWT profile) and Fermilab have quite ambitious plans with token adoption
  • involve people from DUNE computing / Fermilab(?)
    • we'll probably need at some point more detailed description about token flows done by FTS
    • what is the CILogon equivalent of test WLCG IAM instance(?)
• Q: how (many) transfer tokens FTS needs to store?
  • e.g. for RSE level token granularity and storage.read:/RSE it will be just one token for all transfers reading data from one RSE
    • with each new submitted transfer FTS gets new source & destination storage access token
    • FTS can limit number of requests using its own refresh token obtained by token exchange
    • also Rucio rely on caching access token to reduce unnecessary load on IAM infrastructure
• Q: what happens if acquiring refresh token takes too much time
  • transfer job submission access token may become invalid (expired)
  • FTS have no way to get new fresh access tokens
    • such transfer fails
  • again we need quite relliable IAM service to avoid these corner case as much as possible
    • still FTS behavior should be well defined and simple
    • don't plan to come with recovery mechanism for expired token and slow token exchange

17:25 → 17:30 AOB