Notes / discussion

Brian

• Token effort started in 2016 with proposal in 2017
  • it took 2 years to agree on protocol
  • additional 3 years to develop and deploy in production
• 3-4 years if we decide to move to pre-signed URLs
  • Maarten:
    • a lot of work has already been done to implement support for tokens
      • in dCache, EOS, StoRM, XRootD, FTS, Rucio
    • a similar amount of work would be needed for pre-signed URLs
      • and there is not enough time between now and DC24
    • it therefore seems best to do DC24 with tokens and see how that goes
      • pre-signed URLs may get into the picture later
• Pre-signed URLs can come alongside of tokens in the future
  • it would be useful to have a storage that supports pre-signed URLs to be able to test this concept
  • there are a lot of open questions with S3 that are not solved today
    • not everything is solved for tokens either

Andreas (EOS/ALICE)

• ALICE is using pre-signed URLs since ~ 2005 / in production for 20 years
• Our storages are not Amazon
  • Amazon use symmetric keys
  • symmetric keys can't be used securely on 100+ storage sites
  • ALICE is using public/private key to overcome this issues
  • cloud providers don't do third-party-copy and symmetric keys are less issue
  • Chris - may be there is also asymmetrically key option for Amazon / S3 API (to be checked)
• If you leak symmetric key - it's disaster
• Rotating symmetric keys would be a mess / insane
  • currently same proxies are sent everywhere, but they have lifetime
  • thinking about symmetric keys for our infrastructure is generally insane (using Amazon model is bad)
• Performance concerns with IAM as a token issuer
  • Dirac / Rucio could issue their own JWT tokens
    • create access tokens for storage access that will be trusted
    • ALICE already use this approach
  • Brian: this is what OSG does - give service (e.g. Dirac) IAM private key
    • Chris: this can't work for multi-VO token issuers, e.g. EGI CheckIn

Julien (TAPE)

• HTTP REST (for TAPE) standard designed for tokens, changes would require additional effort
• signed url for REST API call - it would be necessary to use e.g. SE-tokens
• Maarten: Dirac could have key to create signed URL for tape

Mihai (FTS)

• FTS must support also non-WLCG communities
  • not everybody use Rucio / Dirac that could be in charge of generating pre-signed URLs
  • FTS already support storing shared keys for cloud storages
    • not optimal for shared multi-VO instances
• Using just pre-signed URLs not possible
  • This would not make FTS devs live easier

Martin (Rucio)

• Even with locally pre-signed URLs we still have to consider overloading Rucio/Dirac with signing
  • ATLAS already faced CPU performance issues with Rucio caused by transfers with cloud storages that rely on pre-signed URLs
    • sub-optimal code used for listing file replicas
    • still needs to be considered also for normal transfers in case all transfers needs to be signed
  • per transfer token / pre-signed URL granularity may be heavy on CPU usage
• ATLAS already store keys in FTS for transfers with cloud storages
• Rucio also have non-WLCG communities - necessary to support OIDC tokens
  • can't focus just on pre-signed URLs - this as in case of FTS doesn't simplify Rucio devs lives

Stephan (CMS)

• CMS already started to deploy WLCG JWT token configuration on their storage sites
• Let's focus on error handling than supporting new features
  • much higher value than exploring / new research with pre-signed URL

Michael Davis

•  what about token for compute
  • Chris: tokens for compute are fine

Petr

• It seems to me that e.g. also Globus rely on (access) tokens instead of pre-signed URLs

IAM performance questions / concerns

• CMS expect to generate ~ 200 tokens per hour for transfers (TPC?)
• max few thousands of tokens per hour for ATLAS with reasonable access token granularity
• LHCb considered only per-transfer tokens
  • this can easily reach IAM performance limits
  • no official numbers from IAM devs / ops team
  • 100Hz access token rate
    • maximum that can be currently reached with CERN IAM instances
    • lead to CERN IAM Halloween incident after 2 days (slide 7)
• access tokens could be generated also by Dirac/Rucio
  • IAM can advertise multiple keys with well-known jwks_uri
    • this may be tricky with multi-VO services & issuers like EGI CheckIn
  • may be useful e.g. for Rucio download / upload use-case
  • concerns about traceability when multiple services can generate valid token
  • can't be used for more complex workflows (token exchange, refresh tokens), because it would be necessary to re-implement OAuth2
    • IAM keeps track of access tokens / store details in DB
    • introspection most probably doesn't work for access token is issued by different party
• IAM built on top of OpenID Connect reference implementation MitreID
  • database design causing performance issuer, e.g. iam#579, iam#580

Summary (by P.V.)

• Chris - come with use-cases that are not well covered (understood) by WLCG JWT tokens (e.g. WLCG-AuthZ-WG/#22)
• At least ATLAS & CMS will use WLCG JWT tokens for DC24 and we'll push service providers in this direction
  • if LHCb / DIRAC cannot make tokens work for data management by that time, that is not a problem and LHCb can participate using X509 / VOMS
  • it would be useful to have in-person meeting / hackathon to move forward with DC24
  • ~ one in summer at CERN and one at the end of calendar year
  • to be discussed in next WLCG DOMA General (May 31, 16:00)
• It is up to LHCb if they choose to investigate different approaches, but that's not a short term project (this should not be at the expense of our WLCG JWT progress)
• IAM performance concerns can be overcome by issuing simple access tokens directly by Dirac or Rucio (shared IAM key)
  • P.V.: I would like to see more activity from IAM dev and ops team in addressing performance issues