Present: John SDS, Tom Dack (Minutes), Stephan L, Dave D, Jim B, Julie M, Dimitrios C, Matthew D, Maarten L, Federice A, Francesco G, Roberta M, SK, 

• Token Profile PRs and Open Issues
  • 2 currently open issues to be discussed today:
    
    • https://github.com/WLCG-AuthZ-WG/common-jwt-profile/pull/27
    • https://github.com/WLCG-AuthZ-WG/common-jwt-profile/pull/45

• PR 27: Clarifies Scope of storage.stage & storage.read
  • The original thinking was that stage should be a superset of read - as typically, you stage files in order to read them
  • The proposal here is that stage should be a separate action, because:
    • it can take many hours or even days to complete;
    • it would be done by agents different from the workflows that will actually read the files.
  • Currently, the definition of stage is inconsistent across the profile
  • It is not believed that anyone currently is implementing stage
  • Maarten suggests following up in the DOMA BDT WG to ensure the correct people can weigh into the discussions
  • Mainly affects usage of tape, and so a discussion with middleware vendors may be needed
  • Francesco to email this issue around to the Tape REST API WG in order to flag this topic there.

•  PR 45: Provide examples of how an "authorized area" is interpreted for HTTP requests
  • Inconsistencies found due to building on historical middleware
  • This issue was initially proposed to fill this gap - originally proposed by Brian B, with following input from Enrico
  • Need for consistent namespacing within tokens
  • Need a description of how tokens should be interpreted to avoid clashes between disc and tape storage elements due to usage of the same/similar paths
  • The StoRM devs should comment in the PR and it probably needs a follow-up in the DOMA BDT WG

• Token Profile Versioning 
  
  • Balance between practicality and acknowledging that changes on production systems are slow
  • But noting that there have been substantial changes - we should not still be on V1. We are not backwards compatible with the 2019 release
  • The next step here seems to be to move to a 2.0 release soon - ideally by the end of the spring.
  • The concept here is to encourage middleware to be flexible in which versions they accept - conceptually, if they accept 2.0, they should accept 2.x
    • They should then ignore anything they do not understand
  • Every minor release should not introduce backwards incompatible changes
  • Francesco questions whether there have been any changes to warrant a 2.0 change - have there been any backwards incompatible changes at this stage?
    • Maarten posits that there hasn't been any notably nasty change which would be backwards incompatible
    • Dave notes that the profile says that anything which does not match should be rejected, and so therefore we are now making an incompatible change when requiring that the minor versions are backwards compatible
    • This means that a 2.0 version should be required.
    • Maarten suggests allowing production to "pretend" it is still using 1.0 tokens, having a different document versioning. We then wait for everyone to be ready before changing the token versions.
    • This would keep things more or less the same for production
    • Acknowledging that if we publish 2.0, middleware changes will take time - so what do we do before then. For changes to move up to speed, it could take half a year or more
    • Allowing the adoption of 2.0 features whilst not changing the token version number
  • Need to agree on what paths mean in tokens. There is some room for a backward incompatible change
    • An example from dCache. Look at the path and see if it doesn't match. If it does not, they check whether the VO root was missing, and so try again with the VO root
  • Stephan notes that we could have a situation where a service provider may need to support multiple token versions simultaneously, so that there is an easy switch. Any time a larger change is made, and will take time to propagate, then there is likely a need to support multiple versions so that we can switch - no way to switch on the experiment side before the sites are ready.
  • Need an intelligent rollout so as to not break production
  • Likely will not be completely solved this year, and will need to evolve over the next few years. The aim for this year should be that the profile document should have caught up.
  • Stephan notes that we should anticipate situations with unsuported profile versions, and so should define the expected error message for this circumstance
  • Dave suggests making a 2.0 tag on github, so as to establish which ones are being considered and are important for 2.0
    • Maarten concurs - there are a number of open issues without pull requests, and we should consider which ones can fit in.
    • The aim however should be move away from 1.0, towards a more correct version recognising lessons learnt over the last 5 years

• High-priority IAM issues working document
  
  • Can be found here
  • These are issues pertaining to VO management, which are higher priority due to the decommissioning of VOMS(-Admin) in ~2 months
  • This google doc bootstrapped from a similar doc, made by Petr, recording issues important to Atlas
  • Maarten hopes that the IAM development team can look and review this and action steps
  • Maarten encourages comments on this
    
    • Francesco thanks the work done for this, and shares that the team have copied this content into a GitHub dashboard and enourages people to look at the open issues shared here: https://github.com/orgs/indigo-iam/projects/8 
    • This should indicate what the team believe are the highest priority tasks, and would appreciate a review to check their thoughts here
    • Then move forward, aiming to be ready for transition
    • Maarten comments that if things are ready to be tried in production, a new tag should be created and then can be tested on the usual machinery
    • Ultimately encourages the IAM team to dictate when is a good time to test - but noting that if something is viewed as fixed and would be profitable, then this should be utilised

• Work underway in parallel to move LHC experiment IAMs from OpenShift to Kubernetes
  
  • An agreement that this work is independent of other activities
  • looking to start gaining value from K8s work as soon as possible - with endpoints available for testing soon
  • This would be a limited setup, not the full HA setup, which will require further R&D 
  • Should be able to be used in parallel, so experiments can switch between the two 
  • Work is planned, but not giving a full firm timeline at this time
  • Berk has recently completed high-rate tests for IAM instances on OpenShift and K8s, using the same database.
  • High token rate (100+ Hz) for 8 hours, saw zero errors
  • This means we have no worries about the DB being shared by 2 instances for the same VO.

• Training for VO administration.
  
  • Maarten is currently working on this, to give documentation for what a person in an experiment secretariat might see, as well as what someone looking to register may see
  • The aim is to publish somewhere convenient for usage

• IAM technical hackathon planned for end of May, at CNAF in Bologna
  
  • 29th - 30th May
  • Will have scheduled slots for remote participation, and offline hackathon working
  • https://indico.cern.ch/event/1401472/