Choose timezone
Your profile timezone:
BiLD-Dev
→
Europe/Zurich
Description
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Join Zoom Meeting
https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09
Meeting ID: 910 8369 4183
Passcode: 12345678
One tap mobile
+41432107108,,91083694183# Switzerland
+41315280988,,91083694183# Switzerland
Dial by your location
+41 43 210 71 08 Switzerland
+41 31 528 09 88 Switzerland
+41 43 210 70 42 Switzerland
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
Meeting ID: 910 8369 4183
Find your local number: https://cern.zoom.us/u/artUfAdNB
Join by SIP
91083694183@188.184.89.188
91083694183@188.185.118.153
Join by H.323
188.184.89.188
188.185.118.153
Meeting ID: 910 8369 4183
Passcode: 12345678
BiLD (Bi-weekly DIRAC Development meeting) – 09/03/2023
On Zoom: Federico, Andrei, André, Alexey, Alexandre, Bertrand, Cedric, Christophe, Christopher, Daniela, Domenico, Ewoud, Guillaume, Igor, Lorenzo, Michel, Simon, Janusz, Ueda, Hideki, Xiaomei
Apologies:
Follow-up from previous meetings
- Last BiLD 2 weeks ago
- all updates below
- Hackathon last week on 8.1.0a10 – https://github.com/DIRACGrid/DIRAC/releases/tag/v8.1.0a10
- Changes added in minutes of https://indico.cern.ch/event/1258505/
- focussed on verifying submission to HTCondor and AREX CEs with tokens (code changes went to v8.0)
- main goal was reached, but can’t get the pilot outputs: https://trello.com/c/79wAKhqE — Andrei created https://github.com/DIRACGrid/DIRAC/pull/6890 for this
- access token also maybe not refreshed frequently enough
- PR https://github.com/DIRACGrid/DIRAC/pull/6874 content hotfixed in, and would need anyway a backport to rel-v8r0
- Maybe problematic anyway
- Issue Mix of groups for ReportGeneratorHandler created
- main goal was reached, but can’t get the pilot outputs: https://trello.com/c/79wAKhqE — Andrei created https://github.com/DIRACGrid/DIRAC/pull/6890 for this
DIRAC communities roundtable
LHCb:
Federico+Alexandre+Christophe+Christopher+Alexey
- Campaigned (through tests and GGUS) for making sure that we would be able to run with OpenSSL 3.0, and with SingularityCE everywhere (the 2 are not connected).
EGI
Andrei
- Playing with several VOs on how to define groups
- Maybe using tokens for users with “entitlements”, mapped to DIRAC groups
- Still on v7.3
ILC/CLIC/FCC/Calice:
André, Lorenzo
- NTR
Belle2
Hideki, Cedric, Michel
- Migration to v7r3 ongoing
- Conf of JobCleaningAgent can be used https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg#L192
- from previous meeting Production to v7r2, but need to move to v8.0 before May, for HTCondor support, potentially also for Rucio support
GridPP:
Daniela+Simon+Janusz
- No updates in production.
Juno:
Xiaomei
- Want to try token submission, first need to move to v8.0
JINR:
Igor
- 400 TB of Raw dataprocessed for BM@N experiment.
- Struggling with xrootd client issue which should be fixed by xrootd 5.5.3
- Federico should be solved by simply updating to the latest DIRACOS2, simply update to latest DIRAC v7.3.35
- Issue with drm -F
Topics from GitHub/Discussions or Google forum
only un-answered topics:
- DIRAC.Interfaces.API.Job.Job.setOutputData::outputSE - replica level specification with inclusion/exclusion
- Should we turn it into issue? (https://github.com/DIRACGrid/DIRAC/discussions/6834#discussioncomment-5062502)
- Update to 8.0.15: M2Crypto.SSL.SSLError: wrong version number
- COMDIRAC profiles & dinit removal
- question for Andrei but not only
DIRAC releases
- v7r3
- no new releases since last BiLD (v7.3.35 is the latest). There are anyway few fixes merged in. We should have a new release after [v7r3] Better handle requests exceptions in AREXCE + minor fix in an error message in ConfigHelper is merged.
- v8r0
- Last one is v8.0.16
- NEW: (#6803) HTCondorComputingElement, ARC(6)ComputingElement and AREXComputingElement enabled to for job operations with tokens
- NEW: (#6803) SiteDirector enabled to select queues by tags
- NEW: (#6803) SiteDirector sets up tokens for ComputingElements configured with the Token tag
- NEW: (#6852) Add setPriority to Job API
- Optimizations for FTS3DB
- Last one is v8.0.16
- v8r1
- Mostly fixes from the hackathon
(Focus of the day) Monitoring:
- Presentation by Ewoud on monitoring evolution – grafana dashboard
- WLCG project, but results will be available to all the DIRAC communities, it’s “enough” to add a Grafana cluster
- (almost) completely separated project from DIRAC line of development
- all dashboard definitions will be added to DIRAC code.
DIRAC projects
- Federico Few more projects “Archived” https://github.com/DIRACGrid/
DIRAC:
Issues by milestone:
- v7r3:
- 3 open issues
- v8.0:
- 26 open issues
- Conflicts when updating JobParameters in ElasticJobParametersDB
- Possible solution proposed in https://github.com/DIRACGrid/DIRAC/pull/6802 but unsure + discussions, but anyway no other solution in sight.
- Expand integration tests for IaM
- Alexandre created PR https://github.com/DIRACGrid/DIRAC/pull/6882
- Tokens: ssid should not be added as a DN
- Conflicts when updating JobParameters in ElasticJobParametersDB
- 26 open issues
- v8.1:
- 21 open issues
- (for PilotLogging) find the VO from the proxy (and from the token)
- re-assigned to Federico for dealing with the “HLT” pilots
- (for PilotLogging) find the VO from the proxy (and from the token)
- 21 open issues
Few new issues:
- Some components still create threads in tornado
- Christophe will have a look
- Jobs stuck in matched when using SingularityCE
- Mix of groups for ReportGeneratorHandler (maybe for v8.0, maybe for Web)
- worrying behavior. Very difficult to reproduce
- It might be only a WebApp-related issue, maybe even only related to Accounting, but unsure about it
PRs discussed:
- [8.0] feat: initial IAM integration tests
- Few questions/comments in there
WebApp:
- Refurbished TokenManager WebApp in PR
Pilot:
- [MERGED in devel, tested in hackathon] Janusz [devel] Remote Pilot Logger to Tornado new JSON format. This is necessary before [integration] Remote Pilot Logger to Tornado could be run.
DIRACOS2:
- from previous meeting Python 3.11 seems to be fine
- Propose to update immediately after OpenSSL 3
- from previous meeting Opened an annoucement about OpenSSL 3
- Included a script to allow others to check if they have affected sites
- Please try it and report the results (and ticket sites if needed)
- Included a script to allow others to check if they have affected sites
- OpenSSL 3 (originally reported by Rucio): important performance degradation: https://github.com/DIRACGrid/DIRAC/discussions/6851#discussioncomment-5190607 also observed by DIRAC, ticket opened in https://its.cern.ch/jira/browse/DMC-1372
- OpenSSL 3.0.1 no ETA.
Documentation:
OAuth2:
- from previous meeting
- WLCG timeline document: https://zenodo.org/record/7014668#.YyLag9JBwQ9
- Meeting note from 8/12/2022: https://demo.hedgedoc.org/cWl2Y5MtTwSn4V5Uic_K5w
- Testing added in https://github.com/DIRACGrid/DIRAC/pull/6882 (unit, and integration tests with IaM)
- In the hackathon we used a flow where a personal token was used, Alexandre suggested to use another flow where the client (DIRAC server) is registered and can ask a Token directly.
- Alexandre Some sites might not accept client access tokens if they check group-based capabilities. Ongoing discussion about that here: https://github.com/WLCG-AuthZ-WG/common-jwt-profile/issues/24
- TLDR: 2 possibilities discussed
- Associate a client to a group (need development)
- Andrei we should think if on our side is needed or not. The mechanism for renewing the token credentials would need to be there anyway. It is anyway worth trying on our side.
- Enforce the fact that sites should support both scope-based (
compute.*scope) and group-based capabilities (wlcg.groups:/dirac/pilot)
- Associate a client to a group (need development)
- Andrei similar discussions are happening in EGI
tornado/HTTPs
- from previous meeting Issue https://github.com/DIRACGrid/DIRAC/issues/6495 keeps track of what can and what will not be moved to https
- no updates for today
- from previous meeting Federico we need to try and propose a coherent update from DIPS to HTTPS services, e.g. for Configuration Services URLs
- PRs from Andrei updating TornadoREST refused (problem with extensions), while there’s https://github.com/DIRACGrid/DIRAC/pull/6874 but maybe also not correct? see comment in https://github.com/DIRACGrid/DIRAC/issues/6877
- Andrei This is just a bug fixing PR, nothing more
- (too) Long discussion on the fact that the code behind Tornado (RPC and REST) is very fragile. The discussion is adjourned into a separate meeting/discussion that will take place probably in April.
- in the meantime, we can collect EGI needs/timeline (issue opened)
management
- from previous meeting 3 issues left, still valid
- NTR
diraccfg
- NTR
COMDIRAC
Daniela
- Issue for the improvements:
- merged [integration] fix: replaced print statments with gLogger
- Input requested on suggestion on how to address disentangling dinit (“session initialization”) from proxy/token generation: (https://github.com/DIRACGrid/DIRAC/discussions/6869)
DB12
Alexandre
- NTR
Rucio
- NTR
Tests
- Moving to HTTPs services as defaults in PR
- Temporarly disabled MultiVoFileCatalog service
- how to make it “equivalent”?
- Temporarly disabled MultiVoFileCatalog service
Release planning, tests and certification
Certification machines
- lbcertifdirac70 machine:
- Daniela: Enabled (and tested, at least with X509) WLCG VO for token testing. Needs update in puppet for /etc/grid-security/vomsdir and /etc/vomses, please.
- Christophe I’ll check and add it
- Daniela Still missing as of 14/12/22 A working config can be found at /cvmfs/grid.cern.ch/etc/grid-security/vomsdir/wlcg/ and /cvmfs/grid.cern.ch/etc/grid-security/vomses/wlcg-voms.cloud.cnaf.infn.it
- Chris https://gitlab.cern.ch/ai/it-puppet-module-voms/-/merge_requests/46 ([CLOSED] see comments)
- Daniela: Enabled (and tested, at least with X509) WLCG VO for token testing. Needs update in puppet for /etc/grid-security/vomsdir and /etc/vomses, please.
- lbcertifdiracoauth machine:
- NTR
Next hackathon(s)
- March 16th on lbcertifdirac70
AOB
Next hackathon: March 16th
Next BiLD: March 23rd
LHCbDIRAC
- v11.0: deploy board in https://trello.com/b/Ep0PAkbv/deploy-110
- Singularity CE everywhere?
- https://lhcb-auth.web.cern.ch/
- from previous meeting Alexandre An admin can allow a user (a client actually) to get access to the
compute.cancel/create/modifyscopes. Chris allowed Andrei and I to get access to the compute scopes, thus we should be able to use our clients to submit pilots.
- from previous meeting Alexandre An admin can allow a user (a client actually) to get access to the
There are minutes attached to this event.
Show them.