BiLD (Bi-weekly DIRAC Development meeting) – 01/12/2022

On Zoom: Federico, André, Andrei, Alexandre, Christophe, Christopher, Daniela, Hideki, Simon, Janusz, Xiaomei, Vladimir
Apologies:

Follow-up from previous meetings

• hackathon November 24th, on 8.1.0a5:
  • Few issues found and solved, not specifically related to this release (e.g. for AREX CE). In general rather smooth.
• Last BiLD 2 weeks ago
  • all updates below

DIRAC communities roundtable

LHCb:

Federico+Alexandre+Christophe+Christopher+Alexey

• Running in production v7.3.34 (latest v7r3 patch)
• LHCbDIRAC hackathon done on top of 8.0.5, no issues found in DIRAC
  • not conclusive though
• applied this change in DFC production and
  • Christophe I would be surprised if anyone else use this
  • Federico should be anyway added to wiki (v8 or no-one will see it)

EGI

Andrei

• struggling to set up a cloud CE (no possibility to specify a project name?)

ILC/CLIC/FCC/Calice:

André

• (Apologies have to leave at 10:30)
• Still working on server updates to 7.3

Belle2

Hideki

• v7r2 in production
• User certificate renewal
  • answered, fine with it
• What is the necessity to migrate from ARC to ARC6?
  • Alexandre maybe you don’t need it, if AREX works fine. ARC6 is an intermediate solution.
  • Daniela WLCG VOs are pushing to move to AREX.
• from previous meeting Deployed ElasticSearch, issues with ComponentMonitoring
  • Federico ComponentMonitoring on ES never fully worked. What’s working:
    • v7r2: WMSHistory
    • v7r3: WMSHistory and JobParameters
    • v8.0: “everything works” (it is documented), and ComponentMonitoring is spilt between Agent and Service Monitoring.

Juno

Xiaomei

• Condor “WholeNode” parameter not considered?
  • Federico please open Discussion

GridPP:

Daniela+Simon+Janusz

• No changes in Production: v7.3.26
• Pre-prod: v7.3.34

Topics from GitHub/Discussions or Google forum

only un-answered topics below:

Error in loading WebApp (5.0.1)

• Should always do pip install WebAppDIRAC[server] ?

MaxJobsInFillMode replacements

• this requires a small fix in Pilot

ERROR: DataString is getting too long

• DIRAC buffering. The coded “solution” was not accepted, a better one should be coded (suggestion is there)

Personal certificates with “sn” and “gn” fields not supported?

• from previous meeting Maybe update to VOMS2CSAgent?

Pilots/ARC “proxy not found”

• from previous meeting Andrei we should simply put by the default the bundling of the proxy in the pilots

…several others in https://github.com/DIRACGrid/DIRAC/discussions?discussions_q=is%3Aunanswered

from the forum: https://groups.google.com/g/diracgrid-forum/c/l1n_M4ZMDnM/m/K1x0ohbGAwAJ?pli=1

• Daniela we used/tested this

DIRAC releases

v7r3

• v7.3.34
  • Resources
    • FIX: (#6584) fix delegation process in the AREXCE
    • FIX: (#6539) multi-node allocations with SLURM
    • FIX: (#6535) AREXCE._getDelegation() returns an error if the process cannot be completed

v8r0

• v8.0.4
  • test
    • NEW: (#6529) added system/rms_script.sh test for basic RMS test
• v8.0.5
  • ConfigurationSystem
    • NEW: (#6555) BDII2CSAgent: allow using AREX or ARC6 Computing Elements instead of ARC, fixes #6541

v8r1

• v8.1.0a5
  • WorkloadManagementSystem
    • CHANGE: (#6568) improve killing of process with psutil
    • CHANGE: (#6556) completely removed old PilotsLogging machinery
  • DataManagementSystem
    • NEW: (#6567) TornadoS3GatewayHandler service
  • ProductionSystem
    • NEW: (#6567) TornadoProductionManagerHandler service
  • StorageManagementSystem
    • NEW: (#6567) TornadoStorageManagerHandler service
  • MonitoringSystem
    • CHANGE: (#6563) remove setup
  • Core
    • CHANGE: (#6561) Swapped the default of EnableSecurityLogging flag
      • NOTE this effectively makes Elastic/OpenSearch obligatory.
    • CHANGE: (#6466) removed DIRAC_USE_JSON_DECODE, flipped DIRAC_USE_JSON_ENCODE

DIRAC projects

DIRAC:

Issues by milestone:

• v7r3:
  • Still 7 open issues
  • Add possibility to specify a DIRACOS2 version in SystemAdministrator and Pilot.
    • often requested
• v8.0:
  • 19 open issues
  • New optimizer tests are flaky
    • almost fully solved
  • Requirements and tasks for token-based pilot submissions
    • PR created as a draft: https://github.com/DIRACGrid/DIRAC/pull/6580
    • oidc-agent packaged with conda?
      • Christopher the packaging on conda was “improved”, so giving at a try in DIRACOS, but still this is “not something that we should be always have”
  • v8.0.1 WebApp ssl issue
    • Merge connected PR https://github.com/DIRACGrid/WebAppDIRAC/pull/699 and close this?
• v8.1:
  • 18 open issues
  • Keep or remove BundleDelivery machinery?
  • Provide minimal support for non-DIRAC DIRAC client
    • Discussion quick revival: https://github.com/DIRACGrid/DIRAC/discussions/6121#discussioncomment-4208019

Other issues:

• Drop DIRAC.Core.Utilities.NTP
  • I guess OK, from v8r0
• Automatically commit pre-commit fixes?
  • Christophe I don’t think it’s a good idea

PRs discussed:

• https://github.com/DIRACGrid/DIRAC/pull/6373
  • breaks WebApp completely
• https://github.com/DIRACGrid/DIRAC/pull/6405
  • Still AREX to be added, others were tried all by Janusz, issue with ARC discovered
• [8.0]Pilot submission with tokens
  • Issue with OpenStack driver, “project” can’t be specified and in a multi-VO environment this is necessary
  • Proxies will be bundled.
  • Christophe you are using the TokenManager, but I think we should NOT have a TokenDB. This is not a discussion for today.
    • the backend DB can go away, the service can/should stay
• [8.1] feat: (Accounting) remove setup
  • Federico question on backward compatibility
    • answer: go for the renaming of the tables
• [8.1] WMS: Working with usernames instead of OwnerDNs
  • Addressing 2 issues at once.
• [8.0] Backporting generic TS scripts from LHCbDIRAC
  • Anyone against this?
• [integration] Remote Pilot Logger to Tornado
  • Passing the integration tests! (but few conflicts)
  • The new JSON format is being coded, and this means also re-thinking a bit where the options are.

WebApp:

• Closed few old issues

Pilot:

• Closed few old issues

DIRACOS2:

• 5 PRs open
• Chris B Fairly sure I have a fix for htcondor 9.0.x with scitokens
  • Will confirm with Andrei before applying

Documentation:

• NTR

OAuth2:

• from previous meeting WLCG timeline document: https://zenodo.org/record/7014668#.YyLag9JBwQ9
• Federico will call a doodle for a meeting on how to continue here

tornado/HTTPs

• from previous meeting Federico for “full production” setup we are not there yet
  • is nginx “mandatory”?
  • can we run more instances?
  • Andrei there’s no “upload” solution for DIRAC SE

management

• from previous meeting 3 issues left, still valid
  • Andrei Updated the script, should be uploaded here

diraccfg

• from previous meeting Christopher Do we want to make a release that drops support for py2?
  • yes

COMDIRAC

• NTR

DB12

• NTR

Rucio

• NTR

Tests

• Federico fixed the submission to SSHBatchCE (code and conf in lbcertifdirac70)

Release planning, tests and certification

Certification machines

• lbcertifdirac70 machine:
  • maybe add an SSHCE site in DIRAC certification submitting to https://batchdocs.web.cern.ch/linuxhpc/index.html cluster ?
    • Alexandre got an access, will be done through diracgri account
  • Computing Element of each supported type should be configured (HTCondorCE, 2 ARCS (Alexandre (maybe use RALPP and Glasgow ? I can talk to them if there are issues --Daniela)), 1 Cloud (Daniela - this is already part of the tests), SSH). Consider OAuth tests as well - what is needed as far as certification servers configuration is concerned.
  • Daniela: Enabled (and tested, at least with X509) WLCG VO for token testing. Needs update in puppet for /etc/grid-security/vomsdir and /etc/vomses, please.
    • Christophe I’ll check and add it
• lbcertifdiracoauth machine:
  • NTR

Next hackathon(s)

• Next week, on lbcertifidirac70 and with v8.1.0aX

AOB

CHEP2023

• abstracts submitted by Federico (generic), Daniela+Simon+Janusz, Alexandre (HPC, LHCb-ish), Xiaomi one for Juno, Andrei one for CTADIRAC, Christopher one LHCb-ish for TS

[ISGC] Andrei abstract for AuthN/Z

Confirmed next workshop as “DIRAC&Rucio workshop 2023”, KEK, 16-20 October 2023

• added to HSF calendar
• no indico yet

Next hackathon: December 8th
Next BiLD: December 15th

LHCbDIRAC

• v10r4: deploy board in https://trello.com/b/kzUKdMts/deploy-v10r3
  • added note for HTTPs CS master deploy
  • DFC update scheduled for next Wed
    • for how long? message to users?
• https://lhcb-auth.web.cern.ch/
  • from previous meeting Andrei this is not properly configured yet. Not usable for Pilot submission yet (compute scopes not added).
• Federico For the certification setup: I have set again dryRun=False for Bdii2CSAgent
• LHCbDIRAC hackathon
  • when? Next Tuesday OK?
• Move from Jira to GitLab for issues: CERN won’t renew the contract with Atlassian/Jira at the current conditions, and if somehow Jira will stay the feeling is that will be only temporary.
  • Anyone would be against to move to using GitLab issues instead?
  • For the existing ones: review, close if you can, or keep it to be moved for later.
  • Jira history: not sure if it will be possible to keep it.