BiLD (Bi-weekly DIRAC Development meeting) – 11/03/2021

At CERN: Still, nobody :/
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Cedric, Christophe, Christopher, Daniela, Hideki, Janusz, Marko, Simon, Vladimir, Xiaomei
Apologies:

Follow-up from previous meeting

• Ran hackathon last week on v7r2-pre37:

  • trello board here
  • VMDIRAC not yet fully installed nor tried
  • ran with CS on https, minor fixes had to be done (mostly doc). Also to puppet module
    • v7r1 clients are not compatible with installations where the CS runs on https, but it was expected
  • Some more fixes had to be done

• PRs merging strategy?

  •  André No time to investigate yet.
  •  Andrei I made some remarks, I’d like an update. If we are switching to this mode, we never come back.
    •  Christopher maybe not too bad. We can go back to old mode if we don’t like it.
    •  Andrei we can’t anyway keep both in parallel, it’s one or the other.
  • “Random” developers need special attention.

DIRAC communities roundtable

GridPP:

 Daniela

• NTR

CLIC:

 André

• Scheduling update to v7r0

LHCb:

 Federico+Chris+Chris+Marko

• We ran yesterday a LHCbDIRAC hackathon with v7r2-pre38. Quite OK, a few PRs with minor fixes created.
  • Managed to run a few services on a kubernetes cluster.
• Using CTA at CERN (replacement of Castor)
  • CTA will also be used at RAL

EGI:

 Andrei

• Still preparing the migration.

France Grilles:

 Andrei

• Started running some SiteDirectors with Pilot3

Belle2

 Hideki

• NTR

BES3/Juno:

 Xiaomei

• Still preparing update to v7r0.

DIRAC releases

• v7r1p31 + v7r1p32:
  • CHANGE: (#5001) JobAgent: if we don’t match a job, independently from the reason, we wait a bit longer before trying again
  • CHANGE: (#5001) JobAgent: moved check of availability of CE before doing anything else
  • NEW: (#4998) JobWrapper - find executables from PATH if not using an absolute filename
• v7r2-pre37 + v7r2-pre38:
  • CHANGE: (#4997) Use prompt-toolkit for password input when generating proxies
  • CHANGE: (#4994) dirac-install.py should now be taken from management
  • CHANGE: (#5010) Use ConfigurationClient in CSCLI and CSShellCLI
  • CHANGE: (#5026) Remove DIRAC.Core.Utilities.ExitCallback
  • CHANGE: (#5019) Remove IRODSStorageElementHandler

DIRAC projects

DIRAC:

By milestone:

• v7r1:
  • almost only documentation is still there to be done
• v7r2:
  • non-urgent issues
• v7r3:
  • Make RSS VO-aware: PR created against v7r1, will need to be ported to v7r3
    • comments in there…
  • OAuth2 will be the main feature (see below)

WebApp:

• nginx configuration here: https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#web-configuration
  • what to do? Just re-phrase it, make sure it’s clear that it is optional.
• Merged a few PRs for v4r2 (SiteSummary page)
• Discussion in https://github.com/DIRACGrid/WebAppDIRAC/issues/427 for Belle2 WebApp application is ongoing. Should not be difficult to fix.

Pilot3:

• Updates for Jenkins (CC7 default, and CS using http) and dirac-install following the version in DIRACGrid/management

DIRACOS:

• NTR

DIRACOS2:

• fts-rest on py3?
  • not yet (soon?)
• DIRACOS2 builds for ARM and Power8 (compatible with Power9?) are there. They have been also tested.
• Question for rucio client… (issue)? (Cedric to answer)
• Review from Andre’?
  • consider it “done”.
• first non-pre-release today.

VMDIRAC:

• NTR
• Almost got it working in Certif server.

Documentation:

• PR [v7r2] Use a conda environment when building the docs

  • biggest problem is in a feature of rtd. Migrating from pip to conda on the main repo should be enough to fix it. We should just create v7r2 and change master to it.

•  Federico I have installed DIRAC for local devs using cond as explained in https://dirac.readthedocs.io/en/integration/DeveloperGuide/DevelopmentEnvironment/DeveloperInstallation/editingCode.html

  • everything was OK (minor: conda/mamba are not in PATH by default)

OAuth2:

• The existing PR will be rebased ASAP (new PR actually will be done.)
• News from issue WLCG and Token transition – reminders and requirements ?
  • Trying to follow all discussions, not yet conclusive
• Is it time for a general discussion? Maybe a brainstorming is needed.

tornado/HTTPs

• NTR (just small PR).

management

• dirac-install and globalDefaults uploaded to CVMFS
  • merged
• 2 tasks left for deploying on CVMFS
  •  Marko should come from DIRAC main repo.

diraccfg

• NTR

other externals, including Rucio

• Rucio File Catalog implementation from Belle
  • Cedric not connected today…
  • Needs (urgently?) a working implementation
  • GridPP has not yet tested it, but some comments went in the PR, especially on the point of configuration.
    • the GridPP Rucio server is multiVO, and not fully working yet.

Release planning, tests and certification

• Certification machine

  • VMDIRAC
    • puppet updated for VMDIRAC. Please don’t update etc/dirac.cfg or bashrc by hand. puppet runs every 30 mins.
    • nearly there
  • Reminder: task for multi-vo
    •  Daniela I will write some multi-vo tests

• Next hackathon

  • next week (18th March)
    • first hackathon for v7r3-pre1

Weekly development(s) focus

upgrading to v7r2

• The structure with src/PROJECTNAME/code... should be adopted
• A setup.py should be written
• For Vanilla projects this affects COMDIRAC
  • VMDIRAC and WebAppDIRAC are only server, so no need to update immediately.
• in LHCb we:
  • created a setup.py, and we can do pip install LHCbDIRAC
  • updated quite some code for python3

AOB and topics from Google forum

Next hackathon on March 18th: https://indico.cern.ch/event/1015938/
Next BiLD on March 25th: https://indico.cern.ch/event/1015936/

DUW10 online: https://indico.cern.ch/e/DUW10 10-11th May 2021:

• announced “officially”, first registrants (not you!)
• Sent a mail to presenters…

LHCbDIRAC

• v10r2 planning in https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/wikis/Releases-planning#based-on-dirac-v7r2
• https://codimd.web.cern.ch/2es8HPUcSmS42jrTxhX3JQ?both
  • MySQL 8: ticket open for upgrade. Now we can upgrade
  • ES7:
    • no centralized logging
    • To be done
• https://gitlab.cern.ch/-/snippets/1547