Bi-weekly DIRAC Development meeting -- 05/03/2020

At CERN: Federico, Zoltan, Christophe, Alexandre, Christopher, Andre', Marko
On Vidyo: Simon, Janusz, Andrei, Andrii, Hideki

Apologies: Ueda

Follow-up from previous meeting

• payload environment isolation
  • See https://github.com/DIRACGrid/DIRAC/issues/4480 (will be a topic for a next BiLD meeting)

DIRAC communities roundtable

• GridPP:
  • NTR
• CLIC:
  • moving to MySQL 5.7
• LHCb:
  • Moved all DIRAC boxes (only 1 left) to CC7, v6r22 + LHCbDIRACOS (based on DIRAC v1r5), and using a new puppet module
• France Grilles:
  • NTR
• EGI:
  • NTR
• Belle2
  • Moving to v6r22

Current situation

DIRAC

• v6r22:
  • fix to avoid trying to compile the web portal when installing DIRAC from the web interface
• v7r0:
  • v7r0p15 is the last one
    • New StorageOccupancy accounting backported from LHCbDIRAC (web part is still missing)
    • fully removed Oracle mentions

VM:

• NTR

WebApp:

• v4r0 is the series for DIRAC versions v6r22, v7r0, v7r1
• v4r1 will be used for DIRAC v7r2
  • several changes for adapting to OAuth2 (see slides in presentation)
  • web.cfg will be made optional, by default use dirac.cfg for all its content

Pilot3:

• 2 PRs for master to evaluate (Andrei)
• should add checksum for pilot.tar when it's uploaded (Federico)

DIRACOS:

• NTR

Documentation

• Missing readthedocs for VMDIRAC (needed? merge in DIRAC?)

OAuth2:

• See pres and weekly focus

tornado and other externals

• M2Crypto and VOMS issues to be sorted out before v7r1 is out

management

• PR for using dirac-distribution container for creating DIRAC extensions releases, now fully working also for DIRAC extensions. Publushed on docker hub and GitHub docker registry, used in LHCb gitlab-CI for creating release tarballs

diraccfg

• published on PyPi and conda forge, used in LHCb gitlab-CI
• python 2 and 3
• can think of being THE only way to call DIRAC CFG

Release planning, tests and certification

Release planning:

• v6r22 series
  • NTR
• v7r0 series
  • only fixes in pipeline
• v7r1 series
  • last hackathon on Tuesday 3/3. 2 issues:
    • PilotWrapper could not find the pilot files, addressed in #4475 (for v7r0)
    • M2Crypto and VOMS extensions: not seen, not working even in v7r0, need to understand why
  • One more will be necessary next Thursday, hopefully the last
  • PRs currently marked for v7r1 can be merged (after review) but no more things than this.
• v7r2 series
  • After discussion and presentation (see below) PRs #4405 and #4421 will be rebased on integration, evaluated and eventually merged
  • pyGSI will be removed, only M2Crypto will stay (#4481)
  • the first steps for JSON will be completed (issues to re-assign the milestones)
  • We will need to decide how to run the DIRAC certification (with X509 proxies?)

Certification process: 

• Working PR for being able to run the integration tests (currently run regularly in Github actions) from DIRAC extensions
  • being tested in LCHb (in GitLab CI)
  • a proper doc+explanation will be needed, one tutorial will be done at the DIRAC workshop
• Github actions now used also by the WebApp

Weekly development(s) focus

• Presentation from Andrii about OAuth (see link in agenda). In general well received, originated several questions/comments. A few issues created:
  • https://github.com/DIRACGrid/DIRAC/issues?q=is%3Aissue+is%3Aopen+label%3AOAuth2

DIRAC: current PRs and tasks being worked on, or topics from Google forum

• v7r0:
  • NTR
• v7r1:
  • Marked PRs that can go in (after review)
• v7r2:
  • 2 PRs (OAuth) to be rebased

#4463 (RSS DB handlers loaded dynamically) can be transformed into a generic mechanism for loading DB classes (https://github.com/DIRACGrid/DIRAC/issues/3848)

AOB

Next week: DIRAC hackathon.

Next BiLD in 2 weeks.

CoronaVirus update and DUW: travels to Japan are for the moment under scrutiny by CERN. DUW confirmed for the moment, but expecting updates.

=================

LHCbDIRAC

LHCbDIRAC hackathon on Tuesday at 14:00

SpaceToken -> StorageOccupancy accounting: need plan