Bi-weekly DIRAC Development meeting -- 19/02/2020

At CERN: Federico, Zoltan, Christophe, Alexandre, Christopher, Andre', Marko
On Vidyo: Janusz, Simon, Wojciech, Andrei, Ueda

Apologies: Daniela

Follow-up from previous meeting.

• payload environment isolation
  • Chris B: will provide LHCb examples, start a forum or issue thread  with them, Igor sent a few by email

DIRAC communities roundtable

• GridPP:
  • NTR
• CLIC:
  • ElasticSearch works fine, using it for WMS Monitoring (StatesAccountingAgent)
• LHCb:
  • Not too far from a production version using v7, but encountered some xrootd issues, partially addressed by a new version (but DIRACOS will be patched for that)
• France Grilles:
  • NTR
  • will invite the Algerian group in
• EGI:
  • NTR, but BDII2CSAgent should start looking for obsolete : pointed to https://github.com/DIRACGrid/DIRAC/issues/1034
• Belle2
  • still on v6r21. Pointed that it's getting outdated.

Current situation

DIRAC

• v6r22:
  • just some minor fixes in the last patch
• v7r0:
  • v7r0p13 is the last patch
    • SubProcess module: function for getting the child PIDs rewritten
    • integration tests can be extended in DIRAC extensions (in GitHub actions and GitLab CI)
  • tarball module is probably buggy, so the pilot wrapper sometimes don't untar the pilot3 tarball. One PR opened for sorting this issue.
• v7r1:
  • last hackathon one week ago on v7r1-pre16
    • One more will be necessary next Thursday, maybe the last
    • main issues discovered are in fact issues on v6r22 or v7r0

VM:

• PR for passing the number of processors to the pilots, and filling mode

WebApp:

• The JavaScript code has a format, mostly made by Zoltan. We should agree if this the "DIRAC standard" (JS doesn't have a standard format)

Pilot3:

• PR https://github.com/DIRACGrid/Pilot/pull/100 merged in master since a few days, all looks good
• PR open for logging (devel branch)
  • tornado part being adapted to using M2Crypto

DIRACOS:

• v1r8 is the last version, but never used in any release. Anyway, v1r9 will be created today or tomorrow.
• Oracle dependency removed (also from DIRAC code, now)
• Tried on CentOS8, broken
  • by that time DIRACOS v2, hopefully

Documentation

• NTR

OAuth2:

• will be discussed in details at the next BiLD meeting

Management:

• dirac-distribution container can handle DIRAC extensions now

diraccfg:

• cfg renamed diraccfg by Chris B
  • tested for several python  versions
  • pip installable
  • can do cfg -> json format

tornado and other externals

• Many JSON encoding issues still open, won't manage to go in v7r1
• OpenSSL version: we are stuck with version 1.0, as 1.1 removed the possibility to specify an environment variable for RFC proxies. A PR has been created but not yet merged.

Release planning, tests and certification

Release planning:

• v6r22 series
  • just a few fixes in PRs
• v7r0 series
  • main fix in PR: pilot wrapper will try to untar pilot files with tar
• v7r1 series
  • PRs with minor fixes, then we can probably release (next week: last? hackathon)
• v7r2?
  • OAuth big PR will go here?
• Integration
  • Some PRs ready to be merged, waiting for Andrei

Certification process: 

• Working PR for being able to run the integration tests (currently run regularly in Github actions) from DIRAC extensions
  • being tested in LCHb (in GitLab CI)
  • a proper doc+explanation will be needed, one tutorial will be done at the DIRAC workshop
• Github actions now used also by the WebApp

Weekly development(s) focus

• we can fully remove the support of Oracle from vanilla DIRAC

DIRAC: current PRs and tasks being worked on, or topics from Google forum

• v7r0:
  • Andrei: PR for escaping "," in CS, for specific use case of a user' DN
    • comment: still ugly, can we convince the CA to not issue such DN?
• v7r1:
  • Discussed above
• v7r2:
  • We will discuss its content at the next BiLD
  • new PR will be done from Janusz, will replace the previous one

AOB

Next week: DIRAC hackathon.

Next BiLD in 2 weeks.