BiLD (Bi-weekly DIRAC Development meeting) – 13/08/2020

At CERN: Nobody, of course!
On Zoom: Federico, Alexandre, Andrii, Aymane, Christophe, Christopher, Daniela, Simon, Vladimir

Apologies: André, Marko

Follow-up from previous meeting

• Error in Pilot related to locale (https://github.com/DIRACGrid/Pilot/issues/112). Deployed 3 weeks ago: can it be confirmed that fixes the original issue by Nica?
  • Nobody to report

After OAuth pres:

• REST interface: RFC for discussion? (see today)
  • Not yet started it [Andrei]
• Basic OAuth library:
  • Chris Burr started drafting something. "More or less I have a proposal, hope to open an issue before holidays
• RFC for semantic of components?
  • not started yet [Andrei/Federico/Andrii]

DIRAC communities roundtable

GridPP:

• NTR

CLIC:

• Nobody there to report

LHCb:

• Ran 2 more certification hackathons using v7r1. Looks OK, just not much time now for production (which will be in September)

France Grilles:

• NTR

EGI:

• NTR

Belle2:

• Nobody there to report

Nica:

• Nobody there to report

Juno:

• Nobody there to report

Current situation

DIRAC

• v6r22:
  • v6r22p33
    • M2Crypto/pyGSI related fixes
• v7r0:
  • v7r0p32
    • JobAgent - define DIRAC_JOB_PROCESSORS environment to number of processors allocated for the user job
    • WebAppDIRAC v4r0p30
• v7r1:
  • v7r1p10
    • NTR
• v7r2:
  • NTR

WebApp:

• Do update to latest version!

Pilot3:

• Merged one PR for better monitoring of scripts
• One draft PR open for tests of various types of CEs

DIRACOS:

• xroot5?
  • ARC6 will need to be fixed first (not for us)
• Marko created one more nightly for HTTPs
  • https://github.com/DIRACGrid/DIRACOS/pull/143 for patched tornado (see below)
• less command fixed (not released yet)

VM:

• NTR

Documentation:

• NTR

OAuth2:

• Discussed last week.
• Documention is in progress. Only done for now for REST.

tornado and other externals

• tornado: today’s topic
• M2Crypto: …end of line for the fixes?
  • still one open PR on M2Crypto side

management

• Marko prepared a container to be used for CVMFS (auto)deploy: https://github.com/DIRACGrid/management/packages/342716 (also in https://hub.docker.com/repository/docker/diracgrid/dirac-cvmfs)
• For using it from GitHub Actions template in: https://github.com/DIRACGrid/management/pull/19 [Andrei + Federico]

diraccfg

• NTR

Release planning, tests and certification

• Big PRs for v7r2 are not yet reviewed. Will need to wait for them before really starting certifications. Given the holiday period, we are talking about September.

Weekly development(s) focus

• HTTPs (presentation on the agenda)
• Previous BiLD/presentations/discussions/links also on the agenda

Summary:

• The advantages are clear (use of Standard library, interrogate services without installing DIRAC).
• Services would need to be migrated by inheriting from the new TornadoService.
• Clients don’t need to be migrated.
• CS protocol flag needs to change to HTTPs
  Dev:
• One route per service, then POST verb with remote method name and parameters
• One tornado process runs all the services
  • Exception for master CS
    Issues:
• Input/output must be JSON-compatible (some tasks still open)
• FileTranfer not fully implemeted
  • Tornado is not for that. Need to use something else
• Threads are not allowed
  • Need to fit into ASync style. Callbacks “poor man” solution.
• Global variables can’t stay (gLogger, gConfig)
• DIRACOS should include:
  • patched Tornado 5.1 (fork will come in GitHub/DIRACGrid)
  • we need to rely on M2Crypto, for now, and there’s also a patch for it

Discussion:

• Tornado is maybe not what we want. But it’s right now the way to have https protocol. Maybe it won’t stay in the future, maybe it could be replaced one day, but in the meantime DIRAC can start speaking HTTPS

Conclusion:

• Merging this PR should be OK. Running https services need to be manual.
• OAuth PR: what’s done here seems to fit there too.

DIRAC: current PRs and tasks being worked on, or topics from Google forum

PRs:

• v7r0:
  • Few PRs ready to be merged, non-problematic
• v7r1:
  • One large-ish PR with simple fixes for scripts
• v7r2:
  • NTR

On issues:

• NTR

Topics from Google forum:

• Little activity lately

AOB

CERN policies still forbid travel. Can’t know yet when there will be next DUW.
EGI workshop in November will go fully virtual. 1 day DIRAC session? probably - AT in contact.

Next BiLD in 5 weeks.

LHCbDIRAC

• Oracle instantclient to be updated in production too.