BiLD (Bi-weekly DIRAC Development meeting) – 22/10/2020

At CERN: A few of us, but each in our office ;-)
On Zoom: Federico, André, Andrii, Anil Panta (Belle2), Cedric, Christophe, Christopher, Daniela, Hideki, Marko, Simon, Vladimir, Xiaomei

Apologies: Alexandre, Andrei

Follow-up from previous meeting

management devs

Marko+AndréWe are discussing moving the DIRAC Monitoring of Components that is used in ILCDIRAC to vanilla DIRAC. Issues with REA.

DIRAC v7r1 hackathons

Ran one hackathon last week on v7r1p14: https://trello.com/b/R6DvV1O0/v7r1p14
- JSON serializations: not all fixed, PR open
- JobParameters on ES looks OK
- Some issues with SingularityCE
  - https://github.com/DIRACGrid/DIRAC/issues/4764 + PR https://github.com/DIRACGrid/DIRAC/pull/4773
- Monitoring on ES looks OK
  - backend ES6.6
- Bdii2CSAgent and Glue2: mostly fine, minor issues in PR

DIRAC communities roundtable

GridPP:

CLIC:

on v6r22 + DIRACOS
enabled “IPv6 ready” for some servers to let the CERN DNS announce the machines with IPv6(?) (disabled by default). All working fine.
- Couldn’t use one IPv6 only CE
Start looking at multi-core usage (mostly >2GB memory requirements)
- Just for ARC CE for the moment
- no PoolCE usage yet
Starting integrating FCC VO and usage.
- Maybe use that for a reason to take over the DIRAC/Workflow/Module developments
- Federicodeveloped long time ago by me, the Script and ModuleBase should be fine. The other 2 modules were never fully tested. There should be no real changes in the last few versions.

LHCb:

Still on v7r0. Hackathons for v7r1 tomorrow, then hopefully production.

France Grilles:

EGI:

Belle2:

Adopting Rucio: https://github.com/DIRACGrid/DIRAC/issues/4495
Cedricwe’ll try with DIRACOS, which we are using with v6r22 in integration server
CedricI will make a PR soon, and then I would like to present it at the BiLD meeting

JINR:

Just 1 VO using JINR service for now, running MC. 500K jobs, 120 TB generated.
NTR on the technical side.

Juno, BES3:

Mostly questions about redundancy

Current situation

DIRAC

v6r22p35:
- exclude fuse from df (watchdog checks)
v7r0p36:
- NTR
v7r1p14:
- BDII2CSAgent: several changes from PR #4752
- Added support for ES7 #4747
- remove backward compatibility for BaseSE defined in StorageElements section #4755
v7r2-pre16:
- Included tornado/HTTPs PR

WebApp:

New PR for PilotMonitor
- failures in this PR should be fixed separately
Use ngnix for data transfer: working on it
- logging to be verified (merge)

Pilot3:

Changed defaults for PoolCE #118
Added test for various CE types (Jenkins, “submitAndMatch” test)

DIRACOS:

xroot5? no news

DIRACOS2:

Presentation at the last BiLD
- André has agreed to take the role of reviewer.
Added repo to DIRACGrid https://github.com/DIRACGrid/DIRACOS2 but right now empty
- ChristopherWill be filled up once https://github.com/DIRACGrid/management/pull/23 is merged
PR to conda-forge about DIRAC in Jupyter notebooks? https://github.com/conda-forge/dirac-grid-feedstock/pull/32

VMDIRAC:

Documentation:

Alexandre pushed tutorial update for installing WMS (v7r0)

OAuth2:

The PR #4650 needs to be rebased and then target v7r3
- Will use HTTPS/tornado as base

tornado/HTTPs

Added
- https://github.com/DIRACGrid/tornado as forked from https://github.com/chaen/tornado
- https://github.com/DIRACGrid/tornado_m2crypto as forked from https://github.com/chaen/tornado_m2crypto
Main PR merged
So basically ready to test

FedericoWhich DIRACOS version should be used?
ChristopheFirst, let’s try a hackathon with the DIRACOS version specific for tornado (already done), to make sure that it doesn’t break anything. Then we can maybe make a new specific version

other externals

management

reminder Marko prepared a container to be used for CVMFS (auto)deploy: https://github.com/DIRACGrid/management/packages/342716 (also in https://hub.docker.com/repository/docker/diracgrid/dirac-cvmfs). For using it from GitHub Actions template in: https://github.com/DIRACGrid/management/pull/19 [Andrei + Federico]

reminder DIRACOS should be expanded on CVMFS [Andrei]

diraccfg

Release planning, tests and certification

Certification machine updated to be multi-VO (task: https://github.com/DIRACGrid/DIRAC/issues/4631)

Daniela

Added the gridpp VO

some issues with RSS, but RSS is not multi-VO
waiting for Janusz

TS should be fine as multi-VO

let’s just run all the tests and we take from there

multi-VO metadata: does it work? specific tests should be added

FedericoMonitoring (on ES) is probably also not multi-VO aware

Next hackathon October 29th ?
- First to verify that the multi-VO part works
  - or at least it doesn’t break anything
- v7r2: no python3 client yet, we will use the DIRACOS for tornado
- Let’s not change the board for now, it will be enough

reminder Jenkins: diracci6-02 “slave” machine will be removed
* this is used by several tests: https://jenkins-dirac.web.cern.ch/label/dirac_ci_6/
* Should move to use CC7 node https://jenkins-dirac.web.cern.ch/computer/diracci7-02/

v7r2:
1. Included HTTPs
2. Include Python 3 PRs (only the most recent one, as the others are included)
3. functional client on python 3 [in dev]
4. Option of DIRACOS2 [in dev]

Weekly development(s) focus

DIRAC: current PRs and tasks being worked on, or topics from Google forum

PRs No other PRs to discuss

Issues

v7r0 (https://github.com/DIRACGrid/DIRAC/milestone/18):
- Nothing in there
v7r1 (https://github.com/DIRACGrid/DIRAC/milestone/19):
- Not discussed
v7r2 (https://github.com/DIRACGrid/DIRAC/milestone/20):
- Not discussed
v7r3 (https://github.com/DIRACGrid/DIRAC/milestone/21):
- Not discussed

Topics from Google forum

NTR (forum quiet lately)

AOB

EGI webinar tomorrow by Andrei.

Next BiLD in 2 weeks.

LHCbDIRAC

ES: before moving to ES7 we need to reduce the number or indices.

Federicodidn’t have time to look into it this week, will try next week.

Hackathon tomorrow, based on a patch of v7r1 that Andrei should create this afternoon