BiLD (Bi-weekly DIRAC Development meeting) – 08/10/2020

At CERN: A few of us, but each in our office ;-)
On Zoom: Federico, Alexandre, André, Anil Panta (Belle2), Christophe, Christopher, Daniela, Igor, Marko, Simon, Vladimir

Apologies: Andrei, Andrii

Follow-up from previous meeting

After OAuth pres (3 BiLD meetings ago):

REST interface:
- Chris Burr opened https://github.com/DIRACGrid/DIRAC/issues/4703 (What RESTful endpoints are used for the new OAuth2 mechanism?)
- Basic OAuth library?
- JWT: suggestion in https://github.com/DIRACGrid/DIRAC/issues/4741 (Session authentication for REST endpoints)

After HTTps presentation (2 BiLD meeting ago)

The PR looks ready to be merged.

management devs

Marko prepared a container to be used for CVMFS (auto)deploy: https://github.com/DIRACGrid/management/packages/342716 (also in https://hub.docker.com/repository/docker/diracgrid/dirac-cvmfs)
For using it from GitHub Actions template in: https://github.com/DIRACGrid/management/pull/19 [Andrei + Federico]

–> Andrei will start using them soon

DIRAC v7r1 hackathons

Ran one hackathon 2 weeks ago on v7r1p12: https://trello.com/b/d3PaISN0/v7r1p12
- MySQL 8 looks fine
- few issues spotted for JSON serializations
- could not test SingularityCE
- Monitoring on ES partly broken
- JobParameters on ES broken

DIRAC communities roundtable

GridPP:

Tested v7r1p12, not yet in production with it.

CLIC:

Still on v6r22
Getting all CE information from GLUE2 now (ported what’s been used in v7r1 via PR)

LHCb:

Still on v7r0. We’ll restart hackathons for v7r1 and move soon (prob in few weeks in production).

France Grilles:

EGI:

Belle2:

v6r22 is finally in production
Transition to v7r0 will not happen this year, will be done after migration to Rucio

Federico that would be an intersting exercise for other users aswell

Nica:

Unpacking tarfile of DIRACOS from CVMFS. Lots of small files cretead, and shared FS don’t like that. Would be better to run it unpacked already (basically to install DIRAC on CVMFS). A post on the google forum was created for that.

Federico this is on the to-do list

Juno, BES3:

Current situation

DIRAC

v6r22p34:
- DMS
  - FIX: (#4748) FTS3Agent - Rotate FTS3Operations list to fetch all. Fixes #4727
v7r0p35:
- Core
  - FIX: (#4751) Do not read suspensionReason from VOMS as it’s not used anywhere and needs special permissions to give this information
- Framework
  - CHANGE: (#4753) NotificationService: if avoidSpam=True the email is now sent immediately and cached for up to one hour, instead of being cached and then sent after up to an hour.
- Resources
  - FIX: (#4743) remove dots in xroot virtual username
v7r1p13:
- JSON serialization fixes
v7r2-pre15:
- Just incorporating all the above

WebApp:

Pilot3:

One draft PR open for tests of various types of CEs, was waiting for v7r1p12 for starting to use it.

Federico didn’t have time yet

DIRACOS:

xroot5?

Christophe I didn’t have time for it yet
https://github.com/DIRACGrid/DIRACOS/issues/148 (firefox in dependency list)
- Created a PR for solving the issue
Marko came out of surprise

VMDIRAC:

Tiny bug fix merged, added to v7r0p35.

Documentation:

Script change for python3 compatibility.

OAuth2:

The PR #4650 will be rebased on top of Tornado PR, which is not yet merged.
The PR will then target v7r3

tornado

JSON serialization: some fixes in a PR after the previous hackathon
Added https://github.com/DIRACGrid/tornado as forked from https://github.com/chaen/tornado

Christophe Missing one package still, will do later.

other externals

management

Containers have been moved to the new GitHub container registry. Not used anywhere yet (not yet in the Actions).

Marko+Andre’ We are discussing moving the DIRAC Monitoring of Components that is used in ILCDIRAC to vanilla DIRAC. Issues with REA.

diraccfg

Release planning, tests and certification

Next hackathon October 15th ?
- still on v7r1 (last, promised!) if it includes #4747 (support for ES7)
Jenkins: diracci6-02 “slave” machine will be removed
- this is used by several tests: https://jenkins-dirac.web.cern.ch/label/dirac_ci_6/
- Should move to use CC7 node https://jenkins-dirac.web.cern.ch/computer/diracci7-02/
v7r2:
1. Include HTTPs
2. Include Python 3 PRs (only the most recent one, as the others are included)

Weekly development(s) focus

Python3, DIRACOS2 (Chris Burr) (presentation on the agenda)

Very interesting presentation attached to the agenda.

Python3:

In v7r2: functional client on python 3.
FTS3 python3 package still missing.

Christophe I have been “promised” it will come this month
Main issues:
- order in dicts
- strings vs unicode vs bytes --> see the pres for extended explanations and examples
  - –> we’re NOT moving to unicode, we’ll just fix the strings that break. Case is CS, and file transfers.
4 PRs, only the last one to be merged, the previous 3 are there for the comments.

DIRACOS2:

based on Conda (advantages in the pres)
there’s a prototype already
you can have it for running an existing DIRAC installation
uses proper GitHub releases
wrote in python3 only

Christophe I completely second everyting
Marko are you the maintainer of the middleware in Conda?
Chris mostly, but by now not only

We can have it in DIRAC v7r2 as an option.

Federico I would like to have a reviewer for DIRACOS2: try to identify with an outside eye what is not clear, or missing, considering also the integration and extensibility issues.

–> André has agreed to take this role as reviewer.
Christophe will also look into it, prob also Marko.

DIRAC: current PRs and tasks being worked on, or topics from Google forum

PRs:

v7r0:
- NTR
v7r1:
- Glue2 and BDII PR #4752 (André):
  - several changes, including moving to Glue2 by default
  - quite difficult, some data published makes no sense and is “filtered” out. See PR.
- ES PR #4747 (Federico):
  - ES7 removed the “types”, which were used for store the Monitoring data, and these have been removed.
  - Trying to use the elasticsearch_dsl library for searching/query through data
  Federico I am not 100% sure that this is compatible with the current production setup, this needs a round of certification
  Federico The elasticsearch python client of version 6 is not guaranteed to be compatible with server version 7, anyway for the tests I used version 6.3.1 of the client with both servers version 6.6.0 and 7.9.1 and so far no problems. I didn’t test version 7 of the client.
  Marko if needed DIRACOS library can be overwritten
v7r2:
- Out of the 4 python3 PRs, only the most recent one will stay (will be rebased after the HTTPs PR)

Issues:

Federico I re-arrenged the issues in milestones (not all of them)

v7r0 (https://github.com/DIRACGrid/DIRAC/milestone/18):
- Nothing in there
v7r1 (https://github.com/DIRACGrid/DIRAC/milestone/19):
- 10 issues, some of them will be closed by open PRs
- #3260 (Make RSS vo-aware) can’t be closed yet
v7r2 (https://github.com/DIRACGrid/DIRAC/milestone/20):
- Not much to report
v7r3 (https://github.com/DIRACGrid/DIRAC/milestone/21):
- Added all OAuth issues

Topics from Google forum:

AOB

CERN policies still forbid travel. Can’t know yet when there will be next DUW.
EGI workshop in November will go fully virtual. No DIRAC on the agenda.

Next BiLD in 2 weeks.

LHCbDIRAC

Let’s not use the “issues” on GitLab, please, to avoid having 2 places where issues are placed.
ES: before moving to ES7 we need to reduce the number or indices. Federico looking into it, some issues found. Still looking for the best way to do it.
We didn’t have a hackathon just after the LHCb computing workshop --> postponed, no need until we get a new DIRAC version with ES7 PR.
We’ll probably get an allocation on NERSC.
- no Singularity but “Shifter” (in-house?) apparently compatible with docker, does it need to be prepared?
Chris to run singularity we only need a few kernel features, I would be surprised if this is not possible.