Focus on:  Hide Contributions
Login

BiLD-Dev

→ Europe/Zurich
Federico Stagni (Conseil Europeen Recherche Nucl. (CERN)-Unknown-Unknown)
Description

Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

Join Zoom Meeting
https://cern.zoom.us/j/91857549918?pwd=S2c1ZmFpYVNDOGk4YjMrZmhOeGROUT09

Meeting ID: 918 5754 9918
One tap mobile
+41315280988,,91857549918# Switzerland
+41432107042,,91857549918# Switzerland

Dial by your location
        +41 31 528 09 88 Switzerland
        +41 43 210 70 42 Switzerland
        +41 43 210 71 08 Switzerland
        +33 1 7037 2246 France
        +33 1 7037 9729 France
        +33 1 8699 5831 France
Meeting ID: 918 5754 9918
Find your local number: https://cern.zoom.us/u/aUn2ex94J

Join by SIP
91857549918@188.184.110.70
91857549918@188.184.89.188

Join by H.323
188.184.110.70
188.184.89.188
Meeting ID: 918 5754 9918
 

working notes
Hide
 
 

BiLD (Bi-weekly DIRAC Development meeting) – 08/10/2020

At CERN: A few of us, but each in our office ;-)
On Zoom: Federico, Alexandre, André, Anil Panta (Belle2), Christophe, Christopher, Daniela, Igor, Marko, Simon, Vladimir

Apologies: Andrei, Andrii

Follow-up from previous meeting

After OAuth pres (3 BiLD meetings ago):

After HTTps presentation (2 BiLD meeting ago)

The PR looks ready to be merged.

management devs

DIRAC v7r1 hackathons

  • Ran one hackathon 2 weeks ago on v7r1p12: https://trello.com/b/d3PaISN0/v7r1p12
    • MySQL 8 looks fine
    • few issues spotted for JSON serializations
    • could not test SingularityCE
    • Monitoring on ES partly broken
    • JobParameters on ES broken

DIRAC communities roundtable

GridPP:

  • Tested v7r1p12, not yet in production with it.

CLIC:

  • Still on v6r22
  • Getting all CE information from GLUE2 now (ported what’s been used in v7r1 via PR)

LHCb:

  • Still on v7r0. We’ll restart hackathons for v7r1 and move soon (prob in few weeks in production).

France Grilles:

  • NTR

EGI:

  • NTR

Belle2:

  • v6r22 is finally in production
  • Transition to v7r0 will not happen this year, will be done after migration to Rucio

Federico that would be an intersting exercise for other users aswell

Nica:

  • Unpacking tarfile of DIRACOS from CVMFS. Lots of small files cretead, and shared FS don’t like that. Would be better to run it unpacked already (basically to install DIRAC on CVMFS). A post on the google forum was created for that.

Federico this is on the to-do list

Juno, BES3:

  • NTR

Current situation

DIRAC

  • v6r22p34:
    • DMS
      • FIX: (#4748) FTS3Agent - Rotate FTS3Operations list to fetch all. Fixes #4727
  • v7r0p35:
    • Core
      • FIX: (#4751) Do not read suspensionReason from VOMS as it’s not used anywhere and needs special permissions to give this information
    • Framework
      • CHANGE: (#4753) NotificationService: if avoidSpam=True the email is now sent immediately and cached for up to one hour, instead of being cached and then sent after up to an hour.
    • Resources
      • FIX: (#4743) remove dots in xroot virtual username
  • v7r1p13:
    • JSON serialization fixes
  • v7r2-pre15:
    • Just incorporating all the above

WebApp:

  • NTR

Pilot3:

  • One draft PR open for tests of various types of CEs, was waiting for v7r1p12 for starting to use it.

Federico didn’t have time yet

DIRACOS:

VMDIRAC:

  • Tiny bug fix merged, added to v7r0p35.

Documentation:

  • Script change for python3 compatibility.

OAuth2:

  • The PR #4650 will be rebased on top of Tornado PR, which is not yet merged.
  • The PR will then target v7r3

tornado

Christophe Missing one package still, will do later.

other externals

  • NTR

management

  • Containers have been moved to the new GitHub container registry. Not used anywhere yet (not yet in the Actions).

Marko+Andre’ We are discussing moving the DIRAC Monitoring of Components that is used in ILCDIRAC to vanilla DIRAC. Issues with REA.

diraccfg

  • NTR

Release planning, tests and certification

Weekly development(s) focus

Python3, DIRACOS2 (Chris Burr) (presentation on the agenda)

Very interesting presentation attached to the agenda.

Python3:

  • In v7r2: functional client on python 3.
  • FTS3 python3 package still missing.

    Christophe I have been “promised” it will come this month

  • Main issues:
    • order in dicts
    • strings vs unicode vs bytes --> see the pres for extended explanations and examples
      • –> we’re NOT moving to unicode, we’ll just fix the strings that break. Case is CS, and file transfers.
  • 4 PRs, only the last one to be merged, the previous 3 are there for the comments.

DIRACOS2:

  • based on Conda (advantages in the pres)
  • there’s a prototype already
  • you can have it for running an existing DIRAC installation
  • uses proper GitHub releases
  • wrote in python3 only

Christophe I completely second everyting
Marko are you the maintainer of the middleware in Conda?
Chris mostly, but by now not only

We can have it in DIRAC v7r2 as an option.

Federico I would like to have a reviewer for DIRACOS2: try to identify with an outside eye what is not clear, or missing, considering also the integration and extensibility issues.

–> André has agreed to take this role as reviewer.
Christophe will also look into it, prob also Marko.

DIRAC: current PRs and tasks being worked on, or topics from Google forum

PRs:

  • v7r0:
    • NTR
  • v7r1:
    • Glue2 and BDII PR #4752 (André):
      • several changes, including moving to Glue2 by default
      • quite difficult, some data published makes no sense and is “filtered” out. See PR.
    • ES PR #4747 (Federico):
      • ES7 removed the “types”, which were used for store the Monitoring data, and these have been removed.
      • Trying to use the elasticsearch_dsl library for searching/query through data

      Federico I am not 100% sure that this is compatible with the current production setup, this needs a round of certification
      Federico The elasticsearch python client of version 6 is not guaranteed to be compatible with server version 7, anyway for the tests I used version 6.3.1 of the client with both servers version 6.6.0 and 7.9.1 and so far no problems. I didn’t test version 7 of the client.
      Marko if needed DIRACOS library can be overwritten

  • v7r2:
    • Out of the 4 python3 PRs, only the most recent one will stay (will be rebased after the HTTPs PR)

Issues:

Federico I re-arrenged the issues in milestones (not all of them)

Topics from Google forum:

  • NTR

AOB

CERN policies still forbid travel. Can’t know yet when there will be next DUW.
EGI workshop in November will go fully virtual. No DIRAC on the agenda.

Next BiLD in 2 weeks.

LHCbDIRAC

  • Let’s not use the “issues” on GitLab, please, to avoid having 2 places where issues are placed.
  • ES: before moving to ES7 we need to reduce the number or indices. Federico looking into it, some issues found. Still looking for the best way to do it.
  • We didn’t have a hackathon just after the LHCb computing workshop --> postponed, no need until we get a new DIRAC version with ES7 PR.
  • We’ll probably get an allocation on NERSC.
    • no Singularity but “Shifter” (in-house?) apparently compatible with docker, does it need to be prepared?

    Chris to run singularity we only need a few kernel features, I would be surprised if this is not possible.

 
There are minutes attached to this event. Show them.
Powered by Indico v3.3.6-pre