Focus on:  Hide Contributions
Login

BiLD-Dev

→ Europe/Zurich
Federico Stagni (Conseil Europeen Recherche Nucl. (CERN)-Unknown-Unknown)
Description

Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

Join Zoom Meeting
https://cern.zoom.us/j/91857549918?pwd=S2c1ZmFpYVNDOGk4YjMrZmhOeGROUT09

Meeting ID: 918 5754 9918
One tap mobile
+41315280988,,91857549918# Switzerland
+41432107042,,91857549918# Switzerland

Dial by your location
        +41 31 528 09 88 Switzerland
        +41 43 210 70 42 Switzerland
        +41 43 210 71 08 Switzerland
        +33 1 7037 2246 France
        +33 1 7037 9729 France
        +33 1 8699 5831 France
Meeting ID: 918 5754 9918
Find your local number: https://cern.zoom.us/u/aUn2ex94J

Join by SIP
91857549918@188.184.110.70
91857549918@188.184.89.188

Join by H.323
188.184.110.70
188.184.89.188
Meeting ID: 918 5754 9918
 

working notes
Hide
 
 

BiLD (Bi-weekly DIRAC Development meeting) – 17/09/2020

At CERN: A few of us, but each in our office ;-)
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Cedric, Christophe, Christopher, Dan Whitehouse (GridPP), Daniela, Marko, Simon, Vladimir, Janusz, Xiaomei

Apologies: Igor

Follow-up from previous meeting

After OAuth pres (2 BiLD meetings ago - 7 weeks ago):

  • REST interface: RFC for discussion? (see today)
  • Basic OAuth library:
    • Chris Burr started drafting something. “More or less I have a proposal, hope to open an issue before holidays”
  • JWT: suggestion in https://github.com/DIRACGrid/DIRAC/issues/4741 (Session authentication for REST endpoints)
  • RFC for semantic of components?
    • not started yet [Andrei/Federico/Andrii]

After HTTps presentation (1 BiLD meeting ago - 5 weeks ago)

A few updates, but the PR looks ready to be merged.

management devs

DIRAC communities roundtable

GridPP:

  • Nothing new.
  • Was waiting for the singularity fixes (merged in in v7r1p12) before putting v7r1 in production.

CLIC:

  • Still on v6r22. Waiting for last moves out of LFC
  • Also have to move from Castor to CTA, in contact with Chris and CERN IT, will wait for LHCb to move

LHCb:

  • Still on v7r0. We’ll restart hackathons for v7r1 and move soon (prob in 3 weeks in production).
  • Castor to CTA: should not need any DIRAC change
  • Except master CS and WebApp we are running with M2Crypto everywhere. Performances to be seen in the long run.

France Grilles:

  • Running v7r0
  • Started looking at ElasticSearch
    • Federico: which version? DIRAC can interact with ES6, not ES7.

EGI:

  • Still struggling with AuthN, CheckIN in conjuction with other community management tools (perun)
  • FileCatalog management for one VO in progress

Belle2:

  • NTR

Nica:

  • Nobody there to report

Juno, BES3:

  • MC transformations running, using MCExtensionAgent for automated extension

Current situation

DIRAC

  • v6r22:
    • No new releases
  • v7r0:
  • v7r1:
    • v7r1p12
      • Fixes for JSON serialization
      • SingularityCE:
        • Use singularity userns flag if possible and allow CVMFS fallback
        • added possibility to run without re-installing DIRAC inside the container
      • NEW: RSS DB schema change to include a VO column (https://github.com/DIRACGrid/DIRAC/pull/4720)
        • Need to add it to the release wiki
        • Andrei: we need to do it “by hand” right now, we can code a simple tool for doing it
          • Chris: we can use external tools for that
          • Andrei: for very simple cases we don’t external tools
  • v7r2:
    • v7r2-pre14 created, just including the above

WebApp:

  • NTR

Pilot3:

  • One draft PR open for tests of various types of CEs, was waiting for v7r1p12 for starting to use it.

DIRACOS:

VMDIRAC:

  • NTR

Documentation:

  • NTR

OAuth2:

  • The PR 4650 will be rebased on top of Tornado PR.
  • The PR will then target v7r3

tornado

  • Tornado PR: ready to be integrated, just one last review needed [Federico]

  • JSON serialization: only 2 tasks left open, rest is in v7r1

  • We should move tornado fork in DIRACGRid repo [Christophe]

other externals

  • M2Crypto:
    • still one open PR on M2Crypto side – but anyway it seems like we don’t need it

management

  • We should to move the containers to new GitHub container registry [Marko]

diraccfg

  • One PR for exposing version parsing (already merged)

Release planning, tests and certification

Federico proposed:

  • Go back to v7r1 for certifying:
    1. MySQL 8 (the DIRAC certification server was updated already, this should not be an issue)
    2. The following flags (added in extra_bashrc) 
      export DIRAC_DEBUG_DENCODE_CALLSTACK=1
export DIRAC_USE_JSON_DECODE=Yes
    3. SingularityCE (~easy to test)
    4. JobParameters on ES (also controlled by a flag, right now turned on)
    5. ElasticSearch 7 (dev to be done, probably)
    6. Move to MultiVO installation, basically what is in https://github.com/DIRACGrid/DIRAC/issues/4631 (this needs work - GridPP?)

Points from 1 to 4 can be tested at the next hackathon that will be next week. Point 5 needs some work. Once done we can do point 6.

  • v7r2:
    1. Include HTTPs
    2. Include Python 3 PRs (only the most recent one, as the others are included)

Start certification with the above 2 once v7r1 is done

Weekly development(s) focus

  • Vladimir: we should suppress Glue1 from Bdii2CSAgent, because ARC CEs are just buggy, and they publish wrong info to Glue1 (see note in https://codimd.web.cern.ch/6r1SApCfTu2DlPYiZGuu_g#)
    • Proposal: v7r1 send notification that it Glue1 will be removed, then later really remove it [Federico]
    • Daniela: We have our extension, we pick things from Glue1 and Glue2
    • Andre’: We use Glue2 only for HtCondorCEs

DIRAC: current PRs and tasks being worked on, or topics from Google forum

PRs:

  • v7r0:
    • NTR
  • v7r1:
    • NTR
  • v7r2:
    • Out of the 4 python3 PRs, only the most recent one will stay (will be rebased after the HTTPs PR)

On issues:

  • NTR

Topics from Google forum:

  • NTR

AOB

CERN policies still forbid travel. Can’t know yet when there will be next DUW.
EGI workshop in November will go fully virtual. 1 day DIRAC session? probably - AT in contact.

Next BiLD in 3 weeks.

LHCbDIRAC

  • ES: before moving to ES7 we need to reduce the number or indices. Federico looking into it, some issues found. Still looking for the best way to do it.
  • Hackathon just after the LHCb computing workshop
 
There are minutes attached to this event. Show them.
    • 10:00 → 10:10
      Items from Previous BiLD-Dev 10m
      Previous BiLD meeting minutes
    • 10:10 → 10:20
      DIRAC Communities roundtable 10m
    • 10:20 → 10:35
      Current situation 15m

      General remarks

      Release notes
      Releases - cfg
      • DIRAC 3m
      • WebApp 2m
      • Pilot 2m
      • DIRACOS 2m
      • VMDIRAC 2m
      • Documentation 2m
      • OAuth2 2m
      • tornado, M2Crypto, other externals 2m
    • 10:35 → 10:50
      Release planning, tests and certification 15m
      DIRAC certification

      Proposal:

      1) Go back to v7r1 for certifying:

      - MySQL 8 (updated already)

      - Flags

      export DIRAC_DEBUG_DENCODE_CALLSTACK=1
      export DIRAC_USE_JSON_DECODE=Yes

      - SingularityCE (~easy to test)

      - JobParameters on ES

      - ElasticSearch 7 (dev to be done, probably)

      - Move to MultiVO installation (this needs work - GridPP?)

       

      2) v7r2:

      - Include HTTPs

      - Include Python 3

      Start certification with the above 2 once v7r1 is done

    • 10:50 → 11:10
      Weekly development(s) focus 20m
      • JSON encode/decode + tests
      • Glue1 vs Glue2
      JSON serialization steps
    • 11:10 → 11:20
      DIRAC: current PRs and tasks being worked on 10m

      Ongoing PRs
      - v6r22 PRs
      - v7r0 PRs
      - v7r1 PRs
      - v7r2 PRs
      Ongoing tasks
      - ?
      Topics from the google forum
      - ?

      DIRAC PRs
      google forum
    • 11:20 → 11:30
      AOB
      Convener: Federico Stagni (CERN)
    • 11:30 → 11:45
      LHCbDIRAC 15m
Powered by Indico v3.3.3-pre