Go to the Indico Home Page Oldest event Older event Up to category Newer event Newest event Printable version
 
Focus on:  Hide Contributions
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    BiLD-Dev

    → Europe/Zurich
    Federico Stagni (CERN)
    Description

    Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

    Join Zoom Meeting
    https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09

    Meeting ID: 910 8369 4183
    Passcode: 12345678
    One tap mobile
    +41432107108,,91083694183# Switzerland
    +41315280988,,91083694183# Switzerland

    Dial by your location
            +41 43 210 71 08 Switzerland
            +41 31 528 09 88 Switzerland
            +41 43 210 70 42 Switzerland
            +33 1 8699 5831 France
            +33 1 7037 2246 France
            +33 1 7037 9729 France
    Meeting ID: 910 8369 4183
    Find your local number: https://cern.zoom.us/u/artUfAdNB

    Join by SIP
    91083694183@188.184.89.188
    91083694183@188.185.118.153

    Join by H.323
    188.184.89.188
    188.185.118.153
    Meeting ID: 910 8369 4183
    Passcode: 12345678

    working notes
    Hide

    BiLD (Bi-weekly DIRAC Development meeting) – 03/03/2022

    On Zoom: Federico, Alexandre, André, Andrei, Cedric, Christophe, Christopher, Daniela, Janusz, Ruben, Simon, Hideki, Xiaomei, Ueda
    Apologies: Andrii

    Follow-up from previous meetings

    DIRAC communities roundtable

    GridPP:

     Daniela+Simon+Janusz

    • NTR
    • ProxyDB issues: upgraded to 7.2.40, will upgrade to 7.3 and then we’ll check again.

    LHCb:

     Federico+Christophe+Christopher+Alexandre+Ruben

    • Moved one production ES to OpenSearch. No issues.

    EGI:

     Andrei

    • from previous meeting Python3 client installation dirac-configure is very confusing, globalDefaults logic should be added (in the works)
      • the suggestion is to make a globalDefaults.json for differentiation (py3 vs py2) and ease of use
    • from previous meeting Python3 pilots can not be configured to install (COMDIRAC) extensions: this feature should be added to Pilot code.
      • Had to switch to Python2 pilots for some communities
    • Config of sites WRT to VOs. BDII not always up-to-date, need to check with others, e.g. GOCDB. In contact with “Vapor” portal https://operations-portal.in2p3.fr/vapor/ (that is supposed to aggregate various sources of information)

    ILC/CLIC/FCC:

     André

    • NTR

    Belle2

     Hideki+Ueda

    Juno

     Xiaomei

    Topics from GitHub/Discussions or Google forum

    DIRAC releases

    • v7r2p41:
      • minor fix for PublisherHandler
      • one commit added to rel-v7r2 for Matcher fix (tags)
    • v7.3.18:
      • Core
        • CHANGE: (#5887) Use importlib.metadata rather than RCSID for getting AgentModule version
        • CHANGE: (#5869) Show callstack in service logs when returning S_ERROR
      • WorkloadManagementSystem
        • NEW: (#5884) Added Account option for SLURM
        • CHANGE: (#5850) HeartBeatTime is set every time a status is changed by the job itself
        • docs changes
    • 8.0.0a15 + 8.0.0a16:
      • CHANGE: (#5870) use DIRAC.Core.Base.Script.Script instead of DIRAC.Core.Utilities.DIRACScript.DIRACScript
      • dirac-login fixes
      • Monitoring
        • NEW: (#5807) Added DataOperation to Monitoring

    DIRAC projects

    DIRAC:

    Issues by milestone:

    Other issues:

    PRs discussed:
    *

    WebApp:

    • Several PRs open by Andrii. They should be reviewed (some reviews done)
      • Andrii will be absent for a while
      •  André we can share the code reviews

    Pilot:

    • from previous meeting  Janusz Revived the code for the remote pilot logger to store log messages in a remote SE. Trying to log as early as possible.
      • Janusz shared the screen, and showed what he achieved up to now.
      • Reused some of the Pilot code that was written in the past
      • The logger sends single (JSON) message to a Tornado area. On the server side an agent is in charge of a first post-processing of the messages. Possibilities (to implement):
        • log files are closed and uploaded to a Storage Element (being S3 or others… does not fully matter)
        • forwards to ElasticSearch the JSON messages as ES documents (or to another backend, but ES seems “natural” now)

    DIRACOS2:

    • from previous meeting We should be able to specify a DIRACOS2 version: in SystemAdministrator and Pilot we should specify the version
      •  Andrei DIRACOS2 installers are all in CVMFS now
      •  Christopher will try that, and code now that everything is there
    • Restored Micromamba, not in a release yet, will do it on Monday

    Documentation:

    OAuth2:

    • from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
      • There will be one this afternoon, probs focus on tokens for Rucio
    • During hackathon, we revived the idea of externalizing groups management to IdP. This was already discussed in BiLD of 11/11/21 (see also slides from 12)
      • has to be exactly 1-to-1 group-IdP
        • the CS/registry would need
      •  Christopher we should probably just make it that the WebPortal behaves like the CLI
      •  Andrei we should then remove the possibility to change easily the DIRAC group from the WebApp drop-down menu
        • basically it’s a logout-login if you want to change the group/scope

    tornado/HTTPs

    • from previous meeting  Federico with py3/tornado, could we potentially get rid of of own tornado fork?
      • it depends on Python 3.10, but of course we never tried it.
      • python 3.10.1 has been released
      • but not a trivial thing (first need to try)
    • from previous meeting  Andrii We can eventually special classes in Configuration and BundleDelivery to pass binary data - to be discussed.
    •  Federico Trying to have ProxyManager on HTTPs, PR should be checked

    management

    •  Andrei pushed DIRACOS2 installers all in CVMFS

    diraccfg

    • NTR

    COMDIRAC

    • NTR

    DB12

     Alexandre

    • Pushed a new version to fix a minor issue with the correction of the norm
    • Asked to check for the case of MP pilots
      • I will check again the norm

    other externals, including Rucio

    • from previous meeting  Janusz https://github.com/DIRACGrid/DIRAC/pull/5302 has been converted to “real” (it was a draft). Linked to a rucio PR, and waiting for a release. Problem was in the rucio configuration.
      • changes in rucio have been merged 2 weeks ago, need to check if they actually made to a release.
      • maybe restart the tests
    • from previous meeting  Andrii there’s no possibility to pass tokens (or proxies) as arguments.
      •  Cedric I will mention
        • no news but there will be a discussion about tokens this afternoon.

    Release planning, tests and certification

    • 8.0 release

      • the release will be done somewhat soon-ish keeping the AuthZ and token developments that have been done as a technology preview. We might do 1 more hackathon on lbcertidiracoath machine but the release will be effectively certified on lbcertifdirac70 using certificates, proxies, and VOMS.
      • we agreed on incorporating the code that has already been written for WebAppDIRAC, and then verify that there is no regression

    • Certification machines

      • lbcertifdirac70 machine:
        • NTR
      • lbcertifdiracoauth machine:
        • machine has been updated (mostly puppet, but also its configuration) and it is now usable.
          • All HTTPs services added
          • Added the configuration for authenticating via external IdPs.
          • There’s no connected ES (no need)

    • Next hackathon(s)

      • Suggestion is that for the next hackathon to use again lbcertifdirac70

    AOB

    Next hackathon on March 10th
    Next BiLD on March 17h

    LHCbDIRAC

    • v10r3: deploy board in https://trello.com/b/kzUKdMts/deploy-v10r3
    • MONIT: added LHCb dashboards from https://monit.web.cern.ch/ (work in progress)
      • added a homepage of organization
      • added the WMSHistory dashboard
        • few suggestions
    • from previous meeting  Christopher the BookkeepingDB assumes a certain detector (Run2). This is wrong because it is not anymore the same detector, but is it useful at all?
      •  Christophe I made a MR changing the way the condition path is constructed. While doing this I found out that there is quite some data that we can’t access
        •  Christopher it might be that some changes that I have started doing locally are fixing the described issue
      •  Christophe we should really make the description unique
    • from previous meeting  Alexandre HPCs status: on BSC we finally have a LHCb box
      • Update  Alexandre Prepared a certification instance (lhcbvoboccertif03) to push jobs in Mare Nostrum: currently resolving issues and monitoring the machine when we push a large number of jobs. Will prepare a PR once everything will be fine.
        • 2 PRs (one in DIRAC, one in LHCb) merged, but no release yet
        • new beefy machine on the way (for production)
    There are minutes attached to this event. Show them.
      • 10:00 → 10:10
        Items from Previous BiLD-Dev 10m
      • 10:10 → 10:20
        DIRAC Communities roundtable 10m
      • 10:20 → 10:30
        DIRAC releases 10m
      • 10:30 → 10:55
        DIRAC projects 25m
        • DIRAC
        • WebApp
        • Pilot
        • DIRACOS
        • DIRACOS2
        • VMDIRAC
        • Documentation
        • OAuth2
        • tornado/HTTPs
        • other externals (include Rucio)
        DIRAC certification
      • 10:55 → 11:00
        Release planning, tests and certification 5m
        DIRAC PRs
        google forum
      • 11:00 → 11:15
        Weekly development(s) focus 15m
      • 11:15 → 11:25
        AOB
        Convener: Federico Stagni (CERN)
      • 11:25 → 11:40
        LHCbDIRAC 15m