Go to the Indico Home Page Oldest event Older event Up to category Newer event Newest event Printable version
 
Focus on:  Hide Contributions
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    BiLD-Dev

    → Europe/Zurich
    Federico Stagni (CERN)
    Description

    Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

    Join Zoom Meeting
    https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09

    Meeting ID: 910 8369 4183
    Passcode: 12345678
    One tap mobile
    +41432107108,,91083694183# Switzerland
    +41315280988,,91083694183# Switzerland

    Dial by your location
            +41 43 210 71 08 Switzerland
            +41 31 528 09 88 Switzerland
            +41 43 210 70 42 Switzerland
            +33 1 8699 5831 France
            +33 1 7037 2246 France
            +33 1 7037 9729 France
    Meeting ID: 910 8369 4183
    Find your local number: https://cern.zoom.us/u/artUfAdNB

    Join by SIP
    91083694183@188.184.89.188
    91083694183@188.185.118.153

    Join by H.323
    188.184.89.188
    188.185.118.153
    Meeting ID: 910 8369 4183
    Passcode: 12345678

    working notes
    Hide
     
     

    BiLD (Bi-weekly DIRAC Development meeting) – 30/09/2021

    At CERN: Federico, Alexandre, Christopher, Christophe, André, Vladimir (in our offices…)
    On Zoom: Andrei, Cedric, Daniela, Hideki, Simon, Janusz
    Apologies:

    Follow-up from previous meetings

    • Christopher On Using black:
      • Tested in DIRACGrid-Test/DIRAC
      • https://github.com/DIRACGrid-test/DIRAC/pull/51/commits/eaab3edf005e84632394566dc50bbd516fefeb7b
      • Qs:
        • Chris We have in LHCBDIRAC big files, and should be checked about merging branches
        • Federico do we still need pycodestyle?
          • Chris B I know more about flake8 (which is a superset of pycodestyle) but there are some things that pycodestyle will complain about that black ignores (like too long comments or strings)
          • Chris B We might want to move pycodestyle/flake8 to pre-commit at some point but that can be a follow up change
        • Federico do I understand correctly that I just need to run pre-commit install? https://pre-commit.com/#quick-start
          • Chris B I’ve updated the docs with pre-commit install --allow-missing-config so you can still work on old branches that don’t have pre-commit configuration
          • Federico should probably be added to README too
        • Federico shall we add a check to GA like suggested in the documentation ?
          • Chris B We run pre-commit run --all-files --show-diff-on-failure in the “basic python 3” github action, so we implictly get the check from that
        • Federico why “130” line length? There are several “line too long”
          • Chris B The CI fails on DIRACGrid-test as we run pycodestyle --diff so we have a handful of bad lines lying around
          • Chris B I propose we ignore the problem for now and move to running pycodestyle without --diff in a later PR
        • Federico I would suggest to add a .git-blame-ignore-revs (to avoid ruining git blame), also including previous massive formatting
          • Chris B I wasn’t aware of that, definitely!
        • Federico is there a hook that stops us from accepting commit messages that would then be declared non-semanticly acceptable?
      • What’s next?
        • Chris B Merge as many PRs as possible (today!)
        • Chris B Propagate all of the branches like normal
        • Chris B I’ll apply pre-commit to each branch individually
        • Chris B (optional) Merge the branches into each other
        • Chris B Fix remaining PRs
        • Chris B Enable the sweeper
    • Christopher Sweeping:
      • Was already in DIRACGrid-Test
      • Federico what adds sweep:failed and why?
        • Chris B The GitHub actions job does when there are conflicts
      • Andrei how does it work in practice?
        • the labels “alsoTargeting:branch” have to be added. This could be automated.
        • if a PR for v7r2 is made a “alsoTargeting:rel-v7r3” label will be added. This will create another PR (once the first is merged), targeting v7r3, with a “alsoTargeting:integration” branch
      • Chris B A couple of things I’ve noticed
        • The branches are pushed to the main repo so the CI will run twice on PRs
          • Suggest we have a “bot account” that owns all of the PRs and makes them from it’s fork
          • Can invite all main developers so they can push branches to this fork
        • Copy the action into the DIRACGrid organisation so we can make edits?
          • yes
        • The commit messages explaining how to cherry pick could be a little clear
      • Consequences:
        • We should not accept anything anymore for rel-v7r1
          • if really needed, 2 PRs would be needed: one for v7r1 and one for v7r2
      • Plan: Chistopher will be given hands free tomorrow, after merging “the current way” as many PRs as possible
    • Andrei New RFC/Discussion to replace ?
      • NTR
    • We canceled the hackathon of 23rd September, because of lacking something to test

    DIRAC communities roundtable

    GridPP:

    Daniela+Simon+Janusz

    ILC/CLIC/FCC:

    André

    • NTR

    LHCb:

    Federico+Christophe+Christopher+Alexandre

    EGI:

    Andrei

    • Tried HTTP service with a FormationFileCatalog. Found that the http URL is not taking a specific service and rather uses the service name as the same as the handler name
      • Christophe please make an issue with that, it’s probably fixable.
    • Testing the v7r3pX service (python2 for the time)

    Belle2:

    Hideki+Cedric+Ueda

    • from previous meeting TS BySize plugin: using floats for size is possible, needs v7r2
    • from previous meeting Observed some finished jobs that are again “Running” - asked to post on forum.
      • answered, basically: update to recent version
    • started v7r1 migration. Probably migrating on October 3rd
    • Optimizer exhausted the FD
      • should have been solved in v7r2 using selectors2
      • there are other patches in v7r2 for issues you’ll hit after applying the file descriptors issues

    DIRAC releases

    • v7r1p47
      • NEW: (#5392) Add AccountingMonitor permission
    • v7r2p28:
      • CHANGE: (#5413) TransformationDB: moved GroupSize from INTEGER to FLOAT
      • NEW: (#5430) add support for execution of pilot wrappers when the python executable is not set
    • 7.3.4:
      • CHANGE: (#5395) RequestTaskAgent uses objectLoader for RequestTasks
      • CHANGE: (#5395) split RequestTasks and WorkflowTasks into distinct module
      • NEW: (#5396) Add Emies endpoint to the ARC CE
      • CHANGE: (#5429) Use python3 in VMDIRAC instances
    • 8.0.0a4
      • release not created yet

    DIRAC projects

    DIRAC:

    Issues by milestone:

    • v7r1:
      • few tasks completed, only 2 issues still open, for documentation
    • v7r2:
      • few tasks completed, several tasks still open
    • v7r3:
      • no new/solved issues since last BiLD
    • v8.0:
      • no new/solved issues since last BiLD

    Other issues:

    PRs discussed:

    • Asked Alexandre for progress on his PRs about HTCondorCE

    WebApp:

    • from previous meetings Problem with empty ComponentMonitoring data - to be checked
      • need to use the DIRAC service, not go directly to ElasticSearch
    • from previous meetings Started to make PRs to use BaseRequesthandler, for WebApp handlers
      • Christophe more documentation is needed

    Pilot:

    • The Pilot run by VMDIRAC has been updated to py3 (from 7.3)
    • Janusz I started working on the Pilot logging, will show some drafts

    DIRACOS:

    DIRACOS2:

    • from previous meetingfts-rest on py3?
      • fts3-rest is added
      • the “official” client will be worked “soon”
    • from previous meetings Still missing rucio
      • 2 issues on their github (1 is solved/closed)
    • few issue with gfal2 and gsoap, working to put in condaforge

    Documentation:

    • several PRs waiting for comments/commits

    OAuth2:

    tornado/HTTPs

    • NTR

    management

    • NTR

    diraccfg

    • NTR

    COMDIRAC

    • NTR

    DB12

    Alexandre

    • from previous meetings Port to python3:
      • In python3: long == int and this generates slightly different scores.
      • We ran DB12 on many different processors: dividing the score by 1.16 seems to correct the issue for Intel processors but not for AMD ones.
      • Update:

        • In python3: long == int and this generates slightly different scores

        • This is actually not the cause because it would generate DB12 py3 scores lower than DB12 py2, this is not the case: probably a mix of many improvements brought with py3
        • Made another analysis based on DB12 runs (done multiple times with py3 and py2 in each job): very similar results (a constant fits well with Intel but not so well with AMD processors + we can observe different scores from a same job/same python version)
      • next is (i) merge the PR (ii) make the package (iii) use it in DIRAC (iv) apply the “fix” 1.16 - 1.19.

    other externals, including Rucio

    • Janusz Current status hanging. Scope resolution should be made pluggable, once that is done we can remove the rucio config file.
      • if there’s a timing issue, a default one could be added for all DIRAC communities
        • that’s probably fine for 90% of VOs

    Release planning, tests and certification

    • Certification machines

      • lbcertifdirac70 machine:
        • NTR
      • lbcertifdiracoauth machine:
        • NTR

    • Next hackathon(s)

      • on lbcertifdirac70, after having a real release (next week)

    AOB and topics from Google forum

    Next hackathon: on October 7th: https://indico.cern.ch/event/1082718/
    Next BiLD on October 14th: https://indico.cern.ch/event/1082717/

    LHCbDIRAC

    • PRs for black can be merged
     
    There are minutes attached to this event. Show them.