BiLD (Bi-weekly DIRAC Development meeting) – 05/11/2020

At CERN: Again, nobody :/
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Cedric, Christophe, Christopher, Daniela, Hideki, Januz, Marko, Ruslan, Simon, Yuji, Ueda

Apologies:

Follow-up from previous meeting

• Quite some work on DIRACOS2
• PR to conda-forge about DIRAC in Jupyter notebooks? https://github.com/conda-forge/dirac-grid-feedstock/pull/32
  • closed
• DIRAC-RucioCatalog plugin coming up (we’ll hear about it today)
• Ran first hackathon last week on v7r2-pre18: https://trello.com/b/vYk0M4KL/v7r2-pre18
  • issues:
    • Few more JSON serialization fixes, PR open by Christophe
    • RSS issues (for v7r1, in fact)
    • Still some SingularityCE issues (v7r1)
    • …
  • goodies:
    • python 2 and python 3 clients look fine
    • no major issues found
    • multi-VO setup looks OK-ish
  • new functionalities not yet tried out

reminder Marko prepared a container to be used for CVMFS (auto)deploy: https://github.com/DIRACGrid/management/packages/342716 (also in https://hub.docker.com/repository/docker/diracgrid/dirac-cvmfs). For using it from GitHub Actions template in: https://github.com/DIRACGrid/management/pull/19 Andrei + Federico

reminder DIRACOS should be expanded on CVMFS Andrei

DIRAC communities roundtable

GridPP:

• Hopefully by the next meeting should run v7r1

CLIC:

• on v6r22 + DIRACOS
• NTR

LHCb:

• Still on v7r0. Will move to v7r1 ASAP (basically only waiting for RSS fix)
• Fully using M2Crypto (also on master CS and Web boxes).

EGI:

• EGI-ace project approved (continuation of EOS-HUB)
• Using the Conda environment in EGI Jupyter notebooks
  • terminal works quite well
    • users don’t have certificate inside the terminal: Andrii created command to transform token in proxy (if proxy is uploaded)
    • scripts don’t quite work

  •  Christopherpackaging is not yet fully finished open issues at if appropriate https://github.com/conda-forge/dirac-grid-feedstock/issues

  •  Andreiit might be DIRAC the issue

  •  Andreiwould be nice to have COMDIRAC too in conda. It’s supported even if there’s not much to do, developer is there.

  •  ChristopherI opened an issue for COMDIRAC for that. Should be pip-installable: https://github.com/DIRACGrid/COMDIRAC/issues/72

• 10 days ago Andrei gave DIRAC EGI Webinar: quite well attended

France Grilles:

• Will be merged in EGI installation at some point
  • will take time

Belle2:

• Started discussing about migration to v7r0. It will not come in this year. Now the focus is on Rucio: migration will be done on v6r22+DIRACOS

JINR:

• NTR

Juno, BES3:

• NTR

Current situation

DIRAC

• v7r0p37:
  • Minor fixes
• v7r1p16:
  • Minor fixes
• v7r2-pre18:
  • Many fixes related to python 3, including switch for installing python 3 client

WebApp:

• fixed the versioning:

  • rel-v4r0 branch for DIRAC versions:
    • v6r22
    • v7r0
    • v7r1
  • rel-v4r2 branch for DIRAC version:
    • v7r2
  • integration for future (v7r3)

• rel-v4r1 branch has been retired

• Use ngnix for data transfer? in progress

Pilot3:

• NTR

DIRACOS:

• xroot5? no news
• ldap3 should be added (PR open)

•  Hidekiis less working? > from v1r14

• next version can include tornado

DIRACOS2:

• Added repo to DIRACGrid https://github.com/DIRACGrid/DIRACOS2

  • now functional
  • last hackathon still used Chris’ DIRACOS repo, need to change dirac-install for update the link: will be done this week

• https://anaconda.org/diracgrid/ is for “specific” packages

  •  FedericoWhy M2Crypto is here?

    • used by the unit tests (for v7r1 CE, for v7r2 no need?)

• Other “standard” packages not distributed via (ana)conda are added to conda-forge.

VMDIRAC:

• NTR

Documentation:

• NTR

OAuth2:

• WIP

tornado/HTTPs

• NTR

management

• PR: Conda recipes for DIRACOS2

diraccfg

• NTR

other externals

• NTR

Release planning, tests and certification

• Certification machine updated to be multi-VO (task: https://github.com/DIRACGrid/DIRAC/issues/4631)

  •  André GridPP VOMS info should be there now (added in central puppet)
  • tests not updated yet to be multi-VO
  •  Federico Will probably update ES to ES7, but first need to sort out a couple things

• Next hackathon November 12th

  • Already called in https://indico.cern.ch/event/970877/

• v7r2:

  • several PRs open, mostly for
    • “minor” fixes
    • removing depracated code
      • e.g. MultiProcessorSiteDirector removed
      • ProxyDB backward compatibility
    • tests updates
      • Removed MySQL 5.6 tests, set MySQL 8 as default

Weekly development(s) focus

topic: DIRAC-Rucio integration

GridPP (pres on the agenda) [Janusz]

• took long time, as was waiting for multi-VO Rucio catalog implementation
• RucioFileCatalog plugin activated for v7r1 (in test)
• nicely integrate with DIRAC tools. Minimal modifications.
• Storage Element definitions (DIRAC-Rucio) not fully sorted out
• to API or not to API?
  • has to include rucio in DIRACOS

   Christopherit’s a no-brainer packaging, as rucio is pure python
   Christopheit’s properly packaged, even DIRACOS 1 can package that easily
   Cedricpython2.7 and 3 supported, 2 might be dropped in the future months

• to REST or not to REST?
  • licensing issues?
  • if we use the python bindings there’s no licensing issues.

General consensus: using python bindings instead of REST look like a better option

Belle2 (pres on the agenda) [Cedric]

• main issue was mapping the Rucio scopes to the namespace
• using python bindings with DIRACOS
• need to install the rucio.cfg (1 per VO) - this can’t be skipped, won’t go through DIRAC
• 1-to-1 mapping between SE and RSE (via agent)

 JanuszConfiguration: can’t I eliminate it completely? It can all go in DIRAC. It’s possible by a configuration parameter in Rucio client
 Andreialso pilots are clients so we would need to ship this file too. We should NOT have it needed
 Cedricit’s still needed ATM. Changes to the API are needed. Alternatively it can be created on the fly.

General questions/comments:

 IgorCan different VO under DIRAC use different single VO Rucios? => should be, but never tried, nor sure
 IgorDoes DIRAC accounting work correctly with Rucio plugin?
 Andrei+Uedashould be yes, it’s just another catalog, for synchronous operations (if you use Rucio only as file catalog)
 Janusz+Uedaif you use Rucio as data manager (for moving datasets) then no.
 ChristopheRucio has their own rules, there’s no way DIRAC knows about them
 Christophewith DIRAC using https the interactions between one and the other will be much simpler. But this is not for tomorrow.

 FedericoMore discussions via mail. Only 1 implementation.

DIRAC: current PRs and tasks being worked on, or topics from Google forum

PRs

• BDII and SE
  • agreed to stop looking for SEs inside BDII  Federico

Issues

• v7r0:
  • Nothing in there
• v7r1:
  • Several tasks in there still, so this is to be considered an actively developed release.
  • URGENT: MultiVO-RSS breaks interface

  •  JanuszI will look into it => put the vo at the end for backward compatibility for a quick fix.

• v7r2:
  • mostly non-urgent
• v7r3:
  • Not discussed

Other issues:

• HTCondor TimeLeft module
  • the information from CERN only are better than nothing  Alexandre will create the PR.
• runit in dirac is old
  • topic will be closed

Topics from Google forum

• NTR

AOB

Next hackathon in 1 week.
Next BiLD in 2 weeks.

LHCbDIRAC

• Waiting for RSS fix on v7r1 before releasing v10r1.
•  Christophe I will circulate a document for changes for the LHCbDIRAC certification machine.