Bi-weekly DIRAC Development meeting -- 07/03/2019

At CERN: Federico, Zoltan, Christophe, Marko, Andre', Sasha
On Vidyo: Andrei, Daniela, Hideki, Simon, Xiaomei, Marcelo, Vladimir, Andrii, Wojciech

Apologies:

Follow-up from previous meeting

• Move of web server: the last file used from AFS (globalDefaults.cfg) will not be used anymore starting from DIRAC v6r21
  • the old one will die together with AFS (end of LHC LS2) so by that time everyone should move to v6r21+

DIRAC communities roundtable

• CLIC:
  • Scaling up (many MC jobs)
  • some services duplicated (TransformationManager, DataStore (helper + master))
    • Exectors chains also duplicated
  • Suggestion: look in https://dirac.readthedocs.io/en/latest/AdministratorGuide/scalingAndLimitations.html#scaling for info on scaling up
    • This guide would need to be expanded
  • needs v6r21
• LHCb:
  • Will use v6r21 in production ASAP (next week)
• GridPP:
  • Stager (AKA StorageManagementSystem) -- tried to use it for a multi-VO installation but it's only single VO
    • will look for making it multi-VO
      • Chris: at some point we'll replace this system with RMS + FTS3, but no deadline yet
    • in the meantime if there's only one VO that needs it can be used already now
• Belle2:
  • Still running very old v6r17
    • going to upgrade soon to v6r20?
      • ...time is ticking!
• Juno:
  • FTS3+TS+RMS being tried out for DDM system
  • VMDIRAC: cloud director gets stuck when trying to connect to one cloud (using deprecated libcloud)
    • need new version
• IGI:
  • Just upgraded to v6r20p26
  • Check in the matcher for pilot version can probably be skipped (especially for Multi-VO)
  • One more experiment joining
• EGI:
  • Smooth operation
    • Working for DIRAC communities, including cloud sites 
  • Ongoing development with the EGI check-in and authentication in the portal. Being tested now.
    • dirac-proxy-init gives you a URL that's passed to browser where the SSO happens. 
  • Working also on using "DIRAC CA" for internal-only authentication
  • Has to consider the changes in M2Crypto PR
• [Related] ESCAPE:
  • new project H2020 (mostly for astrophysics)
  • Marseille will have one person for km3net for setting up their computing
• [Related] DIRAC-RUCIO integration
  • pressure from several users
  • Current DIRAC developers won't do the job but other interested partners might (Belle2, SKA, CTA, ...)

Current situation

DIRAC

• v6r20p26 still the last one
• v6r21:
  • compiled today -- follow instructions in wiki for FTS DB update

VM:

• libcloud being obsoleted
• new connector for OpenStack
• occi connect becoming obsolete too
• can use VM in MP mode (not multi-pilot)

WebApp:

• NTR

Pilot3:

• Wojtek: PR for re-factorization of pilot logger. Hiding the way we do the logging.
• Need to add functionality to replace "request" because it's not available for python 2.6, need to use urllib
  • Chris: in python 2.6 is impossible to connect to HTTPS, would need to use system calls (urllib doesn't work)
• Daniel adapting part of the code using tornado (for syncing DNs from CS).
  • In few weeks it will be hopefully ready

DIRACOS:

• v6r21 can install DIRACOS, but then updates wouldn't work from the SystemAdministrator (for that we would need to wait for v7r0)
• A howto for doing the upgrade will be prepared

Documentation

• Big PR from Andre': https://github.com/DIRACGrid/DIRAC/pull/3980
  • anchors
  • automatically created documentation files
  • error codes documented
• No updates to website

BOINC:

• NTR

DB12:

• NTR

tornado and other externals

• Large PR for M2Crypto created. Have to be re-reviewed carefully, i.e. by Simon
  • could not do an integration test for the executors, need to be checked carefully in certification

Release planning, tests and certification

Release planning:

• v6r20 series
  • v6r20p26 still the last patch of v6r20
• v6r21 series
  • created today, will be announced shortly
• v7r0 series
  • Many BIG PRs, need to merge in a certain order, rebase, etc.
  • Should install on DIRAC certification with:
    • DIRAC v7r0-preX
    • DIRACOS  v1r0
    • WebApp 4.0
    • releases.cfg should be modified accordingly [done?]
  • bashrc will need to be re-done, so a fresh installation is needed
    • Instructions should go to https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r0
  • All Jenkins tests should be updated

Certification process: 

• The DIRAC certification machine should be re-done from scratch
  • Federico created new DB, still to be configured
  • should use CentOS7
• Jenkins pipelines:
  • working fine for LHCbPilot

Weekly development(s) focus

NTR

DIRAC: current PRs and tasks being worked on, or topics from Google forum

• NTR

AOB

DIRAC poster ready, can be downloaded and printed and attached in your institute! https://drive.google.com/file/d/1L0_moK5WZkM7phS7jjxwHY6gocmMIIj1/view

---------------------------------------------------------------------------------------------------------------

LHCb

• Create last pre-release of v9r3 then last tests from Federico plus go in production next week.
• Use DIRACOS in pilots for v9r4?
• Re-organization of ES and Kibana will start to be a topic