BiLD (Bi-weekly DIRAC Development meeting) – 14/01/2021

At CERN: Still, nobody :/
On Zoom: Federico, Alexandre, Andrei, André, Christophe, Christopher, Daniela, Hideki, Janusz, Marko, Simon, Xiaomei

Apologies:

Follow-up from previous meeting

• Ran hackathon last week on v7r2-pre27:
  • trello board here
  • added EnableActivityMonitoring option to REA (using MonitoringSystem)
    • started testing it, some issues discovered, fixed in PR.
  • https functionalities not yet tried out

DIRAC communities roundtable

GridPP:

 Daniela

• NTR

CLIC:

 André

• No change in the iLCDirac setup, still v6r22
• Some minor issues during the break, like dirac-install moving (fixed since then)

LHCb:

 Federico

• Still on v7r0, ran a hackathon last week, all went pretty well, expect:
  • issue with interacting with ES job parameters, fixed in https://github.com/DIRACGrid/DIRAC/pull/4889
• Should install v7r1 next week

EGI:

 Andrei

• Allocating enough resources in Lyon, migrating everything including the databases, and only after starting to merge the services with the ones in France Grilles (no urgency though). Should be an interesting exercise.
• Sorted out the issues with certificates, now will start migrating to v7r1.

France Grilles:

 Andrei

• Still on Pilot2
• Starting to migrate to ES.
  •  Federico: use v7r1 for ES7 compatibility

Belle2:

 Hideki

• DIRACOS migration completed successfully
• Preparing to rucio migration
  • stop tomorrow, make a LFC dump, move to Rucio, restart Monday.
  • will report how it goes

JINR

• NTR

Juno, BES3:

 Xiaomei

• NTR
• Starting migrating to v7.

DIRAC releases

• v7r0p44:
  • Core (dirac-install)
    • CHANGE: (#4873) default generated bashrc does not overwrite X509* variables if already set
    • NEW: (#4873) Add DIRAC_X509_HOST_CERT and DIRAC_X509_HOST_KEY environment variables
  • WMS
    • CHANGE: (#4836) JobWrapper: Add extraOptions only when the executable is a DIRAC script
• *v7r1p24:
  • only fixes
• v7r2-pre27:
  • more fixes

DIRAC projects

DIRAC:

PRs

• [v7r2] Fully support setuptools

  • Large PR because all scripts are changing
  • This is anyway “only code”, logic is staying the same. So it should be “easy” to test
  •  Andrei: each extension will become a pip-installable extension, being this LHCbDIRAC or VMDIRAC
  • pip-install on top of DIRACOS2 or “on-your-own” pip-installable.
  • scripts get renamed, as ‘-’ are not allowed, so moving script names from useing ‘-’ to using ‘_’
    • dirac-service, dirac-agent, etc are hardcoded in the run files
    • will need some transition docs anyway, and some scripts to address it, but for the moment we will keep both names

• [v7r1][WIP] docs: user manage, proxy manager for addressing https://github.com/DIRACGrid/DIRAC/issues/4702

  • some of the documentation requested is not for v7r1, rather for v7r3
  • What’s prepared should be close to final anyway
  • Christophe should look again into it

Issues

• v7r0:
  • 4 issues right now
  • dirac-proxy-init silent fallback to /etc/vomses  Marko will take care ASAP
  • 1 ARC issue  Raja, maybe easy to fix.
• v7r1:
  • Still some tasks in there, some are for documentation
• v7r2:
  • mostly non-urgent ones
  • Make RSS vo-aware
    • Janusz: I will have a look, but up to now I worked on v7r1, will rebase to v7r2
  • addressed and closed PilotCSToJSONSnchronizer is not thread safe by creating a new agent
• v7r3:
  • Not discussed

WebApp:

PRs

• PR documentation on nginx configuration was removed, will create a new one.

Issues

• from hackathon: ComponentMonitoring issue, will need to look into it.

Pilot3:

PRs

• set DIRACSYSCONFIG env variable to pilot.cfg was merged, then all jobs failed in LHCb, the PR was quickly reverted.

Issues

• https://github.com/DIRACGrid/Pilot/issues/120 (Pilot3 adds “pilot.cfg” on to user argument list)
  • fixed by re-opening https://github.com/DIRACGrid/DIRAC/pull/4836 which was merged in v7r0

DIRACOS:

• DIRACOS releases?
  • v1r16 is still the last one (v1r17 was not created in the end)
• 1 prob with rpath that Chris B fixed
• new repository path fixes for test [done]
• new gfal version – pain with SLC6 deprecation
  • these pains will be more and more

DIRACOS2:

• Just waiting for fts and for setuptools PR to be merged
• When installing python 3, we are still installing from Chris B fork

VMDIRAC:

• NTR

Documentation:

• Missing doc for [v7r1] Update the documentation for the new organisation of FC components  Andrei

OAuth2:

• WIP

tornado/HTTPs

• NTR

management

• Task https://github.com/DIRACGrid/management/issues/10  Andrei
  is still to be addressed. Federico thinks it should be high priority.

diraccfg

• NTR

other externals, including Rucio

• Still waiting for changes on rucio side

Release planning, tests and certification

• Certification machine updated to be multi-VO (task: https://github.com/DIRACGrid/DIRAC/issues/4631)

  • NNTR

• Next hackathon

  • next week (21st Jan), already called.

Weekly development(s) focus

• dirac-install: https://github.com/DIRACGrid/DIRAC/issues/4860
  • Plan written in the task is fine
  • We could probably also upload dirac-install somewhere like for globalDefaults.cfg (management task 10)

AOB and topics from Google forum

Next hackathon on January 21st: https://indico.cern.ch/event/994016/
Next BiLD on January 28th.

• External access to computing resources: https://information-technology.web.cern.ch/help/external-collaborators

LHCbDIRAC

• v10r1 next week after v7r1 patch

• proposal for new ProductionManagement

  • https://www.notion.so/LbMCSubmit-Attempt-1-1be8e834f11b41dfa334fabb4a755d6d
  • will look into it.