BiLD (Bi-weekly DIRAC Development meeting) – 25/03/2021

At CERN: Still, nobody :/
On Zoom: Federico, André, Andrei, Andrii, Christophe, Christopher, Daniela, Hideki, Igor, Janusz, Simon, Xiaomei
Apologies: André (will be ~15 minutes late), Alexandre

Follow-up from previous meeting

• Ran hackathon last week on v7r3-pre1:

  • trello board here
  • VMDIRAC not yet fully installed nor tried
  • didn’t go very far (new release!)

• PRs merging strategy?

  • “Sweeeping” is the name that was used by ATLAS in GitLab for doing the action
  •  Marko took that, and started adapting it to GitHub: https://github.com/petricm/sweeper/
  • Testing: https://github.com/DIRACGrid-test/DIRAC/pulls
  • Issues: too many renames for normal git merge, need to increase merge.renamelimit setting for merging to rel-v7r2

DIRAC communities roundtable

GridPP:

 Daniela

• NTR

CLIC:

 André

• Updated to v7r1p34 (directly from v6r22)
  • pilot3=false for now
  • Forgot to adapt to the changes for InputDataQuery -> MetaDataQuery in the TransformationManagerHandler (for v7r0), e.g. createTransformationInputDataQuery removed, no deprecation warning, not enough testing on my side :(
  • (TPC Castor to EOS still doesn’t work with root, gsiftp works, castor will disappear next month anyway) Whatever happened to xrootd 5?

LHCb:

 Federico+Chris+Chris+Marko

• CTA in production at CERN
• We have put Singularity as InnerCE at 1 CE per Tier1.
  • didn’t find specific issues

EGI + France Grilles:

 Andrei

• merged EGI and FG together
  • all preliminary work before Monday (FG hosting)
    • including moving to v7r1 (was v7r0)
  • merged the CFG for users and resources
    • dropped many CREAM CEs in the meanwhile, replaced with ARC and HTCondor CEs
  • then changed the DNS alias
    • and jobs started to flow
  • Issues encountered:
    • robot certificates
      • Some delegations were non-RFC -> Fixed
      • One had a max of delegations to 2 -> path length error
        • problematic for OpenSSL recent versions
      • Another cert needs more investigation, debugging
        • hint: remove the .pyo files at every iteration
        • due to this, running with M2Crypto disabled
    • Many ARC CEs are configured to not deliver the pilot proxy to the current directory
      • as a result, jobs can’t download ISB. Contacted the sites directly.
      •  André: adding XRSLExtraString = (runtimeenvironment="ENV/PROXY") to CE configs also sorts out the issue
  • only ~1000 running jobs for now, few communities have not yet restarted
  • Accounting restarted
  • Migrated the DFC, rather smoothly
• SiteDirectors with Pilot3: not yet switched on for everybody

Belle2

 Hideki

• Still on v6r22
• Frequent heavy load on SandboxStore
  • increased the number of threads but does not look like having the expected result
  •  Andrei have you tried running several instances?
    •  Christophe it was never tried, not sure if it can be done.
  • Suggestion: check https://dirac.readthedocs.io/en/latest/AdministratorGuide/ServerInstallations/scalingAndLimitations.html?highlight=limitations#servers and work on the sysAdmin part rather than DIRAC only.
    • Another suggestion is to look at the disk activity

BES3/Juno:

 Xiaomei

• running transformations:
  • how to use FailoverRequest module?
    • Federico will answer by mail
  • how to use FailoverSEs?
    • Chris wants to backport a function from LHCbDIRAC (which is also in ILCDIRAC).

JINR:

 Igor

• No changes in DIRAC infrastructure
• Can we monitor the resources usage of user jobs?
  •  Federico In DIRAC the JobWrapper starts a Watchdog that is based on psutil (from v7r1) for internal monitoring. There’s a potential of enhanced it for reporting info to an external DB (e.g. into ES or InfluxDB).
  •  Christopher suggestion to look into https://github.com/HSF/prmon
• in NICA: tried with sending info to InfluxDB with separate script
  • some performance issues, so limited.
• JINR switched all CREAM CEs to ARC

DIRAC releases

• v7r1p33 + v7r1p34:
  • NEW: (#4960) add --FTSOnly and --ExcludeSE options to dirac-dms-protocol-matrix
• v7r2:
  • Created first release
    • Using WebAppDIRAC:v4r2-pre6, will be using v4r2 final from v7r2p1
• v7r3-pre1:
  • first pre-release
  • still using OAuthDIRAC:v0r1-pre6, time to remove it

DIRAC projects

DIRAC:

Issues by milestone:

• v7r1:
  • almost only documentation is still there to be done
• v7r2:
  • a few issues still there, not urgent
• v7r3:
  • all issues are still to be addressed

PRs discussed:

Discussed PR #5035 on DIRAC scripts re-writing. The suggestion is to split this PR in 2 parts, one with the logic changes and a second with the bulk changes.

WebApp:

• OAuth2 PR created
• One PR to be rebase to v4r2 (with hackathon fixes)
• Christophe opened ticket for v4r0 and accounting WebApp

Pilot3:

• https://github.com/DIRACGrid/Pilot/pull/129
  • added a switch to pilot for installing a python3 DIRAC
  • found a couple issues in DIRAC, added them to https://github.com/DIRACGrid/DIRAC/pull/5039
  • also found that extra are not there
    •  Christopher it’s a bug in pip, fixing upstream

DIRACOS:

• xroot5? (last issue there)
  • to be done, still. It’s in DIRACOS2.
• also should check for adding arc6 (right now still on arc5 client, this poses a few issues with some sites)

DIRACOS2:

• from previous meetingfts-rest on py3?
  • No news
• from previous meeting Question for rucio client… (issue)?
  • NTR
• first non-pre-release (2.0) is there since a few weeks

VMDIRAC:

• NTR

Documentation:

• Using a conda environment when building the docs (PR merged)
• Building docs on RTD for PRs
  • using the same thing used for the publication
    • so, you can browse the docs inside a PR

OAuth2:

• from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
  • not discussed today
• Main PR not discussed today, it will need a dedicated meeting

tornado/HTTPs

• refactored the DFC handler into 1 (part of PR)

management

• from previous meeting only one task left for deploying on CVMFS

  •  Marko should come from DIRAC main repo.

•  Federico we need to start taking care of the deployments of DIRACOS2 and of pip-installable DIRAC clients

  • for DIRACOS2, what is in https://github.com/DIRACGrid/DIRACOS2/releases is fine for interactive usage, not for pilots. Releases should always be:
    • on CVMFS
    • maybe also on https://diracos.web.cern.ch/diracos/
  • for pip-installable packages:
    • on CVMFS …?
      • maybe a different way is possible, using virtualenv
    • maybe using proxies like https://lhcb-repository.web.cern.ch/ …?
      • that should not be needed

COMDIRAC

• Adapting to py3 setup

other externals, including Rucio

• from previous meetings Rucio File Catalog implementation from Belle
  • Needs (urgently?) a working implementation
• PR Reviewed a few times, not all the “editorial” comments have been replied
  • Cedric busy with other business
• It situation persists, then it’s maybe better to branch off
•  Janusz: I am in the process of interfacing to our own Rucio server
  • started from the changing the client a bit

Release planning, tests and certification

• Certification machine

  • VMDIRAC
    • NTR
  • Reminder: task for multi-vo

• Next hackathon

  • next week (1st April)
    • v7r3-pre2?

Weekly development(s) focus

NTR

AOB and topics from Google forum

Next hackathon on April 1st: https://indico.cern.ch/event/1022567/
Next BiLD on April 8th: https://indico.cern.ch/event/1022566/

LHCbDIRAC

• v10r2 planning in https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/wikis/Releases-planning#based-on-dirac-v7r2
• https://codimd.web.cern.ch/2es8HPUcSmS42jrTxhX3JQ?both
  • MySQL 8:
    • almost there
  • ES7:
    • done?
• https://trello.com/b/ECkoWbmk/deploy-v10r1