BiLD (Bi-weekly DIRAC Development meeting) – 27/05/2021

At CERN: Still, nobody :/
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Christophe, Christopher, Daniela, Hideki, Igor, Janusz, Marko, Simon, Yuji
Apologies: Christopher

Follow-up from previous meetings

Ran hackathon last week on 7.3.0a8:
- trello board here
- ComponentMonitoring still looks problematic.
  - Christophe needs to be further developed to be really used
    - we should screen it out (remove it from the WebApp and maybe remove it from docs)
- VMDIRAC not tested ( Daniela would like to note that any member of the dteam VO can run the VMDIRAC certification test)
  - Related to this: Daniela would like to install the VMDIRAC extension to the WebInterface on the certification machine, but hasn’t got a clue how to go on about this. Any takers ?
    - VMDIRAC has the Web Extension, so it should show up?
      - maybe it’s just missing in the list in local dirac.cfg
        
        https://gitlab.cern.ch/ai/it-puppet-module-dirac/-/blob/qa/code/templates/diraccertif.cfg.erb#L145
        
        André volunteers
      - Andrei Q: has it been migrated to ExtJS6 version?
        
        not sure
  - Would like to introduce a test to make sure the VM Monitor is running, to avoid surprises like this one: https://github.com/DIRACGrid/VMDIRAC/pull/150 and the easiest way would be a web check.
    - can be just a note in the trello card
PRs merging strategy?
- Marko done…?
  - PR request created: https://github.com/DIRACGrid/DIRAC/pull/5139
    - Discussion on who’s in charge of making the sweeper-created Pull Request. Everyone thinks it should be the developer of the original PR.
  - Federico we should tried it out in https://github.com/DIRACGrid-test/DIRAC first
  - discussion about when to start applying it.
    - conflicting ideas if using black or not.
      - main points against (from Andrei and André):
        
        the advantage is not very clear
        
        in a while we might need to update the style again
    - make a poll on using black or not?

DIRAC communities roundtable

GridPP:

Daniela

Running v7r1p39 in production.
User issues: "Creating new processes randomly fails with EPoll: Bad file descriptor polling for events, opened in May 2020 by Chris B: https://github.com/xrootd/xrootd/issues/1198 This seems to be becoming more prevalent, could we revisit this issue ?
- Christophe there’s an env variable to get around - it should be removed by the bashrc created in the pilot by dirac-install
  - suggestion is to get DIRAC from CVMFS

User issues: We managed to get a dodgy file entry in the DFC (file only present in one of two tables) and Simon would like a cleaner way to remove these entries and/or a clearer error message than hacking the database and this:

> dirac-dms-remove-files /gridpp/user/d/daniela.bauer/api/specialpath/testfile.1592236837.LCG.RAL-LCG2.uk.txt
Failed to remove replica. /gridpp/user/d/daniela.bauer/api/specialpath/testfile.1592236837.LCG.RAL-LCG2.uk.txt 
{'FileCatalog': "Server error while serving removeReplica: 'FileID'"} Failed to remove 1 files with error:  {'FileCatalog': "Server error while serving removeReplica: 'FileID'"}

Simon would like to discuss this briefly in this meeting.

Administrator issues: Does DIRAC still issue warning for short lifetime certificates (we had this problem originally with short autogererated certificates from DUNE) and if yes, how is this going to work for token?

CLIC:

André

Finally made the PR with “ComponentSupervisionAgent” f.k.a. “MonitoringAgent”

LHCb:

Federico+Chris+Chris+Marko

Using v7r1 with DIRACOS v1r21 for a while now, and we have seen no problems
from last LHCbDIRAC hackathon, it seems that we are ready to move to v7r2 in production (will be done next week)

EGI (+ France Grilles):

Andrei

from previous meeting M2Crypto and VOMS extension follow-up (issue)
*
from previous meeting problem with retrieving the output for ARC
- DIRACOS v1r21 OK?
Federico I see (from DUW) ESCAPE is part of the EGI users. At CHEP they showed a “data lake” with no mention of DIRAC…?

Belle2

Hideki

Ready to move on to v7r0, prob next week.

JINR

Igor

TS was successfully configured. MC + analysis, looks good
idea of providing DIRAC for wider range of users. App for sending jobs?
- recompiled WebApp, using docker

DIRAC releases

v7r1p40:
- CHANGE: (#5123) Read X509_CERT_DIR from central place
- Some HTCondorCE fixes
v7r2p7:
- FIX: (#5133) The RPC Client class should now be threadsafe
v7r3-pre8 (7.3.0a8):
- NEW: Added WMS State Machine

DIRAC projects

DIRAC:

Issues by milestone:

v7r1:
- Closed a couple issues for PilotSubmission accounting (PRs created and approved, not merged yet)
v7r2:
*
v7r3:
- Closed https://github.com/DIRACGrid/DIRAC/issues/3260 (RSS multi-VO)
- Closed https://github.com/DIRACGrid/DIRAC/issues/2219 (WMS State Machine)
https://github.com/DIRACGrid/DIRAC/issues/4574 Chris would like to bring this up

PRs discussed:

[v7r1] Add multi-node allocation support in the CEs (srun with LocalCE and SSHCE)
- Federico Several changes went in, not super-easy to test. I would suggest to rebase to v7r2
[v7r2] Helper functions to make working with errors easier
- several comments to it, deserves some discussion
- not discussed
[v7r2] ComponentSupervisionAgent : import from ILCDIRAC
- doc in the code. Everything is configurable
- only for the dips, not sure how to interact with https (not tried)
  - Christophe use Client instead of RPCClient. I will comment how it should be done.
[v7r3] move Script goods to DIRACScript(wait #5061)
- delayed a bit
login with tokens
- first PR for this
- used to store tokens
- DIRAC CS will be added as an IdP

WebApp:

Pilot3:

DIRACOS:

xroot5?
- no updates
condor 9.0?
- 9.0 presentation
  - File transfer (input, output): new plugins
  - https, Amazon S3…
  - Add information about GPU usage
  - More security (ID Tokens…)
  - Tool for monitoring jobs in real time: condor_watch_q
    - by the way, the HTCondorCE calls the pilotDB to get information about status of the pilots in a specific Site.
      Is there any particular reason? I wonder if it wouldn’t be better to get the status of the jobs/pilots directly from HTCondor, seems to dependent to the pilots.
- 9.1 presentation
  - New command names: condor <noun> <verb>
    - condor job status, condor job submit
  - Legacy tools (condor_q, condor_submit, …) not going anywhere…
  - Might need to take into account these new commands in the future relases?
  - Briefly talked about HTCondor interacting with HPC with no external connectivity (CMS on Mare Nostrum)
  - New things probably not useful for DIRAC (at least for the moment):
    - Possibility to create set of jobs but do not see any use-case with DIRAC
    - Better integration of GPU
    - Kubernetes support
    - Docker universe becomes Container universe: more general
- Alexandre Overall, HTCondor 9.0 brings many new features but most of them are probably not really useful for DIRAC.

DIRACOS2:

from previous meetingfts-rest on py3?
- NTR
from previous meeting Question for rucio client… (issue)?
- NTR
updating to python3.9

VMDIRAC:

Documentation:

some issues with Sphinx 4, should be fixed by now.

OAuth2:

from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
*
from previous meeting Andrii forwarded https://docs.google.com/document/d/1ZQzElD866yV6t9AomeW6r-L9s-64pNB95OFDd1EZEbA/edit?ts=60814a09#heading=h.ur0csvxkk8tc which contains technical drawings about the proposed implementation.
- anyone can comment on it as now. Please do read it!

tornado/HTTPs

management

from previous meeting 1 task left for deploying on CVMFS
- Not yet done Marko I will come back to it
from previous meeting Add same check to VOMS_DIR and VOMSES as there is for CERT
- This PR was merged
- Asked to add the created bashrc to qa branch of puppet modules
  - also done, not propagated to master

COMDIRAC

other externals, including Rucio

Release planning, tests and certification

Federico approved a true multiVO test for certification: https://github.com/DIRACGrid/DIRAC/pull/5157 (–> trello card)
Certification machine(s)
- authcertif machine:
  - https://github.com/DIRACGrid/DIRACOS/issues/178 (py3 only ?)
  - We install DIRAC py3 on authcertif
  - from previous meeting we should probably need admin access to dteam IaM – probably also managed by Andrea
    - Andrii I am in VO and I have an account
Next hackathon
- next week (3rd June) – regular
- “oath+py3” hackathon in next 2 weeks - tbd exactly when

Weekly development(s) focus

DUW 2 weeks ago
- I have not received any comment
  - any comment from you, right now?
  - Federico I am personally rather satisfied
- Some notes taken here: https://demo.hedgedoc.org/Z7ET6a8VRPeVaVJhKJ2qTA
- Some interest in Rucio, but the integration is not fully complete.
- AuthZ is also a topic that received interest.
- Most wanted features:
  - Pilot logs is very wanted feature by many communities and installations
    - GridPP also proposing implementation
  - Component supervision Agent ported from ILC
- Reminder: don’t hesitate to use GitHub for issues!
  - Federico I have not created any issue, if you have some, please do.
- Recordings added to the Indico page

AOB and topics from Google forum

Next hackathon on June 3d
Next BiLD on June 10th

LHCbDIRAC

No news
python3 client on CVMFS
- LHCbPilot would need to add a flag