BiLD (Bi-weekly DIRAC Development meeting) – 11/02/2021
At CERN: Still, nobody :/
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Christophe, Christopher, Daniela, Hideki, Marko, Simon, Vladimir
Apologies: Cedric
Follow-up from previous meeting
- Ran hackathon last week on v7r2-pre32:
- trello board here
- One issue with a script used by the pilots made it such that we could not test many things
- We fixed everything that was “not passing”
- VMDIRAC
- added to instructions
- dirac-installed but no services/agents running
- Need configuration
- https services tested (DFC on https)
- issues found, PR with fix
DIRAC communities roundtable
GridPP:
Daniela
- Upgraded to v7r1p26. No problems observed.
CLIC:
André
LHCb:
Federico
- MP queues, mostly from Arc CEs. Still a few issues being addressed.
- Switched to Singularity InnerCE at one CE at CERN and ended up with issues (image could not be found in CVMFS). Not clear why, opened PR for debugging
- On a somewhat related note: through the LHCb Pilot we are adding tags for each CVMFS location mounted, and use that for matching jobs. Can be backported to vanilla Pilot if there’s an interest.
EGI:
Andrei
- Slowly preparing the migration of the services. WebPortal already done.
France Grilles:
Andrei
Belle2:
Hideki
- Started migration to v7r0. Not yet in certification, but installed “locally”. Have not seen issues up to now.
- Security inspector: complain that M2Crypto server still accepts TLSv1. Check if can be disabled.
- Christophe we would need everyone to be on v7r1
- DIRAC Issue will be re-opened
- Rucio: few struggles with APIs, new version of
rucio will be
DIRAC releases
- v7r0p48:
- v7r1p28:
- CHANGE: Revert #4919 - absolute imports in several modules
- CHANGE: (#4926) JobAgent: flipped the default for fillingMode
- CHANGE: (#4926) SiteDirector: The max number of jobs in filling mode is not set anymore by the SiteDirector. The option MaxJobsInFillMode was removed
- v7r2-pre33:
DIRAC projects
DIRAC:
PRs
- Fixes for scripts and docs that were not propageted from v7r1: https://github.com/DIRACGrid/DIRAC/pull/4958
- better documentation from DIRAC scripts
- suggestion is to merge ~quickly. Federico and Christopher looked into it, seems fine. Andre’ will also look.
- Andrii wants to also register arguments not only options
- Federico please do it another PR once this is merged.
Issues
- v7r0:
- v7r1:
- almost only documentation needed
- v7r2:
- discussed at previous meeting
- v7r3:
WebApp:
- python3 and WebApp changes: to be done, should target branch v4r2
PRs
- 2 PRs open. Looks like gitlab-CI workflows are broken.
- json linter broken? Andrii
- Christopher the workflow is just broken, should copy from LHCbWebAppDIRAC (issue opened).
Issues
- 4 issues opened in the last 2 months, all by Philippe.
- Mostly for the plotting of the accounting
- Christophe the plotting system would probably need to be fully replaced. There’s an issue in DIRAC for that: https://github.com/DIRACGrid/DIRAC/issues/4574
- won’t discuss today but please collect there your thoughts and requirements.
Pilot3:
DIRACOS:
- New DIRACOS release v1r17
- new
gfal2 version + davix
gfal2: last release candidatedavix: using a development version. Chris used for testing HTTPs endpoints, working fine.
- adds
rucio
- package
cachetools was removed (was only used by LHCb, which added it to LHCbDIRACOS)
- LHCb using v1r17 in production and working fine.
- Hideki can we update the packages of DIRACOS in our extension (need to do it for latest
rucio)?
- Christophe sure, but you can do it in vanilla DIRACOS for everyone to benefit
- Federico Belle2 is using v6r22, who is still on DIRACOS v1r3 (very old). We can’t fully say that bumping to lastest DIRACOS would come without issue.
DIRACOS2:
- Still waiting for
fts-rest on py3
- When installing python 3, we are still installing from Chris B fork?
Reminders:
- DIRACOS2 is for python3 only. No efforts will be ever put into python2.
- It will be offered for client installations only starting from DIRAC v7r2
- Future versions should flip the flag of the client, and offer a server python3 installations (which right now is broken, won’t look for fixing it in v7r2)
- for v7r3 we maybe make it as default for client installations
VMDIRAC:
- Maybe some dependencies are not in DIRACOS, like
libcloud. With certification we’ll find out.
- python3?
- it’s through the py3 compatibility test
- Christopher worth running py3 pylint directly.
Documentation:
OAuth2:
- WIP
- In issue WLCG and Token transition – reminders and requirements there are a few links of meetings and docs that WLCG is doing. As DIRAC we should be present there (DIRAC is also mentioned). Now that we are approaching DIRAC v7r3 we need to make sure that what’s done is complying with the requirements being discussed at WLCG level.
- Andrei I will try to be present there
tornado/HTTPs
- Couple fixes added after last hackathon.
management
diraccfg
other externals, including Rucio
- Daniela
- Follow up wt Janusz’ email and Federico’s comment. Imperial is under immense political (and funding) pressure to deliver (as in “released”) something for DIRAC-RUCIO that is multi-VO aware very soon, even if it is a minimal implementation and we have to hack it into shape later.
- For everyone else, here are the pertinent points from Janusz’ proposal:
- Since it has been decided that Belle’s implementation of a Rucio file catalog is the way to go we would like to start work on making it VO-aware and integrate it with our Multi-VO Rucio server at RAL.
- The current RucioFileCatalogClient init() signature is flexible enough, so we would come up with a small PR showing how we could eliminate the need of the Rucio config file or any env vars and give preferences to arguments passed into the Rucio client() call. The current PR gives us no chance to do it since the Rucio client() is called w/o the arguments and passes the proxy location to the client() in the environment.
- In the original client() implementation, which is a part of Rucio API the input parameters take precedence over other configuration options, so once we provide those in Dirac config we should be ready to go. A list of allowed arguments in the Rucio client() call looks sufficient for a generic integration w/o a config file. If the input arguments are not provided by Dirac, they would be provided by the config file - Belle way.
- As mentioned in the introduction, Imperial is asking for an imminent integration of the Belle code into code DIRAC, so we can start adapting it.
- Federico: There are two existing PRs:
- (Federico cont’): From the report that Cedric gave at the previous BiLD meeting (https://demo.hedgedoc.org/KnAARicoTsi-8Cr-V0NFvQ#Belle2) it was not so clear if Belle’s implementation if fully shareable. I would anyway ask Cedric to put in DIRAC, even if it contains specific Belle’s code. The target branch does not really matter, it can even be v6r22, it does not need to be v7r2.
- Janusz would prefer to develop against v7r1 as this is what Imperial has set up everywhere.
- Cedric is implementing the suggestions and updating the PR today.
Release planning, tests and certification
-
Certification machine
- VMDIRAC
- Daniela I am working on it - need some input from Federico (appcredential/“software not installed error”)
- I am updating the documentation as I go along.
- Reminder: task for multi-vo
-
Next hackathon
- next week (18th Feb)
- Federico Most probably I won’t be there, I will prepare for the hackathon to happen but you do it without me.
Weekly development(s) focus
PRs merging strategy
Issue opened: https://github.com/DIRACGrid/DIRAC/issues/4941
- André
- Using something similar for gitlab
- Needs to sort out how to make the release notes
- each branch would have its own release notes
- conclusions for now:
- needs more test of the available github actions, maybe invest time in expanding/fixing one of the existing ones.
- the proposal is interesting and not rejected, but not yet possible to adopt it.
HTCondorCE submission and usage
- Issue opened https://github.com/DIRACGrid/DIRAC/issues/4935 by GridKA admins in response to LHCb’s “misusage” (in GridKA’s terms). Basically: since we are not running a local schedd, the jobs (the pilots) stay in “completed” and don’t get removed. After 100K accumulates, pilots submission halts.
- in GGUS ticket we discussed an operational solution. It will be tested and maybe added as possibility when local schedd is not used. Maybe it’s just a matter of changing the defaults.
- Federico I sent around a mail asked basically what to do:
- run our own
schedd
- do something different, but with the net result of basically “download all your pilot outputs as soon as you are done”: development in DIRAC that, independently of what’s the CE type (HtCondor, Arc, SSH, vac, cloud etc), would store 100% of pilots outputs somewhere for some time. This would be nice but I don’t know if anyone would have time to develop that. Certainly not tomorrow, and not without planning.
- there’s an option to store the pilot output as blobs in MySQL, but this is not scalable. This is used for caching as of now.
- the SiteDirector has an option for that already but not used as of now
- ideally you would store the pilot outputs in ElasticSearch and do search in it.
- Daniela GridPP does not care too much, as long as you leave the current system in place and don’t make the running of a local schedd compulsory - so far we had no complaints. Also: The pilots on our (Imperial’s) HTCondorCEs show as “Done” - is that different to “Completed” ? Shouldn’t the pilot logs get auto deleted after a limited time anyway, i.e. when the pilots get removed ? I (Daniela, because Simon might know/remember more than me :-S ) thought that was the case anyway? We rather not download the pilot logs unless we have to and we rarely if ever look at logs that are more than a day old.
- CLIC runs one
schedd for all the sites. Looks like running without much issues. It downloads all outputs on the file system, gets removed periodically.
- Chris B A naive estimate of the pilot log size in LHCb would be 180GB a week (1.9 million files, 1.2 billion lines, easily compressed to 30GB)
AOB and topics from Google forum
Next hackathon on February 18th: https://indico.cern.ch/event/1008132/
Next BiLD on February 25th: https://indico.cern.ch/event/1008133/
DUW10 online: https://indico.cern.ch/event/852597/overview 10-11th May 2021.
will announce it in 2 weeks if there are no further comments.
LHCbDIRAC