BiLD-Dev
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Join Zoom Meeting
https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09
Meeting ID: 910 8369 4183
Passcode: 12345678
One tap mobile
+41432107108,,91083694183# Switzerland
+41315280988,,91083694183# Switzerland
Dial by your location
+41 43 210 71 08 Switzerland
+41 31 528 09 88 Switzerland
+41 43 210 70 42 Switzerland
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
Meeting ID: 910 8369 4183
Find your local number: https://cern.zoom.us/u/artUfAdNB
Join by SIP
91083694183@188.184.89.188
91083694183@188.185.118.153
Join by H.323
188.184.89.188
188.185.118.153
Meeting ID: 910 8369 4183
Passcode: 12345678
BiLD (Bi-weekly DIRAC Development meeting) – 22/04/2021
At CERN: Still, nobody :/
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Christophe, Christopher, Daniela, Hideki, Igor, Janusz, Marko, Simon, Xiaomei
Apologies:
Follow-up from previous meeting
- Ran hackathon last week on v7r3-pre4:
- trello board here
- VMDIRAC is installed. Looks ~OK just some questions on the configuration, some already fixed (https://github.com/DIRACGrid/VMDIRAC/pull/149)
- PRs merging strategy?
- NTR
DIRAC communities roundtable
GridPP:
Daniela
- Nothing to report wrt the production instance.
- We have advertised the DIRAC workshop to our users, but they are unlikely to engage directly with DIRAC.
- On Monday we saw an overload of the file catalog, which is rare, but we were wondering what ‘typical’ access rate for e.g. LHCb is ?
- sockets tuning + system parameters might help (https://dirac.readthedocs.io/en/latest/AdministratorGuide/ServerInstallations/scalingAndLimitations.html#servers)
- check for long-running DB queries
CLIC:
André
- NTR
LHCb:
Federico+Chris+Chris+Marko
- Nothing interesting to report
EGI + France Grilles:
Andrei
-
Users started to submit long jobs (longer than the VOMS extension). This is an issue at the moment as the payload renewal inside the CE was failing (M2Crypto error message). Issue created: https://github.com/DIRACGrid/DIRAC/issues/5114. Asked VOMS to increase the default length, but for the moment put back a pre-M2Crypto version
-
André ARC Information not in BDII (glue2): example ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=150495 with that problem at KEK, where they had to fix something in the ARC system
Belle2
Hideki
- v7r0 migration ongoing.
- Trying to use webdav protocol
- some sites are crushing.
libcryptissue - the suggestion is to try with DIRACOS v1r18
- some sites are crushing.
- Andrei I got rumors that you are thinking to submit DIRAC pilots to Panda for accessing HPC sites.
- Hideki not that I know
JINR
Igor
- Also seen proxies renewal issues (looks very similar to EGI issues)
- Tried to use TransformationSystem for MC
- looks promising.
- next step is to start running jobs according to input data
- Looking into DB12. Some tests will be shared.
BES3+JUNO
Xiaomei
- Doing upgrade. Also trying webdav
- How to remove replicas?
- there are a few ways
- Plugin that choses files which have a descendent in the filecatalog https://gitlab.cern.ch/CLICdp/iLCDirac/ILCDIRAC/-/blob/Rel-v31r0/ILCTransformationSystem/Agent/TransformationPlugin.py#L80-114 . used to transfer files to tape after reconstruction transformation ran
DIRAC releases
- v7r1p38:
- Just fixes
- *v7r2p4:
- New integration tests. As PR 5094 has been merged, it means that you should update your integration tests accordingly, in case you were using them in your extenion. E.g. LHCb GitLab-CI tests
- v7r3-pre5:
- Just fixes on top
- Using explicitely DIRACOS v1r20
DIRAC projects
DIRAC:
Issues by milestone:
PRs discussed:
- just briefly discussed some of the open PRs.
WebApp:
- Merged 2 small PRs for fixes
- Preparing for pip package. PR is almost ready to be merged. Not yet publishing to pypi, will do later.
Pilot3:
- Daniela https://github.com/DIRACGrid/Pilot/pull/129 (fairly large update to pilot code): It didn’t break anything (this time?) but should this have tags so we can pull specific versions ?
- maybe use a commit hash?
- for pulling specific version there should be already the option, but not so trivial
develbranch to be actually used, e.g. in certification?
- maybe use a commit hash?
DIRACOS:
xroot5?- NTR
arc6- created a v1r19 and tested in DIRAC certification. Looked OK, but v1r20 had to be created anyway for other issues with the SQLAlchemy version. This was done.
condor 8.8?- Marko tried several times to have it working. Several dependencies needed.
DIRACOS2:
- from previous meeting
fts-reston py3?- NTR
- from previous meeting Question for
rucioclient… (issue)?- NTR
- N else TR
VMDIRAC:
- Using Pilot/Version instead of Cloud/Version
Documentation:
- NTR
OAuth2:
-
from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
- Foreseen to have a hackathon at the beginning of June, for using tokens in the pilot framework
-
from previous meeting
- some discussions from meeting with few devs, that happened a few days ago: https://demo.hedgedoc.org/38mzdSQpT36MifxBp7xX8g
- anyone is welcome to read and participate to it
- In general, at some point each and every VO of DIRAC will need to move to interact with IAM or Check-In.
- some discussions from meeting with few devs, that happened a few days ago: https://demo.hedgedoc.org/38mzdSQpT36MifxBp7xX8g
-
We’ve decided to create a second certification setup where to test these OAuth interactions.
-
Andrii forwarded https://docs.google.com/document/d/1ZQzElD866yV6t9AomeW6r-L9s-64pNB95OFDd1EZEbA/edit?ts=60814a09#heading=h.ur0csvxkk8tc but there was no time to discuss it.
tornado/HTTPs
- NTR
management
- from previous meeting 1 task left for deploying on CVMFS
- Not yet done
COMDIRAC
- NTR
other externals, including Rucio
- PR: https://github.com/DIRACGrid/DIRAC/pull/5067
- failing tests
- no updates since last time
- from previous meeting Janusz I tried integrating in our setup. Didn’t try the agents.
- configuration: Rucio insists on having a config file, but all these configs should (could) be put in DIRAC CS. We should insist on Rucio to not needing this config file.
Release planning, tests and certification
-
Certification machine
- Reminder: task for multi-vo
-
Next hackathon
- next week (29th April)
- v7r3-preX
- next week (29th April)
Weekly development(s) focus
NTR
AOB and topics from Google forum
- DUW in less then 3 weeks
- Timetable is shaping: https://indico.cern.ch/event/852597/timetable/#all.detailed
- Monday: “developments” sessions
- Tuesday: “experiment reports” session. Prepare your pres, answer the pre-set questions and add whatever content you want. 15min + 5.
- Igor I would like to ask 2 more questions:
- number of jobs ran in the last 2 years.
- CPU (or Wall) time used by jobs in the last 2 years.
- Igor I would like to ask 2 more questions:
- Timetable is shaping: https://indico.cern.ch/event/852597/timetable/#all.detailed
Next hackathon on April 29th:
Next BiLD on May 06th: https://indico.cern.ch/event/1032098
LHCbDIRAC
- v10r2 planning in https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/wikis/Releases-planning#based-on-dirac-v7r2
- https://codimd.web.cern.ch/2es8HPUcSmS42jrTxhX3JQ?both
- No news
- https://trello.com/b/ECkoWbmk/deploy-v10r1 --> all postponed to v10r2
- At some points we should install a python3 client on CVMFS
- in the hands of lb-dirac (then LHCbPilot would need to add a flag)