Bi-weekly DIRAC Development meeting -- 26/09/2019

At CERN: Federico, Zoltan, Christophe, Alexandre, Andre', Christopher, Sasha, Xiaomei, Joshoua [For VyPR presentation]
On Vidyo: Andrei, Andrii, Daniela, Simon, Janusz, Hideki, Ueda, Yugi Kato

Apologies: 

Follow-up from previous meeting

• NTR

DIRAC communities roundtable

• CLIC:
  • NTR
• LHCb:
  • NTR
• GridPP:
  • Testing v7r0
  • Using VMDIRAC for running on Openstack clouds
  • Started looking into using SingularityCE
• Belle2:
  • 2 issues reported on SiteDirector #4262 an #4263
  • still on v6r21
• EGI
  • pressure to use Check-IN ASAP
  • requests for applications packed as singularity containers
    • not fully clear how to proceed

Current situation

DIRAC

• v6r21
  • will need a patch for fixing #4262 an #4263 [Andrei]
• v6r22:
  • v6r22p11 created, only minor bug fixes
• v7r0:
  • a tag for v7r0p1 created, but not announced. LHCb will try a release on top of DIRAC v7r0 first (tomorrow)
• v7r1:
  • second hackathon next week
  • PR #4261 will be targeting v7r1
  • PR #4286 (run Integration tests in CI) will be rebased on v7r1 if possible, and then included

VM:

• Should use Pilot3 instead of Pilot2, and use a new version together with DIRAC v7r0

WebApp:

• Allowed possibilities to use cookies (v4r0)

Pilot3:

• Pilot logging: needs update
• PR for MultiProcessor handling (not MultiLauchAgent) has been merged in master (production now)
• adding shellCheck

DIRACOS:

• One more release to come (today) for including newest release of XRootD [Chris H]
  • and this should be used for DIRAC v7r0
• Discussed if it's worthy (or not) to add Singularity to DIRACOS
  • mostly: how big is that?
  • and also, need to consider what's solving: e.g. is it for the payload?
• MySQL 8 support can't be assured now because the MySQL client need to be updated
  • mostly question if it's already RPMed for SLC6

Documentation

• PR #4266 with restructuring of admin doc: can be merged
  • further changes in other PRs

OAuth2:

• Several fixes after hackathon

tornado and other externals

• PR #4261 (which will be rebased on v7r1) is a first step in the direction of using tornado and HTTP
  • there's a jenkins test (run with a flag) used for showing what's failing atm (e.g. tuples can't be serialized in JSON) but still a long way to go
• M2Crypto: we can reverse the M2Crypto flag in v7r1, and remove it (and pyGSI) from v7r2 

Release planning, tests and certification

Release planning:

• v6r22 series
  • NTR
• v7r0 series
  • we'll announce it next week
• v7r1 series
  • second hackathon next week, first stopped rather quickly due to issues with the ProxyManager
  • we'll only certify that nothing is broken, and we'll consider that all the OAuth2 part is a technology preview
• Integration (and v7r2)
  • Work of the GSoC students merged in integration, more PRs to be merged

Certification process: 

• Next week hackathon of v7r1-preX
  • developers of OAuth2 MUST be there

Weekly development(s) focus

• Joshua Dawes gave a presentation on VyPR, a python performance analysis tool (attached to the agenda)
  •  a tool for making performance assessment, and for checking under which conditions certain python functions take longer than expected
  • applied right now in CMS for conditions software (a computing seminar have been given about)
  • a first test has been done using a DIRAC RPC call, and this is also the first time that it's used outside of CMS and for a Grid software
  • A possible usage is: sit on a DIRAC server (e.g. one of the DFC services) and verify what's going on in a production environment
  • This is a developer/engineering tool, not a monitoring tool
  • It's a pip-installable package, backend can be postgresql
  • In DIRAC we might consider investing time in it, but need to understand what we are looking for

DIRAC: current PRs and tasks being worked on, or topics from Google forum

• https://github.com/DIRACGrid/DIRAC/pull/4286 is for running integration tests in CI
  • Github actions (a gitlab-CI for github) is a beta product (for which we subscribed) has been tried out
  • (docker) containers have been taken up to now from Chris B repo on gitlab
    • dockerhub for DIRAC: need to add Chris B as admin [Andrei]

AOB

Next BiLD in 2 weeks from now

---------------------------------------------------------------------------------------------------------------

LHCb

• https://docs.google.com/document/d/1U4DR-n7Omv4oKOmUaqvXkCTazsgGJG1ybfZYmat8Z9Y/edit?ts=5d8241f4
  • document with a collection of operations, mostly for DB
  • MySQL: several things to do (see doc), the first can be using the strict mode everywhere, also on MySQL 5.6
    • The migration to 5.7 will need a downtime, to be planned
  • Oracle will also need to be updated, maybe a downtime is needed [Zoltan to verify and add it to the doc]
  • In the meantime: migrate slowly all our voboxes to CentOS7, with v6r22+DIRACOS
• LHCbDIRAC v10r0:
  • based on v7r0
  • first hackathon tomorrow