Bi-weekly DIRAC Development meeting -- 06/06/2019

At CERN: Federico, Zoltan, Christophe, Andre', Marko, Vladimir, Christopher, Anton
On Vidyo: Simon, Andrei, Xiaomei, Marcelo, Ueda, Hideki

Apologies:

Follow-up from previous meeting

• GS DOCS: https://hepsoftwarefoundation.org/gsdocs/2019/proposal_DIRAC.html
  • several users contacted us
• SiteDirector fix (originally in https://github.com/DIRACGrid/DIRAC/issues/4014) should (still) be applied to v6r21 too
  • Federico will take care
• DIRAC certification machine has been re-done and started to be used
• WebApp 4.0 updated for v6r22 for PilotManagerHandler
• PR for refactoring the scripts for creating the docs is merged
• OAuthDIRAC:
  • created extension
  • PR for Web prepared
  • changes to ProxyDB infrastructure in a very much discussed PR

DIRAC communities roundtable

• CLIC:
  • NTR, not moved yet to v6r21
• LHCb:
  • DIRACOS machine restored, up and running. Also tried running the RequestExecutingAgent there and DMS operations are OK.
• GridPP:
  • Moving Centos7 servers. (known) issues found using the current lcgBundle for glite software. Trying to re-package.
• EGI:
  • installation is the same as some time ago, stable. Working on OAuth2 integration.
• Juno
  • Moving to v6r21 in a few weeks
• IGI
  • NTR

Current situation

DIRAC

• v6r21:
  • v6r21p7 is the last patch release
    • Mostly minor fixes.
    • Some fixes for calculating the space occupancy, not final
• v6r22-pre2:
  • Testing with certification process for:
    • WebAppDIRAC 4
    • PilotManagerHandler service (looks OK)
    • Better use of Pilot 3 (TBC)

VM:

• Several updates lately.

WebApp:

• Web 4.0 in testing, several fixes done.
• Added tests for each PR via Travis
  • pylint
  • format (pep8)
  • pytest doesn't run because there are no unit tests
  • Zoltan will look if it's possible to add some tests for the JS part

Pilot3:

• NTR

DIRACOS:

• xroot updated, created a new release candidate

Documentation

• PR for refactoring the scripts for creating the docs is ready to be merged

BOINC:

• NTR

DB12:

• NTR

tornado and other externals

• NTR

Release planning, tests and certification

Release planning:

• v6r21 series
  • NTR
• v6r22 series
  • Jenkins tests ran
  • pre2 has been put on the DIRAC certification machine
    • which is CC7, and use DIRACOS too
• v7r0 series
  • NTR

Certification process: 

• The DIRAC certification machine has been re-done from scratch, with CentOS7
• No LHCb VO there, so for the moment there are no resources.
• Federico will check about the dteam vo to have some jobs running there, and a few resources for testing

Weekly development(s) focus

Heated discussion about testing, its role, and its impact on the releases.

The general agreements are:

• unit tests and integration tests are great and we need more but can't test everything, and they are limited to the interactions that happen only inside DIRAC
  • everything that can be tested with the above (unit + integration) should be tested with the above
• for everything else there are system tests
  • and there are system tests that are coded in DIRAC/tests/System
  • the system tests are run with a "real" setup, meaning using the certification machine (lbcertifdirac7.cern.ch)
• the certification process is the sum all the above (unit + integration + system)
• the certification process is summarized in the trello board
  • cards in trello (should) contain everything that need to be done. Some are just instructions for not forgetting anything.
• the fastest we are in running the certification process, the faster we are in including developments
• If a pre-release passes all the trello cards (so all the steps of the certification process) then such release becomes a release.
  • everything that is maybe waiting will be left for the next.

DIRAC: current PRs and tasks being worked on, or topics from Google forum

• PR for docker compose can be merged

AOB

- after DUW: Federico will send a questionnaire like other years

- vyper (python tool from IT) will be evaluated by Andre'

Next BiLD in 2 weeks from now.

---------------------------------------------------------------------------------------------------------------

LHCb

• v9r4 will be based on v6r22
  • most urgent matter is the use of LbEnv+LbPlatformUtils for running applications on containers
    • Chris B is looking into it, it should be matter of few weeks to have it in production
  • We'll most probably stick with (new) lcgBundle that Chris B will try to create starting from the current production one. Chris H + Zoltan will also try the DIRACOS solution for the client.
• https://its.cern.ch/jira/browse/LBCORE-1420 : no progress yet