See https://twiki.cern.ch/twiki/bin/view/LCG/WLCGContainers for working group page and actions. Agreed baseline doc is here.

WLCG Containers Working Group

→ Europe/Zurich
513/1-024 (CERN)

513/1-024

CERN

50
Show room on map
Hide

Containers Working Group Meeting
2 July 2018

Present

  • Vincent, Maarten, Dave, Alessandra, Andrej, Olga, Gavin, Jakob, ...

 

Previous minutes

Minutes accepted with no comments, some typos corrected.

 

Update on Singularity and discussion

Dave gave an update on whats been happening with Singularity (see slides). Main points and discussion:

 

Singularity anad EPEL

  • Singularity 2.5.2-rc3 in preparation
  • Sites should move to the 2.5 series, which was an update for a security issue
  • Brian has now taken ownership of EPEL package. Dave removed the old unreferenced patches and reset it onto the current upstream Singularity version.
  • Singularity has been removed from WLCG repo: sites should now get it automatically from EPEL instead

 

New "underlay" feature

  • Underlay feature has been written by Dave, with pull-request to Singularity [ref]
  • Concerns from Dave that it will not get into the 2.6 version of singularity which will be the last of the 2.x releases. Sylabs already have some stuff for it, and may not want to devote much time testing what is quite a new feature. We should test it and enthusiastically express support for it being added to the pending 2.6 release in the linked pull request. Timescale are likely quite soon - once 2.5.2 is out of the way, Sylabs will be looking to collect what they have for the 2.6 release.
  • If we do have to wait for the 3.0 version, it is likely to come out as production at the end of the year (though an alpha version is expected in a couple of months). This is a new implementation and will likely require the underlay feature to be re-implemented in Go (Dave reckons a fairly easy translation).
  • It was generally agreed that most people would like the underlay feature asap, both experiments and sites, so we should test it. Looking for volunteers:
     
    • [ACTION] Alessandra agreed to test for ATLAS and comment on the pull request
       
    • [ACTION] Olga agreed to test for CERN and comment on the pull request
       
    • A site admin from CMS has already commented positively on the PR
       
    • ALICE (Maarten) reckons it's still a bit early, more likely looking to integrate in Autumn. LHCb not present.
       
  • Discussion about maintaining a patched version ourselves, either inside EPEL or WLCG repo with the underlay feature added. Some discomfort at doing this, and questions as the whether this would be allowed given EPEL's policies: it's a feature, not a bug, with a not-yet-accepted upstream pull-request). It was noted that we could turn the underlay feature off by default (it's currently on by default as fallback in the current code), which would make it less risky to deploy to EPEL. We concluded that the option is there if we need it, but we still preference to have it tested by us and to encourage Sylabs to accept it for 2.6.

 

SINGULARITY_BINDPATH environment feature

  • Dave outlined the (existing, but poorly documented) way of adding arbitrary bind-mounts to singularity when using the --contain option (the issue is that when using the --contain option, the singularity.conf mountpoints are ignored). Some discussion as to why singularly behaves like this, but that's the way it is.

 

Sec review for new version 3.0 of Singularity

  • ... which is a significant rewrite (in Go rather than C).
  • Kicked off by Jeff Templon at NIKHEF on WLCG list. Suggestion to have the Wisconsin team look do a sec review of the new 3.0 Go code. Sylabs are interested and have even offered to fund the review. We should encourage this.

 

Discussion on timelines of unprivileged user namespaces

  • Generally thought unlikely that RH 7.x series will include the user namespaces as anything other than Tech Preview. More likely production for RH 8.0. It's somewhat edgy to use this for production since RH don't reliably fix sec bugs on tech previews.
  • RH8.0 expected "soon", and fc27 from which it is expected to be cut from already has unprivileged user namespaces enabled by default.
  • Maarten noted there will be a long-tail of systems on RH6 and RH7 for many years to come, though noted that container technology should generally make this easier to handle
  • Question of other container technologies - we noted that once we are using unprivileged user namespaces, the migration to other container technologies should be rather straightforward should it become necessary.

 

Testing of EPEL releases

  • We agreed that it would be helpful to have regular testing on these releases - but in case no karma, the maintainer can advance to production anyway after 1 week.
  • CERN volunteered to have HammerCloud-based smoke-test on their "batchtest" cluster (~1% of lxbatch) which includes the epel-testing repo, and to look at adding a Singularity-based job-profile for that. [ACTION].

 

Review actions

Existing actions:

  • WC1 Done : We now have a good release in EPEL.
  • WC6 Updated : "Underlay" pull-request ready to test, see new actions.
  • WC4 Closed : Overlay/CVMFS issue is understood, and the workaround is WC6.
  • WC3 Closed : Andrej / Alessandra confirm that WC6 underlay would help them out and have taken a new action to test it.

New actions:

  • WC7: Alessandra / Andrej: ATLAS to test "underlay" feature works for them and comment on pull-request.
  • WC8: Olga / Gavin: CERN to test "underlay" feature works for them and comment on pull-request.
  • WC9: Olga / Gavin: CERN to setup HammerCloud based CI test for epel-testing including Singularity based test

 

AOB

  • Next meeting after the summer, tbd
  • Noted also report to GDB after the summer

 

Testing Refs

Pull request in Singularity (to enthusiastically update!):

Test releases with "underlay" feature to test:

Underlay is enabled by default in /etc/singularity/singularity.conf if
overlay does not work.  On EL7 you can disable overlay by setting the
environment variable SINGULARITY_DISABLE_OVERLAYFS=1, by setting enable
overlay = no in singularity.conf, or by using the exec "-u" option to
run unprivileged (assuming you have enabled unprivileged user namespaces).

 

There are minutes attached to this event. Show them.