WLCG Containers Working Group

Monday 2 Jul 2018, 16:00 → 17:30 Europe/Zurich

513/1-024 (CERN)

Containers Working Group Meeting
2 July 2018

Present

• Vincent, Maarten, Dave, Alessandra, Andrej, Olga, Gavin, Jakob, ...

 

Previous minutes

Minutes accepted with no comments, some typos corrected.

 

Update on Singularity and discussion

Dave gave an update on whats been happening with Singularity (see slides). Main points and discussion:

 

Singularity anad EPEL

• Singularity 2.5.2-rc3 in preparation
• Sites should move to the 2.5 series, which was an update for a security issue
• Brian has now taken ownership of EPEL package. Dave removed the old unreferenced patches and reset it onto the current upstream Singularity version.
• Singularity has been removed from WLCG repo: sites should now get it automatically from EPEL instead

 

New "underlay" feature

• Underlay feature has been written by Dave, with pull-request to Singularity [ref]
• Concerns from Dave that it will not get into the 2.6 version of singularity which will be the last of the 2.x releases. Sylabs already have some stuff for it, and may not want to devote much time testing what is quite a new feature. We should test it and enthusiastically express support for it being added to the pending 2.6 release in the linked pull request. Timescale are likely quite soon - once 2.5.2 is out of the way, Sylabs will be looking to collect what they have for the 2.6 release.
• If we do have to wait for the 3.0 version, it is likely to come out as production at the end of the year (though an alpha version is expected in a couple of months). This is a new implementation and will likely require the underlay feature to be re-implemented in Go (Dave reckons a fairly easy translation).
• It was generally agreed that most people would like the underlay feature asap, both experiments and sites, so we should test it. Looking for volunteers:
   
  • [ACTION] Alessandra agreed to test for ATLAS and comment on the pull request
     
  • [ACTION] Olga agreed to test for CERN and comment on the pull request
     
  • A site admin from CMS has already commented positively on the PR
     
  • ALICE (Maarten) reckons it's still a bit early, more likely looking to integrate in Autumn. LHCb not present.
     
• Discussion about maintaining a patched version ourselves, either inside EPEL or WLCG repo with the underlay feature added. Some discomfort at doing this, and questions as the whether this would be allowed given EPEL's policies: it's a feature, not a bug, with a not-yet-accepted upstream pull-request). It was noted that we could turn the underlay feature off by default (it's currently on by default as fallback in the current code), which would make it less risky to deploy to EPEL. We concluded that the option is there if we need it, but we still preference to have it tested by us and to encourage Sylabs to accept it for 2.6.

 

SINGULARITY_BINDPATH environment feature

• Dave outlined the (existing, but poorly documented) way of adding arbitrary bind-mounts to singularity when using the --contain option (the issue is that when using the --contain option, the singularity.conf mountpoints are ignored). Some discussion as to why singularly behaves like this, but that's the way it is.

 

Sec review for new version 3.0 of Singularity

• ... which is a significant rewrite (in Go rather than C).
• Kicked off by Jeff Templon at NIKHEF on WLCG list. Suggestion to have the Wisconsin team look do a sec review of the new 3.0 Go code. Sylabs are interested and have even offered to fund the review. We should encourage this.

 

Discussion on timelines of unprivileged user namespaces

• Generally thought unlikely that RH 7.x series will include the user namespaces as anything other than Tech Preview. More likely production for RH 8.0. It's somewhat edgy to use this for production since RH don't reliably fix sec bugs on tech previews.
• RH8.0 expected "soon", and fc27 from which it is expected to be cut from already has unprivileged user namespaces enabled by default.
• Maarten noted there will be a long-tail of systems on RH6 and RH7 for many years to come, though noted that container technology should generally make this easier to handle
• Question of other container technologies - we noted that once we are using unprivileged user namespaces, the migration to other container technologies should be rather straightforward should it become necessary.

 

Testing of EPEL releases

• We agreed that it would be helpful to have regular testing on these releases - but in case no karma, the maintainer can advance to production anyway after 1 week.
• CERN volunteered to have HammerCloud-based smoke-test on their "batchtest" cluster (~1% of lxbatch) which includes the epel-testing repo, and to look at adding a Singularity-based job-profile for that. [ACTION].

 

Review actions

Existing actions:

• WC1 Done : We now have a good release in EPEL.
• WC6 Updated : "Underlay" pull-request ready to test, see new actions.
• WC4 Closed : Overlay/CVMFS issue is understood, and the workaround is WC6.
• WC3 Closed : Andrej / Alessandra confirm that WC6 underlay would help them out and have taken a new action to test it.

New actions:

• WC7: Alessandra / Andrej: ATLAS to test "underlay" feature works for them and comment on pull-request.
• WC8: Olga / Gavin: CERN to test "underlay" feature works for them and comment on pull-request.
• WC9: Olga / Gavin: CERN to setup HammerCloud based CI test for epel-testing including Singularity based test

 

AOB

• Next meeting after the summer, tbd
• Noted also report to GDB after the summer

 

Testing Refs

Pull request in Singularity (to enthusiastically update!):

• https://github.com/singularityware/singularity/pull/1638

Test releases with "underlay" feature to test:

• EL7: https://koji.chtc.wisc.edu/koji/taskinfo?taskID=273473
• EL6: https://koji.chtc.wisc.edu/koji/taskinfo?taskID=273472

Underlay is enabled by default in /etc/singularity/singularity.conf if
overlay does not work.  On EL7 you can disable overlay by setting the
environment variable SINGULARITY_DISABLE_OVERLAYFS=1, by setting enable
overlay = no in singularity.conf, or by using the exec "-u" option to
run unprivileged (assuming you have enabled unprivileged user namespaces).

 

16:00 → 16:05 Review of minutes from previous meeting

See https://indico.cern.ch/event/710207/note/

Speaker: Gavin McCance (CERN)

16:05 → 16:30 Review of Singularity developments

Speaker: Dave Dykstra (Fermi National Accelerator Lab. (US))

WLCGContainers180702.pdf

16:30 → 16:50 Discussion and testing

EPEL testing pending:

A singularity-2.5.2 release candidate has been tagged:
   https://github.com/singularityware/singularity/releases/tag/2.5.2-rc1
The packages that are built from it are marked as version 2.5.1.99.
Please test it with your workflows and let me know any successes or
failures.

I have built rpms to make it easier to test.  For those in OSG you
can get them from the osg-development yum repo.  For other people
you can download the rpms from here:
   EL6: https://koji.chtc.wisc.edu/koji/taskinfo?taskID=274571
   EL7: https://koji.chtc.wisc.edu/koji/taskinfo?taskID=274572

 

"Underlay" pull-request testing:

My PR for the singularity "underlay" feature is now submitted:
   https://github.com/singularityware/singularity/pull/1638

I have been told by the singularity core development team that they
might not have the resources to adequately review this for a 2.x release
and might have to wait until it is rewritten for 3.0.  If you believe
this is important to get into the 2.6 release, please comment in the PR
your experiences with testing it, and carefully review the code as much
as you can, so the singularity core team can feel confident enough in it
without having to put in a lot of their resources for review.  2.6 is
likely to be the last 2.N (feature release) before 3.0, and this is
definitely more than a bug fix so it can't go into a 2.6.X. 

It would also be helpful to make comments in the ticket about how strong
your desire is that this get into the 2.6 release.

Worst case, if they reject it or put it off for too long, we have the
option of adding it as a patch to the OSG & EPEL singularity-2.6
release.

In order to make it easier for you, I did an OSG scratch build including
this PR and everything else currently in the development-2.x branch and
you can find the 2.5.1-3 rpms here:
   EL7: https://koji.chtc.wisc.edu/koji/taskinfo?taskID=273473
   EL6: https://koji.chtc.wisc.edu/koji/taskinfo?taskID=273472

underlay is enabled by default in /etc/singularity/singularity.conf if
overlay does not work.  On EL7 you can disable overlay by setting the
environment variable SINGULARITY_DISABLE_OVERLAYFS=1, by setting enable
overlay = no in singularity.conf, or by using the exec "-u" option to
run unprivileged (assuming you have enabled unprivileged user namespaces).

16:50 → 17:00 Review of actions

https://twiki.cern.ch/twiki/bin/view/LCG/WLCGContainers#Actions