See for working group page and actions. Agreed baseline doc is here.

WLCG Containers Working Group

513/1-024 (CERN)



Show room on map

WLCG Containers Meeting - 3 July 2019

 * Local: Maarten Litmaath, Lukas Heinrich, Ricardo Rocha, Vincent Brillault, Maxim Storetvedt, Gavin McCance
 * Remote: Alessandra Forti, Andrej Filipcic, Dave Dykstra, Emmanouil Vamvakopoulos, Igor Sfiligoi, 
           Jakob Blomer, Mathieu ?, Marco Mambelli, Sebastien Gadrat, Simone Mosciatti


Previous minutes: no comments, accepted.


Dave presented the current status of Singularity:

 * Current is 3.2-1 from epel. 3.2.1-1.1 coming very soon fixing a couple of relevant bugs
 * 3.3.0 RC1 available offering fakeroot option, allows building without sudo
 * 3.4 (expected ~August) is expected to have fusecmd option for fuse mountpoint with fuse3, inside container.
    - Needed for WLCG? Expected to be very helpful for HPC sites that have singularity
   but not CVMFS
    - noted that it would miss out on the shared caching, but can be useful sometimes
 * unpriliveged user mount in RHEL8
  --> trying to convince Sylabs to have this as default in Singularity
 * Security assessment of Singularity 3 (albeit earlier version) due next week


Jakob reviewed the CERNVM/CVMFS workshop and,

Ricardo reviewed the container software distribution event,

Discussion on future direction. It was noted that all these tools seem to be moving in the same general direction, many
of them based of the same underlying technologies, notably toward unprivileged and daemonless modes of operation.
It was expressed that Singularity probably has the best set of features for us at the moment (e.g. unprivileged underlay), not least because we have a direct way (via Dave et al) of influencing the feature-set. In the future, as unprivileged user namespaces, mounts and overlay gradually appear, other tools, based on the wider-used codebase will likely become interesting. It was also noted that, given the end goal is to run these tools out of CVMFS by unprivileged users, it doesn't really matter to the site what experiments use (and they can use different things).


Dave presented a proposal for a baseline and Alessandra presented the ATLAS view.

ACTION: Maarten and Gavin took the task to write a clear version of the below for iteration and agreement by the working group over mail:

[[ There was good agreement for what is needed, focussing on a strong recommendation (though not requirement) that all WLCG sites enable unprivileged user namespaces on EL7. This would allow experiments to immediately use Singularity directly from CVMFS, which would generally both work for ATLAS and CMS. In this case, sites who have no other local use, could uninstall the local Singularity. If kept, the preference is to enable unprivileged modes of operation, with underlay, but note that experiments will likely *default* to use their own version in CVMFS if they can (assuming the site has enabled unprivileged user namespaces). Sites should still be allowed to disable unprivileged user namespaces - in this case, the local Singularity installation would need to be used in 
setuid mode, with the correct config (notably underlay enabled). ]] 



 * Discussion on user-code distribution. Dave noted a Fermilab mechanism of distributing user-code based on tagged tarballs + basic container image, that limits the number of distinct images in CVMFS. ATLAS noted their preferring option of packaging user-code directly in containers in CVMFS (built off a standard ATLAS base image) and relying on the dedup of CVMFS - thus making the tagged container the instrument of analysis preservation. It still needs to be demonstrate how to effectively clean up many user containers in CVMFS.



 - Clean up of some obsolete actions on CERN

 - Add action on Gavin and Maarten to write up initial version of proposal

There are minutes attached to this event. Show them.
    • 4:00 PM 4:05 PM
      Review of minutes from previous meeting 5m


      Speaker: Gavin McCance (CERN)
    • 4:05 PM 4:20 PM
      Singularity update and status 15m
      Speakers: Dave Dykstra (Fermi National Accelerator Lab. (US)), Igor Sfiligoi (UCSD)
    • 4:20 PM 4:45 PM
      Review of container SW distribution event and review of CERNVM workshop 25m
      • Key points from Container Software distribution event: (Ricardo)

      • Key points from CERN VM workshop: (Jakob)

      • Where do we see this all going?

      • Is this the future?

      Speakers: Jakob Blomer (CERN), Ricardo Brito Da Rocha (CERN)
    • 4:45 PM 5:10 PM
      (Un)priviliged discussion - can we agree a baseline yet? 25m

      Unprivileged vs privileged container runtimes.

      • Can we agree on the options we recommend to sites?

      • Do we agree that sites are allowed, if they wish, to have only unprivileged containers?

      • (How) do we want to start writing this up?

      Speakers: Alessandra Forti (University of Manchester (GB)), Dave Dykstra (Fermi National Accelerator Lab. (US))
    • 5:10 PM 5:20 PM
      Current status and review of actions 10m