See for working group page and actions. Agreed baseline doc is here.

WLCG Containers Working Group Meeting

31/S-028 (CERN)



Show room on map

The current situation wrt EPEL packing was discussed - the main issue being the various patches (without upstream reference) kept by the current maintainer and the slow response in generating new versions. While it was agreed that the current situation of using the OSG built version and distributing using the WLCG yum repo is fine for now, getting back to a good EPEL version would be desirable. [ACTION] Maarten agreed to contact David Love in Manchester (the current maintainer) to discuss our community's needs.

Dave mentioned the new version being tested by OSG, 2.4, is not quite ready for wider use, and efforts will now focus on the new upstream 2.4.1 version.

Alessandra outlined the current status of ATLAS' use of Singularity. Discussion focussed on ATLAS' current need for an expanded PWD directory inside the image. There was some technical disagreement as to whether the PWD of the pilot could be mounted at an arbitrary point within the image, without overlayfs - if overlayfs is needed, this likely implies that sites that support ATLAS would not be able to use Singularity in non-setuid mode, which some participants expressed significant disappointment at. Andrew suggested a possible workaround and got the [ACTION] to describe it more fully for testing, with an [ACTION] on Andrej see if it's workable for ATLAS.

CMS noted that they always mount the PWD of the pilot at /srv inside the container and the payload code running inside the container has been modified to allow this. ATLAS noted that this is possible, but changing the payload and pilot code to do that would be rather expensive.

Dave outlined the desirability of running Singularity in non-setuid mode and the steps necessary to enable it (boot param and sysctl param on RH7.4) and discussed bind-mount dropping bug in CVMFS (fixed in the new cvmfs-2.4.3-1 version). The bug only affects sites running EL7.4 kernel AND CVMFS version 2.4.0->2.4.2 AND running singularity with overlay disabled - considering this, the participants felt there was no need to rush the roll-out of the new CVMFS version.

Singularity issue 943 was discussed (where the underlying image is unpacked in cvmfs, overlayfs has trouble creating a subdirectory of an existing directory). Jakob found that CVMFS_HIDE_MAGIC_XATTRS=yes avoids the problem, but that's not a good general solution. The problem is probably a limitation of overlayfs so there's probably not much that singularity can do about it. [ACTION] Dave (subsequently) opened an issue on CVMFS to discuss the issue and see whether a workaround in CVMFS makes sense or not.

The analysis use-case of ATLAS was discussed - the question of whether this requires fat images (image files) was debated. Typically the images are around 2GB in size and their use requires Singularity in setuid-mode. ATLAS suggested that a Singularity Hub service could be launched at CERN to help distribute these images, though the impact on the site's local node storage is not clear, and depends on the churn rate. The alternative of unpacking the analysis images and deploying in CMVFS (as per production images) was discussed. Two cases were looked at:

  1. that of standard, blessed analysis images - it was generally agreed that these would be amenable to standard unpacked CVMFS distribution, assuming a reasonably prompt garbage collection policy in CVMFS (given the potential churn);
  2. development images actively being worked on by analysis dev teams - here the concern was that the CVMFS deploy time (currently 1 hour, but improving) was not sufficiently fast to give an acceptable turnaround for analysis dev groups to test their images on the grid. Some participants expressed skepticism about this use-case and suggested that local testing might be more appropriate.

It was agreed to better track the emerging baseline recommendation and issues in a live Google Doc [ACTION: Gavin to set up].

It was agreed to meet once every two weeks, on Wednesday at 16.00, starting on Wed 6th December.

There are minutes attached to this event. Show them.
    • 3:00 PM 3:10 PM
      Singularity RPM packaging status and directions discussion 10m
      Speaker: Gavin McCance (CERN)
    • 3:10 PM 3:40 PM
      ATLAS update 30m
      Speaker: Alessandra Forti (University of Manchester (GB))
    • 3:40 PM 4:10 PM
      Singularity EL7 bindmount issue and non-setuid mode 30m
      Speaker: Dave Dykstra (Fermi National Accelerator Lab. (US))
    • 4:10 PM 4:30 PM
      Discussion: summary of open points for singularity 20m
      Speaker: Gavin McCance (CERN)
    • 4:30 PM 4:40 PM
      Future meetings and regular slot 10m
      • Aim to find a slot every two weeks slot where a reasonable number of interested people can often make it
        • Not critical if you miss one
        • Max 1 hour
        • Typically just Vidyocon


      • Reasonable time for US, >=15h / 16h CET?
      • Tuesday out for me every week (Gavin)
      • Thursday clashes with CERN IT weekly technical meeting + misc WLCG, would prefer to avoid
      • Leaves Monday, Wednesday, (Thursday), Friday