Packaging Tools Discussion

Europe/Zurich
40-R-A10 (CERN)

40-R-A10

CERN

20
Show room on map
Slides
Videoconference Rooms
HSF_Packaging_Tools
Name
HSF_Packaging_Tools
Description
HSF Packaging Tools Discussion
Extension
10373973
Owner
Benedikt Hegner
Auto-join URL
Useful links
Phone numbers

Discussion contributions prepared by
  Ben Couturier
  Giulio Eulisse
  Andrea Valassi
  Patrick Gartung

During the discussion we identified four more or less connected problem areas:

Package building:
  o Multiple solutions are used in production (UPS, CMS environment, lcgcmake, orch+waf,...).
  o Common agreement that we need not a single software stack for everybody, but tools to allow people to assemble whatever they need.
  o Common agreement that source code of software build needs to be preserved/kept as copy.
  o Broad consensus that the ability to patch software is a requirement. Patches may be needed for urgent fixes, adaption of packages to new platforms or compilers, or for ensuring relocatability. HSF should act as point of exchange for these patches
  o Giulio presented the way of CMS. One git repo per package is used for the bookkeeping of patches. This triggered a discussion on the need to push patches upstream and whether the CMS system isn't over-engineered.
  o Marco, Ben, Giulio proposed to set up a system of largely independent layers - build layer, dependency handling, packaging. With an agreed 'protocol' between these layers. This triggered a longer discussion with the following outcome:
   - The build layer should allow the build of a single package in a configure+make style; this would allow existing systems to take advantage of it. CMS expressed interest in this.
   - for newly starting efforts one of the existing systems could be extended/adapted to cover all three layers.
   - An open question is how relocation scripts affect the decoupling of these layers
   - Discussion on whether the packaging step should be a 'catch-all' or specific to the target system. Target oriented seems to be the best idea.

Deployment:
  o Support for multiple platforms required
  o Should take advantage as much as possible of the target system; allow to mix local installations and custom build packages
  o Deployment should be possible entirely in user land
  o Relocation needs to be supported
  o Andrea Valassi presented the current status of HEP_OSSlibs, which ensures a minimum host environment on RH/Centos systems
    - raised the question how it interconnects with HSF releases software

Runtime environment:
  o Solutions like chroot, RH software collections, python's virtualenv were presented
  o Little experience in the community on this.
  o Agreed that we need to look into the potential of these solutions
 
Dockers:
  o The usage of dockers was seen as a possibility to tackle multiple issues - runtime environment, deployment in the Grid, ...
  o Pere Mato and others proposed the concept of layered containers - base HEP system, commonly shared packages, experiment specific container
  o It can be used to run one linux flavour on a differently flavoured host or other systems using boot2docker
  o Dockers does not support diamond like container setups. So only simple setups can be provided here.
  o Containers are per default stateless. Needs change to be useful for local/interactive usage.
  o With little experience on all sides this needs closer investigation.

Possible follow ups:
  a few people volunteered to follow up with the package build 'protocol'
  looking into existing solutions for virtual environments
  investigate docker potential and use cases
 
We plan to have a follow up meeting in ~1 month from now.

There are minutes attached to this event. Show them.
    • 13:30 13:35
      Ideas 5m
      Speaker: Ben Couturier (CERN)
      Slides
    • 13:35 13:40
      Ideas 5m
      Speaker: Mr. Giulio Eulisse (Fermi National Accelerator Lab. (US))
      Slides
    • 13:40 13:45
      Ideas 5m
      Speaker: Dr. Andrea Valassi (CERN)
      Slides
    • 13:45 13:50
      ideas 5m
      Speaker: Dr. Patrick Gartung (Fermilab (US))
      Slides
Your browser is out of date!

Update your browser to view this website correctly. Update my browser now

×