Key4hep Discussion

Tuesday 17 Nov 2020, 09:00 → 10:00 Europe/Zurich

Vidyo

Live Notes

Notes Archive

# Key4hep Live Notes

This is a document for taking notes during Key4hep meetings.

Date: November 17, 2020

Agenda: https://indico.cern.ch/event/975840/

Connected: Joseph, Tao, Wenxing, Andre, Frank, Benedikt, Placido, Teng, Sang Hyun, Jiaheng, Xiaomei, Thomas, Valentin, Clement, Weidong, Gerri, Pere
Apologies: Paolo

## Communities Round Table

### New collaborator

* Teng Li from Shandong University

## Transition of FCCSW to EDM4hep started
* https://github.com/HEP-FCC/FCCSW/tree/edm4hep

### Raised some questions and issues:
* Duplicated GeoSvc
   - https://github.com/cepc/CEPCSW/issues/64
* Some mention of key4hep probably good, https://github.com/key4hep/Pandora
    - [x] renamed to k4Pandora
* Some tests build with spack are taking 10x more time to run
    * Units missmatch https://github.com/HEP-FCC/FCCSW/issues/418
    - [x] now handles different hepmc unit conventions gracefully
    - [ ] down to three failing tests, all connected to the same HepMC file  

## Naming scheme for common key4hep repositories
- start packages with lowercase k4
- GMP -> **k4MarlinWrapper**
    - WIP https://github.com/key4hep/GMP/issues/20
    - FG: propose to call it `k4MarlinWrapper` (agreed, also ask Andre)

- Renamed `k4-spack` to `key4hep-spack` (similar to `key4hep-doc`) meta package

## k4 delphes interface
- its own repository: https://github.com/key4hep/k4SimDelphes
- EDM4hep_Delphes (pythia8 dependence), also moved there
- package would depend on Gaudi?
- Eventually move EDM4hep interface to Delphes?

* Thomas fixed seg-fault (https://github.com/key4hep/EDM4hep/issues/89)

## Abstracts for HSF/WLCG meeting
* https://indico.cern.ch/event/941278/
* November 19-24

* Contributions
    * Spack usage (Valentin)
    * podio Developments (Thomas, Benedikt)

## Podio

### CI

* EDM4hep and podio in LCG releases?
    * LCG_97a_FCC_4
    * dev3, dev4 (currently having python3 issues)

### Discussion: Interplay between Readers, Writers, EventStore and Collections
* https://github.com/AIDASoft/podio/issues/140
* Have to define ownership. Create a consistent interface.
    * EventStore was meant to own things, take things from reader
    * EventStore should pass data to writer and forget about it
        * Writer acts like a sink
        * What about writing to multiple files?
            * Create an entity seperate from the EventStore for this purpose
* Const correctness
    * Metadata implementation not good at keeping it
* CollectionIDTable can change from file to file, but is only read once
* Separate 2 use-cases: users of the data, developers and integrators

* EventStore, at the moment
    * Is filled at the beginning, written out at the end
* TODO (Benedikt): sketch out the model

### Different IO backends
* https://github.com/AIDASoft/podio/issues/131
* How to best pass information/code/libraries to downstream packages?
* Add a factory for the reader?
    * Add SIOReader::getEntries
* Abstract interface for the writers?
    * no reason not to have one

## EDM4hep

## Build system / Target compiler / dependencies

* Full Spack installation now available under /cvmfs/sw.hsf.org
* [x] update documentation
    * see https://github.com/key4hep/key4hep-doc/pull/11

* docker container working with spack at

    https://github.com/joequant/key4hep-container

* CI running
    * Incremental build every hour
    * Full build 8 hours each day
    * Now making webpage public
    * Send out notices when something breaks
    
* Nightlies can now be built from commits (avoid full rebuilds if nothing changes)
    * being run on https://gitlab.cern.ch/key4hep/k4-deploy
* Automatic generation of setup script
    * [x] done in key4hep-stack
* Added printout to show software version being used
    * /cvmfs/sw.hsf.org/key4hep/setup.sh
* cvmfs publisher now registered as a gitlab runner
* "pre-configured" spack from nightlies available for download
 

### Problem with Root (e.g., 6.20.04) and spack installed gcc

* Cannot create dictionaries
    * std headers not found
    * https://github.com/spack/spack/issues/17488
    * work in progress ...

* Thomas: this seems to be a general issue w/ spack installed compiler
    * Valentin: could be worked around by disabling compiler wrappers

* TODO: document the workaround of adding the compiler symlink

### CMake Template

* Valentin: need to update the HSF CMake project template
    * https://github.com/HSF/tools
    * could be a nice student project
    * [ ] need a project description

----

## k4FWCore

* Problem with writing additional TTree: https://github.com/key4hep/k4FWCore/issues/10
    * work in progress

* Valentin developed branch to updated the CMake configuration for k4FWCore

## MarlinWrapper

* rename to `k4MarlinWrapper`

## Delphes "Algorithm" to run from Gaudi

* Investigating services and algorithms to use
* Possibility found for going over all collections
    * Conversion service in Gaudi: convert arbitrary objects into other arbitrary objects
    * E.g., convert EDM4hep to LCIO , FCCedm to EDM4hep, etc.
    * work in progress
    * needs the Delphes2EDM4hep converter merged
* B-tagging information hard to use
    * Similar to metadata problem

* -> no news

## Distributed Computing, Workload Management, Data Management

### Grid production system
* Clement: discussion needed on what should/could be taken out of ILCDirac into Dirac to be also usable for everyone else
    * Andre/Andrea have started to include FCC-VO in ILCDirac
* Gerri: should have discussion about how to go about this in next meeting

* Xiaomei: IHEP experiments using a DIRAC instance. Share common pieces in DIRAC.
* Weidong: DataManagement with Rucio in DIRAC

* Gerri: Andrea Stano adding wrapper for LEP MC programs to iLCDirac
    * KKMC, BHLumi
* Clement: Share MC generator settings
    * Invole MC generator experts to see how much can be shared

* Share common modules used in transformation or user jobs?

* Weidong: Is Ganga used for iLCDirac?
    * Python scripts calling the interfaces to submit jobs

* -> Schedule dedicated discussion for distributed computing and Key4hep
    * Second week of December

* Clement: Common generation interfaces?
    * for DIRAC? For the Generator? For the Output?
    * Slightly getting out of scope :)

 

### Validation system
* Teng/Weidong: Validation system for key4hep
* LHCb CI & Validation: https://indico.ihep.ac.cn/event/11444/session/12/contribution/173/material/slides/0.pdf

## Gaudi

### Gaudi v35
* cmake for dependent packages has to change
* See for example
    * https://github.com/key4hep/k4FWCore/pull/19
* plan to add some documentation to
* https://gitlab.cern.ch/gaudi/Gaudi/-/merge_requests/1152
* [ ] How to approach transition?
    * concerns: CEPCSW, FCCSW, k4FWCore, k4GaudiMarlinWrapper. (k4SimDelphes ... yet to be written)
    * Can provide Gaudi v35 on CVMFS
    * changing the CI setup scripts introduces a lot of noise though
    * Could disable packages that have not transitioned yet
    * test locally, and make sure the nightlies run correctly once all packages have updated
    * Move forward to Gaudi 35, and don't look back? Don't worry about keeping something compatible with earlier version

 

## Geometry packages

## First release

* Content:
    * EDM4hep/PODIO
        * need new tags
    * K4FWCore: dataservice, +++
    * Marlin Wrapper
    * DD4hep plugin for EDM4hep output
    * Example
        * [x]: Simulate a few muons and reconstruct with MarlinWrapper
        * [x] need to find someone working on this (from CLIC ?)
        * Clement: could do sth. based on Delphes output and running ilcsoft vertexing in MarlinWrapers
            * might be available by end of month
* How to tag releases
    * using release script from iLCSoft for individual packages

## AOB

* Proposal: discuss simulation programs
    * Gaussino
    * Gaudi integrations
    * ddsim

## Next meeting

* December 1, 2020

## Call for Logos

 

09:00 → 09:05 Introduction

09:50 → 10:00 Discussion

Speakers: Everyone, Valentin Volkl (University of Innsbruck (AT))

Call for Logos