EDM4hep Discussion

Tuesday 12 Jan 2021, 09:00 → 10:00 Europe/Zurich

Zoom

EDM4hep Live Notes
==================

Date: January 12, 2021
Indico: https://indico.cern.ch/event/983626/

This is a document for taking notes during EDM4hep meetings.

Connected: Joseph, Andre, Tao, Weidong, Teng, Jiaheng, Frank, Birgit, Placido, Sang Yun, Thomas, Xingtao, Valentin, Clement, Brieuc, Gerri, Hyeonja

Apologies:

## Introduction and General Points

* Happy new Year!

### Key4hep presentation at Epiphany

* https://indico.cern.ch/event/934666/contributions/4154229/

* Add to https://key4hep.github.io/key4hep-doc/talks-and-presentations/README.html
    * Also add a link to the FCC presentation https://indico.cern.ch/event/934666/contributions/4154228/

### Virtual Chep

* papers(!) due February 28, 2021
    * Submissions should consist of a paper of 6-10 pages, following the instructions for authors. Paper submission is open until 28 February 2021.
    * https://indico.cern.ch/event/948465/page/21562-instructions-to-submitters

* Contribution:
    * Key4hep overview
        * Few paragraphs per contribution?
    * Delphes parts?
    * podio+edm4hep!

* TODO: share the overleaf from the last CHEP proceedings
* TODO: create repository for published documents?

* Why not CERN gitlab/DESY stash?
    * Need cern or desy computing accounts to access non-public repositories
    * (At least at the moment for CERN gitlab, might change)
* Private github repository free now?
  * Unlimited public/private repositories
  * See: https://github.com/pricing

## Progress and Discussion

## Podio

### Benchmarking
[Thomas] Added some scripts and tools to have almost automatic benchmarking and some results using k4SimDelphes to "generate" data for I/O. Could be made fully automatic with a bit more work and if we want it to be fully automatic.
* https://github.com/tmadlener/podio_benchmarking
* Example results: https://github.com/tmadlener/podio_benchmarking/tree/master/results/k4SimDelphes/ee_Z_bbbar

* Teng: Add to github actions, upload plots as artifacts?
* Thomas: takes about 1 hour to run (depending on number of events)
* Teng: Can set schedule time in the yaml file. Very useful for validation

* Andre: The root peak structure understood?
* Thomas: Not yet fully understood/investigated. "working hypothesis" is that root has to resize some internal buffers if an Event/Collection has a certain size, resp. when it hits an Event with a certain size for the first time.

### Plans for multi-threading testing of EDM4hep for CEPC

* Using Gaudi-hive functionality to speed up garfield simulation of the driftchamber
* Any documentation sugestions for how this can be done?
* Valentin: sounds very useful, not much experience in FCC
* Andre: What is the status of multi-threading for podio?
* Thomas: Reading is safe by design, writing has to be synchronised (e.g. by Gaudi)
* Frank: Root or also SIO backend?
* Thomas: Difference between standalone readers and what happens in the Gaudi world
* Valentin: Inside Gaudi it should work

### Issues

#### more fixes to unsigned and shadowed variables
* https://github.com/AIDASoft/podio/pull/158

#### c++ concepts
* BH: add compile time checks for class behaviours: e.g., movable

#### CollectionIterator does not fully satisfy iterator concept
* https://github.com/AIDASoft/podio/issues/150
* For const collections, would be useful, e.g. using std::find on the collection

#### Idea: True reference collections
* https://github.com/AIDASoft/podio/issues/146

#### issue w/ ROOT and (vectors of) non-copyable collections
* happens in ROOT 6.22
* PM: there is a patch available in LCG repository
    * ROOT team is working on a general solution

#### What are the different branches in the root file?

* Related to use in RDataFrame
* Encode more information in the _relation_ branch names?

#### Multi-Threading

See minutes of https://indico.cern.ch/event/969468/
--> Open issue(s) for further discussion

* Thomas and Benedikt discussing and started to work, nothing to report yet

#### "event class" in podio

* Currently being perceived

#### Writing second file with another tree

* https://github.com/key4hep/K4FWCore/issues/10
* this problem only happens in the GAUDI framework
    * if user tries to write an additional ROOT file
    * to be addressed
* Solved

### PRs
* https://github.com/AIDASoft/podio/pulls

#### Disable collection operator=
* https://github.com/AIDASoft/podio/pull/154
* Review: Benedikt

 

### Meta Data

#### Usage of "metadata" for user defined data
* need to check if current implementation addresses all use cases
* need test use-cases

### Issues following MetaData Developments

* cannot write out event data previously read from file
    * Issue: https://github.com/AIDASoft/podio/issues/103
    * Test: https://github.com/AIDASoft/podio/pull/102
    * TM: have fix, but break framework core handling of collections

* there seems to be a fix in the Gaudi ROOTWriter in FCC-EDM !?
    * historical reason, fixes and developments should be back ported to PODIO
    * ...

### EventStore

### Schema Evolution

- Version for object descriptions, etc.
- Open issue: https://github.com/AIDASoft/podio/issues/86

### Features

* Subset collections?

## LCIOConverters

* k4LCIOReader:
    * https://github.com/key4hep/k4LCIOReader
    * Moved k4LCIOReader code to key4hep project

## EDM4hep
https://github.com/key4hep/EDM4hep/pulls

### EDM4hep tools

* Thomas, Clement:
* Where to put tools using EDM4hep? In EDM4hep or separate repository?
* VV: Would be nice to keep EDM4hep stable, so separate repository.
* FG: Concur. Many releases for LCIO just to update tools.

### Issues

#### k4FWCore: cannot read collections back
* https://github.com/key4hep/k4FWCore/issues/27

### DD4hep output plugin

--> Moved to DD4hep

### Repository is getting too big

* github pages branch is getting large
* Can they be created on the fly?
* Use a different host for the website

### Tracker Hit

* Q: Tracker hit input to tracking algorithms?
* A: In LCIO different traker hits: Planar and Cylindrical using inheritance. Still needs to be adressed how to do this in EDM4hep.
* Q: Is inheritance needed?
* Q: What to use for Driftchambers?
* Open issue:

### Need review of EDM4hepDelphes output

* Output is not stored if no tracks or towers created (?)
    * Allow also other types to be part of the list
    * Allow construction in a more general way (VV)
* Need to assume there is a complete list of reconstructed particles?

* Relations:
    * FCCSW: Issue with more than one RecoParticle pointing to one MCParticle
    * In LCIO to Delphes, conversion happens later than in edm4hep, maybe delphes does de-duplication?
    * TODO: open issue in edm4hep with reproducer (VV)

--> Moving to separate repository

#### Reworking output handling

- TM: better address edge cases
- TM + CH: FCC cases, not sure if delphes or output
- https://github.com/key4hep/k4SimDelphes/pull/5
- TM: How and what to test?
-

### PRs
https://github.com/key4hep/EDM4hep/pulls/

### Release 1.0

* Need:
    * ~~Plugin~~
    * ~~Eventheader~~
    * ~~Meta Data (Event / Run Parameters)~~

## AOB

### Generator infrastructure in FCCSW moved to EDM4hep
* VV: show at next key4hep

### Framework integration of k4SimDelphes

* On hold until delphes output reworked
* Started with updating the Gaudi documentation
    * New cmake in Gaudi v35

### Dual Read-out calorimeter for FCC

* Special data structure used for that simulation
* Tried to use edm4hep, some issues
* Present in a future meeting

### Conditions handling in Belle2
* Benedikt, or Martin Ritter

### Feedback from FCC tutorial for snowmass

* Common question: What are the different branches in the root file?
* ``uproot`` was also used to access the root file
    * Basically re-implemented event store
        * Associations
    * Try to get this into the, e.g., podio repository

### Next meeting:

* January 26, 2020

 

09:00 → 09:05 Introduction

Speakers: Andre Sailer (CERN), Frank-Dieter Gaede (Deutsches Elektronen-Synchrotron (DE))

EDM4hep Github

Podio Github

09:25 → 10:00 Discussion

Speaker: Dr All