Choose timezone
Your profile timezone:
Powered by Indico
v3.3.4-pre
Live nodes and previous meeting summary are stored in codimd.
Zoom link in announcement email; please contact rootdev@cern.ch if you did not receive it!
RNTuple unaligned friends. Concrete ideas for the design. Discussions with Vincenzo and Jakob.
Discussion about RNTupleProcessor, to have friends/chains in a composable way.
Ultimately the indexed RNTuple that can be joined should play together with the RNTupleProcessor.
RNtupleProcessor: interface that is basically an iterator. Event-by-event look over the composed RNTuple (horizontal/vertical joined RNTuple).
[Philippe] You said something like the current code does not live in the processor, why?
[Florine] At this point the two designs are not there, ultimately will depend on each other or be the same thing. I just want to approach initially as the same thing.
[Philippe] The processor currently is focusing on chain, then the index
[Florine] For the initial prototype of the processor we use the friends as they are now, completely aligned, then we can introduce concept of chains.
[Philippe] The index is usually covering one tree. By definition the index is all in memory, the intent is to avoid having to scan through a file to scan it. Technically you can choose to have an index that spans a full chain if you so wish (if it fits in memory). For an arbitrary length chain, you usually create one index per tree of the friend chain, but that leaves the question of in which file is event number 10.
It is assumed somewhere that the event number is monotonally increasing per file. So that they are mixed up within a chain but each file is greater than the previous. So you can a simple forward linear search for which file it is.
[Florine] It is persistified in a way right? You build an index and it can be persistified to a TTree
[Philippe] Yes, but when you load a TTree you load the entire index into memory
Running benchmarks for parallel writing.
Skimming of the AGC data only taking the branches that the AGC actually needs, applying some loose cuts, then use those cuts to drop some electrons and jets would not be used by the AGC. THis seems to scale pretty nicely in ntpl-perf01 from 900GB to 19GB for the RNTuple dataset.
MiniAOD to NanoAOD. THe benchmark is working, testing on 300GB of opendata CMS Run 2. This brings it down to 3 GB, only writing very few branches. Right now debugging the TTree reading code that is passing through many locks due to the way I'm reading MiniAOD without the CMSSW framework. Found a part that is taking far too many locks on the file, hopefully this should show nice scaling behaviour of the writer. Ideally every thread opens a file separately and they can write into one RNTuple safely.
[Philippe] Where are the locks you are seeing?
[Jonas] I manage to read enough data out of MiniAOD by using MakeClass. If you happen to do it wrong when it tries to find dictionaries for every class, it will take a lock for every branch of every file opened and find if the class has a dictionary or not. By limiting to only the branches needed then I don't hit the lock.
Initially I got the TTree and I called SetMakeClass on all branches, which I don't need. Also there are many branches in a MiniAOD. What I do now instead is I call SetMakeClass on the leaf branches.
[Philippe] A priori you don't need SetMakeClass. It should now be implicit when you set the address of the branch to a numerical type it should automatically call SetMakeClass on that branch.
[Jonas] Initially when I tried it I got some segfaults, I also need to read the leaf count. But I need to double check. For now I'm happy I get the data, even if slowly and figure out where the locks are and which options I can set to improve.
I have been debugging an issue with an ATLAS analysis use case with TTreeIndex. It seems the Streamer method triggers an integer overflow at https://github.com/root-project/root/blob/3fa99d2122581a8ef0133e4ea6ac53fe46a7d64f/tree/treeplayer/src/TTreeIndex.cxx#L607
[Philippe] Yes, that and the 1GB limit which is even stricter. So this is very much intertwined with https://github.com/root-project/root/pull/14627
Action items:
* Change the signature of all TBuffer*::WriteFastArray to take a Long64_t instead of Int_t so to avoid truncations happening at the call site
* Make sure we launch Fatal instead of silently returning when the method is writing more than 1GB
* Change the implementation of RDataFrame IMT so that the TTreeIndex code does not trigger I/O, currently happening because of a call to TTreeIndex::Clone at https://github.com/root-project/root/blob/570341049a21a7c9f1423947021990c3e3096b49/tree/tree/src/InternalTreeUtils.cxx#L400
Distracted with reading PRs and fixing the CI.
[Philippe] How was the Fixathon?
[Others] It went very well, lots of nice contributions from outside the team!
[Philippe] Are we tracking those?
[Vincenzo] Yes, at https://github.com/orgs/root-project/projects/8.
[Florine] About this issue https://github.com/root-project/root/issues/14555 , what do you think Philippe?
[Philippe] We need to understand how much the feature of printing STL containers is important/used
[Florine] It's implemented in RNTuple via Show
[Philippe] Then it is fixed!
[Vincenzo] About this https://github.com/root-project/root/issues/14567, can we close it?
[Philippe] It seems that libraries are properly found in both build/install, do we use any command line tool?
[Vincenzo] We don't install command line tools anymore
[Philippe] Then we can close that!