Choose timezone
Your profile timezone:
185th ROOT Parallelism, Performance and Programming Model Meeting
ROOT PPP Room
PPP meeting 07.08.2025
[Stephan]: So without calling CommitRange the attribute range is not committed automatically?
[Giacomo]: Indeed. In some of my dev iterations I had the range automatically commit on destruction but this involved certain implicit behaviours/assumptions between the user and the AttributeSet ownership.
[Stephan]: You can still let the user know it's not being committed with a warning
[Giacomo]: Unclear whether we want this
[Jonas]: Only the pending range would have the flag if it is commited. In the destructor only of the appending range you need to check. Doing thing in a destructor, probably we don't want this to be an exception just a warning so it's fine, exceptions would be a bit ugly.
[Stephan]: slide 7, you pass by value. That means that the user can commit a range without moving?
[Giacomo]: A pending range is non-copyable, this is done on purpose so that it must be moved-from.
[Jonas]: Same thing with std::unique_ptr, you must always accept it by value.
[Vincenzo]: Slide 9, when calling CommitRange this will check that both the range and the entry are consistent with the attribute set?
[Giacomo]: Yes indeed
[Vincenzo]: Stepping through entries between the main RNTuple and the attribute sets is detached?
[Giacomo]: Everytime we call Fill on the main writer we know the current entry index, so attribute set will know at which specific entry it is before being commmited.
[Giacomo]: The reason we have two calls to BeginRange and CommitRange is so that the user can't just pass a (begin, end) range pair. That could lead to inconsistencies.
[Jonas]: We may need another sentence on thread-safety in our docs. Usually we say conditionally thread-safe. I guess here we can't any two objects that come from the same writer. e.g. commit two ranges from two different threads because we need a consistent number of entries. And even for writing to the underlying TFile, although we could probably do it.
[Giacomo]: when you create the attribute set you also create the header for the attribute RNTuple, then you do everything, then you close both RNTuples at the same time so not thread-safe per se. In fact, one thing we need to understand how to do is how to mix all of this with the parallel writer.
[Vincenzo]: slide 12, GetAttributes returns keys/names of the attributes for the set?
[Giacomo]: No, all the API returns iterators over indices of atttribute entries you need to load to get actual data.
[Vincenzo]: Have you thought about aggregations that might happen on the attribute sets after merging? Not saying ROOT should do these, but maybe we could help users. For example, I imagine that one way to start this aggregation is being able to distinguish which attribute sets belong to which original RNTuple, i.e. with some provenance information.
[Giacomo]: we would like to provide a default attribute set for each RNTuple that would give some basic metadata e.g. its uuid. This could help with establishing the provenance of different RNTuples that get merged together. We know users will want to squash attrsets after merging, but we have no clear idea of what to do yet. Maybe we want to provide some tool that gives an interface to modify the attributes according to certain schemas
[Jonas]: Not sure. Need to rewrite at least the footer, which needs rewriting at least the anchor. Doing this after the fact might be difficult. Generally the design of RNTuple is that once it's written the RNTuple is immutable. We would need to see how to make it work with the on-disk format.
[Stephan]: Probably RNTuple attributes are gooing to be very few w.r.t. data, so even if squashing is a costly operation, it will happen on a few entries compared to the total number of entries of the dataset
[Giacomo]: Deduplicating attributes for alternating entry ranges is something that CMS is interested in. We might decide to handle that. One idea is to simply make the entry range a list rather than a simple range. Before doing anything we need to double check with the experiment if it's exactly what they need.
[Stephan]: Is there a way to get ranges of attributes so that you can open an RNTuple and just see the ranges of the attributes and what their values are? Just to check that everything works as intended (interactive exploration of RNTuple attributes).
[Giacomo]: That would work similarly to a regular RNTuple.
[Stephan]: Can you get to the auxiliary RNTuple and just print it?
[Giacomo]: Right now yes, but that's actually we would like to not allow. In principle the attribute RNTuple should not show up as TKeys in a TFile. There was a point in time where they were showing up and it was a problem with merging. In any case, user would not be able to get the RNTuple directly
[Jonas]: In principle you could have a Show method on the RNTupleAttSetReader.
[Giacomo]: This would also print the internal entry (start and lenght).
[Jonas]: Slide 4 think about changing the name of the untyped record so that it is not the same as the AttrSet_Name.
[Stephan]: I think it's interesting for the user to see range start and length
[Jonas]: Yes we should print it in a way that the user understands
[Stephan]: I guess the most frequent case is one where users walk through an RNTuple (e.g. via RDataFrame). If you call GetAttributes in a hot event loop is it slow or acceptable?
[Giacomo]: Also Jakob pointed it out, currently it's potentially slow. In the typical case it should be fast but could be faster. Currently the API is optimized for random access whereas in many cases we have sequential access. Trying to envision something that works with the RNTuple data source for RDF might be a good idea.
[Jonas]: If you want to optionally place an additional restriction on the attribute sets. e.g. if it has at most one attribute at a given global index. That might allow us to do some optimizations when reading back. On the other hand, you can re-establish if the attribute set has that property when reading back and thus do the optimizations. So maybe the restriction does not help. We could just add writing checks
[Giacomo]: Even taking one step back, did we ever really assess if it's worth supporting overlapping ranges.
[Jonas]: My motivation was in the beginning that if you actually want to use RNTuple attributes as a key-value store than you must have overlapping ranges in order to be able to store multiple keys for the same entry.
[Stephan]: Can't you create more fields in the same range?
[Jonas]: If you know what fields you need in the beginning. Otherwise you need key-value store.
[Jonas]: This is sort of late model extensions without late model extension. We told CMS that for example they can store trigger fields but then they need to store the string every time for every trigger field, even though compression will take care of it.
[Philippe] I don't see why CommitRange doesn't happen automatically. Since we don't support appending to the same RNTuple.
[Giacomo]: One interface reason and one implementation reason. Interface reason is we would like to make the committing explicit, like when calling Fill on an RNTupleWriter which also is explicit. Implementation reason is annoying because we need to change several assumptions on lifetime of objects, even the types exposed to the users. Initially I was doing it, but that implied handing a weak handle to the user, to be used carefully to avoid committing past the end of life of the main writer. The current approach seems like a better trade off and features a very similar interface as the one of RNTupleWriter.
[Philippe]: In the current implementation, there is nothing that prevents the user from making a mistake and having the attribute[set/range] survive the main writer.
[Giacomo]: That actually was also changed recently. There is now in place something that prevents users from using wrongly. The writer handle is not a raw pointer but a weak pointer, so we throw if it's not valid.
[Philippe]: For the writing, you create a model, create attributes sets, then begin range. Before committing the range, is there any way to add additional fields? I guess the answer is creating another attribute set.
[Giacomo]: Attribute sets are indepent of each other.
There are minutes attached to this event.
Show them.