Jonas Hahnfeld: About slide 25. modules.idx is regenerated once per installed wheel. Autoparsing is paid once per ROOT session, so it cumulates. Thus the PCH-based approach is inherently more costly than the module-based one.
Vassil: Another metric to look into for comparison is peak RSS. Modules are fixing mostly peak memory use of PCH. The part about having the forward-declarations of types in the libraries is not always reliable. For instance, we cannot autoload standalone function declarations, any code with concepts does not work very well etc.
Stephan: What happens with the module of the standard library? I guess it's generated, otherwise we may run into similar issues that were described in the past with different build and runtime glibc/standard C++ libs.
Vincenzo: That's an orthogonal problem, out of scope for this work.
Stephan: What was harder to achieve between the two split wheel approaches you tried, PCH or PCM?
Jake: modules took longer but also because I didn't know anything about it. For the PCH part I was assisted by LLM, but the only issue I found was I had to remove the writeemptyPCM option in CMake for the feature libraries. In terms of complexity, they are similar but the modules option is more hacky at the moment.
Vincenzo: Thanks to Jake we now have a working demonstrator, can we think of a path towards production?
Jonas Hahnfeld: Related to the modules-based approach, I believe you can actually do without the modules.idx file completely. Today I revived an environment variable that if detected can instruct cling to avoid using modules.idx, but instead preloading all modules. It has a cost (~30 MB in my tests), but would avoid the need to deal with that in the packaged wheels.
Vincenzo: I am worried about the amount of patching with string-based manipulation that Jake had to do. One potential improvement is using CMake install components more concretely, does it make sense?
Stephan: In the past we have used install components, but they currently don't make any sense (see usage of INSTALL components HEADERS). I believe it would be some work to clearly define them, but there's no major showstopper.
Vassil: I think we can make it so that no string patching is required for modules. One thing we did but didn't merge was in that direction. The idea was to have a single modulemap with no header dependency on the components but then each library would also provide its own payload modulemap.
Vassil: For the longer vision, what we had written in the very old modularization strategy of ROOT. Because we have a compiler in the infrastrucutre, nothing prevents to get the core installed and rebuild everything on top.
Vincenzo: my mental model after Jake's presentation and listening to the opinions here is that the CMake build of ROOT could have clearly defined INSTALL COMPONENTS, so that smaller separate wheels could be produced. Then at runtime, import ROOT would need to check which parts of ROOT are present in site-packages. ROOT clang modules would be rebuilt, and thanks to a modular distributed set of .modulemap file we would avoid the need to patch a single file every time. This at least in my mind sounds like it would provide a stable runtime system irrespective of how many ROOT components are present.
Vassil: what kind of support you need from the team to land your work?