# Summary of build issues in >1.5G compilation units ## Overview of recurring problems - The "ordered sublist" variant used for measurements is very costly * For fitter examples, accounts for ~50% of template processing time * The reason it's so costly is that every method code and std::visit lambda is duplicated 63x. * Paul think we can get this down to 6x, which would be helpful. * See https://github.com/acts-project/acts/issues/350 . * Quantitative impact on CKF tests: - 3.0s of Source time (eager InstantiateFunction) - 0.9s of ParseClass time (InstantiateClass) - 0.2s of eager InstantiateClass time - 0.4s of early CodeGen Function time (DebugType) - 1.5s of MeasurementHelpers::getSurface (std::visit) - 1.6s of `~_Variant_storage` (visit impl) * Extra 100ms comes from instantiating multiple operator== of matrix - 1.8s of `vector::emplace_back` * 1.2s of allocator_traits::construct, 0.6s of `_M_realloc_insert` - Some ~100ms of misc overhead from std::variant that includes it - 0.7s of `std::visit` at CKFSourceLinkSelector * 0.3s of which is instantiating various Matrix classes, will stay - 0.3s of `std::visit` at CombinatorialKalmanFilter * If we sum all of that up, we get 10.1s of overhead which is directly attributable to the use of that large `std::variant`. For comparison, total frontend time is 30s, of which 26s is clearly spent instantiating stuff. - The `debugLog` mechanism used throughout Propagator seems surprisingly costly * Hard to quantify exact impact as it's spread out across lots of tiny `std::function` constructor instantiations. * `debugLog` impl itself is ~0.1s per instance when using boost string algs. * See https://github.com/acts-project/acts/issues/347 for details. * Paul is on it: https://github.com/acts-project/acts/pull/348 . - Some Eigen geometry methods must be unbloated like matrix methods were * These methods show up in every compilation profile: - `Transform::rotation` (0.7s in CKF tests) - `Transform::inverse` (0.7s in CKF tests) - `Transform::operator*(Transform)` (0.3s in CKF tests) - `Transform::operator*(Matrix)` (0.3s in CKF tests) * These ones are a bit less frequent, but still common - `Quaternion::operator*` (9/24) (0.2s in CKF tests) * There ones are rare, but super-expensive - `Quaternion::setFromTwoVectors` (4.2s in BoundingBoxTests) * Based on JacobiSVD, so will also need to reimplement that one. * These ones are rare and mildly expensive - `Transform::pretranslate` (0.3s) - `Translation::operator*(AngleAxis)` (0.3s) - `Transform::prerotate` (0.2s) - `Transform::operator*(AngleAxis)` (0.1s) - More Eigen matrix operations could also use a reimplementation * Unfortunately, more complex than a matrix sum/product... * These operations show up in lots of compilation profiles: - `Matrix::inverse` (11/24) (1.3s in CKF tests) - `LDLT` aka Cholesky (6/24) (0.7s in CKF tests) - `Matrix::determinant`, common in vertexing (0.6s) * These operations show up sometimes, but rarely and remain quite cheap - `JacobiSVD` (0.2s in CKF tests) - Identity, setIdentity (0.2s in CKF tests) - Norm, normalized, lpnorm (0.1s in CKF tests) - Matrix constructors (0.1s in CKF tests) - Matrix type instantiations & debug info (0.1s in CKF tests) - The use of Map and Array in MultiTrajectory is not free (0.9s in CKF tests) ## Per compilation unit breakdown - CKF Tests (4.3G) * Why it's using lots of RAM: Code bloat from lots of small functions * Main culprits from time profile: - Measurement "ordered sublist" variant (top priority, ~40% of time spent instantiating stuff aka 10.1s) * Not just creating it: everyone who uses it suffers, detail above. - Eigen geometry implementation (total: 2.2s) * `Transform::rotation` (0.7s) * `Transform::inverse` (0.7s) * `Transform::operator*(Transform)` (0.3s) * `Transform::operator*(Matrix)` (0.3s) * `Quaternion::operator*` (0.2s) - Eigen matrix transformations (total: 2.2s) * `Matrix::inverse` (1.3s) * `LDLT` aka Cholesky (0.7s) * `JacobiSVD` (0.2s) - Map and Array in MultiTrajectory reistantiate a lot (total: 0.9s) - Misc Eigen matrix ops (total : 0.5s) * Identity, setIdentity (0.2s in CKF tests) * Norm, normalized, lpnorm (0.1s in CKF tests) * Matrix constructors (0.1s in CKF tests) * Matrix type instantiations & debug info (0.1s in CKF tests) - KalmanFitterTests (4.1G) * Mostly the same as CKF tests - FittingAlgorithmFitterFunction (3.8G) * Mostly the same as CKF tests - EventDataViewTests (3.8G) * Mostly the same as CKF tests - TrackFindingAlgorithmTrackFinderFunction (3.7G) * Mostly the same as CKF tests - PropagationTests (2.5G) * Why it's using lots of RAM: Code bloat from lots of small functions * Main culprits from time profile: - Overuse of std::function in logging (top priority) * Lots and losts of std::functions everywhere in test * Precise impact hard to estimate, but omnipresence is suspicious * Also makes rest of the profile hard to read - Eigen geometry overheads: As in CKF, plus... * `Transform::pretranslate` (0.3s) * `Transform::prerotate` (0.2s) - boost::unit_test * Test data generator produces gigantic complex types * Internally uses other costly stuff like std::tuple and std::bind - Maybe use of boost/string * Header takes a suspicious 0.9s to parse * Some traces in the profile, but not huge - Overuse of std::variant here and there - Traces of interesting Eigen stuff, but hard to see due to the omnipresent std::function overhead. - PropagationExampleBase (2.4G) * Mostly the same as PropagationTests + overuse of std::variant for BField - FatrasSimulationBase (2.4G) * Mostly the same as PropagationTests, but also... - Use of Eagen::applyOnTheLeft (0.1s) - Use of boost::container::flat_multiset (0.1s) - BoundingBoxTests (2.3G) * Why it's using lots of RAM: Code bloat from lots of small functions * Main culprits from time profile: - Eigen geometry implementation * `Quaternion::setFromTwoVectors` (4.2s, top priority) - This thing is huge, it takes 4.2s to build on its own - Reason is that it calls JacobiSVD, and Eigen decompositions are a huge source of code bloat. * `Transform::rotation` (0.6s) * `Transform::inverse` (0.4s) * `Transform::operator*(Matrix)` (0.4s) * `Translation::operator*(AngleAxis)` (0.3s) * `Transform::operator*` (0.2s) * `Transform::operator*=(AngleAxis)` (0.2s) * `Transform::translate` (0.2s) * `Quaternion::operator*` (0.2s) * `Transform::operator=(AngleAxis)` (0.2s) * `Rotation2D::operator*` (0.1s) * `Transform::operator=(Rotation2D)` (0.1s) - Use of Eigen::Array in AxisAlignedBoundingBox::intersect (0.1s) - MultiTrajectoryTests (2.1G) * Mostly a subset of CKF overhead, including - Measurement variant overhead - Eigen geometry overhead - Map and Array overhead - About 0.6s of Boost.Test overhead coming from use of BOOST_TEST macro * Solution: Don't use BOOST_TEST, its equality semantics are a mess - GainMatrixUpdaterTests (2.1G) * Mostly a subset of CKF overhead, including - Measurement variant overhead - Eigen geometry overhead - Matrix inversion overhead - MaterialValidationBase (2.0G) * Profile is very similar to PropagationTests, fixing it will likely help - AdaptiveMultiVertexFinderTests (2.0G) * Some overheads similar to CKF - Eigen geometry - Matrix::inverse * Some overheads similar to Propagator (e.g. debugLog) * Large contribution from `Matrix::determinant` (0.6s) * Use of `std::regex` in VertexingDataHelpers is a bit expensive (0.5s) - This function really doesn't need to be inline, move it to cpp file. - MeasurementHelpersTests (1.9G) * Mostly a subset of CKF overhead, including - Measurement variant overhead, obviously (3.3s worth of it) - Eigen geometry overhead - Map and Array overhead, amplified by equality w/o eval (1.5s) - MaterialCollectionTests (1.8G) * Similar "big things" as in AdaptiveMultiVertexFinderTests, but no regex. * A bit more geometry overhead - `Translation::operator*(AngleAxis)` (0.3s) - `Transform::operator*(AngleAxis)` (0.1s) - IterativeVertexFinderTests (1.7G) * Profile is quite similar to AdaptiveMultiVertexFinder tests - GainMatrixSmootherTests (1.7G) * Mostly a subset of CKF overhead, including - Measurement variant overhead - Eigen geometry overhead - `LDLT` and `JacobiSVD` overhead - TrackingVolume (1.6G) * Mostly a subset of BoundingBoxTests overhead, also bound by the `Quaternion::setFromTwoVectors` monster. - KalmanExtrapolatorTests (1.6G) * Mostly a subset of PropagatorTests overhead, including - debugLog mechanism overhead - Eigen geometry overhead - FullBilloirVertexFitterTests (1.6G) * Profile is quite similar to AdaptiveMultiVertexFinder tests, mostly bound by determinants, inverse, and Eigen geometry. But no Propagator here. - FatrasSimulationTests (1.6G) * Mostly a subset of PropagationTests overhead - ExtrapolatorTests (1.6G) * Mostly a subset of PropagationTests overhead - DirectNavigatorTests (1.6G) * Mostly a subset of PropagationTests overhead * A bit more geometry overhead - `Translation::operator*(AngleAxis)` (0.3s) - `Transform::operator*(AngleAxis)` (0.1s) - AdaptiveMultiVertexFitterTests (1.6G) * Similar to FullBilloirVertexFitterTests