# 2020-08-27@unleash-eagen (post rebase) Large compilation units from master that are still a concern - CombinatorialKalmanFilterTests: 4.4G (-1.0G), 2m7s (-38s) - KalmanFitterTests: 4.0G (-1.4G), 1m53s (-69s) - TrackFindingAlgorithmTrackFinderFunction: 3.8G (-1.9G), 2m1s (-61s) - EventDataView3DTests: 3.7G (-2.1G), 1m39s (-55s) - FittingAlgorithmFitterFunction: 3.6G (-1.9G), 1m50s (-55s) - BoundingBoxTest: 2.3G (-0.2G), 54s (-4s) - PropagationExampleBase: 2.2G (-0.3G), 1m24s (-24s) - FatrasSimulationBase: 2.1G (-0.5G), 1m11s (-15s) - GainMatrixUpdaterTests: 2.1G (-1.3G), 49s (-40s) - MultiTrajectoryTests: 2.1G (-0.2G), 47s (-3s) - AdaptiveMultiVertexFinderTests: 2.0G (-0.8G), 55s (-26s) - MeasurementHelpersTests: 1.9G (-0.1G), 38s (-1s) - StepperTests: 1.9G (-0.4G), 1m11s (-18s) - GainMatrixSmootherTests: 1.7G (-0.3G), 37s (-3s) - MaterialValidationBase: 1.7G (-0.7G), 1m1s (-11s) - AdaptiveMultiVertexFitterTests: 1.6G (-0.9G), 34s (-22s) - DirectNavigatorTests: 1.6G (-0.2G), 35s (-8s) - ExtrapolatorTests: 1.6G (-0.5G), 43s (-10s) - FatrasSimulationTests: 1.6G (-0.4G), 40s (-9s) - IterativeVertexFinderTests: 1.6G (-1.1G), 50s (-15s) - MaterialCollectionTests: 1.6G (-0.6G), 47s (-9s) - PropagationDenseConstant: 1.6G (-0.4G), 39s (-11s) - TrackingVolume: 1.6G, 34s (-4s) Large compilation units from master that are now OK - AdaptiveMultiVertexFinderAlgorithm: 1.5G (-1.1G), 35s (-22s) - FullBilloirVertexFitterTests: 1.5G (-0.6G), 34s (-15s) - IterativeVertexFinderAlgorithm: 1.5G (-0.6G), 32s (-15s) - KalmanExtrapolatorTests: 1.5G (-0.1G), 33s (-9s) - KalmanVertexTrackUpdaterTests: 1.4G (-0.8G), 28s (-18s) - PropagationEigenConstant: 1.4G (-0.2G), 30s (-11s) - VertexFitAlgorithm: 1.4G (-0.5G), 31s (-12s) - KalmanVertexUpdaterTests: 1.3G (-0.6G), 29s (-13s) - LinearizedTrackFactoryTests: 1.3G (-0.3G), 34s (-3s) - PropagationAtlasConstant: 1.3G (-0.3G), 28s (-7s) - PropagationCompareAtlasEigenConstant: 1.3G (-0.3G), 36s (-2s) - PropagationStraightLine: 1.3G (-0.3G), 32s (-4s) - ZScanVertexFinderTests: 1.3G (-0.4G), 30s (-5s) # 2020-08-26@master (commit b244d0d5f1af7f2c30a108e3448cb932c0ebb9fd) Seen >1.5G on system monitor, checked with GNU time - EventDataView3DTests: 5.8G **(+0.7G)**, 2m34s (+1s) - TrackFindingAlgorithmTrackFinderFunction: 5.7G (-0.1G), 3m2s (-4s) - FittingAlgorithmFitterFunction: 5.5G (+0.1G), 2m45s (-1s) - CombinatorialKalmanFilterTests: 5.4G **(-0.6G)**, 3m13s (+2s) - KalmanFitterTests: 5.4G (+0.2G), 3m02s (+9s) - GainMatrixUpdaterTests: 3.4G (-0.1G), 1m29s (+2s) - AdaptiveMultiVertexFinderTests: 2.8G (~same), 1m21 (-1s) - IterativeVertexFinderTests: 2.7G (~same), 1m5s (-6s) - AdaptiveMultiVertexFinderAlgorithm: 2.6G (~same), 57s (-2s) - FatrasSimulationBase: 2.6G (+0.1G), 1m26s (+4s) - AdaptiveMultiVertexFitterTests: 2.5G (-0.1G), 56s (-3s) - BoundingBoxTest: 2.5G (~same), 58s (+2s) - PropagationExampleBase: 2.5G (-0.1G), 1m48s (+9s) - MaterialValidationBase: 2.4G (-0.1G), 1m12s (~same) - MultiTrajectoryTests: 2.3G (~same), 50s (+1s) - StepperTests: 2.3G (-0.2G), 1m29s (+6s) - KalmanVertexTrackUpdaterTests: 2.2G (-0.1G), 46s (-4s) - MaterialCollectionTests: 2.2G (-0.2G), 56s (-3s) - ExtrapolatorTests: 2.1G (-0.2G), 53s (-2s) - IterativeVertexFinderAlgorithm: 2.1G (-0.1G), 47s (-2s) - FullBilloirVertexFitterTests: 2.1G (-0.1G), 49s (-1s) - FatrasSimulationTests: 2.0G (-0.2G), 49s (-6s) - GainMatrixSmootherTests: 2.0G (~same), 40s (~same) - MeasurementHelpersTests: 2.0G (~same), 39s (+1s) - PropagationDenseConstant: 2.0G (NEW!), 50s (NEW!) - KalmanVertexUpdaterTests: 1.9G (~same), 42s (-1s) - VertexFitAlgorithm: 1.9G (-0.1G), 43s (-1s) - DirectNavigatorTests: 1.8G (-0.1G), 43s (-5s) - ZScanVertexFinderTests: 1.7G (-0.1G), 35s (-2s) - TrackingVolume: 1.8G (+0.1G), 38s (~same) - KalmanExtrapolatorTests: 1.6G (-0.2G), 42s (-3s) - LinearizedTrackFactoryTests: 1.6G (~same), 37s (-4s) - PropagationAtlasConstant: 1.6G (NEW!), 35s (NEW!) - PropagationCompareAtlasEigenConstant: 1.6G (NEW!), 38s (NEW!) - PropagationEigenConstant: 1.6G (NEW!), 41s (NEW!) - PropagationStraightLine: 1.6G (NEW!), 36s (NEW!) # 2020-07-26@unleash-eagen (post rebase) ## Usual results Large compilation units from master that are still a concern - CombinatorialKalmanFilterTests: 4.3G (-1.7G), 1m56s (-75s) - KalmanFitterTests: 4.1G (-1.1G), 1m48s (-65s) - EventDataViewTests: 3.8G (-1.3G), 1m41s (-52s) - FittingAlgorithmFitterFunction: 3.8G (-1.6G), 1m52s (-54s) - TrackFindingAlgorithmTrackFinderFunction: 3.7G (-2.1G), 1m55s (-71s) - PropagationTests: 2.5G (-0.2G), 1m19s (-16s) - FatrasSimulationBase: 2.4G (-0.1G), 1m8s (-14s) - PropagationExampleBase: 2.4G (-0.2G), 1m26s (-13s) - StepperTests: 2.4G (-0.1G), 1m12s (-11s) - BoundingBoxTests: 2.3G (-0.2G), 51s (-5s) - GainMatrixUpdaterTests: 2.1G (-1.4G), 47s (-40s) - MultiTrajectoryTests: 2.1G (-0.2G), 44s (-5s) - AdaptiveMultiVertexFinderTests: 2.0G (-0.8G), 54s (-5s) - MaterialValidationBase: 2.0G (-0.5G), 58s (-14s) - MeasurementHelpersTests: 1.9G (-0.1G), 36s (-2s) - MaterialCollectionTests: 1.8G (-0.6G), 50s (-9s) - GainMatrixSmootherTests: 1.7G (-0.3G), 35s (-5s) - IterativeVertexFinderTests: 1.7G (-1.0G), 49s (-22s) - AdaptiveMultiVertexFitterTests: 1.6G (-1.0G), 34s (-25s) - DirectNavigatorTests: 1.6G (-0.3G), 37s (-11s) - ExtrapolatorTests: 1.6G (-0.7G), 46s (-9s) - FatrasSimulationTests: 1.6G (-0.6G), 41s (-14s) - FullBilloirVertexFitterTests: 1.6G (-0.6G), 33s (-17s) - KalmanExtrapolatorTests: 1.6G (-0.2G), 33s (-12s) - TrackingVolume: 1.6G (-0.1G), 32s (-6s) Large compilation units from master that are now OK - AdaptiveMultiVertexFinderAlgorithm: 1.5G (-1.1G), 35s (-25s) - IterativeVertexFinderAlgorithm: 1.5G (-0.7G), 33s (-16s) - KalmanVertexTrackUpdaterTests: 1.4G (-0.9G), 28s (-22s) - LinearizedTrackFactoryTests: 1.4G (-0.2G), 27s (-14s) - VertexFitAlgorithm: 1.4G (-0.6G), 28s (-16s) - KalmanVertexUpdaterTests: 1.3G (-0.6G), 26s (-17s) - ZScanVertexFinderTests: 1.3G (-0.5G), 23s (-14s) No new compilation units seen >1.5G on system monitor ## Clang results What if we built the big ones using clang instead? - CombinatorialKalmanFilterTests: 2.4G, 1m35s - KalmanFitterTests: 2.2G, 1m30s - EventDataViewTests: 2.1G, 1m19s - FittingAlgorithmFitterFunction: 2.2G, 1m22s - TrackFindingAlgorithmTrackFinderFunction: 2.1G, 1m22s - PropagationTests: 1.2G, 1m1s - FatrasSimulationBase: 1.2G, 58s - PropagationExampleBase: 1.3G, 1m2s - StepperTests: **FAILS TO BUILD** - BoundingBoxTests: 1.2G, 37s - GainMatrixUpdaterTests: 1.1G, 36s - MultiTrajectoryTests: 1.2G, 39s - AdaptiveMultiVertexFinderTests: 1.1G, 45s - MaterialValidationBase: 1.0G, 44s - MeasurementHelpersTests: 1.1G, 30s - MaterialCollectionTests: 1.0G, 40s - GainMatrixSmootherTests: 1.0G, 27s - IterativeVertexFinderTests: 1.0G, 41s - AdaptiveMultiVertexFitterTests: 0.8G, 28s - DirectNavigatorTests: 0.9G, 29s - ExtrapolatorTests: 0.9G, 36s - FatrasSimulationTests: 0.9G, 32s - FullBilloirVertexFitterTests: 0.8G, 28s - KalmanExtrapolatorTests: 0.8G, 27s - TrackingVolume: 1.0G, 30s # 2020-07-26@master (commit 99facf77edf6b85d2776512fbfd960d2ffbb6b37) Seen >1.5G on system monitor, checked with GNU time - CombinatorialKalmanFilterTests: 6.0G, 3m11s - TrackFindingAlgorithmTrackFinderFunction: 5.8G, 3m6s - FittingAlgorithmFitterFunction: 5.4G, 2m46s - KalmanFitterTests: 5.2G, 2m53s - EventDataViewTests: 5.1G, 2m33s - GainMatrixUpdaterTests: 3.5G, 1m27s - AdaptiveMultiVertexFinderTests: 2.8G, 1m22s - IterativeVertexFinderTests: 2.7G, 1m11s - PropagationTests: 2.7G, 1m35s - AdaptiveMultiVertexFinderAlgorithm: 2.6G, 60s - AdaptiveMultiVertexFitterTests: 2.6G, 59s - PropagationExampleBase: 2.6G, 1m39s - BoundingBoxTests: 2.5G, 56s - FatrasSimulationBase: 2.5G, 1m22s - MaterialValidationBase: 2.5G, 1m12s - StepperTests: 2.5G, 1m23s - MaterialCollectionTests: 2.4G, 59s - ExtrapolatorTests: 2.3G, 55s - KalmanVertexTrackUpdaterTests: 2.3G, 50s - MultiTrajectoryTests: 2.3G, 49s - FatrasSimulationTests: 2.2G, 55s - FullBilloirVertexFitterTests: 2.2G, 50s - IterativeVertexFinderAlgorithm: 2.2G, 49s - GainMatrixSmootherTests: 2.0G, 40s - MeasurementHelpersTests: 2.0G, 38s - VertexFitAlgorithm: 2.0G, 44s - DirectNavigatorTests: 1.9G, 48s - KalmanVertexUpdaterTests: 1.9G, 43s - KalmanExtrapolatorTests: 1.8G, 45s - ZScanVertexFinderTests: 1.8G, 37s - TrackingVolume: 1.7G, 38s - LinearizedTrackFactoryTests: 1.6G, 41s # 2020-07-22@unleash-eagen (post rebase) Seen >1.5G on system monitor (may understimate max-RSS by a few 100MBs) - CombinatorialKalmanFilterTests: 4.1G - FittingAlgorithmFitterFunction: 3.9G - EventDataViewTests: 3.7G - KalmanFitterTests: 3.7G **(300MB+ regression?)** - TrackFindingAlgorithmTrackFinderFunction: 3.5G - PropagationExampleBase: 2.4G - AdaptiveMultiVertexFinderTests: 2.3G - PropagationTests: 2.3G - StepperTests: 2.3G - BoundingBox: 2.2G - FatrasSimulationBase: 2.2G - GainMatrixUpdaterTests: 2.1G - MultiTrajectoryTests: 2.1G **(100MB+ regression?)** - MaterialValidationBase: 1.8G - MeasurementHelpersTests: 1.8G - MaterialCollectionTests: 1.7G - ExtrapolatorTests: 1.6G - GainMatrixSmootherTests: 1.6G # 2020-07-22@master (commit 9b858dcf5d5b106) Seen >1.5G on system monitor (may understimate max-RSS by a few 100MBs) - CombinatorialKalmanFilterTests: 5.9G - KalmanFitterTests: 5.7G - TrackFindingAlgorithmTrackFinderFunction: 5.6G - EventDataViewTests: 4.9G - FittingAlgorithmFitterFunction: 4.8G - GainMatrixUpdaterTests: 3.3G - AdaptiveMultiVertexFinderTests: 3.1G - PropagationTests: 2.6G - AdaptiveMultiVertexFitterTests: 2.5G - AdaptiveMultiVertexFinderAlgorithm: 2.5G - MaterialValidationBase: 2.5G - PropagationExampleBase: 2.5G - BoundingBoxTest: 2.4G - FatrasSimulationBase: 2.4G - IterativeVertexFinderTests: 2.4G - MaterialCollectionTests: 2.4G - StepperTests: 2.4G - ExtrapolatorTests: 2.2G - KalmanVertexTrackUpdaterTests: 2.2G - MultiTrajectoryTests: 2.2G - FatrasSimulationTests: 2.1G - FullBilloirVertexFitterTests: 2.1G - IterativeVertexFinderAlgorithm: 2.1G - DirectNavigatorTests: 1.9G - GainMatrixSmootherTest: 1.9G - MeasurementHelpersTest: 1.9G - VertexFitAlgorithm: 1.9G - KalmanVertexUpdaterTests: 1.8G - KalmanExtrapolatorTests: 1.7G - TrackingVolume: 1.7G - ZScanVertexFinderTests: 1.7G # 2020-07-17@unleash-eagen (block redesign, reimpl of various operations) So, with the new build optimizations, how close are we to our goal of using a maximum of 1.5GB of RAM per build process? Let's see what goes >1.5 GB of RSS in my system monitor and compare that to our previous Eagen-based build's memory usage. - CombinatorialKalmanFilterTests: 4.3GB (-0.6GB), 2m00s (-59s) - FittingAlgorithmFitterFunction: 3.9GB (-1.2GB), 1m54s (-41s) - TrackFindingAlgorithmTrackFinderFunction: 3.7GB (-1.3GB), 2m05s (-52s) - EventDataViewTests: 3.7GB (-1.4GB), 1m37s (-43s) - KalmanFitterTests: 3.4GB (-1.7GB), 1m51s (-46s) - AdaptiveMultiVertexFinderTests: 2.5GB (-0.6GB), 1m12s (-28s) - PropagationExampleBase: 2.5GB (-0.3GB), 1m27s (-16s) - FatrasSimulationBase: 2.4GB (-0.1GB), 1m12s (-11s) - PropagationTests: 2.4GB (-0.3GB), 1m20s (-26s) - BoundingBoxTests: 2.3GB (-0.1GB), 52s (NEW!) - StepperTests: 2.3GB (-0.1GB), 1m08s (-26s) - GainMatrixUpdaterTests: 2.1GB (-0.7GB), 47s (-31s) - MaterialValidationBase: 2.0GB (-0.4GB), 59s (NEW!) - MultiTrajectoryTests: 2.0GB (-0.3GB), 44s (-5s) - MeasurementHelpersTests: 1.9GB (NEW!), 36s (NEW!) - MaterialCollectionTests: 1.8GB (-0.5GB), 49s (NEW!) - GainMatrixSmootherTests: 1.7GB (NEW!), 35s (NEW!) - ExtrapolatorTests: 1.6GB (-0.7GB), 46s (NEW!) Clearly, <1.5GB/process is still a bit of a stretch, but we're getting closer. # 2020-07-08@unleash-eagen (rebased Eagen) Former >3GB build hogs: - EventDataViewTests: 5.1GB (-0.6GB), 2m20s (-15s) - FittingAlgorithmFitterFunction: 5.1GB **(+0.3GB)**, 2m35s (-15s) - KalmanFitterTests: 5.1GB (-0.6GB), 2m37s (-17s) - TrackFindingAlgorithmTrackFinderFunction: 5.0GB (-0.6GB), 2m57s (-10s) - CombinatorialKalmanFilterTests: 4.9GB (-1.0GB), 2m59s (-16s) - AdaptiveMultiVertexFinderTests: 3.1GB (-0.2GB), 1m40s **(+3s)** - GainMatrixUpdaterTests: 2.8GB (-0.6GB), 1m18s (-19s) Former watchlist: - PropagationExampleBase: 2.8GB **(+0.2GB)**, 1m43s **(+3s)** - PropagationTests: 2.7GB _(~same)_, 1m46s **(+10s)** - FatrasSimulationBase: 2.5GB _(~same)_, 1m23s **(+3s)** - AdaptiveMultiVertexFitterTests: 2.4GB (-0.1GB), 55s (-4s) - StepperTests: 2.4GB (-0.1GB), 1m34s **(+12s)** - MultiTrajectoryTests: 2.3GB _(~same)_, 49s _(~same)_ Other system monitor peaks >= 2.0GB (not checked w/ GNU time): - BoundingBoxTest (NEW): ~2.4GB - MaterialValidationBase (NEW): ~2.4GB - AdaptiveMultiVertexFitterALgorithm (NEW): ~2.3GB - ExtrapolatorTests (NEW): ~2.3GB - IterativeVertexFinderTests (NEW): ~2.3GB - MaterialCollectionTests (NEW): ~2.3GB - FatrasSimulationTests (NEW): ~2.1GB - FullBilloirVertexFitterTests (NEW): ~2.0GB - IterativeVertexFinderAlgorithm (NEW): ~2.0GB - KalmanVertexTrackUpdaterTests (NEW): ~2.0GB # 2020-07-08@master (new master) Former >4GB build hogs: - CombinatorialKalmanFilterTests: 5.9GB (-1.5GB), 3m15s (-23s) - KalmanFitterTests: 5.7GB (-1.1GB), 2m54s (-27s) - TrackFindingAlgorithmTrackFinderFunction: 5.6GB (-1.3GB), 3m07s (-37s) - FittingAlgorithmFitterFunction: 4.8GB (-1.6GB), 2m50s (-38s) - GainMatrixUpdaterTests: 3.4GB (-0.7GB), 1m37s (-5s) - AdaptiveMultiVertexFinderTests: 3.3GB (-0.8GB), 1m47s (-21s) New >4GB build hogs: - EventDataViewTests: 5.7GB, 2m35s (uses KalmanFitter) Other processes that may be worth watching in the future (seen >3GB before): - PropagationTests: 2.7GB, 1m36s - PropagationExampleBase: 2.6GB, 1m40s - AdaptiveMultiVertexFitterTests: 2.5GB, 59s - FatrasSimulationBase: 2.5GB, 1m20s - StepperTests: 2.5GB, 1m22s - MultiTrajectoryTests: 2.3GB, 49s # 2020-06-22 (minimal Eagen) Initially identified build hogs (verified with GNU time): - CombinatorialKalmanFilterTests: peak-RSS 6.6GB (-0.8GB), build time 3m18 (-20s) - TrackFindingAlgorithmTrackFinderFunction: peak-RSS 6.4GB (-0.5GB), build time 3m30 (-14s) - FittingAlgorithmFitterFunction: peak-RSS 6.1GB (-0.3GB), build time 3m06 (-22s) - KalmanFitterTests: peak-RSS 5.8GB (-1.0GB), build time 3m04 (-17s) - GainMatrixUpdaterTests: peak-RSS 3.7GB (-0.4GB), build time 1m30 (-12s) - AdaptiveMultiVertexFinderTests: peak-RSS 3.4GB (-0.7GB), build time 1m46 (-22s) Other >3GB processes (system monitor estimates): - PropagationExampleBase: ~3.9G - PropagationTests: ~3.8GB - MultiTrajectoryTests: ~3.8GB - StepperTests: ~3.4GB - FatrasSimulationBase: ~3.2GB # 2020-05-25 (pre Eagen) Build hogs above 4GB, verified with GNU time. 1. Tests/UnitTests/Core/TrackFinder/CombinatorialKalmanFilterTests.cpp (peak-RSS: 7.4 GB, build time: 3m38s) 2. Examples/Algorithms/TrackFinding/src/TrackFindingAlgorithmTrackFinderFunction.cpp (peak-RSS: 6.9GB, build time: 3m44s) 3. Tests/UnitTests/Core/Fitter/KalmanFitterTests.cpp (peak-RSS: 6.8GB, build time: 3m21s) 4. Examples/Algorithms/Fitting/src/FittingAlgorithmFitterFunction.cpp (peak-RSS: 6.4GB, build time: 3m28s) 5. Tests/UnitTests/Core/Vertexing/AdaptiveMultiVertexFinderTests.cpp (peak-RSS: 4.1GB, build time: 2m08s) * Note: Strangely, the closely related AdaptiveMultiVertexFitterTests takes "only" 2.8GB and 1m10s. This difference may be worth investigating. 6. Tests/UnitTests/Core/Fitter/GainMatrixUpdaterTests.cpp (peak-RSS: 4.1GB, build time: 1m42s) * Note: Strangely, the closely related GainMatrixSmootherTests takes "only" 2.3GB and 53s. This difference may be worth investigating.