ROOT Team Meeting

Friday 23 Nov 2007, 11:00 → 12:50 Europe/Zurich

32-1A24

11:00 → 11:02 News

11:05 → 11:10 QA for TMVA

Development of stressTMVA.cxx validated with CINT, the compiler, valgrind memcheck and massif.

Speaker: Joerg Stelzer

Slides TMVA_massif.pdf

general remarks: ============ the purpose of the stressTMVA program is to track the stability of the classifier performance over the different releases. Hence testTMVA performs the training and evaluation of all (by our judgment) important classifiers and compares the resulting signal selection efficiencies (at given background efficiency) for each classifier with a reference value. status: ===== Currently the output of the program is ****************************************************************** * TMVA - S T R E S S suite * ****************************************************************** Test 1: Data Preparation ............... OK Test 2: Training Methods ............... OK Error in <TWebFile::TWebFile>: http://root.cern.ch/files/TMVAtestRef.root does not exist Test 3: Compare to Reference ........... FAILED ERROR: could not find reference file http://root.cern.ch/files/TMVAtestRef.root since I have not yet implemented the reference comparison completely. The training and testing happens on 1000events each, the time spent is about 1 minute. What other information would you like to get from these tests? valgrind: ====== I tested TMVA with valgrind (valgrind --tool=memcheck --leak-check=yes --trace-children=yes --num-callers=12 --show-reachable=yes stressTMVA) ==2999== LEAK SUMMARY: ==2999== definitely lost: 569,946 bytes in 9,904 blocks. ==2999== indirectly lost: 19,016,348 bytes in 47,505 blocks. ==2999== possibly lost: 988,600 bytes in 19,470 blocks. ==2999== still reachable: 16,750,241 bytes in 87,899 blocks. ==2999== suppressed: 0 bytes in 0 blocks. and (valgrind --tool=massif --trace-children=yes --num-callers=12 stressTMVA) Plot is attached. The memory leaks reported by valgrind are significant, however I am not sure yet how that scales with the number of events. Most of the leaks appear (at first glance) during the setup, and should not scale with the number of events. However, I have called on the four main developers to spend the weekend to remove all leaks and we should be done within a week.

11:10 → 11:15 QA for ROOFIT

Development of $ROOTSYS/test/stressRoofit.cxx validated with CINT, the compiler, valgrind memcheck and massif

Speaker: Wouter Verkerke

****************************************************************** * RooFit - S T R E S S suite * ****************************************************************** ****************************************************************** * Starting S T R E S S basic suite * ****************************************************************** Test 1 : Generate,Fit and Plot on basic p.d.f...................OK Test 2 : Addition operator p.d.f................................OK Test 3 : Product operator p.d.f.................................OK Test 4 : Conditional product operator p.d.f.....................OK Test 5 : Analytically convolved p.d.f...........................OK Test 6 : Simultaneous operator p.d.f............................SKIPPED Test 7 : Addition oper. p.d.f with range transformed fractions..SKIPPED Test 8 : Multiple observable configurations of p.d.f.s..........OK Test 9 : Formula expressions....................................OK Test 10 : Functions of discrete variables........................OK Test 11 : Workspace persistence..................................SKIPPED Test 12 : Extended likelihood constructions......................OK Test 13 : Data import and persistence............................OK Test 14 : Plotting with variable bin sizes.......................OK Test 15 : Likelihood ratio projections...........................OK Test 16 : Complex slice projections..............................OK Test 17 : Interactive fitting....................................OK Test 18 : Lagrange multipliers...................................OK Test 19 : Chi^2 fits.............................................OK Test 20 : Use of conditional observables.........................OK Test 21 : Constant term optimization.............................SKIPPED Test 22 : Numeric Integration....................................SKIPPED ****************************************************************** * Darwin macbrun.local 8.10.1 Darwin Kernel Version 8.10.1: W ****************************************************************** ****************************************************************** * ROOTMARKS = 747.9 * Root5.17/05 20071016/1544 ****************************************************************** Q A S t r a t e g y f o r R o o F i t ******************************************** * stressRooFit package for QA testing (now operational) ---------------------------------------------------- - Replace set of private macros that were used so far for testing. - Monolithic version in $ROOTSYS/test, modular development version in $ROOTSYS/roofitcore/test. - Now implements 17 tests excercising most aspects of RooFit. Another 5 tests are foreseen by the end of next week to extend coverage - Several of these tests excercise very complex projections and other advanced manipulations of p.d.f.s. Suite timing is kept short by reducing default number of input events for fits and projections. - Input data is kept to minimum by generating data for fitting and plotting from p.d.f rather than taking it from external source. Testing framework sets seed of random generator to standard value at begin of each test to ensure reproducibility. - Basic test suite takes ~30 seconds, reference file ~100 Kb, mostly plots and fit results. - Each test is based on RooFitUnitTest utility class that can perform automated regression tests on - RooPlots with (multiple) data and p.d.f projections, - RooTables - RooFitResults (full output including correlation matrix is tested) - Double_t values - Test will fail if runTest() function returns false (currently not used), any ERROR or FATAL messages are logged through RooMsgService, or any of the regression tests fail. - Reference file automatically produced by running stressRooFit with '-w' option. The reference file itself is easily validated manually as all plots and tables are easily browsed or printed interactively. * Memory leaks are checked by running valgrind on stressRooFit. ------------------------------------------------------------- - I run each test separately using the '-n <#> -mc' arguments where <#> is the test number and -mc activates the memory check mode. - In memory check mode - RooTrace memory tracing is activated (keeps track of all RooFit objects) - Post-run regression tests are not performed as reference file I/O appears to leave some objects in memory after file is closed that clutter valgrind output. Objects offered for regression test are immediately deleted by test framework to avoid false leak reports. - I run valgrind with following arguments valgrind --suppressions=valgrind-suppression_ROOT.supp --suppressions=valgrind-suppression_ROOT_optional.supp \ --tool=memcheck --leak-check=full --show-reachable=yes then I apply a home grown awk-based filter on output to weed out CINT-related leak reports that involve RooFit objects in the trace as well as couple of other related patterns (TClass etc, full list below) . I also filter out any leak pattern that does not involve any RooFit class. Here is the relevant fragment of the filter script. buf[] is the array containing each leak report token with buf[1] being the 'at' line of the report. --- CINT leak ignore rules based on line 2 matching --- if (match(buf[2],"G__store_dictposition")) doprint=0 ; if (match(buf[2],"G__call_setup_funcs")) doprint=0 ; if (match(buf[2],"G__search_tagname")) doprint=0 ; if (match(buf[2],"G__allocvariable")) doprint=0 ; if (match(buf[2],"G__memfunc_para_setup")) doprint=0 ; if (match(buf[2],"G__add_ipath")) doprint=0 ; if (match(buf[2],"G__getparameterlist")) doprint=0 ; if (match(buf[2],"G__params::operator")) doprint=0 ; if (match(buf[2],"G__stack_instance")) doprint=0 ; if (match(buf[2],"G__createfuncmacro")) doprint=0 ; if (match(buf[2],"G__malloc")) doprint=0 ; if (match(buf[2],"G__createtemplateclass")) doprint=0 ; if (match(buf[3],"_GLOBAL__I_gCINTMutex")) doprint=0 ; if (match(buf[2],"G__add_setup_func")) doprint=0 ; if (match(buf[2],"G__search_typename")) doprint=0 ; if (match(buf[2],"G__herits::operator")) doprint=0 ; if (match(buf[2],"G__savestring")) doprint=0 ; if (match(buf[2],"G__set_class_autoloading_table")) doprint=0 ; --- Other ROOT leak ignore rules based on line 2/3 matching --- if (match(buf[2],"TFormula::Compile")) doprint=0 ; // Note that TFormula seems to have a leak if (match(buf[2],"TFormula::Optimize")) doprint=0 ; if (match(buf[2],"TCint::CreateListOfBaseClasses")) doprint=0 ; if (match(buf[2],"TClassTable::TClassTable")) doprint=0 ; if (match(buf[2],"TCint::SetClassInfo")) doprint=0 ; if (match(buf[2],"TStringRef::GetRep")) doprint=0 ; if (match(buf[2],"TGuiFactory::CreateApplicationImp")) doprint=0 ; if (match(buf[3],"TApplication::CreateApplication")) doprint=0 ; --- ROOT leak ignore rules based on any line matching --- for (j=0 ; j<iline ; j++) { if (match(buf[j],"GenerateInitInstance")) doprint=0 ; if (match(buf[j],"ROOT::TGenericClassInfo::Init")) doprint=0 ; if (match(buf[j],"ROOT::TDefaultInitBehavior::CreateClass")) doprint=0 ; if (match(buf[j],"TClass::GetStreamerInfo")) doprint=0 ; if (match(buf[j],"TClass::Streamer")) doprint=0 ; if (match(buf[j],"TClass::CalculateStreamerOffset")) doprint=0 ; } The stressRooFit suite shows no valgrind memory access errors or conditional jumps on uninitialized values. Also, with this level of filtering stressRooFit currently runs clean of leaks. To achieve this I fixed O(10) small leaks in various places. In addition I have installed an automatic cleanup procedure that destroys all singleton instances that RooFit creates using an atexit() handler which show up as leaks in --show-reachable=yes mode. Massif profiling of stressRooFit shows a mostly flat time profile. The total amount allocated is dominated by the large 2D lookup tables used for the fast complex error functions that are used by some of the B physics p.d.f.s in the tests (which are also deleted by the atexit() handler) * To be done ----------- - Short term: Still need to tune tolerance settings for automated comparisons to avoid false alarm due to numeric noise variations on different platforms. - Short term: Finish 5 additional test in basic suite. Time scale one week. - Long term: Implement 'Extended' suite of tests with another O(10) tests that are too CPU intensive to be run routinely (couple of months)

11:25 → 11:30 QA for the next Development and Production releases: CORE

Development of a procedure such that: -configure --prefix=xxx, make, make install is certified for each installation with a procedure running the test suite on all binary files. -procedure to verify that the tar.gz or .msi files are OK for all platforms -It would be nice to have the two above procedures run by the nightly builds. -make sure that the tar,msi files contain all the files to run the tutorials and the test suite and that these files do not contain unnecessary files (not deleted by make distclean). -ROOT validated with gcc4.2

Speaker: Fons Rademakers

11:35 → 11:40 QA for the next Development and Production releases:CINT

Description of the procedure (via roottest). It would be good to have a $ROOTSYS/test/stressCint as well (preferred).

Speaker: Axel Naumann

pictures CINTtiming.gif

My "task": "Description of the procedure (via roottest). It would be good to have a $ROOTSYS/test/stressCint as well (preferred)." No stressCint for now. roottest run with "make valgrind", roottest run with "make TIME=1" to study CINT. Roottest now runs stress suite (interpreted) and some tutorials. Works fine except for known problems with stressGeometry and stressGraphics. ROOT QA (and esp CINT tests, roottest, valgrind, timing, etc) now documented in the Wiki. Also contains new valgrind suppression file. Should be checked into $ROOTSYS/etc at some point. Rene Brun wrote: > > Please send me the status of your QA procedure and work still to be done > > as a simple text file before the meeting tomorrow. All of roottest and cint test passing on all testable platforms. Roottest with valgrind reported only a few valgrind errors in CINT - and some of these might be caused by the same issue: G__allocvariable G__conditionaloperator G__exec_asm G__letvalue G__rate_parameter_match G__templatesubstitute G__type2string It is not clear whether fixing them before the December release is a good idea. Usually it is not possible to fix issues in CINT without side-effects - and be it for progress with CINT 7.1. We will evaluate these errors in the coming weeks (during the "workshop"?). Timing performance studies with optimized linux 32 bit on roottest/cint, comparing v5-14-00-patches with trunk: see attached picture. Only two tests show deterioration - not much. Average: 25% improvement - probably mostly due to startup time of root. Cheers, Axel. as requested: an updated version of the valgrind report on some of roottest, now using the suppression files from the ROOT Wiki. <http://axel.web.cern.ch/axel/valgrind-20071121-192637.log.summary.txt> aka ~axel/public_html/valgrind-20071121-192637.log.summary.txt and <http://axel.web.cern.ch/axel/valgrind-20071121-192637.log> aka ~axel/public_html/valgrind-20071121-192637.log Cheers, Axel.

11:40 → 11:45 QA for the next Development and Production releases:I/O

make sure that the existing roottest runs with no errors on Linux, Windows and MAC.

Speaker: Philippe

11:45 → 11:50 QA for the next Development and Production releases:Graphics

stressGraphics extended to test ALL graphics primitives including cloning pads and canvases, test of SavePrimitive with cross-check that teh filse generated produce the same results as the parent script. test of all output systems (.ps,.gif,etc) in batch mode. stressGraphics running in interpreted mode and with ACLIC, validated with valgrind with memcheck and massif.

Speaker: Olivier

stressGraphics status: ====================== stressGraphics has been extended to test almost ALL graphics primitives including cloning pads and canvases. stressGraphics has now 39 tests each of them testing one particular graphics primitive or graphics feature. stressGraphics is meant to be run in batch mode. The procedure: ============== For each test, stressGraphics does the following: 1) Run the test and generate the corresponding PS file. 2) Compare the size of the PS file with a reference size stored in the file stressGraphics.ref. An error is allowed because some tests use random numbers and also the size may depend on which machine the test runs. The error for each test is also stored in stressGraphics.ref. 3) The .C file corresponding to the test is generated (SavePrimitives). 4) The PS file is generated form the .C file obtained. For points 2) and 4) the test is considered as OK if the PS file size correspond to the file size stored in stressGraphics.ref (within the error) otherwise the test is marked as FAILED. When a test FAILED all the generated PS and .C files are kept for further investigation otherwise (if the test is OK) they are removed. stressGraphics should be run as standalone compiled (option -g -O2) program and also in the CINT context: $ ./stressGraphics root [0] .L stressGraphics.cxx root [1] stressGraphics() root [0] .x stressGraphics.cxx These three ways should be also run under valgrind. This is where most time is spent. These procedure should be done on all platforms having valgrind installed. We found different result on 32bits linux and 64 bits linux. To be added: ============ For each test try also other output formats and do the same kind of size comparison now done for PS. The other formats we should try are: pdf, png, gif, jpg. May be also some printing test (cf the current problem with PDF printing with CUPS on SLC4). Test the memory usage (leaks). A first attempt was done two weeks ago but it was not completed because it was more urgent to set up the basic framework described above.

11:50 → 11:55 QA for the next Development and Production releases:GUI

Implementation of a web page describing the certification tests: a set of actions/sessions to be run to systematically exercise all GUI widgets. This includes runs with valgrind memcheck and massif. Testi of memory leaks when using the mouse extensively with the FitPanel or the various GedEditors using sliders.

Speaker: Ilka

Here is the link of the page I am working on. http://antcheva.web.cern.ch/antcheva/GUITests.html Cheers, Ilka

11:55 → 12:00 QA for the next Development and Production releases:PROOF

Development of a stressProof and stressXrootd -stressProof including -the lightweight Proof (by Long or similar) -a test with chains and entrylists. -stressXrootd: setup an xrootd daemon at pcroot

Speaker: Fons

(macbrun) [209] stressProof ****************************************************************** * Starting P R O O F - S T R E S S suite * ****************************************************************** * Log file: /tmp/ProofStress_W62DuT ****************************************************************** Test 1 : Open ............................................. OK * Test 2 : GetLogs .......................................... OK * Test 3 : Simple ........................................... OK * Test 4 : H1:http .......................................... OK * * All registered tests have been passed :-) * ******************************************************************

12:00 → 12:05 QA for the next Development and Production releases: MATH

Development of stressMathCore running with CINT and ACLIC, validated with valgrind memcheck and massif. About 40 tests a la stressLinear to be implemented.

Speaker: Lorenzo

12:20 → 12:40 QA for the Geometry package

Extend stressGeom with many more cases and run it under valgrind memcheck and massif.

Speaker: Andrei Gheata

For TGeo, QA is done via stressShapes.cxx and stressGeometry.cxx - stressShapes.cxx tests the navigation functions Contains() and DistFromInside/Outside() against reference data. It generates random points and directions that may differ on different testing machines and therefore the comparison with the reference is done on statistical basis. The limitation of the test is that not all shape topologies can be really stresses (there are too many possibilities) This test works well on all platforms as far as I know. - stressGeometry.cxx is a much more complex test checking most global navigation functions for several real experiment geometries using the same reference input data + results. This test currently shows some differences if run on different machines which will be fixed one by one. This test is now extremely powerful and I estimate that once it will produce consistent result on all machines will really be a reliable QA proof of the stability of the system. Andrei

12:40 → 12:45 QA for GL

Checklist (web page) describing the validation procedure for all GL operations that can be run by any member of the team

Speaker: Matevz Tadel

Unfortunately I didn't get very far with stressGL today: http://mtadel.home.cern.ch/mtadel/stressGL/stressGL.html I'll come to the group meeting tomorrow ... Cheers, Matevz

12:45 → 12:50 AOB