templateMI92017-07-20T10:55:25.0396813521P0DLibreOffice/4.2.8.2$Linux_X86_64 LibreOffice_project/420m0$Build-2marian ivanov
-1096
-2923
30350
19699
view1
false
false
true
true
true
true
false
false
true
1500
false
//////////////////////////////////////////8=
//////////////////////////////////////////8=
false
true
false
0
8
false
true
true
4
0
0
1
-1084
-2923
31360
22793
2540
2540
254
254
254
1
254
1
false
1500
false
true
$(user)/config/standard.sob
0
$(user)/config/standard.soc
$(user)/config/standard.sod
1270
false
en
US
$(user)/config/standard.sog
true
$(user)/config/standard.soh
false
false
true
true
false
true
false
false
true
false
false
false
false
false
$(user)/config/standard.soe
false
4
true
0
low-resolution
false
6
true
<number><number>
<number>
10th May 2017
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
<number><number>
User defined tests in AliRoot/AliPhysics
Outlook
Continuous testing - motivation
CTest
Log parsing and regression tests
alihadd(.sh) and corrupted file monitoring
Continuous testing
Anything that can go wrong, will go wrong.
Murphy's law
Continuous testing - execute tests automatically after changes in code
Why?
For example...
A real entertaining example(s) of unexpected failure in ALICE code
Continuous testing using CTest
CTest is a testing tool distributed as a part of CMake.
can be used to automate updating, configuring, building, testing, performing memory checking, performing coverage, and submitting results to a CDash or Dart dashboard system.
Two basic modes of operation for CTest.
CMake is used to configure and build a project, using special commands in the CMakeLists.txt file to create tests.
CTest runs a script
CTEST integrated into build of AliRoot/AliPhysics.
Automatically executed within each pull request
CTest in AliRoot/AliPhysics
CTEST integrated into build of AliRoot/AliPhysics
Basic test in the pull request
currently typical time to run a test O(5-30s)
usually library loading
Can we use CTEST for user defined tests ?
Can we use CTEST for regression tests ?
User defined test within CTEST
[AliPhysics/latest-alicesw] /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot $> cd $AliRoot_BUILD/
[AliPhysics/latest-alicesw] /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot $> ctest -R func_STAT_*
Test project /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot
Start 96: func_STAT_AliTreePlayer
1/2 Test #96: func_STAT_AliTreePlayer ........... Passed 2.52 sec
Start 97: func_STAT_AliTMinuitToolkitTest
2/2 Test #97: func_STAT_AliTMinuitToolkitTest ... Passed 4.26 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 6.79 sec
User defined test within CTEST
[AliPhysics/latest-alicesw] /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot $> cd $AliRoot_BUILD/
[AliPhysics/latest-alicesw] /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot $> ctest -R func_STAT_*
Test project /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot
Start 96: func_STAT_AliTreePlayer
1/2 Test #96: func_STAT_AliTreePlayer ........... Passed 2.52 sec
Start 97: func_STAT_AliTMinuitToolkitTest
2/2 Test #97: func_STAT_AliTMinuitToolkitTest ... Passed 4.26 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 6.79 sec
CTest: Verbose mode
[AliPhysics/latest-alicesw] /data/alicesw/sw/BUILD/AliRoot-latest/AliRoot $> ctest --verbose -R func_STAT_*
….
97: Info in <AliTMinuitToolkitTestLinear>: FitLogLike:Gaus(80%)Cauchy(20%)- MISAC(50)
97: Info in <TCanvas::Print>: png file AliTMinuiToolkit.TestLinearFitStatExample.png has been created
97: Info in <TCanvas::Print>: pdf file AliTMinuiToolkit.TestLinearFitStatExample.pdf has been created
97: [2017-07-20 11:28:55 CEST] [ivanov-laptop] [SUCCESS] statTest.testAliTMinutiToolkitTestLinear: All OK
2/2 Test #97: func_STAT_AliTMinuitToolkitTest ... Passed 4.68 sec
Current status
First 2 user defined tests integrated into AliRoot
see example in pull request https://github.com/alisw/AliRoot/pull/326
func_STAT_AliTreePlayer
func_STAT_AliTMinuitToolkitTest
It works!
Next in queue
tests for STAT classes - AliElasticSearch, AliDrawStyle, AliNDLocalRegression
extension of existing tests AliExternalInfo and AliTreePlayer tests
User defined CTest
Questions:
maximal limit of the CPU time?
O(1-10 min)
naming convention for tests ?
running different parts of a test on a pull request and full test once per (day/release)
optional archiving of the log files ?
EOS
log file indexing ?
json documents for further regression test
Regression test
Regression testing: verify that software still performs the same way (including CPU and RAM consumption)
What if the results of an algorithm is not well defined resp. not known?
Dead bands to raise alarm ?
can not be used in CTEST directly
Alarms based on the difference to reference more appropriate
Proposal: Regression tests in ALICE
Integrate tests into standard CTEST
Export critical variables for regression tests into a DB
variables should be part of the log file
using log parser to create json files
logstash and grok currently investigated
Elastic stack (ELK, ElasticSearch, logstash, Kibana) should be well suited for regression testing
ELK (to be) used for GRID monitoring
https://www.elastic.co/blog/grid-monitoring-at-cern-with-elastic
I plan to use ELK for monitoring of user jobs at GSI farm resp. on GRID
e.g corrupted files/cpu/memory/user defined tag/values
alihadd(.sh) and corrupted file monitoring
alihadd for merging and corrupted file monitoring
Recently: problems observed merging trees from a user grid filtering production
3% of the Filtered.root files corrupted
Generic solution:
Prepare a list of corrupted files
investigate reasons of failures parsing the log files for affected production
Merge only good files
Status:
code developed with TPC GIT
alihadd.sh and alihadd.test.sh
successfully used for merging of the production above
To use the code in GRID → code into $ALICE_ROOT/STEER/Utilities
pull request https://github.com/alisw/AliRoot/pull/331#issuecomment-316670080
Plans: alihadd for merging and corrupted file monitoring
Optional use alihadd in user defined grid jobs
bad/corrupted files to be registered in DB (e.g ELK)
merging to be done only for Good files
Investigate option to use “robust” merging in AlienPlugin/Lego train
alihaddsh -k -s 200000000 -checkKeys highPt -prefetch 1 -v 1 -timeOut 30 test.root @input.list 2>&1 |tee testBadChunks5.log
# check output - valid only for particular input data
status=0
cd $wdir/testBadChunks5/
[ `wc -l <input.list.Good` != 5 ] && alilog_error "alihadd.test.testBadChunks5: Invalid number of good files input.list.Good!=5"
[ `wc -l <input.list.Bad` != 5 ] && alilog_error "alihadd.test.testBadChunks5: Invalid number of bad files input.list.Bad!=5"
Example usage of alihadd.sh as in test
$AliRoot_SRC/STEER/Utilities/test/alihadd.test.sh;