Summary of the ATLAS IO performance investigations
==================================================

Here is a summary (seen from the ROOT side) of the progress with ATLAS IO
in the past 2 weeks. From Gerhard Brandt I received 1 file (file1)
  -AOD.067184.big.pool.root (1.3 GBytes) with 9705 branches (split=99)

We have run the read tests on this file, with and without the TreeCache
with versions 5.22/00d and 5.25/03. The results for 5.25/03 are shown in the 
table below when reading local disk files.
Thanks to these files and discussions with Gerhard and Ilija, we came with 
several improvements in 5.25/03 that I presented at the meeting last week
organized by Dirk. Meanwhile, we came with one more improvement in the SVN trunk.
When writing files with 5.25/03 we now flush the basket buffers at regular 
intervals and the first time we do so, we also optimize the size of branch buffers.
This process is automatic.

I converted file1 above using the latest development and created
the corresponding file2.
In order to measure the pros and cons of the compression algorithm, I also
created a copy (file3) of file2 with no compression.
In table1 below, I show the size occupied by CollectionTree only in the 3 files:

    file1:  1265 MBytes, compression = 3.21
    file2:  1070 MBytes, compression = 3.68
    file3:  4069 MBytes, compression = 1.00
                    Table 1
                    -------

In table 2 below, I report the real time, CP time and raw disk time to read all 
branches from these files, making sure that the files are not in the system 
cache (cold files).

    ***********************************
    * file  * TreeCache  * TreeCache  *
    *       *     0          30 MB     
    ***********************************
    * file1 *  RT=658s   *  RT=183s   *
    *       *  CP=166s   *  CP=126s   *
    *       *  IO=510s   *  IO= 60s   *
    ***********************************
    * file2 *  RT=117s   *  RT=109s   *
    *       *  CP=102s   *  CP= 99s   *
    *       *  IO= 15s   *  IO= 12s   *
    ***********************************
    * file3 *  RT=160s   *  RT=155s   *
    *       *  CP= 97s   *  CP= 96s   *
    *       *  IO= 66s   *  IO= 62s   *
    ***********************************
                    Table 2
                    -------

NOTE 1
======
From Table2 one sees that we have gained a factor 6 in real time when
comparing the original ATLAS AOD file with no TreeCache (what ATLAS has been
using till now) and the same file rewritten with 5.25/03 and read with
the default TreeCache. Files written with the new version are between 10 and 20%
smaller.
Using file3, we see that ROOT I/O can achieve 61 MBytes/s for pure raw disk IO
and about 25 MBytes/s to read and recreate the full event model in memory.
When using a compressed file, we achieve 9.4 MBytes/s, but this correspond
to about 34 MBytes/s of final event data in memory.
From these results, I recommend ATLAS to write new files with 5.25/03 (will be 5.26
in about 3 weeks) as soon as possible. Reading old data with
the new version will be faster, but not as fast as if you write the data with
the new version of ROOT.

NOTE 2
======
Although the gains with the TreeCache seems less spectacular when reading
the new files, the TreeCache is absolutely essential when running with xrootd
or dCache. The TreeCache reduces (typically by a factor 10000) the number
of network transactions. The improved TreeCache also removes backward seeks
and improves the performance on the server side. No changes are required
on the xrootd side.

NOTE3
=====
We are currently running similar tests with the CMS AOD files and the first
results also indicate substantial gains in realtime with all the files
topologies tested so far.

NOTE4
=====
All the results presented above have been obtained on my MacBookPro and
the same behavior has been observed on 2 Linux systems (32 and 64 bits).

NOTE5
=====
Understanding the behavior of ROOT IO is now starightforward thanks to the 
new class TTreePerfStats (see http://root.cern.ch/root/html/TTreePerfStats.html)