\documentclass{CHEP2006}
\setlength{\paperheight}{297mm}
\setlength{\paperwidth}{210mm}

%%
%%  Use \documentclass[boxit]{JAC2003}
%%  to draw a frame with the correct margins on the output.
%%
%%  Use \documentclass[acus]{JAC2003}
%%  for US letter paper layout
%%

\usepackage[latin1]{inputenc} % For Latin 1 character usage
\usepackage{graphicx}

%%
%%   VARIABLE HEIGHT FOR THE TITLE BOX (default 35mm)
%%

\setlength{\titleblockheight}{35mm}

\renewcommand{\floatpagefraction}{0.95}
\renewcommand{\topfraction}{0.95} 
\renewcommand{\bottomfraction}{0.95} 
\renewcommand{\textfraction}{0.05}

\begin{document}
\title{VALIDATION OF GEANT4 BERTINI CASCADE NUCLIDE PRODUCTION USING
PARALLEL ROOT FACILITY}

\author{A. Heikkinen and T. Lindén, Helsinki Institute of Physics, Helsinki, Finland}

\maketitle

\begin{abstract}

We present an investigation to validate Geant4 \cite{geant4} Bertini
cascade nuclide production by proton induced reactions on
various target elements \cite{bertinivalidation}. The production of
residual nuclides is calculated in the framework of an intra-nuclear
cascade, pre-equilibrium, fission, and evaporation model \cite{bertini}.

A 132 CPU Opteron Linux cluster running the NPACI Rocks Cluster
Distribution \cite{rocks} based on Red Hat Enterprise Linux has
been used to compute proton induced isotope production cross-section
results for the Bertini cascade. We have used the new features of the
Parallel ROOT Facility (PROOF), distributed with ROOT version 5, to
analyse the isotope cross-section data. Automatic class generation for
PROOF event data-analysis has been used on the Rocks cluster
\cite{root,proof}. Performance results for the cluster as measured with
the ProofBench package are also presented \cite{proofbench}.\end{abstract}

\section{INTRODUCTION}
Cross sections for nuclide production by proton induced nuclear reactions
at low and intermediate energies are of importance in many fields of science.
They are needed in the design and operation of accelerator based experiments, 
space technology, and cosmic ray physics.  
They are essential for radiation protection studies, 
for radioactive waste transmutation and for simulation of 
radionuclide production in nuclear medicine \cite{schiekel96}.

Careful validation of hadronic shower models in Geant4 is needed to
ensure a successful analysis phase of the future LHC experiments \cite{chep04}. 
As part of a considerable coordinated effort put into physics validation, 
we study here isotope production from semi-empirical and theoretical Bertini models. 
%Contrary to data oriented models, also available in Geant4, 
%we need to perform 
A full Monte Carlo experiment, requiring significant machine resources, 
is needed to study rare isotope production channels. 

\section{CLUSTER HARDWARE}

\begin{figure}[htb]
\centering
\includegraphics*[width=60mm]{ametisti_small_600x800.eps}
\caption{The 132 CPU Opteron cluster at the University of Helsinki
is used for physics-, chemistry- and HEP-applications.
The {\em Ametisti} cluster is part of the Finnish
M-grid, which currently has 1118 AMD Opteron CPUs connected with the
Advanced Resource Connector (ARC) middleware \cite{arc}.}
\label{ametisti}
\end{figure}

The 132 CPU {\em Ametisti} cluster at our campus (see Fig.
\ref{ametisti}) is shared between the the Department of Physical
Sciences, the Department of Chemistry and Helsinki Institute of Physics.
It consists of a HP Proliant DL-585 frontend, which has a shared 2 TB HP
SA-6402+MSA-30 SCSI disk system. HP Procurve Gb/s and fast ethernet
switches have been used for the three networks on the cluster. The two
separate Gb/s networks for communication and NFS-traffic have been
implemented to enhance the performance of the shared NFS disk system.
The fast ethernet network is used for remote management, allowing remote
booting of the nodes and the frontend. The cluster nodes are managed
remotely trough a single CPU HP Proliant DL-145 G1 administration
node. Table \ref{node} describes the cluster nodes in detail.

\begin{table}[hbt]
\begin{center}
\caption{Components of the cluster compute nodes:}
\begin{tabular}{|l|l|}
\hline 
Nodes & 1U HP Proliant DL-145 G1 \\
CPU    & dual 1.8--2.2 GHz AMD Opteron \\
RAM   & 2--4 GB ECC registered DDR \\
Disk  & 2*(80 or 160) GB 7200 rpm PATA \\ 
Networks & dual Gb/s, management 100 Mb/s\\
Display card & PCI ATI Rage XL 8MB \\
PCI & one 133 MHz 64 bit slot \\ 
USB & CDROM / \\
 & flash memory stick, if needed\\ \hline
\end{tabular}
\label{node}
\end{center}
\end{table}




\section{CLUSTER SOFTWARE ENVIRONMENT}

We use a significantly patched version 3.2 of NPACI Rocks Cluster
Distribution as the cluster operating- and management system on
{\em Ametisti}. The batch queue system has been upgraded to Sun N1 Grid
Engine (SGE) release 6 update 4. As cluster application software Geant4
v4.7.0.p01 and ROOT v5.08/00b have been used.

\subsection{Cluster operating- and management system Rocks}

NPACI Rocks Cluster Distribution is a RPM based cluster management
software for scientific computation based on the source code of Red Hat
Enterprise Linux, supporting cluster installation, configuration,
monitoring and maintenance. Several Rocks
based clusters have made it to the Top500 list \cite{top500}, for a
current list of the installed Rocks clusters, see the Rocks website
\cite {rocksweb}. 

Version 3.2. used in this work with kernel
2.4.21-27.0.4 is based on Red Hat Enterprise Linux 3.0.
The monitoring of the {\em Ametisti} cluster is
performed using the web-based Rocks Ganglia tool (see Fig.
\ref{ganglia}). 

\begin{figure}[htb]
\centering
\includegraphics*[width=83truemm]{ametisti_ganglia.eps}
\caption{Rocks Ganglia monitoring page for {\em Ametisti}. The plot shows
that sometimes the batch queues have unused CPUs, that could be used for
interactive PROOF sessions.}
\label{ganglia}
\end{figure}

\subsection{Batch queue system SGE}

Rocks comes equipped with the SGE batch queue system. A test queue for
very short jobs or installation configuration development has been set
up on {\em Ametisti}. This queue has been used in the static PROOF
configuration. Larger jobs can be handled by adjusting the number of
CPUs in the queue accordingly. The PROOF team has worked on a dynamic
configuration file that can interact with different batch queue systems,
but there is not yet any implementation for SGE.


\subsection{Data analysis system ROOT/PROOF}

PROOF is an extension to ROOT which allows the parallel
analysis of large ROOT trees distributed over many computers. PROOF can
be used on a set of inhomogenous workstations or in a homogenous cluster
environment. It uses a master slave architecture with possible layers of
submasters and load balancing through pull mode task distribution. It
has been designed to be used on systems ranging from single or multi
core workstations to large clusters with $\cal{O}$(100) CPUs
\cite{proofbench} or even
collections of clusters connected with grid middleware. The best
performance is obtained when the data set to be analyzed is distributed
over the local disks of the cluster nodes.

We installed ROOT v5.08/00b on the frontend node and it was distributed with NFS to all
compute nodes. The PROOF configuration was implemented in the
Rocks XML configuration tree \cite{chep04}. 
The NFS performance is not a problem, because of the dedicated Gb/s NFS-network, 
and because only a small part of the cluster is usually available for PROOF usage.
%Since usually only a small
%part of the cluster is available for PROOF usage, the NFS performance
%should not be a problem, especially because of the dedicated Gb/s
%NFS-network.

\section{RESULTS}
We have validated nuclide production by proton-induced reactions on various elements. 
In order to evaluate the code, we have generated $\sim 10^8$ single interactions at proton energies
of 20 MeV to 8.1 GeV.
The experimental data are compared with semi-empirical and theoretical
models available in a Bertini cascade hybrid model based on intra-nuclear cascade, 
preequilibrium, fission, and evaporation reactions (see Fig. \ref{data}). 
In many cases the performance was found to be comparable to codes 
such as HETC, CEM, LAHET, or CASCADE \cite{bertinivalidation}.

The number of different isotopes the Geant4 Bertini code can produce is very high, 
and the results are limited by statistics for the rarely produced isotopes (see Fig. \ref{simulation}).
In addition to nuclide production cross sections used for validation, 
our Monte Carlo experiment has provided information on the energy and 
direction of the projectiles.

The computations in this work were done on the {\em Ametisti} cluster
described previously, building on the experiences of a small test setup
reported in \cite{chep04}. 
\begin{figure}[htb]
\centering
\includegraphics*[width=65mm]{exitationResult.eps}
\caption{Example of Geant4 Bertini cascade code simulation. Proton induced
isotope production is studied for the case of 39-Y-89(p,X)40-Zr-88 and
validated against experimental data \cite{data}.}
\label{data}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics*[width=65mm]{exitationResultO.eps}
\caption{In the cluster environment we are able to study rare Monte Carlo
events with a good statistical accuracy. Here proton absorption is
demonstrated for the case 8-O-16(p,X)9-F.}
\label{simulation}
\end{figure}

We used the {\em Run\_Simple\_Test.C} -script available in ProofBench
package \cite{proofbench} to verify the setup of our PROOF cluster and
compare the relative PROOF performance of {\em Ametisti} with {\em
Mill}, which is a 32-bit 2.133 GHz AMD Athlon Rocks cluster with 64
CPUs. Processing eight million events with four nodes (eight slaves) with
one file per slave (14 GB of data in total) with PROOF was 1.8 times
faster on the 64-bit 1.8 GHz Opteron {\em Ametisti}-nodes than on the
32-bit 2.133 GHz {\em Mill}-nodes. The corresponding ratio for the
packet processing CPU time per slave was 2.1.

\section{CONCLUSIONS}

The PROOF package develops at a rapid place with many useful features
added during the year 2005, like the graphical user interface to PROOF
clusters, using stateless connections (for details see Ref.
\cite{proof2}). The authentication has developed and the default SSH
passphraseless authentication used on Rocks works in spite of the warning
messages given, so getting it to work was easier than on our test
cluster \cite{chep04}. PROOF is a useful and powerful tool with a lot of
potential and it would deserve more extensive documentation.

Utilizing the parallellism of present and future computers is a
challenge to data analysis software in HEP. ROOT and PROOF seems to
develop in a very promising way to be able to harness the CPU-power
needed for analysis of large data samples. We have found these tools
very useful in Geant4 physics validation studies.

As a continuation of this work, 
we are currently parallellizing the Geant4 validation code using the 
TOP-C library. 

\section{ACKNOWLEDGEMENTS}

We highly appreciate the PROOF team who has been very helpful in
debugging and finding fixes for problems found.

% \begin{thebibliography}{9}   % Use for  1-9  references
\begin{thebibliography}{99} % Use for 10-99 references

\bibitem{geant4}
    Geant4 Collaboration, ``GEANT4: A Simulation Toolkit'',
    Nuclear Instruments and Methods in Physics Research,
    NIM A 506 (2003), 250-303.

\bibitem{bertinivalidation}
A. Heikkinen, ``Validation of Geant4 Bertini cascade nuclide
production'',
    Proceedings of FrontierScience 2005, Milan, Italy, September 12-17,
2005.

\bibitem{bertini}
    A. Heikkinen, N. Stepanov, and J.P. Wellisch,
    ``Bertini intra-nuclear cascade implementation in Geant4'',
    arXiv: nucl-th/0306008.

\bibitem{rocks}
    P. Papadopoulos, M. Katz, and G. Bruno, ``NPACI Rocks: Tools and
    Techniques for Easily Deploying Manageable Linux Clusters'',
    Concurrency Computat: Pract. Exper. 2002; 00:1-20.

\bibitem{root}
    F. Rademakers, M. Goto, P. Canal, R. Brun, ``ROOT Status and Future
    Developments'', arXiv: cs.SE/0306078.

\bibitem{proof}
    M. Balli et al., ``Parallel Interactive and Batch HEP-Data Analysis
    with PROOF'', Proceeding of ACAT05, May 22 - 27, 2005
    DESY, Zeuthen, Germany.

\bibitem{proofbench}
    M. Ballintijn et al., ``Super scaling PROOF to very large
clusters'', Proceedings of Computing in High Energy Physics,
    CHEP'04, Interlaken, Switzerland,  27 Sep -- 01 Oct 2004.

\bibitem{schiekel96}
    Th.~Schiekel et al., ``Nuclide production by proton-induced reactions 
on elements (6 $\leq$ Z $\leq$ 29) in the energy range from 200 MeV to 400 MeV'', 
Nucl. Instr. and Meth. in Phys. Res. B 114 (1996) 91-119.
  
\bibitem{chep04}
    A. Heikkinen and T. Lindén, ``Validation of the GEANT4 Bertini
    Cascade model and data analysis using the Parallel ROOT Facility on
    a Linux cluster'', Proceedings of Computing in High Energy Physics,
    CHEP'04, Interlaken, Switzerland, 27 Sep -- 01 Oct 2004.

\bibitem{arc}
   www.nordugrid.org/middleware/.

\bibitem{top500}
The list of the 500 fastest computers according to the Linpack
benchmark, www.top500.org.

\bibitem{rocksweb}
The homepage of NPACI Rocks cluster distribution, www.rocksclusters.org.

\bibitem{data}
M. Khankhasayev ed., ``Nuclear Methods for Transmutation of Nuclear Waste'',
Proceedings of the Workshop, Dubna, Russia, 2931
May, 1996.

\bibitem{proof2}
   G. Ganis, ``PROOF : status and developments'', Root
   2005 users workshop.

\end{thebibliography}
\end{document}
