\documentclass{CHEP2006}
\setlength{\paperheight}{297mm}
\setlength{\paperwidth}{210mm}

%%
%%  Use \documentclass[boxit]{JAC2003}
%%  to draw a frame with the correct margins on the output.
%%
%%  Use \documentclass[acus]{JAC2003}
%%  for US letter paper layout
%%

\usepackage{graphicx}


\chardef\til=126

%%
%%   VARIABLE HEIGHT FOR THE TITLE BOX (default 35mm)
%%

\setlength{\titleblockheight}{35mm}
% declarations for front matter
\title{LONG-TERM EXPERIENCE WITH GRID-BASED MONTE CARLO MASS PRODUCTION FOR THE ZEUS EXPERIMENT}
\author{H. Stadie\thanks{Hartmut.Stadie@desy.de},
        M. Ernst,
        R. Mankel,
        K. Wrona, DESY, Hamburg, Germany \\
	J. Ferrando, University of Glasgow, United Kingdom
}       
\begin{document}

\maketitle
\begin{abstract}
The detector and collider upgrades for HERA-II have considerably increased the 
demand on computing resources for Monte Carlo production for the ZEUS 
experiment. In order to close the gap, the existing production system was 
extended to be able to use Grid resources as well. This integrated system has 
been successfully in production since November 2004. A total of 400 million 
events have been simulated  by forty different Grid sites within 15 months 
exceeding the capacity of the old system. We present the production setup and 
and its implementation which is based on the ZEUS Grid-toolkit. Our setup 
includes an elaborate system to monitor the participating sites and every 
submitted Grid job.
%introduce the toolkit that has been developed by ZEUS for efficient use of 
%the existing Grid middleware.
\end{abstract}

\section{Introduction}
The ZEUS experiment is a multi-purpose high-energy physics detector at the HERA
collider. HERA is an electron-proton collider located at the DESY laboratory 
in Hamburg, Germany. The ZEUS detector recorded its first collisions in 1992.
After the luminosity enhancement of the collider, subsequently named HERA-II,
 the considerably upgraded ZEUS detector started data taking in 2002.
\par
 A large amount of simulated data is needed to analyze the events taken by 
the ZEUS experiment. This Monte Carlo simulation is very compute-intensive. 
\subsection{HERA-I Monte Carlo production \mbox{system}}\label{sec1.1}
A distributed Monte Carlo production system had been developed by the ZEUS 
collaboration already in the early 1990s to ensure adequate production 
capacity. This system, called {\it Funnel}~\cite{Funnel}, manages all Monte 
Carlo requests from the physicists centrally and distributes them to compute
farms at the collaborating institutes. A ``simulation request'' consists of 
the input file for the detector simulation, which has been produced using a 
high-energy physics generator, and the selected simulation version. Three 
executables are run to complete a request: the detector simulation, the 
trigger simulation, and the reconstruction program. Its output files are 
centrally stored in the DESY tape systems.  Based on the production in 2004, 
this production system reached a capacity of 330 million HERA-I events per 
year using around 200 CPUs worldwide.
\subsection{Monte Carlo demand for the \mbox{HERA-II} running period}
The computing needs for Monte Carlo production are largely increased by the 
upgrades of the detector and the luminosity upgrade of the collider. 
Figure~\ref{fig:lumi} compares the HERA-I and HERA-II running and shows the 
much higher luminosity for HERA-II, which increases the number of Monte Carlo 
events needed.
\begin{figure}[tb]
  \includegraphics[viewport=72 410 496 785,width=0.48\textwidth]{img/lumi00_0}
  \caption{Integrated luminosity over the number of running days for the HERA-I and the HERA-II running. The curve for HERA-II has a much steeper slope due to the luminosity upgrade.}
  \label{fig:lumi}
\end{figure}
According to an evaluation based on the expected integrated luminosity, the 
demand for Monte Carlo might reach as high as 700 million \mbox{HERA-II} events
 per year in 2007. 
\par
Furthermore, the CPU time for simulation and reconstruction of a HERA-II event 
is larger compared to HERA-I by approximately a factor of two due to the 
additional detector components. This effectively halves the capacity of the 
\mbox{HERA-I-style} production  system. Therefore, a more powerful Monte Carlo 
processing system is vital to meet the requirements of ZEUS physics analyses. 
%\subsection{Grid resources}
%Fortunately, the advent of Grid technology opens an elegant solution to 
%this dilemma: Computing resources for 
%%current and future 
%high-energy physics 
%%experiments 
%are emerging in large national centers and at universities and 
%research institutes. Grid technology should allow use of the services of the 
%different computing sites in a standardized way. Two main projects 
%implementing this idea are the LHC Computing Grid (LCG)~\cite{LCG,LCGwww} 
%and Grid2003~\cite{Grid2003,Grid2003www}. 
%Through its member institutes, the ZEUS collaboration has access 
%to many of these evolving computing resources.
%\par
%In addition, the DESY IT group have set up Grid infrastructure~\cite{Gellrich}
%which serves as the hub of the ZEUS Grid activities. 
%This includes the hosting of the ZEUS virtual organization, 
%a resource broker, a powerful storage element, and local computing resources. 
%Of similar importance for this project has been the acquirement of Grid 
%expertise and the support for our project by the DESY IT Grid group.

\section{The integrated Monte Carlo production system}
We have developed a new production system to fulfill the Monte Carlo 
production requirements of the ZEUS collaboration by integration of Grid 
resources~\cite{stadie}. Through its member institutes, the ZEUS collaboration 
has access to many evolving computing resources that are accessible via Grid 
technology. The access is based on the middleware used for either the LCG/EGEE 
grid~\cite{LCGwww} or the Grid3/OSG~\cite{OSGwww} infrastructure.
%Two main projects building up Grid infrastructure are the LHC 
%Computing Grid (LCG)~\cite{LCGwww} and Grid2003~\cite{Grid2003www}.
\par
In addition, the DESY IT group have set up Grid infrastructure~\cite{Gellrich,Gellrichb}
which serves as the hub of the ZEUS Grid activities~\cite{Ernst:2005vg}.
This includes the hosting of the ZEUS virtual organization, operation of
a resource broker, a powerful storage element, and local computing resources. 
Of similar importance for this project has been the acquirement of Grid 
expertise and the support for our project by the DESY IT Grid group.
%This should be achieved by the opportunistic usage of the available Grid resources.
\subsection{Design goals}
The following criteria were set for the design of the integrated production 
system:
\begin{itemize}
\item The existing resources outside of the Grid should still be usable.
\item The inclusion of Grid resources should be completely transparent to
the physicists requesting a Monte Carlo sample. A request is formulated 
in the same way as before %(see~\ref{sec1.1}) 
and the same bookkeeping 
information about the requests is returned. Output files created with the new 
system can be accessed in the same way as for the files produced with the old 
system.
\item The production system should run almost fully automatically.
\item ZEUS Grid jobs should run without pre-installed software.
This enables us to add new sites quickly and use resources opportunistically. 
%\item Grid jobs must run with a high efficiency.
%As no intervention is possible, while a job is executing remotely on the 
%Grid, the jobs must be able to run autonomously and handle errors correctly. 
%Prevalent sources of errors are, for example, failed data transfers or 
%problems with the setup of the worker node. 
%A good error log is important to identify problems and work on their solution.
%\item It should be easy to optimize the usage of Grid resources.
%Maintaining a high efficiency requires good monitoring of all Grid 
%operations.  The monitoring should allow the identification of bottlenecks, 
%badly configured sites, and other problems that might reduce the overall 
%performance. Furthermore, the design of our system should be flexible enough 
%to allow for quick modifications to optimize the setup. 
\item The Grid jobs should be able to run on different middleware flavors.
The Grid middleware is evolving rapidly. In fact, completely new 
%middleware 
projects are emerging and even the behavior of existing client tools changes 
from release to release. Therefore, our system should allow us to add the 
support for a new middleware version easily.
\end{itemize}
\subsection{The ZEUS Grid-Toolkit}
The ZEUS Grid-toolkit~\cite{ZEUSGridToolkit} is the basic toolkit for the implementation of the new 
production system. It is written in object-oriented Perl and consists of a 
set of classes for basic data structures, job submission, data transfer, 
and output logging and validation. A main advantage of Perl is that the same 
toolkit can be used for both the production system and for the production 
scripts running on the worker nodes. The toolkit itself is completely 
independent of other ZEUS software.
\par
%The parts of the toolkit that use Grid client tools directly are encapsulated, 
%because a variety of client tools to access Grid services exist on different 
%sites and new projects are being developed. 
%However, the main concepts for 
%data handling and job submission do not change. For example, a file can be 
%copied and registered in a catalog, replicated and deleted, while a job can 
%be submitted, its state can be queried, and its output can be retrieved. 
A variety of client tools to access Grid services exist on different 
sites and new projects are being developed.
To support different projects simultaneously, the ZEUS Grid-toolkit has been 
designed using the {\it Strategy pattern} and adds an additional layer of 
abstraction to the usage of Grid client tools. Abstract 
interfaces are defined for data handling and job operations, and the 
appropriate middleware implementation is chosen at run time based on a 
configuration file and the installed middleware packages. These 
implementations use fault tolerant methods, e.g. automatic retrial of failed 
data transfers, and have largely increased the overall job efficiency.
%\par In addition, the encapsulation enables us to fix known deficiencies of 
%the Grid tools. In earlier versions of the LCG middleware, failed or never 
%ending data transfers were the main cause for job failures. In the ZEUS Grid 
%toolkit, all data transfer commands are, therefore, run with a timeout and 
%are retried automatically a configurable number of times in case of failures. 
%Furthermore,  the size of the file and a checksum are used to validate the 
%file after every transfer to insure the integrity of the data. These measures 
%have been found to reduce the failure rate considerably. 

\subsection{Layout of the integrated Monte Carlo production system} 
To include Grid resources in the Monte Carlo production system completely 
transparently for the users, the existing mature interfaces for the submission
of a Monte Carlo request %(see~\ref{sec1.1})
 and querying its state are reused. 
As can be seen in Figure~\ref{fig:layout}, a central scheduler distributes 
incoming Monte Carlo requests either to the traditional Monte Carlo sites or 
to a gateway to the Grid resources. This setup allows us to preserve the 
resources of the HERA-I production system and, in fact, to reuse most of the 
existing scheduler.
\begin{figure}[htb]
  \includegraphics[viewport=0 408 610 787,width=0.48\textwidth]{img/system_layout}
  \caption{Layout of the integrated Monte Carlo production system. The ``gLite'' depicts a placeholder for any new middleware project.}
  \label{fig:layout}
\end{figure}
\par
The ``gateway'' node is a LCG-2 user interface and acts as a bridge between 
the production system and the Grid world. Cron jobs process the incoming 
requests and keep track of the individual Grid jobs. A database is used to 
store the state of the Monte Carlo requests and their associated Grid jobs. 
%The number of Grid jobs handled simultaneously by the system can 
%exceed a thousand jobs without perceptible deterioration of performance. 
All the code is written in object-oriented Perl and the ZEUS Grid-toolkit is 
used to copy files to Grid and submit jobs. 
\begin{figure*}[tb]
\vspace{-1.6cm}
\includegraphics[viewport=30 105 767 566,width=0.98\textwidth]{img/zeusgridprodsites}
\vspace{1.6cm}
\caption{Integrated overall production of ZEUS Monte Carlo events on the 
Grid by sites.}
\label{fig:prod}
\end{figure*}
\par
For any new request, a cron job translates this request into a set of Grid 
jobs and copies the input file to the storage element at DESY. The jobs 
assigned to the LCG/EGEE sites are submitted using the resource broker at 
DESY. Since the ZEUS Grid-toolkit is able to simultaneously support different 
middleware projects, we were able to establish submission to a Grid3/OSG 
site using an implementation of our job submission interface based on the 
Globus toolkit~\cite{Globus}.
%Jobs are submitted directly to this site using an implementation 
%of our job submission interface based on the Globus toolkit~\cite{Globus}.
Every Grid job processes 
between 1000 and 2000 events which corresponds to a run time of around three 
hours. The status of the jobs is updated regularly.
%and failed jobs are automatically re-submitted. 
When a Grid job has finished, its output sandbox 
is retrieved and the log files are checked for error conditions. The result of 
these checks is stored in an additional database table. This database is used 
by our monitoring system and enables us to identify problems quickly.
If the job has passed the check, the Monte Carlo output file is transferred 
from the DESY storage  element to the final tape storage pool. As both systems 
use the dCache mass storage subsystem~\cite{dCache,dCachewww}, this a very 
fast operation. Grid jobs that encountered an error are automatically 
resubmitted. This guarantees an efficiency, defined as  the number of produced 
events divided by the number of requested events, of very close to 100\%.  
\par
When all Grid jobs for one request have finished, the gateway returns the 
request to the production system with all necessary bookkeeping files and 
an archive containing all log files of the individual jobs.
\subsection{The Grid jobs}
The Grid Jobs that simulate the Monte Carlo events get all 
the run scripts and the calibration constants from two archives that are copied
from the local storage element to the worker node. As the three executables 
belonging to the requested Monte Carlo version are also copied from Grid 
storage, no pre-installed software is needed to run the production jobs.
This allows us to add new Grid sites quickly to our production.
\par All commands on the worker node are run with an enforced time-out using
a special class of the ZEUS Grid-toolkit to avoid run-away jobs. Run-away jobs
would be killed by the local batch system and would return no output to the 
user making it impossible to identify the cause of the problem. All errors 
can be identified easily with this measure and the logging of all commands and 
data handling operations to standard output.
%When a Grid job starts running on a worker node, it first copies two 
%archives containing all the run scripts for the Monte Carlo production and the 
%calibration constants and the three executables belonging to the requested
%Monte Carlo version from a Grid storage element to the local disk. This 
%avoids the requirement of pre-installed software. As each Grid job only 
%processes part of the input file, the events needed for this job are then 
%extracted from the input file. The executables are run consecutively and 
%their log files are checked after each run. If no errors are found, the 
%output file is finally copied to the DESY storage element. 
%\par
%The execution of data handling operations and all commands are logged to the
%standard output. The same is done with errors that occurred
%during the data validation or were spotted in the log files. This output is 
%analyzed by the production system after the job output was retrieved.
%\par  A crucial problem for diagnostics on the Grid is the fact that a job 
%waiting for data for a long time or hitting an endless loop in an executable 
%is generally killed by the local batch system and delivers no output. If this 
%happens, one does not obtain any information about the cause of the failure. 
%Therefore, it is important to guarantee that the job finishes within the queue
% time limit to identify these problems. This is achieved with the ZEUS Grid-toolkit by imposing a timeout on all commands executed on the worker node.
\section{Operation of the Grid production}
Figure~\ref{fig:prod} shows the integrated Grid production of ZEUS Monte 
Carlo events over time. As can be seen, the presented production system has 
been in continuous operation since November 2004 and used forty different 
Grid sites in eight different countries. Time periods with a small number of 
produced events are usually related to a temporary absence of Monte Carlo 
requests. This indicates that the overall capacity of the production system 
exceeds the demands.% and offers a very fast turnaround.
\subsection{Monitoring}
Good monitoring is needed to use the Grid resources efficiently. 
Our monitoring system regularly contacts the 
information services of the participating sites. For every queue, 
%that allows submission for members of the ZEUS virtual organization 
the number of free and used CPUs and  the number of 
waiting and running jobs are stored with RRDtool, 
a system to store and display time-series data~\cite{rrdtool}.
Furthermore, it queries our production database and stores for each queue the 
number of waiting and running jobs known to our production system.
\par
A site with temporary problems might e.g. accept Grid jobs, but erroneously 
never submit them to its local batch system. This causes the site to always 
publish available resources and attract more and more jobs like a 
``black hole''.
This problem can be detected by comparing the number of scheduled and running 
jobs as reported by the site with the numbers from our production database that
 are obtained from the resource broker. The monitoring system offers this 
information via a web interface. A monitoring plot for a site that acted like a
black hole is shown in Figure~\ref{fig:mon}. Not a single job was running or 
being scheduled at this site although the resource broker had sent more than 
six hundreds jobs to this site. Although our production system cancels jobs that
are not running after a certain amount of time, this problem could only be 
cured by manually removing the site temporarily from our production.
% One can 
%examine the hourly, daily, monthly, and yearly distributions of these and 
%other quantities.
\begin{figure}[htb]
\includegraphics[width=0.48\textwidth]{img/blackhole}
\caption{Example of a monitoring plot showing a site acting as a ``black hole''. The site is publishing free resources and is receiving Grid jobs without submitting them to its local batch system.}
\label{fig:mon}
\end{figure}
\par
\par A further source of information for the monitoring system are the 
database entries for every 
finished Grid job. They contain, for example, the site to which the job was 
sent, the number of produced events, and  the error message, in case of a 
failure. The failure modes and the production rate for every site can be 
studied separately with this information. In addition, one can view  the 
Grid job efficiency for every site in real-time. A web page lists all errors
that occurred in the last 24 hours.
\section{Status and conclusions}
The integrated ZEUS Monte Carlo production system that utilizes both the 
traditional  Monte Carlo production resources and Grid resources started 
production in November 2004. Since then, 40 different LCG sites in eight
different countries have contributed to the ZEUS Monte Carlo production.
Biggest individual contributor has been the computing center at the 
Rutherford Appleton Laboratory in the United Kingdom, followed by the 
clusters at DESY in Hamburg, Germany,  and at Padua in Italy. In May 2005, 
the Grid3 site of the University of Wisconsin in the USA has been added using 
the Globus implementation in the ZEUS Grid-toolkit, which highlights the 
flexibility of our design.
\par 
A total of 400 million Monte Carlo events have been simulated and 
reconstructed on the Grid until February 2006 and are now being used in the 
ongoing ZEUS physics analyses. The Grid production by now exceeds the number 
of events produced by traditional (non-Grid) sites. In fact, daily Grid 
production rates four times above the \mbox{HERA-I} Monte Carlo production rates were 
attained. This illustrates that the inclusion of Grid resources 
largely increased the Monte Carlo production capacity for ZEUS.
\section*{Acknowledgment}
We would like to thank the administrators of all participating sites for 
their support.
% and their hard work.
% on setting up and maintaining  their site.  
\begin{thebibliography}{99}
\bibitem{Funnel} B. Burow, ``Funnel: Towards Comfortable Event Processing'', proc. International Conference on Computing in High Energy Physics 1995, DESY-95-236D (1995).
\bibitem{stadie} H. Stadie et al., ``Monte Carlo Mass Production for the ZEUS Experiment on the Grid'', Nucl. Instrum. Meth. A, article in press.
%\bibitem{LCG} M. Lamanna, ``The LHC Computing Grid Project at CERN'', Nucl. Instrum. Meth. A534 (2004) 1-6.
\bibitem{LCGwww} LCG: \texttt{http://lcg.web.cern.ch/LCG/}.
%\bibitem{Grid2003} I. Foster et al., ``The Grid2003 Production Grid: Principles and Practice'', proc. 13th International Symposium on High-Performance Distributed Computing (2004).
\bibitem{OSGwww} Open Science Grid: \texttt{http://www.opensciencegrid.org}.
\bibitem{Gellrich} A. Gellrich et al., ``Grid Technology in Production at DESY'', Nucl. Instrum. Meth. A, article in press. 
\bibitem{Gellrichb} A. Gellrich, ``Deploying an LCG-2 Grid Infrastructure at DESY'', these proceedings.
\bibitem{Ernst:2005vg} M.~Ernst, A.~Gellrich and R.~Mankel, ``DESY becomes hub for Grid-based HERA events,'', CERN Cour. 45N3 (2005) 19-20.
\bibitem{ZEUSGridToolkit} K. Wrona et al., ``The ZEUS Grid-Toolkit - an experiment independent layer to access Grid services'', these proceedings.
\bibitem{Globus} I. Foster, C. Kesselmann, ``Globus: A Metacomputing Infrastructure Toolkit'' Intl. J. Supercomputer Applications 11(2) (1997) 115-128. 
\bibitem{dCache} P. Fuhrmann, ``dCache, LCG Storage Element and enhanced use cases'', proc. International Conference on Computing in High Energy Physics 2004.
\bibitem{dCachewww} dCache: \texttt{http://www.dcache.org/}.
\bibitem{rrdtool} RRDtool: \texttt{http://people.ee.ethz.ch/{\til}oetiker/
webtools/rrdtool/}.
\end{thebibliography}

\end{document}
