21-25 May 2012
New York City, NY, USA
US/Eastern timezone

The Fermi-LAT Dataprocessing Pipeline

22 May 2012, 13:30
4h 45m
Rosenthal Pavilion (10th floor) (Kimmel Center)

Rosenthal Pavilion (10th floor)

Kimmel Center

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session


Mr Stephan Zimmer (OKC/ Stockholm University, on behalf the Fermi-LAT Collaboration)


The Data Handling Pipeline ("Pipeline") has been developed for the Fermi Gamma-Ray Space Telescope (Fermi) Large Area Telescope (LAT) which launched in June 2008. Since then it has been in use to completely automate the production of data quality monitoring quantities, reconstruction and routine analysis of all data received from the satellite and to deliver science products to the collaboration and the Fermi Science Support Center. In addition it receives heavy use in performing production MonteCarlo tasks. In daily use it receives a new data download every 3 hours and launches about 2000 jobs to process each download, typically completing the processing of the data before the next download arrives. The need for manual intervention has been reduced to less than <.01% of submitted jobs. The Pipeline software is written almost entirely in Java and comprises several modules. The software comprises web-services that allow online monitoring and provides AIDA charts summarizing work flow aspects and performance information. The server supports communication with several batch systems such as LSF and BQS and recently also Sun Grid Engine and Condor. This is accomplished through dedicated JobControlDaemons that for Fermi are running at SLAC and the other computing site involved in this large scale framework, the Lyon computing center of IN2P3. While being different in the logic of a task, we evaluate a separate interface to the Dirac system in order to communicate with EGI sites to utilize Grid resources, using dedicated Grid optimized systems rather than developing our own. More recently the pipeline and its associated data catalog have been generalized for use by other experiments, and are currently being used by the Enriched Xenon Observatory (EXO), Cryogenic Dark Matter Search (CDMS) experiments as well as for MonteCarlo simulations for the future Cherenkov Telescope Array (CTA).
Student? Enter 'yes'. See http://goo.gl/MVv53 yes

Primary author

Mr Stephan Zimmer (OKC/ Stockholm University, on behalf the Fermi-LAT Collaboration)


Dr Andrei Tsaregorodtsev (Universite d'Aix - Marseille II (FR)) Dr Claudia Lavalley (IN2P3, LUPM) Dr Luisa Arrabito (IN2P3, LUPM) Dr Tom Glanzmann (SLAC, Fermi-LAT Collaboration) Dr Tony Johnsson (SLAC, Fermi-LAT Collaboration)

Presentation Materials