21-25 May 2012
New York City, NY, USA
US/Eastern timezone

AutoPyFactory: A Scalable Flexible Pilot Factory Implementation

22 May 2012, 13:30
4h 45m
Rosenthal Pavilion (10th floor) (Kimmel Center)

Rosenthal Pavilion (10th floor)

Kimmel Center

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session


Dr Jose Caballero Bejar (Brookhaven National Laboratory (US))


The ATLAS experiment at the CERN LHC is one of the largest users of grid computing infrastructure, which is a central part of the experiment's computing operations. Considerable efforts have been made to use grid technology in the most efficient and effective way, including the use of a pilot job based workload management framework. In this model the experiment submits 'pilot' jobs to sites without payload. When these jobs begin to run they contact a central service to pick-up a real payload to execute. The first generation of pilot factories were usually specific to a single VO, and were bound to the particular architecture of that VO's distributed processing. A second generation provides factories which are more flexible, not tied to any particular VO, and provide new or improved features such as monitoring, logging, profiling, etc. In this paper we describe this key part of the ATLAS pilot architecture, a second generation pilot factory, AutoPyFactory. AutoPyFactory has a modular design and is highly configurable. It is able to send different types of pilots to sites and exploit different submission mechanisms and queue characteristics. It is tightly integrated with the PanDA job submission framework, coupling pilot flow to the amount of work the site has to run. It gathers information from many sources in order to correctly configure itself for a site, and its decision logic can easily be updated. Integrated into AutoPyFactory is a flexible system for delivering both generic and specific job wrappers which can perform many useful actions before starting to run end-user scientific applications, e.g. validation of the middleware, node profiling and diagnostics, and monitoring. AutoPyFactory now also has a robust monitoring system and we show how this has helped establish a reliable pilot factory service for ATLAS.

Primary author


Graeme Andrew Stewart (CERN) John Hover (Brookhaven National Laboratory (BNL)-Unknown-Unknown) Dr Jose Caballero Bejar (Brookhaven National Laboratory (US)) Peter Love (LANCASTER UNIVERSITY)

Presentation Materials