Speaker
Prof.
Kaushik De
(UNIVERSITY OF TEXAS AT ARLINGTON)
Description
A new offline processing system for production and analysis, Panda, has been
developed for the ATLAS experiment and deployed in OSG. ATLAS will accrue tens of
petabytes of data per year, and the Panda design is accordingly optimized for data
intensive processing. Its development followed three years of production experience,
the lessons from which drove a markedly different design for the new system.
Key design features include
- data-driven workflow with managed placement of datasets (file collections) at
processing sites; job-placement that assures jobs are dispatched to sites holding
their input data
- tight integration with the ATLAS DDM system Don Quijote 2 which provides all data
management services
- late binding of jobs to worker nodes for dynamic and flexible prioritization and
scheduling, and to isolate workloads from latencies and failure modes in acquiring
processing resources
- a service oriented architecture with a fast and lightweight communication layer
based on REST-style web services
- queue management, data placement, job dispatching, and processor acquisition
operate as asynchronous services, with support for multiple instances, for fast
throughput and maximum scalability.
In this paper we motivate and describe the design and implementation of the system,
the current state of its deployment for production and analysis operations, and the
work remaining to achieve readiness for ATLAS datataking.
Primary authors
Prof.
Kaushik De
(UNIVERSITY OF TEXAS AT ARLINGTON)
Dr
Torre Wenaus
(Brookhaven National Laboratory)
Co-authors
Dr
Alexei Klimentov
(Brookhaven National Laboratory)
Dr
David Adams
(Brookhaven National Laboratory)
Dr
Horst Severini
(University of Oklahoma)
Mr
Karthik Arunachalam
(University of Oklahoma)
Dr
Marco Mambelli
(University of Chicago)
Dr
Mark Sosebee
(UNIVERSITY OF TEXAS AT ARLINGTON)
Dr
Nurcan Ozturk
(UNIVERSITY OF TEXAS AT ARLINGTON)
Dr
Pavel Nevski
(Brookhaven National Laboratory)
Mr
Tadashi Maeno
(Brookhaven National Laboratory)
Dr
Wensheng Deng
(Brookhaven National Laboratory)
Dr
Xin Zhao
(Brookhaven National Laboratory)
Dr
Yuri Smirnov
(Brookhaven National Laboratory)
