21–25 May 2012
New York City, NY, USA
US/Eastern timezone

End-To-End Solution for Integrated Workload and Data Management using glideinWMS and Globus Online

22 May 2012, 17:25
25m
Room 914 (Kimmel Center)

Room 914

Kimmel Center

Parallel Distributed Processing and Analysis on Grids and Clouds (track 3) Distributed Processing and Analysis on Grids and Clouds

Speaker

Parag Mhashilkar (Fermi National Accelerator Laboratory)

Description

Grid computing has enabled scientific communities to effectively share computing resources distributed over many independent sites. Several such communities, or Virtual Organizations (VO), in the Open Science Grid and the European Grid Infrastructure use the glideinWMS system to run complex application work-flows. GlideinWMS is a pilot-based workload management system (WMS) that creates on demand, dynamically-sized overlay Condor batch system on Grid resources. While the WMS addresses the management of compute resources, however, data management in the Grid is still the responsibility of the VO. In general, large VOs have resources to develop complex custom solutions, while small VOs would rather push this responsibility to the infrastructure. The latter requires a tight integration of the WMS and the data management layers, an approach still not common in modern Grids. In this paper we describe a solution developed to address this shortcoming in the context of Center for Enabling Distributed Petascale Science (CEDPS) by integrating glideinWMS with Globus Online (GO). GO is a fast, reliable file transfer service that makes it easy for any user to move data. The solution eliminates the need for the users to provide custom data transfer solutions in the application by making this functionality part of the glideinWMS infrastructure. To achieve this, glideinWMS uses the file transfer plug-in architecture of Condor. The paper describes the system architecture and how this solution can be extended to support data transfer services other than GO when used with Condor or glideinWMS.

Primary author

Parag Mhashilkar (Fermi National Accelerator Laboratory)

Co-authors

Dr Burt Holzman (Fermi National Accelerator Lab. (US)) Cathrin Weiss (UW Madison) Dr Gabriele Garzoglio (FERMI NATIONAL ACCELERATOR LABORATORY) Lukasz Lacinski (University of Chicago) Raj Kettimuthu (Argonne National Laboratory) Mr Xi Duan (Illinois Institute of Technology) Zach Miller (UW Madison)

Presentation materials