12-16 April 2010
Uppsala University
Europe/Stockholm timezone

Parameter sweep job submission to Eucalyptus clouds

Apr 13, 2010, 4:00 PM
20m
Room IV (Uppsala University)

Room IV

Uppsala University

Oral Emerging technologies (cloud, virtualization etc) Novel Architectures and Technologies

Speaker

Peter Kacsuk (SZTAKI)

Description

Large parameter sweep applications require large number of resources. Unfortunately the average number of processors in EGEE VOs is between 500-800 processors that is far not enough for large parameter sweep applications. The situation can be improved if jobs of such parameter sweep applications can be distributed to available cloud resources, too. The interconnection of P-GRADE portal with clouds enable this execution scenario. The talk explains how the interconnection of P-GRADE with Eucalyptus has technically been solved and how this new system can be used by the end-user communities.

Impact

The mixed use of gLite-based academic grids and Eucalyptus-based clouds is enabled for large parameter sweep simulations by interconnecting P-GRADE with both gLite grids and Eucalyptus clouds. As a result any existing VO of EGEE can be extended with cloud resources in a seamless way.

Conclusions and Future Work

Future work will include the designation of a smart scheduler than can optimize the number of jobs to be sent to the clouds and to the grid according to the available grid and cloud resources. Furthermore, the model will be extended for commercial models including the consideration of payment for cloud resources.

Detailed analysis

The 3G Bridge is an extensible generic grid-to-grid bridge developed in the EDGeS project. Its modular plugin
architecture allows extending it to connect arbitrary grid middleware. The Eucalyptus/ Amazon EC2 plugin for the 3G-Bridge allocates resources from the cloud to a pool accessible via a standard job manager (e.g. Condor) for job
submission. The resources are allocated into a single pool (cluster) where the size can be dynamically adjusted up to a pre-configured limit (for security reasons). Theoretically this allows to grow the cluster to an arbitrary size, but since cloud resources are charged by hour this is undesirable. The goal is to find the "sweet spot" the balance between price and performance. Not utilized resources need also to be removed from the pool after a period of inactivity, taking into account that adding a resource to the pool has a high overhead (cost and time required for network transfer and deployment). Recently P-GRADE portal has been connected to 3G Bridge. As a result of this new development jobs of a parameter sweep applications can be seamlessly distributed between EGEE resources, desktop grid resources and cloud resources.

Keywords clouds, parameter sweep applications, portal, scheduler
URL for further information http://www.lpds.sztaki.hu/

Primary authors

Co-author

Presentation materials