Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

Context-aware distributed cloud computing using CloudScheduler

Oct 12, 2016, 12:45 PM
GG C2 (San Francisco Mariott Marquis)


San Francisco Mariott Marquis

Oral Track 3: Distributed Computing Track 3: Distributed Computing


The distributed cloud using the CloudScheduler VM provisioning service is one of the longest running systems for HEP workloads. It has run millions of jobs for ATLAS and Belle II over the past few years using private and commercial clouds around the world. Our goal is to scale the distributed cloud to the 10,000-core level, with the ability to run any type of application (low I/O, high I/O and high memory) on any cloud. To achieve this goal, we have been implementing changes that utilize context-aware computing designs that are currently employed in the mobile communication industry. Context-awareness makes use of real-time and archived data to respond to user or system requirements. In our distributed cloud, we have many opportunistic clouds with no local HEP services, software or storage repositories. A context-aware design significantly benefits the reliability and performance of our system by locating the nearest or optimal location of the required services. We describe how we are collecting and managing contextual information from our workload management systems, the clouds, the virtual machine and our services. This information is used not only to monitor the system but also to carry out automated corrective actions. We are incrementally adding new alerting and response services to our distributed cloud. This will enable us to scale the number of clouds and virtual machines. Further, a context-aware design will enable us to run analysis or high I/O application on opportunistic clouds. We envisage an open-source HTTP data federation (for example, the Dynafed system at CERN) as a service that would provide us access to existing storage elements used by the HEP experiments.

Primary Keyword (Mandatory) Cloud technologies

Primary author

Dr Randall Sobie (University of Victoria)


Mr Colin Roy Leavett-Brown (University of Victoria) Mr Darryl Ring (University of Victoria) Dr Kevin Casteels (University of Victoria) Mr Matthew Murray (University of Victoria) Mr Michael Paterson (University of Victoria) Dr Rolf Seuster (University of Victoria) Mr Ryan Taylor (University of Victoria)

Presentation materials