Speaker
Description
As results of the excellent LHC performance in 2016, more data than expected has been recorded leading to a higher demand for computing resources. It is already foreseeable that for the current and upcoming run periods a flat computing budget and the expected technology advance will not be sufficient to meet the future requirements. This results in a growing gap between supplied and demanded resources. Physics is likely to be limited by the available computing and storage resources.
One option to reduce the emerging lack of computing resources is the utilization of opportunistic resources such as local university clusters, public and commercial cloud providers, HPC centers and volunteer computing. However, to use opportunistic resources additional challenges have to be tackled.
The traditional HEP Grid computing approach leads to a complex software framework that has special dependencies in operation system and software requirements, which currently prevents HEP from using these additional resources. To overcome these obstacles the concept of pilot jobs in combination with virtualization and/or container technology is the way to go. In this case the resource providers only needs to operate the “Infrastructure as a Service”, whereas HEP manages its complex software environment and the on-demand resource allocation. This approch allows us to utilize additional resources in a dynamically fashion on different kind of opportunistic resource providers.
Another challenge that has to be addressed is that not all workflows are suitable for opportunistic resources. For the HEP workflows the deciding factor is mainly the external network usage. To identify suitable workflows that can be outsourced to external resource providers, we propose an online clustering of workflows to identify those with low external network usage. This class of workflows can than be transparently outsourced to opportunistic resources dependent on the local site utilization.
Our approach to master opportunistic resources for the HEP community in Karlsruhe is currently evaluated and refined. Since the general approach is not tailored to HEP, it can be easily adapted by other communities as well.
