Speaker
Description
Current computing paradigms often involve concepts like microservices, containerisation and, of course, Cloud Computing.
Scientific computing facilities, however, are usually conservatively managed through plain batch systems and as such can cater to a limited range of use cases. On the other side, scientific computing needs are in general orthogonal to each other in several dimensions.
We have been operating the Open Computing Cluster for Advanced data Manipulation (OCCAM), a multi-purpose heterogeneous HPC cluster, for more than one year adopting a cloud-like paradigm. Each computing application is mapped to a dynamically expandable virtual farm, tuned and configured to the application’s needs and able to access special hardware like GPU accelerators or low-latency networks as needed, thus delivering computational frameworks that are well consolidated within the community (for a smooth end user experience) while leveraging modern computing paradigms.
By using mostly mainstream software tools like Docker (used throughout our architecture to run both service and computational tasks), Calico for virtual network management, Mesos and Marathon for orchestration and by exploiting some of the work made in the context of the INDIGO-DataCloud project, we aimed at minimising the development and maintenance effort, while using a high-quality software stack.
In this work we present the status of the system, operational experience, lessons learnt, and our outlook for further development. We will also present some preliminary performance comparisons between containerized and bare-metal scientific computing applications in an HPC environment.
