Workshop on adapting applications and computing services to multi-core and virtualization

24 Jun 2009, 13:30 → 26 Jun 2009, 18:00 Europe/Zurich

40-S2-A01 (CERN)

Preparatory workshop towards drawing up a roadmap for providing IT services which allow LHC experiment applications to exploit multi-core and virtualization capabilities. Applications being developed to exploit new hardware architectures and virtualization technology impose new requirements on the computing services provided by the local computer centers or by the Grids. The goal of this workshop is to discuss these new requirements and to have a first exploration of the possible solutions that service providers could offer to the LHC experiments. Contacts: Pere Mato (CERN) , Predrag Buncic (CERN) , Vincenzo Innocente (CERN) , Denise Heagerty (CERN) EVO conference has been booked at Universe community 24-June: http://evo.vrvs.org/evoGate/koala.jnlp?meeting=M2MvMB2a2iDaDn9I9lDM92 25-26 June: http://evo.vrvs.org/evoGate/koala.jnlp?meeting=M9MIMi2e2BDeD89e99DD99

Summary Document WS09-Summary-final.pdf

Wednesday 24 June

Welcome and Technology session

1 Introduction to the Workshop

Speaker: Pere Mato

Slides Introduction_WS_20090624.pdf Introduction_WS_20090624.ppt

2 VMware virtual datacenter operating system and cloud computing

- Virtual Datacenter OS - Foundation for the Cloud - VMware vSphere Architecture advantages and benefits - How to transform your datacenters to vSphere Architecture

Speaker: Ralf von Gunten (VMware (Sr. Systems Engineer))

Slides vSphere-CERN_2009-06-24.pdf vSphere-CERN_2009-06-24.ppt

3 Cisco Unified Computing System

- Cisco Datacenter Vision - Unified Computing Systems overview - Unified Computing Systems business benefits and technical advantages

Speaker: Vincent Meoni (Cisco (DataCenter Systems Engineer))

Slides 2009_06_CERN_vmeoni_UCS.pdf 2009_06_CERN_vmeoni_UCS.ppt

4 From Application to Appliance - Tackling the “Last Mile” of IT Process Automation

Speaker: Erik Troan (rPath (CTO))

Slides rpath_ppt_for_cern.pdf rpath_ppt_for_cern.ppt

16:00 coffee break

5 Using Linux KVM and KSM in scientific computing environments

Linux as Hypervisor with KVM is particularly performant virtualization technology to use in scientific computing environments, thanks especially to its (out of sync) secondary MMU management of the shadow page tables and it support for CPUs with EPT or NPT. In addition KVM leverages: the efficiency, testing and reliability of the huge pool of Linux Kernel device drivers, the close integration of the KVM shadow page tables management with the Linux Virtual Memory subsystem that allows KVM to take advantage of kernel features like swapping and KSM in a fully transparent way and much more.

Speaker: Andrea Arcangeli (RedHat)

Slides kvm-ksm.pdf

6 Scheduling for mixed parallel and sequential workload

The talk will cover various techniques within the LSF Workload Management system for promoting higher efficiency with a non-homogeneous workload: - How to guarantee that parallel jobs can run in a mixed sequential/parallel environment (slot reservation /memory reservation/backfill) - How to maximize the efficiency on multicpu/multicore machines by using CPU binding - How to maximize the efficiency of very short jobs by using sub-scheduling (Session Scheduler) - How to maximize the efficiency of memory usage - How to maximize the efficiency of distributed parallel jobs: taking in account the compute farm topology

Speaker: Mr Robert Lurin (Platform)

Slides CERN_LSF_Scheduling_for_mixed_workload.pdf CERN_LSF_Scheduling_for_mixed_workload.ppt

7 Discussion

Thursday 25 June

New Application Requirements

8 Adapting LHC Applications to Multi-core architectures

Speaker: vincenzo innocente (CERN)

Slides VIatWS0609.pdf VIatWS0609.ppt

9 Experience with multi- and many-core at openlab

The presentation would focus on the current hardware, future hardware, scalability issues and the experience we've had with both software and hardware in the area of multi-/many-core

Speaker: Andrzej Nowak (CERN)

slides slides

10 Virtualizing LHC Applications

Speaker: Predrag Buncic (CERN)

Slides VM-Workshop-Buncic.pdf VM-Workshop-Buncic.ppt

10:45 coffee break

11 ATLAS experience with multi-core and virtualization

Speaker: Paolo Calafiura (LBNL)

Slides CERNWorkshopJul09_ATLASExperience.pdf

12 LHCb End-User Experience

Speaker: Thomas Ruf (CERN)

Slides MultiCoreAndVM.pdf

Computing Services

13 Next generation of virtual infrastructure with Hyper-V

Internet Services group provides the hardware and OS layers for services operated by different CERN organizational units. Within this framework, a user can ask for a new virtual server from a web browser and have it ready within 15 minutes. The latest release of the virtual infrastructure is based on Hyper-V, Microsoft Virtual Machine Manager, management SOAP web services and user web interface. This talk will present this approach and will discuss the evolution of the Microsoft Hypervisor from the perspective of both the end-user and the system administrator.

Speaker: Juraj Sucik (CERN)

Slides 20090606_-_Virt_Workshop_at_CERN.pdf 20090606_-_Virt_Workshop_at_CERN.pptx

14 Virtualisation from a security perspective

A look at some of the security issues around virtualization, covering both newly-added concerns as well as potential security benefits from using virtualization.

Speaker: Jan Iven (CERN)

Slides virtsec.odp virtsec.pdf

15 Possibilities and limitations of virtualization from a networking perspective

This talk will address: types of networks for data centers and their impact on VM deployment, IPv4 address space limitation, VM migration and network infrastructure/services to support VMs at CERN.

Speaker: Jose Carlos Luna Duran (CERN)

Slides IT-CS-Virtualization.pdf IT-CS-Virtualization.pptx

16 Virtualisation of Batch Services at CERN

In this talk we will present a load-driven virtual machine provisioning mechanism based on Platform Virtual Machine Orchestrator (VMO). LSF and VMO are coupled to manage virtual machines that meet the requirements of the jobs. The underlying infrastructure uses Quattor/LANDB/CDB and LEMON monitoring which makes this virtual machine prototype compatible with the existing operations procedures.

Speaker: Sebastien Goasguen (Clemson University)

Slides batchvirt.pdf

15:30 coffee break

17 Practical aspects of multi-core job submission at CERN

Speaker: Mr Ricardo Manuel Salgueiro Domingues da Silva (IT-FIO)

Slides PraticalMultiCoreAtCERN.pdf PraticalMultiCoreAtCERN.ppt

18 Discussion

Workshop Dinner

Please use the following doodle poll to register to the dinner http://www.doodle.com/qkkbscitdac3gh6q
Transport can be organized.

Friday 26 June

Grid Services

19 Nimbus: Open Source IaaS Cloud Computing Software

Nimbus is an open source cloud computing toolkit that allows you to turn your cluster into an Infrastructure-as-a-Service (IaaS) cloud.

Speaker: Kate Keahey (Mathematics & CS Division, Argonne National Laboratory)

Slides nimbus.ppt.pdf nimbus.ppt.ppt

20 EGEE MPI working group

In the original LHC Computing Grid, it was not possible to run parallel jobs which rely on communication between concurrent processes. Upon request of the user community, an MPI implementation on the Grid has been created. Recently, an MPI working group has been setup to investigate the bottlenecks with this MPI implementation both for system administrators and for users. Another task of this working group is to write recommendations for improvements and extensions. One of the extensions is a the mechanism for the user to request all cores on a single node.

Speaker: Jeroen Engelberts

Slides MPI_WG_Proceedings.pdf MPI_WG_Proceedings.ppt

21 Evolution of OSG to support virtualization and multi-core applications

Speaker: Daniel Charles Bradley (High Energy Physics)

Slides DanOSGMultiVirt.pdf DanOSGMultiVirt.ppt

22 The INFN experience with virtualized computing services and perspectives for integration of Grid and Cloud services in EGI (European Grid Initiative)

In this talk we'll describe how INFN approached the problem of virtualizing existing computing farms, the technology and performance studies that were carried out, and a framework for the integration of both Grid and Cloud virtualized services into a distributed computing infrastructure like EGI.

Speaker: Davide Salomoni (INFN-CNAF)

Slides ds_cern_virtualization_0906.pdf ds_cern_virtualization_0906.ppt

11:00 coffee break

Summary, final discussion and wrap-up

23 Summary of the workshop

Speaker: Denise Heagerty (CERN)

Slides Discussion-Summary.pdf Discussion-Summary.ppt

24 Dynamic Cluster Partitioning for Worker Nodes

Speaker: Yves Kemp (DESY)

Slides DESY-virtual-WN.pdf

25 Discussion