Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

INDIGO-Datacloud: a Cloud-based Platform as a Service oriented to scientific computing for the exploitation of heterogeneous resources

Oct 13, 2016, 11:15 AM
Sierra B (San Francisco Mariott Marquis)

Sierra B

San Francisco Mariott Marquis

Oral Track 6: Infrastructures Track 6: Infrastructures


Patrick Fuhrmann (Deutsches Elektronen-Synchrotron (DE))


INDIGO-DataCloud (INDIGO for short, https://www.indigo-datacloud.eu) is a project started in April 2015, funded under the EC Horizon 2020 framework program. It includes 26 European partners located in 11 countries and addresses the challenge of developing open source software, deployable in the form of a data/computing platform, aimed to scientific communities and designed to be deployed on public or private Clouds, integrated with existing resources or e-infrastructures.

In this contribution the architectural foundations of the project will be covered, starting from its motivations, discussing technology gaps that currently prevent effective exploitation of distributed computing or storage resources by many scientific communities. The overall structure and timeline of the project will also be described.

The main components of the INDIGO architecture in the three key areas of IaaS, PaaS and User Interfaces will then be illustrated. The modular INDIGO components, addressing the requirements of both scientific users and cloud/data providers, are based upon or extend established open source solutions such as OpenStack, OpenNebula, Docker containers, Kubernetes, Apache Mesos, HTCondor, OpenID-Connect, OAuth, and leverage both de facto and de jure standards.

Starting from the INDIGO components, we will then describe the key solutions that the project has been working on. These solutions are the real driver and objective of the project and derive directly from use cases presented by its many scientific communities, covering areas such as Physics, Astrophysics, Bioinformatics, Structural and molecular biology, Climate modeling, Geophysics, Cultural heritage and others. In this contribution we will specifically highlight how the INDIGO software can be useful to tackle common use cases in the HEP world. For example, we will describe how topics such as batch computing, interactive analysis, distributed authentication and authorization, workload management and data access / placement can be addressed through the INDIGO software. Integration with existing data centers and with well-known tools used in the HEP world such as FTS, Dynafed, HTCondor, dCache, StoRM, with popular distributed filesystems and with Cloud management frameworks such as OpenStack and OpenNebula as well as support for X.509, OpenID-Connect and SAML will also be discussed, together with deployment strategies. A description of the first results and of the available testbeds and infrastructures where the INDIGO software has been deployed will then be given.

Finally, this contribution will discuss how INDIGO-DataCloud can complement and integrate with other projects and communities and with existing multi-national, multi-community infrastructures such as those provided by EGI, EUDAT and the HelixNebula Science Cloud. The importance of INDIGO for upcoming EC initiatives such as the European Open Science Cloud and the European Data Infrastructure will also be highlighted.

Primary Keyword (Mandatory) Cloud technologies
Secondary Keyword (Optional) Distributed data handling
Tertiary Keyword (Optional) Computing middleware

Primary authors

Davide Salomoni (INFN CNAF) Giacinto Donvito (INFN-Bari) Isabel Campos Plasencia (Consejo Superior de Investigaciones Cientificas (CSIC) (ES)) Jesus Marco (Universidad de Cantabria (ES)) Jorge Oliveira Gomes (LIP Laboratorio de Instrumentaco e Fisica Experimental de Particulas) Luciano Gaido (Universita e INFN (IT)) Lukasz Dutka (Cyfronet) Marcin Plociennik (PSNC) Marcus Hardt (Kalrsruhe Institute of Technology) Patrick Fuhrmann (DESY) Roberto Barbera (Universita e INFN, Catania (IT))


Alessandro Italiano Italiano (INFN-CNAF) Mr Alvaro Lopez Garcia (CSIC) Alvise Dorigo (Universita e INFN, Padova (IT)) Andrea Ceccanti Bartosz Kryza (ACC Cyfronet-AGH) Bas Wegh (KIT) Christian Bernardt (Deutsches Elektronen-Synchrotron (DE)) Diego MICHELOTTO (INFN CNAF) Doina Cristina Aiftimiei (INFN) Elisabetta Ronchieri Emidio Giorgio (Istituto Nazionale Fisica Nucleare (IT)) Enrico Fattibene (INFN - National Institute for Nuclear Physics) Federica Fanzago (Unknown) Frederic Schaer (CEA) Germán Moltó (Universidad Politécnica de Valencia) Giovanni Aloisio (Unknown) Giuseppe Andronico (INFN SEZIONE DI CATANIA) Ignacio Blanquer (UPV) Jiri Sitera (Unknown) Joao Pina (LIP, Lisbon) Lionel Schwarz (IN2P3) Lisa Zangrando (Universita e INFN, Padova (IT)) Ludek Matyska (Unknown) Marco Fargetta (INFN Catania) Marco Verlato (Universita e INFN, Padova (IT)) Marica Antonacci (INFN Bari) Mario Jorge Moura David (LIP Laboratorio de Instrumentaco e Fisica Experimental de Particulas) Massimo Sgaravatto (Universita e INFN, Padova (IT)) Mathieu Velten (CERN) Matthew James Viljoen (STFC - Rutherford Appleton Lab. (GB)) Michal Owsiak (PSNC) Michał Urbaniak Miguel Caballer (UPV) Milos Mulac (Unknown) Pablo Orviz Fernandez (Universidad de Cantabria (ES)) Paul Millar Peter Solagna (EGI.eu) Pierre-Francois Honore (Instituto de Fisica Corpuscular (IFIC) UV-CSIC) Ricardo Brito Da Rocha (CERN) Riccardo Bruno (INFN Catania) Sandro Fiore (Unknown) Sara Vallero (Universita e INFN Torino (IT)) Sonia Taneja (Universita e INFN, Bologna (IT)) Stanisław Jankowski (PSNC) Stefano Bagnasco (I.N.F.N. TORINO) Stefano Dal Pra (INFN) Sylvain Reynaud (CNRS) Tim Bell (CERN) Tomasz Szepieniec Vincent Llorens (CNRS) Vladimir Sapunenko (INFN-CNAF (IT)) Zdenek Sustr (Czech Technical University (CZ))

Presentation materials