Sep 24 – 27, 2019
EC-JRC Ispra
Europe/Rome timezone

Building a European wide, Bioinformatics jobs execution network

Sep 27, 2019, 9:00 AM
25m
EC-JRC Ispra

EC-JRC Ispra

European Commission – Joint Research Centre Via Enrico Fermi, 2749 I - 21027 Ispra (VA) Italy N45° 48' 36.09'' E008° 37' 16.72'' N45° 48.601 E008° 37.278 45.80998, 8.62135 https://www.openstreetmap.org/#map=17/45.80998/8.62135
HTCondor presentations and tutorials Workshop presentations

Speaker

Gianmauro Cuccuru (University of Freiburg)

Description

With more than 2,000 bioinformatics tools available, Usegalaxy.eu (https://usegalay.eu) is the biggest Galaxy instance in Europe covering most of the hottest bioinformatics topics and communities.
After one year from its public launch into March 2018, Usegalaxy.eu has reached the important milestone of 5 million jobs executed and over 6 thousand registered users.

Several computer centers across Europe are currently sharing their remote computation power to support the Usegalaxy.eu load: IT, UK, CZ, DE, PT, ES,...
To create this network of shared computational resources, we leverage:
Pulsar (https://pulsar.readthedocs.org), a TES-like service written in Python that allows a Galaxy server to automatically interact with those remote systems,
VGCN (https://github.com/usegalaxy-eu/vgcn), a virtual image which has all of the required components to act as a galaxy compute node as part of an HTCondor cluster.
Terraform (https://github.com/usegalaxy-eu/terraform), a set of scripts for safely and efficiently building the infrastructure into a modern cloud environment.
Galaxy’s job destination framework allows job execution parameters to be determined dynamically at runtime, offering a flexible option for choosing the job endpoints, and the Pulsar layer ensures execution details are correctly exchanged to correctly perform jobs into the local and/or remote HTCondor clusters.

Speaker release Yes

Primary authors

Gianmauro Cuccuru (University of Freiburg) Bjoern Gruening (University of Freiburg) Helena Rasche (University of Freiburg)

Presentation materials