Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

IceProd Supercomputer Mode: How IceCube Production Runs on Firewalled Clusters

Nov 5, 2019, 11:15 AM
15m
Riverbank R3 (Adelaide Convention Centre)

Riverbank R3

Adelaide Convention Centre

Oral Track 3 – Middleware and Distributed Computing Track 3 – Middleware and Distributed Computing

Speaker

David Schultz (University of Wisconsin-Madison)

Description

For the past several years, IceCube has embraced a central, global overlay grid of HTCondor glideins to run jobs. With guaranteed network connectivity, the jobs themselves transferred data files, software, logs, and status messages. Then we were given access to a supercomputer, with no worker node internet access. As the push towards HPC increased, we had access to several of these machines, but no easy way to use them. So we went back to the basics of running production jobs, staging data in and out and running offline jobs on the local queue. But we made sure it still integrated directly with our dataset management and file metadata systems, to not lose everything we had gained in recent years.

Consider for promotion No

Primary author

David Schultz (University of Wisconsin-Madison)

Presentation materials