24th International Conference on Computing in High Energy & Nuclear Physics

IceProd Supercomputer Mode: How IceCube Production Runs on Firewalled Clusters

5 Nov 2019, 11:15

15m

Riverbank R3 (Adelaide Convention Centre)

Oral Track 3 – Middleware and Distributed Computing Track 3 – Middleware and Distributed Computing

Speaker

David Schultz (University of Wisconsin-Madison)

Description

For the past several years, IceCube has embraced a central, global overlay grid of HTCondor glideins to run jobs. With guaranteed network connectivity, the jobs themselves transferred data files, software, logs, and status messages. Then we were given access to a supercomputer, with no worker node internet access. As the push towards HPC increased, we had access to several of these machines, but no easy way to use them. So we went back to the basics of running production jobs, staging data in and out and running offline jobs on the local queue. But we made sure it still integrated directly with our dataset management and file metadata systems, to not lose everything we had gained in recent years.

Presentation materials

IceProd Supercomputer.pdf