24th International Conference on Computing in High Energy & Nuclear Physics

Scheduling, deploying and monitoring 100 million tasks

5 Nov 2019, 11:30

15m

Riverbank R3 (Adelaide Convention Centre)

Oral Track 3 – Middleware and Distributed Computing Track 3 – Middleware and Distributed Computing

Speaker

Prof. Andreas Wicenec (International Centre of Radio Astronomy Research)

Description

The SKA will enable the production of full polarisation spectral line cubes at a very high spatial and spectral resolution. Performing a back-of-the-evelope estimate gives you the incredible amount of around 75-100 million tasks to run in parallel to perform a state-of-the-art faceting algorithm (assuming that it would spawn off just one task per facet, which is not the case). This simple estimate formed the basis of the development of a prototype, which had scalability as THE primary requirement. In this talk I will present the current status of the DALiuGE system, including some really exciting computer science research.

Presentation materials

100 Million CHEP.pdf

100 Million CHEP.pptx

100 Million Tasks