10-14 October 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

“Big Data” in HEP: A comprehensive use case study

13 Oct 2016, 14:00
GG A+B (San Francisco Mariott Marquis)


San Francisco Mariott Marquis

Oral Track 5: Software Development Track 5: Software Development


Oliver Gutsche (Fermi National Accelerator Lab. (US))


Experimental Particle Physics has been at the forefront of analyzing the world’s largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems collectively called “Big Data” technologies have emerged to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (skimming and slimming experiment-specific data formats), these new technologies use different approaches and promise a fresh look at analysis of very large datasets and could potentially reduce the time-to-physics with increased interactivity.
In this talk, we present an active LHC Run 2 analysis, searching for dark matter with the CMS detector, as a testbed for “Big Data” technologies. We directly compare the traditional NTuple-based analysis with an equivalent analysis using Apache Spark on the Hadoop ecosystem and beyond. In both cases, we start the analysis with the official experiment data formats and produce publication physics plots. We will discuss advantages and disadvantages of each approach and give an outlook on further studies needed.

Primary Keyword (Mandatory) Analysis tools and techniques
Secondary Keyword (Optional) Data processing workflows and frameworks/pipelines
Tertiary Keyword (Optional) Software development process and tools

Primary author

Oliver Gutsche (Fermi National Accelerator Lab. (US))


Dr Alexey Svyatkovskiy (Princeton University) Dr Bo Jayatilaka (Fermi National Accelerator Lab. (US)) Cristina Ana Mantilla Suarez (Fermi National Accelerator Lab. (US)) James Kowalkowski (Fermi National Accelerator Laboratory (FNAL)) Jim Pivarski (Princeton University) Matteo Cremonesi (Fermi National Accelerator Lab. (US)) Saba Sehrish (Fermilab)

Presentation Materials

Your browser is out of date!

Update your browser to view this website correctly. Update my browser now