Speaker
Description
LUX-ZEPLIN (LZ) is the world’s most sensitive WIMP dark matter direct-detection experiment, acquiring petabytes of data per year using a dual-phase xenon time projection chamber (TPC) with a seven tonne active mass. User-facing metadata related to TPC conditions and data processing environments are stored in six different SQL and NoSQL databases, which historically were accessed by five bespoke programmatic and graphical interfaces that connect to unique subsets of these databases. The fragmented nature of these interfaces was difficult to maintain and confusing for end-users.
This work details how we used Helm to deploy Trino and Superset to respectively provide a unified SQL and graphical interface to access LZ metadata databases. This allowed us to immediately deprecate one graphical interface we maintain, and improve response times by up to a factor of 100 for common queries. We will also discuss our adoption of Ibis to complement Trino and Superset through its ability to create composable SQL queries with a dataframe-like python API. With minimal effort, Ibis has superseded a number of our other programmatic database interfaces, improving query expressiveness with a smaller code footprint.