JCOP Framework Working Group, 25 March 2025
Slides and meeting notes are at INDICO: https://indico.cern.ch/event/1525671/
Participants: Shkarovskiy Sergey,Piero Giorgio Verdini,Martin Zemko,Pavel Fiala,Ivan Shvetsov,Luis Miguens Fernandez,Alexandru Savulescu, Andre Augustinus, Michele Donadoni,Wassef Karimeh,Dominika Liskova,Jean-Charles Tournier,Peter Bond,Piotr Golonka,Rafal Kulaga,Geraldine Thomas,Paris Moschovakos,Adrien L, Mariusz Suder,Mateusz Lechman,Peter Chochula,Clara Gaspar, Christoph Pires, Luis Granado, Cristina Vazques, Ombretta Pinazza, Valeri Falaleev, Fernando Varela,Frank Glege, Enrico Gamberini, Adrien (?)
NextGen Archiver
presented by Martin Zemko and Rafal Kulaga
- Question from the presenters: is there any interest amongst the users to have "wrappers" for the database API. e.g. for CTRL++, Python, REST ?
- clarification: as of now this is not included in the plan of work for this year...
- Frank: Has the storage for TimescaleDB been compared with what Oracle schema consumes?
- Space usage without compression is almost the same. Once compressing isacivated on TimescaleDB one gets up to 95% space savings.
- Frank: are there any High-Availability features in TimescaleDB?
- Yes, all of the solution available already for PostgreSql should be applicable.
However, currently the CERN DBoD systems do not support them.Discussions are ongoing with IT/DA to provision this for the TimescaleDB instances to be used for WinCC OA. However, this requires resources to be assigned it IT..
- Technically, we should be able to reach a similar level of availability to what Oracle offers
- Frank: sampling of data may be dangerous if applied blindly. Needs to be taken into account when applied. For some cases constant-rate sampling may not even be the right solution
- Frank advocates the CMS schema design as a "verbose" solution with self-explaining structure, allowing "any" user to query data avoiding the complexity of the standard RDB Archiver schema. "Verbose" schema could allow to provide an API for data queries.
- Rafal: TimescaleDB schema is designed around the same concepts as the "standard" RDB schema: data is in the "EVENTS" table, and meta-data (DPE names, aliases, comments) the "ELEMENTS". It is however extremely much simpler than the original schema of the RDB Archiver (500 lines vs 30000+ lines of code). Non-CMS users seem to be OK with this separation of meta-data from events data, as is shown by the variety of web tools (such as grafana frontends)
- Fernando clarified that a user-friendly and easy to use API is anyway proposed as a part of solution, as presented in the slides
- Discussion had to be stopped here to allow for the other topic to be presented. Could be continued if enough interest.
ALICE Data Management Interfaces
presented by Dominika Liskova, ALICE
- Frank: is AGRANA visible on the Internet? Is there any access control? How is it done? How do people modify their dashboard? Explained by Dominika with an example. CERN SSO is integrated.
- Frank: is there a "pre-publication" mechanism for dashboards? No, but one could always restore the previous version of a dashboard. On modification a notification is shown to all users
- Fernando: how is the scalability of the presented solution, with respect to a number of users?
- Still undergoing process; Currently working OK open to users of the ALICE collaboration
- P.Bond: the backend is containerized (Docker). Grafana is on bare metal but could use the one in containers. NGINX as reverse proxy -> plans to kubernetize everything.
- It would be great to have the architecture/deployment showed at the FWWG
- Fernando: AGRANA is connected to the DB replica, but is there any control on queries executed to prevent resource-starvation or DoS?
- Backend runs only queries coded explicitly and strictly by the developer, no arbitrary SQL
- There are protections and limits on the number of rows that could be fetched from DB.
- Rafal: a lot of inspiration and advanced techniques showb - would be keen to learn from experience.
- The code of Rafal's initial prototype has obviously been completely refactored - would be great to see how/what was done
- Rafal: could one use variables in the dashboards? Yes, even though many dashboards are static. For DP Selector, everything is based on variables, and users may use variables.
- Rafal: Is there any kind of downsampling? No, all data is always served. Possibly a future extension.
AOB, Next meetings
There are minutes attached to this event.
Show them.