Knowledge Bases and Agents for CERN
TBD
CERN
Welcome to the Indico page for the Knowledge Bases and Agents workshop! Organized by members from different experiments and CERN, both from the management and technical side, for a complete perspective on this matter and to move forward on the same page.
The goal of this workshop is two-fold:
- To showcase and discuss the various efforts across CERN and beyond on knowledge bases and agents, generically speaking, as a bed for cross-contamination and collaboration on common issues/goals.
- To assess the current landscape concerning this matter, identify what the physics community would like to come out of it, and make plans to achieve it as a community.
We encourage you to register for any of the following reasons:
- You are interested to learn or contribute to what people are working on with respect to agents, knowledge bases, and related infrastructure in real use cases at CERN
- You would like to be a user of some of the deployments being worked on.
- You have your own use case you would like to deploy an agent or a knowledge base for, whether you are looking for (a) to deploy our/a system with your own collection of data, or (b) you have a more complex use case you’d like to brainstorm about, or collaborate with us more directly on. Submit an abstract!
- You work on a project in this space and would like to present it at the workshop
You can also submit an abstract. We encourage you to submit anything from ongoing projects to loosely formed ideas. There will be time for talks as well as a session dedicated to brainstorming, discussing plans–more open. After, there will be time to do some work, be it for formalizing said plans with others for continued collaboration, dedicated sessions for work or continued discussion, whatever is deemed useful by participants and organizers alike during the workshop.
Some of the organizers have experience in data retrieval and setting up knowledge bases, and also in the running of production-level agents, through the Archi project which has served as a testbed for agentic use in Computing Operations.
-
-
10:00 AM
→
10:30 AM
Welcome, introduction, workshop plans and goals 30m 503/1-001 - Council ChamberSpeaker: Pietro Lugato (Massachusetts Inst. of Technology (US))
-
10:30 AM
→
12:30 PM
Knowledge bases plenary, CERN AI strategy, LLM infrastructure 503/1-001 - Council Chamber
-
10:30 AM
Knowledge bases plenary 1hSpeaker: Jason Mohoney (Massachusetts Inst. of Technology (US))
-
11:30 AM
Update from CERN AI Strategy 30m
Update from CERN AI Strategy.
Speaker: Maurizio Pierini (CERN) -
12:00 PM
AIGW: CERN's LLM Service 30m
Large language models are becoming part of everyday work at CERN, from writing code to analysing documents. The CERN AI Gateway (aigw.cern.ch) provides a single, CERN-managed entry point to these models. This talk introduces the platform and its main features. Through a live demo, we will show how to create an API key and call a model. Attendees will leave being able to start using the AI Gateway on their own. The talk is aimed at all CERN users, and no prior experience with LLM APIs is required.
Speaker: Paulo Guilherme Pinheiro Pereira
-
10:30 AM
-
12:30 PM
→
2:00 PM
Lunch 1h 30m TBD
TBD
CERN
-
2:00 PM
→
3:00 PM
Security and AI tools for exploration and development 40/S2-C01 - Salle Marie Sklodowska-Curie
-
2:00 PM
Agents gone wild: You said what, somebody else said how, nobody said stop 30m 40/S2-C01 - Salle Marie Sklodowska-Curie
Agents gone wild: You said what, somebody else said how, nobody said stop
Speaker: Jose Carlos Luna Duran (CERN) -
2:30 PM
AI tools for exploration and development 30m 40/S2-C01 - Salle Marie Sklodowska-CurieSpeaker: Josh Bendavid (CERN)
-
2:00 PM
-
3:00 PM
→
3:30 PM
Coffee break 30m TBD
TBD
CERN
-
3:30 PM
→
5:00 PM
Evaluation for Operations and MCP 40/S2-C01 - Salle Marie Sklodowska-Curie
-
3:30 PM
Building and Evaluation AI Assistance for CMS Computing Operations 45m
CMS Computing Operations runs the experiment's data management and processing on the WLCG. Operators answer a steady stream of questions about Rucio rules, datasets, sites, workflows and the tools behind them, drawing on scattered documentation, code, tickets and live service data. We built an AI assistant for CMS Computing Operations on the Archi framework and deployed it for operators. It combines retrieval over this operational knowledge with read-only access to live systems such as Rucio and DBS.
To learn whether its answers can be trusted, and which design choices matter, we built an evaluation from real operator questions and expert-written cases. Each gold answer is split into facts, an LLM judge grades every answer, each question is asked five times, and setups are compared with paired confidence intervals. We compared different knowledge stores (a vector database, a knowledge graph and their combination), different language models and agents, and the production configuration. We present what we learned about building and evaluating such an assistant for day-to-day operations.
Speakers: Antonio Battaglia (Chulalongkorn University (TH)), Dmytro Kovalskyi (Massachusetts Inst. of Technology (US)), Hasan Ozturk (CERN), Jason Mohoney (Massachusetts Inst. of Technology (US)), Juan Pablo Salas Galindo (University of Wisconsin (US)), Luca Lavezzo (MIT), Pietro Lugato (Massachusetts Inst. of Technology (US)), viphava khlaisuwan -
4:15 PM
The Facility Is the Context: Building an MCP Platform for Scientific Computing 45m
Large language model agents are becoming useful for software development and scientific analysis, but making them useful on shared computing infrastructure requires more than the model itself. Agents need reliable access to the software, data, workflows, and identities that researchers already use.
This talk introduces the Model Context Protocol (MCP) as a practical interface between AI agents and scientific software. I will briefly discuss how MCP can expose existing Python libraries, databases, experiment services, Jupyter environments, and batch systems, and then show how these pieces come together in an MCP platform developed for the UChicago Analysis Facility.
The platform provides a common gateway for MCP services while handling authentication, authorization, and credential brokering separately from the individual services. This allows agents to work with resources such as Rucio, Jupyter, filesystems, and HTCondor without directly handling users' long-lived credentials.
Using examples from the ATLAS Analysis Facility and work extending this approach across experiments, I will discuss what works today, the infrastructure challenges that remain, and what an MCP-based approach could mean for making agentic computing portable across scientific facilities.
Speaker: Dr Giordon Holtsberg Stark (University of Chicago (US))
-
3:30 PM
-
10:00 AM
→
10:30 AM
-
-
9:00 AM
→
10:30 AM
Ongoing efforts in knowledge bases 40/S2-A01 - Salle Anderson
-
9:00 AM
LHCb Brain: Knowledge Base 20m
The LHCb Brain Knowledge Base is the searchable corpus that grounds LHCb Brain reasoning. CDS documents, GitLab code, merge requests and issues, TWiki, Indico, and LHCb HTML domains are collected by dedicated scrapers running as cron jobs on Kubernetes in the CERN PaaS. For PDF documents, Docling on GPU extracts the text and reconstructs LaTeX formulas. The text is chunked, and a header is written at the top of every chunk, recording the document title, URL, section, date, and privacy constraints. The full chunk is embedded with EmbeddingGemma 300M, so a later RAG search can match the title and the URL as well as the prose. An alternative keyword search path is available with BM25. The vectors are stored in a FAISS index. Every time the Brain's router chooses to query the Knowledge Base, it searches the indexed data, passes the matching body text and the date to the language model, and keeps the URL for the numbered source list, so the user can trace which sources the Brain is using. There is ongoing work to fine-tune the embedding model on PhD theses with benchmark evaluation.
Speaker: Pawel Kopciewicz (CERN) -
9:20 AM
ATLAS OpenSearch: A Knowledge Base for Agents & People 20m
ATLAS OpenSearch meshes over twenty CERN sources — TWiki,
Zensical/MkDocs, GitLab, Indico, CDS, CDS Videos, arXiv, Inspire,
JIRA, Confluence, analysis notes, and more — into aliased,
independently indexed knowledge bases, searchable through the ATLAS
documentation site and used by over 600 collaboration members. Agent
and MCP interfaces came later, built on top of that same index.Each source has its own automated ingestion pipeline, combining
keyword search with automatically generated embeddings behind models
hosted in-cluster, swappable without downstream change. Dense jargon
and acronyms expand via a token graph at query time, so the same
concept is found however it's written, and relevance changes are
paired-tested against real, human-sourced queries before shipping.MCP agents expose search, semantic retrieval, and graph lookups to
external clients, and dedicated agents are built into individual
result types. ATLAS OpenSearch backs agent platforms inside and
outside ATLAS, including an in-cluster MCP server and UChicago
Analysis Facility's MCP.Next: combining sources into additional knowledge graphs, assistants
for remaining content types, and integration with Chatlas.Speaker: Jason Oliver (Adelaide University (AU)) -
9:40 AM
chATLAS: Knowledge Retrieval and Agent Infrastructure for ATLAS 20m
chATLAS is a deployed Retrieval-Augmented Generation assistant providing source-attributed conversational and search access to more than 30,000 documents from several ATLAS documentation systems. It combines dense and keyword retrieval with domain-adapted embeddings, contextual retrieval, and ranking. A benchmark constructed from ATLAS documentation is used to evaluate retrieval and answer quality and guide the system’s design.
First deployed in late 2023, chATLAS now includes CERN authentication, intent routing, safety handling, Mattermost integration, and a Model Context Protocol server supporting its use in agentic workflows. We will present the system’s architecture, evaluation results, and operational lessons, and discuss opportunities to share retrieval infrastructure, interfaces, and evaluation approaches with other knowledge-base and agent projects at CERN.
Speakers: Alex Sopio (The University of Edinburgh (GB)), Daniel Thomas Murnane (Niels Bohr Institute, University of Copenhagen), Gabriel Facini (UCL (GB)), Joe Egan (University College London), Juerg Beringer (Lawrence Berkeley National Laboratory (LBNL)), Ki Ryeong Park (Columbia University (US)) -
10:00 AM
The CMS programme, structured and searchable 30m
Information about the CMS physics programme is spread across hundreds of publications and internal analysis notes and is accessible mainly through manual search. We present two synergistic approaches developed at UW–Madison, both running on-premises with open models.
MITRA is a retrieval-augmented assistant over CMS analysis documentation - internal and public. A two-tier vector database first identifies the relevant analysis and then searches its full documentation, answering context-specific questions with citations.
An AI-driven Cartography for Discovery at the LHC extracts selection logic, object definitions and kinematic regions from published analyses into a structured, queryable database. On the Run-2 CMS BSM searches, it reaches about 90% accuracy across around 150 papers, validated against a hand-built ground truth. The database answers questions that retrieval alone cannot: which analyses cover a given final state, where they overlap, which triggers they rely on, and which regions remain unexplored.
The two layers reinforce each other. Retrieval over internal notes can verify and complete details missing from publications, and the structured database can serve as a knowledge source for agents.
We will present both systems and use cases including overlap studies and cross-checks of observed excesses.
Speakers: Abhishikth Mallampalli (University of Wisconsin (US)), Deborah Pinna (University of Wisconsin (US)), Sridhara Dasu (University of Wisconsin Madison (US)), Tulika Bose (University of Wisconsin (US))
-
9:00 AM
-
10:30 AM
→
11:00 AM
Coffee break 30m TBD
TBD
CERN
-
11:00 AM
→
12:00 PM
Ongoing efforts in knowledge bases 40/S2-A01 - Salle Anderson
-
11:00 AM
Keeping LEP-era knowledge usable: a DELPHI knowledge base for physicists and AI agents 30m
The DPHEP initiative preserves collider data, but they remain useful only as long as the tacit knowledge surrounding them survives. While the JADE reanalysis succeeded years after PETRA shut down because former collaborators were available to explain the detector, software, and analysis choices, that live expertise is now rapidly disappearing for the LEP experiments.
As a pilot project, we are constructing a comprehensive knowledge base for DELPHI to serve both human physicists and AI agents. The corpus comprises roughly 3,300 documents (113,000 pages)—including 1,560 internal notes, 920 conference contributions, 460 publications, and 320 theses—with about 40% digitised from scans. The public DELPHI internal notes are particularly critical, capturing detector and analysis details omitted from formal papers. All documents are converted to Markdown and indexed for keyword and semantic search. An LLM-generated wiki of linked topic pages, to be reviewed by former collaborators, is hosted on the web, while a Model Context Protocol (MCP) server enables agents to navigate legacy FORTRAN code stored in PATCHY format via call trees and subroutine descriptions.
We evaluate the system using tasks drawn from published DELPHI analyses: recovering measurement conventions, tracing selection cuts from notes to implementation code, and guiding an agent to reconstruct D*± mesons using DELPHI data converted to the EDM4hep event data model. Through layered ablation tests—evaluating models without the archive, with papers only, with internal notes added, and with the wiki — we isolate the contribution of each layer. Our findings contribute to establish a scalable methodology for long-term physics data preservation across the other LEP experiments.
Speakers: Dietrich Liko (Austrian Academy of Sciences (AT)), Dr Ulrich Schwickerath (CERN) -
11:30 AM
What OpenSearch offers & how CERN runs it 30m
Behind most knowledge bases and LLM agents there is a retrieval problem: finding the right information quickly, and only what the user is allowed to see. OpenSearch, an open-source (Apache 2.0) search and analytics engine, has grown from a scalable full-text engine into a platform that covers this ground in one place. This talk gives an overview of what OpenSearch offers for knowledge bases and agents. We start from keyword search and its limits, then cover dense and sparse vector search with several engines and compression options, hybrid search, and models hosted in the cluster or reached through external connectors such as the CERN AI Gateway. We show how ingest and search pipelines extract, chunk and embed documents, and then rewrite queries, fuse scores and re-rank results, without application code. On the agent side we present the available agent types, agentic search that turns natural-language questions into queries, built-in and external MCP servers, agentic memory, and Agent Health for observing and evaluating agents.
We then present the CERN OpenSearch service: over a hundred clusters and more than a petabyte of data, run as a service for the CERN and WLCG communities, with single sign-on, e-group-based access control and a support model for users. We close with why OpenSearch is a good fit for knowledge bases and agents.
Speaker: Sokratis Papadopoulos (CERN)
-
11:00 AM
-
12:00 PM
→
1:00 PM
Ongoing efforts in physics analysis 40/S2-A01 - Salle Anderson
-
12:00 PM
License to Fit 20m
Large Language Models (LLMs) and LLM based agentic systems have transformed how work is done in many industries, especially software development and mathematics.
As physicists we have to ask ourselves how we can benefit from these developments, and what the best use cases are.
Previous case studies have shown that agents are generally capable of using HEP tooling and supporting researchers.
But, especially in more autonomous setups, agents struggle with nuances and conventions, which can lead to subtle mistakes that invalidate results.
This makes these tools too unreliable for unsupervised use in active research, as validation and error hunting can quickly become as much or more work than the task itself.
Our goal is to provide agents with physics-specific knowledge bases which provide detailed, traceable and searchable background information.
This should help make agents more reliable and their work more verifiable.
In our current project we use agents for deep literature research to construct a taxonomy of background estimation methods.
The key feature of the workflow is that every piece of information that lands in the final taxonomy is traceable back to its original source.
The taxonomy itself can then be used as a grounding knowledge base for agents that conduct HEP analyses.Speakers: Anna Hallin (University of Hamburg), Dr Carsten Burgard (Hamburg University (DE)), Gregor Kasieczka (Hamburg University (DE)), Sarah Heim (Deutsches Elektronen-Synchrotron (DE)), Tim Lukas (University of Hamburg) -
12:20 PM
Using MLFlow for Developing Agentic Applications 20m
With version 3 of MLFlow, there is now some significant support for developing agentic applications. It helps with evaluating prompts, skills, models, and MCP servers and helps developers evaluate agent response quality.
We will describe some of the more important features and show how the Agentic IRIS-HEP project is making use of it.
Speaker: Benjamin Galewsky (Univ. Illinois at Urbana Champaign (US)) -
12:40 PM
RooAgent: An LLM Agent for Root-Based High Energy Physics Analysis 20m
We present RooAgent, a Python package that provides an LLM-based interface for ROOT-based high energy physics data analysis. Analysis functions covering histogram inspection, event selection, visualisation of kinematic distributions, fitting, and significance estimation are exposed as tools that an LLM agent invokes, supplying the required arguments in response to plain-language prompts. The analysis logic is implemented in PyROOT. Two operating modes are supported: a Model Context Protocol (MCP) server for use with the Claude CLI or the OpenAI Codex CLI, and a LangGraph-based agent that selects its LLM provider, Claude or OpenAI, at runtime. We test RooAgent on multi-step workflows, including simulated pp → ZH (Z → ℓ⁺ℓ⁻, H → bb̄) events and ATLAS open data for H → ZZ* → 4ℓ. The package is available on PyPI, and the source code is hosted on GitHub.
Speaker: Aman Desai (Adelaide University (AU))
-
12:00 PM
-
1:00 PM
→
2:20 PM
Lunch 1h 20m TBD
TBD
CERN
-
2:20 PM
→
4:00 PM
Ongoing efforts in physics analysis 40/S2-A01 - Salle Anderson
-
2:20 PM
FLUKA coding Agent 20m
Language models process code as a flat token window, but the information flow of a large codebase is obfuscated from this viewpoint: call structure, control flow, and shared state are graph-shaped and repository-wide. Our project proposes coupling a graph transformer over the property graph of such a codebase to a pretrained decoder LLM. Our target is FLUKA, a general-purpose Monte Carlo code for particle transport and interactions developed by a collaboration lead by CERN. FLUKA consists of validated physics embodied in a large FORTRAN77 codebase, which is captured as a graph. The graph is built by a deterministic parser complemented by LLM annotations for variable and code descriptions. We've spent some time trying a VLM-like coupling, where the graph is read by an independently pretrained model and injected into the LLM. We've found that graph models have great difficulty carrying interpretable information between nodes, especially across multiple layers. This is why we're now shifting to a graph as attention-mask approach.
Speakers: Noah Julian Stauble (ETH Zurich (CH)), Stefano Marin -
2:40 PM
From reproduction to discovery: benchmarking AI-assisted workflows with collider resonance searches 20m
The rapid development of knowledge bases and AI agents offers new opportunities within the increasingly complex workflows of experimental particle physics. Their use in physics analyses, however, requires establishing how such systems can be benchmarked, made reproducible and scientifically validated, and how improvements over expert-designed analyses can be understood.
This contribution proposes dijet and paired-dijet resonance searches as realistic benchmark environments for exploring these questions. A possible roadmap would begin with reproduction of established analysis workflows and results and progress toward optimization of concrete analysis problems. Examples include jet pairing in paired-resonance searches and robust multidimensional dijet search strategies exploiting both invariant mass and angular information, where alternative approaches can be evaluated through signal efficiency, background discrimination, mass resolution and ultimately expected sensitivity, while understanding the origin and robustness of any improvement.
Run-3 data and more complex topologies, including asymmetric resonance decays, could subsequently test whether validated AI-assisted approaches can progress from reproducing established analyses toward improving and expanding existing search strategies.Speaker: Niki Saoulidou (National and Kapodistrian University of Athens (GR)) -
3:00 PM
Towards Agent-Driven HEP Analysis: Physics Closure and Harness Development for a RAG-Based AI Co-scientist 20m
We present an update on our prototype Retrieval-Augmented Generation (RAG) and agentic LLM tool designed to accelerate and support high-energy physics analyses. As a case study, we applied the system to the published 2016 Λb → Λγ Run 2 LHCb analysis. Reproducing legacy workflows is often slow and error-prone due to fragmented code, dispersed documentation, personnel turnover, and software evolution over multiple data-taking periods. Our tool integrates semantic retrieval, code-understanding capabilities, and autonomous agent actions to navigate these challenges.
The system ingests heterogeneous knowledge sources, including archived analysis scripts, notes, published documentation, and the modern LHCb software stack, and uses a vector database to build a unified retrieval layer. Built on general-purpose LLMs, the assistant can propose analysis strategies, generate or repair scripts, and autonomously perform several reconstruction and analysis tasks. With the current prototype we reach near physics closure with the published analysis. In internal tests, the prototype demonstrated the potential to compress workflows that normally require months or years of manual effort.
We have also used the framework to revive a stale analysis, driving it to near closure in a matter of months, pending physics committee review and approval.
We are now extending the tool with an agent harness that records the analysis state (selections, cuts, fit configurations, intermediate outputs), so the agent can resume from a checkpoint, compare runs, and roll back after a failed step instead of restarting from scratch. On top of this harness we are implementing an autoresearch-style loop in which the agent proposes, tests, and evaluates candidate selection cuts against a defined figure of merit, starting with the optimisation of the Λb → Λγ selection.
We discuss the tool's architecture, agentic capabilities, and safeguards, as well as its current limitations. Verification is ongoing to evaluate reliability, detect errors, and benchmark performance against non-RAG LLM baselines. These case studies highlight the promise of RAG-augmented, agent-driven systems in restoring, verifying, and accelerating complex HEP analyses, pointing toward future end-to-end AI-assisted analysis pipelines.
Speakers: Alexia Martorell I Granollers (La Salle, Ramon Llull University (ES)), Arantza De Oyanguren Campos (Univ. of Valencia and CSIC (ES)), Carlos Abia Sarria (Univ. of Valencia and CSIC (ES)), Cilicia Uzziel Perez (La Salle, Ramon Llull University (ES)), Miriam Calvo Gomez (La Salle, Ramon Llull University (ES)), Xavier Vilasis Cardona (La Salle, Ramon Llull University (ES)) -
3:20 PM
Agentic AI for HEP Analysis: From Autonomous Measurements to Verifiable Ones 20m
Experimental high energy physics analysis is traditionally a multi-year effort dominated by repetitive implementation that demands little physics insight. LLM-based AI agents can now autonomously execute substantial portions of this pipeline, collapsing the implementation bottleneck to roughly ten hours of wall-clock time. This talk surveys the rapidly developing landscape of agentic AI in high energy physics, anchored by Just Furnish Context (JFC), a framework built on Claude Code that integrates autonomous analysis agents, literature-based knowledge retrieval, and a multi-agent review system mirroring the tiered review structure of a real collaboration. Given only a short physics prompt and a dataset, JFC plans and executes the full chain: event selection, background estimation, systematic uncertainty quantification, blinded statistical inference, and publication-grade documentation, with human oversight concentrated at a single unblinding gate. Demonstrations on ALEPH, DELPHI, and CMS open data include a CMS Run-1 $H\to\tau\tau$ measurement consistent with the published value and the first primary Lund jet plane density measured in $e^+e^-$ collisions, to our knowledge the first novel HEP measurement produced autonomously by an AI agent.
I will situate these results among other recent agentic efforts across the community and confront the question they force: how do we know an agent’s analysis is correct? Published measurements sit in every training corpus, so recovering a known value proves little, and reading the output, whether by an expert or an LLM judge, does not scale. I will preview ongoing work that holds agents to the standard physicists apply to their own methods: injection and closure tests on samples where the truth is known but hidden from the agent. As implementation cost collapses, the bottleneck shifts from coding capacity to physics ideas, review bandwidth, and trust.
Speakers: Andrzej Novak (Massachusetts Inst. of Technology (US)), Dolores Garcia (Massachusetts Inst. of Technology (US)), Eric Anton Moreno (Massachusetts Institute of Technology (US)), Philip Coleman Harris (Massachusetts Inst. of Technology (US)), Samuel Kai Bright-Thonney (Massachusetts Inst. of Technology (US)), Yiyang Zhao (Massachusetts Inst. of Technology (US)) -
3:40 PM
Reward-Driven HEP Challenges for Self-Improving Agentic AI via Reinforcement Learning 20m
Recent advances in agentic AI have demonstrated the potential to automate complex high-energy physics (HEP) workflows. We focus on the regime of HEP tasks where frontier models still struggle to provide correct and complete solutions, and introduce an open-ended optimization task format with continuous, automatically evaluated rewards. Unlike conventional benchmarks focused on task completion, these tasks target reinforcement learning (RL) environments to generate agentic improvement on state-of-the-art HEP challenges. We benchmark frontier reasoning models on several HEP tasks and demonstrate an initial proof of concept in which a small language model improves through RL using automated feedback without human intervention. By connecting challenging HEP problems with verifiable scientific rewards and RL, we aim to advance agentic AI from reproducing established workflows to autonomously discovering better scientific solutions, paving the way toward self-improving scientific agents.
Speakers: Alexander Yue, Julia Lynne Gonski (SLAC National Accelerator Laboratory (US)), Qibin Liu (SLAC National Accelerator Laboratory (US))
-
2:20 PM
-
4:00 PM
→
4:30 PM
Coffee break 30m TBD
TBD
CERN
-
4:30 PM
→
5:30 PM
Ongoing efforts in knowledge bases: open discussion and decide on focused sessions (Thursday) 40/S2-A01 - Salle Anderson
-
4:30 PM
→
5:30 PM
Ongoing efforts in physics analysis: open discussion and decide on focused sessions (Thursday) 40/S2-A01 - Salle Anderson
- 5:30 PM → 5:50 PM
-
9:00 AM
→
10:30 AM
-
-
9:00 AM
→
10:30 AM
-
9:00 AM
A dedicated agent for ATLAS ITk pixel module production: LocalDB Agent 30m
The ATLAS ITk pixel-module local database (LocalDB) is a critical MongoDB system that tracks thousands of pixel modules and their components (bare modules, flex PCBs, and frontend chips) throughout complex assembly and testing workflows.
We have developed and deployed an agent that enables researchers to query this database in natural language. The agent routes queries through specialized skills to a suite of 38 domain-specific tools exposed via MCP.
The LocalDB Agent follows a decoupled Agent–LLM–MCP–MongoDB architecture and comprises two specialized agents: a main agent for database queries and a diagnostic agent for system debugging. Eight reusable skills address recurring use cases such as sign-off reports, production rates, and stage censuses, while a unified tool layer and strict guardrails improve efficiency and prevent uncontrolled query loops.
The system has been validated against benchmark queries and is now actively used to monitor ATLAS ITk pixel-production metrics on the shared LocalDB instance used by three institutes located the Paris area: CEA IRFU, CNRS IN2P3 LPNHE and IJCLab.
This presentation will describe the architecture, lessons learned, and performance of a real-world agent deployment in a particle-physics detector production environment. We will discuss the trade-offs between generality and specialization, the role of structured prompts and reusable skills, and opportunities for cross-experiment collaboration on shared challenges in knowledge bases and agentic workflows.
The authors acknowledge the use of AI-based tools for language editing and formatting assistance of this abstract.
Speakers: Francesco Crescioli (Centre National de la Recherche Scientifique (FR)), Matthias Saimpert (CEA - IRFU - Université Paris-Saclay (FR)) -
9:30 AM
Orbit: An agentic RAG assistant that keeps sensitive data inside CERN 20m
Members of CERN's Occupational Health & Safety and Environmental Protection (HSE) Department work with data that is often sensitive, and personal correspondence. General-purpose AI assistants are not an option for this material, because they send it to external providers and give little control over what gets read. Orbit is a desktop application for macOS, Windows and Linux built to close that gap. It answers questions in natural language by searching the user's mailbox and calendar and CERN's own systems (EDMS, CDS, Indico, Confluence, Jira, ServiceNow, RAISIN, REMUS and the CERN documentation sites), and every answer cites its sources.
Data stays inside CERN. There is no central server. The mailbox is indexed on the user's own computer, the server listens only on localhost, and the models run through the CERN AI Gateway or locally through Ollama or oMLX. The retrieval is agentic but bounded. Code decides which services are searched and runs them in parallel. The model may ask only for a fixed list of read-only follow-up reads within the same service, for a limited number of rounds. Users can attach their own files, ask for charts, or turn an answer into a PDF, Word, PowerPoint or Excel documents. They can also create email drafts, which open in Outlook unsent, and schedule recurring reports. The result is a working assistant that HSE staff can use every day and that respects the confidentiality of their work.
Speakers: Adrien Ledeul (CERN), Alex Savulescu (CERN), David Villamil Fernandez (CERN), Gustavo Segura (CERN) -
9:50 AM
Orbit’s Brains: how an AI assistant at CERN remembers what you decide 20m
AI conversations often start from scratch. Decisions, corrections and preferences get lost between chats, while a team’s knowledge remains scattered across mailboxes and documents.
Brains, Orbit’s memory, keeps that knowledge as readable Markdown notes, one per concept. You can organize brains around your own work, a team or a project. Notes link to each other, helping Orbit retrieve related context, and several brains can contribute to the same answer. A shared brain gives colleagues a common record of decisions, procedures and lessons learned.
The rule is simple: the model proposes, you decide. Orbit drafts what is worth keeping from a conversation. You choose the destination brain, review the draft, edit it and select what to save. Nothing is written without your approval. When answering from memory, Orbit cites the notes it uses and identifies their brain. It treats them as working context, not official documentation, and updates preserve original authorship and contribution history.
You control which brains a conversation can consult and which brain receives a reviewed update. This lets you keep personal context separate from knowledge intended for a team or project. Notes are stored and indexed locally, sharing follows the access permissions of each brain’s folder.
In this talk, we follow one decision from a chat into a note and back into a later answer. We show how brains support individual work and collaboration, and discuss what can still go wrong: outdated knowledge, sensitive information and two people editing the same note at once.
Speakers: Adrien Ledeul (CERN), Alex Savulescu (CERN), David Villamil Fernandez (CERN), Gustavo Segura (CERN) -
10:10 AM
Eigentic 20m
-
9:00 AM
-
10:30 AM
→
11:00 AM
Coffee break 30m TBD
TBD
CERN
-
11:00 AM
→
12:00 PM
-
11:00 AM
WisDQM Evolution: Towards Benchmarking, Knowledge Graphs, and More 20m
Efficient and effective operation and data quality monitoring (DQM) are crucial to the success of high energy physics experiments. These important functionsoften fall to shifters who range from students to experts who work with varied understandings of the complicated subsystems they oversee. Agentic LLM-powered chatbots are particularly well suited for the structure of shift taking as they can synthesize data and documentation to assist human decision making in the operation of these detectors and their DQM processes.
Our group has used the Archi framework to deploy an agentic retrieval-augmented generation (RAG) chatbot called WisDQM with access to tools and documentation to allow it to assist shifters in the operation of the CMS experiment. During the conclusion of Run 3 in 2026, WisDQM was available for shifter usage at FNAL Remote Operations Center primarily for online DQM shifters and later expanded to trigger and offline tracker DQM shifters. Since the end of CMS operations due to the LHC Long Shutdown, our team has focused on offline benchmarking of WisDQM as well as incorporating a WisDQM Operational Knowledge Graph (OKG).
We will present the updated architecture of WisDQM including its framework using Archi, its database, toolset, and interface, as well as our benchmarking efforts and development towards utilizing our WisDQM OKG and Archi v3. We will also present plans for expanding WisDQM’s capabilities with access to other CMS data sources as well as future integration of the CMS DQM Vision efforts.Speakers: Aron Soha (Fermi National Accelerator Lab. (US)), Gabriele Benelli (Brown University (US)), Jessica Tang (Brown University (US)), Juan Pablo Salas Galindo (University of Wisconsin (US)), Samantha Andrea Sunnarborg (Brown University) -
11:20 AM
CMS CRAB analysis user support agents 20m
Archi-CRAB: a knowledge-graph-grounded, tool-integrated assistant for CMS CRAB Analysis user support
Archi-CRAB is an AI assistant for analysis user support in CRAB, the CMS Remote Analysis Builder. It builds on two open-source projects, OKG, a knowledge-graph engine, and Archi, its OKG distribution for HEP and operational use, taking their CERN-team bundle (connectors, schemas, skills) as given and evolving the CRAB knowledge graph and the engine together, release by release. The graph ingests the CRAB server and client code as a code graph of files and symbols, the CRAB documentation and user guide, sanitized TWiki pages and the surrounding stack (Rucio, DBS, DAS, WMCore, FTS3, CMS monitoring, with CMSSW and the HTCondor manual next), published atomically under CI-checked invariants. It is served as an MCP endpoint with runtime skills for retrieval planning, traversal and evidence discipline, and sits with the live operational tools behind the CRAB MCP gateway on Kubernetes (ToolHive virtual MCP, CERN SSO, Cedar policies), so Claude Code and the OpenWebUI CRAB Assistant see one curated tool surface and frozen graph builds (v3, v4) are benchmarked side by side.
To measure usefulness on real work, we built the Archi-CRAB Gold Dataset from a frozen corpus of 526 CMS Talk CRAB tickets in 27 clusters. 245 tickets from 13 clusters are curated so far (176 active), plus a 69-question FAQ control set. Each item carries the user's question, the expert answer, required and optional expected atoms with quoted sources, the error origin, tool fixtures and a reproducibility recipe; atom lint, quote checks and LLM review run in CI, and releases are versioned. The hardest required atom sets the tier: Tier 1 is answerable from one documentation or code lookup; Tier 2 needs live calls to one system (CRAB, Rucio, DBS, site status, MONIT); Tier 3 chains two or more systems; Tier 4 is operator-only and is scored on correct diagnosis and escalation. The higher the tier, the more a correct answer depends on live-tool integration rather than retrieval.
Tool coverage at evaluation time: CRAB task, job, log, FJR, transfer and publication introspection (25 tools), site status, readiness and downtime (8), and, through the CompOps gateway, Rucio (36), DBS (13) and MONIT (12), plus the 6 OKG graph tools. Ablations are composed by which MCP servers the harness connects: grep only, vector only, OKG only, grep+vector (the v2 baseline) and combinations.
The first evaluation wave ran the 132 polished Tier-1 tickets three times each against the v2 vectorstore arm and OKG v4. Both arms pass about 39% of tickets with 0.54 to 0.56 average atom recall, up from 14% and 0.3 to 0.37 before curation; OKG v4 is level with the v2 baseline on correctness, faster per ticket (79 s against 94 s) but reads about 1.9 times the input tokens, and roughly a quarter of answers in either arm still contain a false claim. Every missed atom is diagnosed by owner: a retrieval miss, an OKG content gap (the per-cluster registry names the source that closes it), expired live data, a missing tool, or the dataset itself. Across the corpus, Rucio location and rule reads (about 215 tickets), site status (160) and CRAB logs (150) are the largest unblockers; xrootd path-to-site tracing, CRIC/VOMS identity and HTCondor ClassAds are the missing integrations. Tiers 2 to 4 are not yet run.
In short, knowledge-driven questions are already answered from the graph alone, with both arms level and the remaining misses traceable to named content gaps; complex live investigations remain the open problem, and the tool-demand ranking sets the order in which to close it.
Speakers: Krittin Phornsiricharoenphant (Chulalongkorn University (TH)), Hsuan-Chi Liu, Vijay Chakravarty (CERN), Katy Ellis (Science and Technology Facilities Council STFC (GB)), Stefano Belforte (Universita e INFN Trieste (IT)) -
11:40 AM
Towards autonomous Data Taking systems 20m
EP R&D WP9.1 is a cross-experiment collaboration, and its objective is to investigate the potential of Artificial Intelligence (AI) solutions to improve automation and the overall efficiency of experiment operations. Currently, data-acquisition systems for LHC experiments rely on rule-based expert systems and complex event processing for tasks such as error handling and anomaly detection. However, AI systems could offer a valid alternative, providing real-time analysis and more flexible decision-making capabilities. As part of the work package activities, the results obtained across two specific focus areas are presented: Generative AI and autonomous decision-making for experiment autopilot operations.
The Generative AI activity explores the application of Large Language Models (LLMs) in control rooms to assist Shifters with complex operational documentation and agentic applications to interact with existing applications using natural language. A Retrieval-Augmented Generation (RAG) pipeline simplifies document interaction and has been evaluated with the ALICE FLP and ATLAS Trigger operational knowledge bases. In another development, an agentic framework uses Model Context Protocol (MCP) servers to interact with existing applications via REST APIs and CLI commands. The first example is the ATLAS ELISA logbook use case, in which an LLM agent automates message retrieval, thread reconstruction, and automated message processing before running a multi-stage summarization to publish structured daily report entries directly to the logbook during the last months of Run 3. A second example is an agentic application for ATLAS Run Control, which enables natural language interaction to schedule the start and stop times of runs.
Another ongoing activity is currently investigating Reinforcement Learning (RL) approaches for making autonomous decisions within Run Control operations. To navigate systems governed by Finite State Machines (FSMs), Reinforcement Learning (RL) policies are trained across various FSM graph structures to learn minimum-path traversal.
All these activities share the common goal of reducing the need for human intervention and improving overall operational efficiency through AI-based automation. A detailed overview of the current development status, obtained results, and future objectives is presented.
Speakers: Alina Corso Radu (University of California Irvine (US)), Antoine Marzin (CERN), Giuseppe Avolio (CERN), Joaquin Hoya (Argonne National Laboratory (US)), Luigi Podda (CERN), Sylvain Chapeland (CERN), Walter Hopkins (Argonne National Laboratory (US))
-
11:00 AM
-
12:00 PM
→
1:30 PM
Lunch 1h 30m TBD
TBD
CERN
-
1:30 PM
→
2:40 PM
-
1:30 PM
OWL: a curated, self-maintaining knowledge base of ATLAS/WLCG distributed computing, served to AI agents over MCP 30m
The operational knowledge behind ATLAS distributed computing (how Rucio, PanDA, FTS, CRIC and the sites behave, and why) is spread across TWiki pages, GitHub, Indico slides, GGUS tickets, mailing lists and people's heads. Much of it contradicts itself or is out of date. ChATLAS, an ATLAS project, answers questions by retrieval-augmented generation over such documents. OWL complements it by curating the knowledge itself when it is written, rather than when a question is asked.
OWL (the ATLAS AI Librarian) is part of AEGIS (AI Experimental Grid Infrastructure Sentinel), an effort to improve the monitoring and operation of ATLAS distributed computing with AI. AEGIS runs a team of agents on the OpenClaw platform, and a set of Model Context Protocol (MCP) servers that give these agents and interactive assistants authenticated access to CRIC, Rucio, GGUS, Elasticsearch monitoring and facility documentation. OWL is the shared memory of that ecosystem. It breaks documents and conversations into atomic claims. Each claim links back to the exact text it came from, and records separately when the fact was true and when OWL learned it. New claims are compared with existing knowledge: duplicates are dropped, outdated claims are closed, and real conflicts become disputes, routed first to their authors. Where an entity is registered in CRIC, OWL uses its CRIC identity, which links knowledge to live operational state. All writes are attributed through CERN SSO.
We will describe the design and current deployment, and discuss how a shared, citable, continuously maintained knowledge base could serve agents across experiments.
Speaker: Ilija Vukotic (University of Chicago (US)) -
2:00 PM
A Code Factory for End-to-End Computing Systems Engineering and its application to the ATLAS Level-1 Monitored Drift Tube Project 20m
codefactory is designed to reduce the time spent reconstructing context, locating authoritative sources, and setting up tools when moving between repositories. The goal is practical: make engineering work more efficient without allowing a coding agent to decide what it may change or whether a result is correct. \codefactory separates orchestration, context and evidence, reusable capabilities, and project-specific effects. It records why work starts, allows one writer per mutable surface, stops when authority is unclear, and leaves validation to project tools and human review. Its memory consists of typed records scoped to a family, project, task, or private machine. Instead of loading the whole knowledge base, it follows bounded links and selects records relevant to the current task. The implementation supports onboarding, Packs and Profiles, governed Codex and oh-my-pi (OMP) campaigns, validation, remediation, integration, and recovery, but it does not grant production authority. The same model is applied to the ATLAS Level-1 Monitored Drift Tube (L1MDT) and related UMass projects. The work covers FPGA firmware and generated data, project and system network architecture, test-stand nodes such as gateways, monitoring services, microcontroller units (MCUs), and switches, and text publications including papers, reports, and slides. Repository evidence shows inspectable intent, explicit source authority, bounded effects, and cross-repository coordination, but not yet causal productivity gains.
Speaker: Guillermo Loustau De Linares (University of Massachusetts (US)) -
2:20 PM
LHCb Brain: Agents 20m
The LHCb Brain Agents (LbAgents) is the set of AI agents that assist LHCb physicists and operators with the experiment's computing systems. The first use case targets Analysis Productions, where physicists request the processing of data and simulation by opening a GitLab merge request, and a CI pipeline runs test jobs on the grid before the production is submitted. When the pipeline fails, users often cannot tell whether the problem lies in their request or in the infrastructure, and operators investigate each merge request by hand. A LangGraph agent, deployed in production, now performs this first investigation: with the read-only tools of the Analysis Productions MCP server, it reads the pipeline results, job logs and merge request changes, and drafts a root-cause report for the author. Any tool call that writes, such as posting the report, is guarded by a human-in-the-loop mechanism: the agent pauses until an operator approves or rejects it. The agent first ran on models we deployed on the CERN ML service and now uses those of the AI Gateway (AIGW) that CERN IT is setting up. As new models appear every few weeks, we are building a benchmark from operator-corrected runs traced in Langfuse, replayed with mocked tools and scored with deterministic and LLM-as-judge metrics, to find the best trade-off between accuracy, cost and latency.
Speakers: Alexandre Franck Boyer (CERN), Ryunosuke O'Neil (CERN), The LHCb Brain
-
1:30 PM
-
2:40 PM
→
3:10 PM
Coffee break 30m TBD
TBD
CERN
-
3:10 PM
→
3:50 PM
Ongoing efforts: AI+autonomous ops TBD
TBD
CERN
-
3:10 PM
Towards a Standardized Benchmark for Vision-Language Models in CMS Data Quality Monitoring 20m
The Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider generates a large volume of complex data, making manual inspection of Data Quality Monitoring (DQM) plots by shift takers increasingly difficult. Agentic AI systems built on vision-language models (VLMs) are rapidly being developed to address this, as they can interpret DQM plots against shift instructions and deliver actionable diagnostic guidance. Their growing role in detector operations makes rigorous benchmarking essential, yet no common evaluation standard exists to compare such systems or to quantify their reliability. We present ongoing work toward a benchmark framework for VLM-based DQM assistance in CMS. The framework combines a curated dataset of DQM plots from multiple subsystems, the associated shift instructions as a knowledge base, and human-written reference assessments. Model outputs are scored against these references for plot interpretation, reasoning, and the final diagnostic decision, together with operational metrics such as latency. We evaluated several VLMs within the framework and recorded the best score obtained for each. The benchmark follows community standards for scientific machine learning benchmarks. We provide this benchmark, with these best scores as reference baselines, for agentic systems under development for CMS such as Archi, to be compared against.
Speaker: Sreevani Jarugula (Fermi National Accelerator Lab. (US)) -
3:30 PM
A Multi-Source Agentic Knowledge Base for CERN's Geodetic Metrology Group 20m
The Geodetic Metrology (BE-GM) group manages more than a decade of operational knowledge scattered across heterogeneous storage systems: JIRA for day-to-day survey tickets, Confluence for policies and standard procedures, GitLab for software and infrastructure-as-code, MkDocs sites for the user-facing manuals of in-house tooling such as LGC2, and shared file stores holding survey reports, measurement logs and PDFs. The content is intrinsically multimodal - plain text sits side-by-side with coordinate tables, instrument plots, scanned sketches and photographs within a single document. BE-GM users need to be able to ask natural-language questions ("what is the meaning and use of *FRAME in an .lgc file?", "what did SUS-2988 conclude about HLSR observations?") and receive accurate, source-cited answers in seconds, regardless of which silo the answer lives in.
Taking the Archi framework - the agentic testbed developed for CMS Computing Operations - as our baseline, we have built a BE-GM-oriented deployment that extends Archi with on-demand, PAT-authenticated REST integrations for JIRA tickets, Confluence pages (including tinylink multi-hop resolution) and GitLab merge requests, together with an admin-curated web-crawl pipeline for public documentation such as lgc2.docs.cern.ch. Each integration surfaces a dedicated LangChain tool to a single ReAct-style agent, which routes queries using an explicit decision tree over source types (jira, confluence, gitlab_mr, web, local_files). Careful HTML-to-text normalisation and source-aware prompting ensure the LLM receives clean, attributable context. The system already answers heterogeneous BE-GM queries end-to-end against the live sources.
Looking ahead, we would like to use this workshop to brainstorm three open problems with the wider CERN community: (i) scale - moving from a single-user prototype to a shared BE-GM service with per-user ACL enforcement propagated across every upstream source; (ii) latency - bringing worst-case response time to a few seconds while keeping the corpus refreshed at human timescales; and (iii) multimodality - extending retrieval beyond text so the agent can reason over the engineering drawings, plots and photographs that are inseparable from survey documentation. We expect the lessons to generalise well beyond geodesy, and welcome collaboration with groups facing the same shape of problem.
-
3:10 PM
-
3:50 PM
→
4:50 PM
Ongoing efforts: AI+autonomous ops: open discussion and decide on focused sessions (Thursday) 13/2-005
-
4:00 PM
→
5:30 PM
How collaborative softwares have adapted to AI contributions TBD
TBD
CERN
https://indico.cern.ch/event/1739582/
Plan is to focus on how collaborative software projects have adapted to AI contributions (to raise awareness around being mindful towards maintainers of software).
-
9:00 AM
→
10:30 AM
-
-
11:00 AM
→
1:00 PM
Discussion session on shared tools: MCPs, KBs, harnesses etc. 6/R-012 - conference room
-
1:00 PM
→
2:00 PM
Lunch 1h TBD
TBD
CERN
-
2:00 PM
→
4:00 PM
Discussion session on agents for physics analysis, benchmarking, RL, etc. 2h 6/R-012 - conference room
-
4:00 PM
→
5:30 PM
Discussion session on community-building, proceedings 1h 30m 6/R-012 - conference room
-
5:30 PM
→
6:30 PM
Discussion on Archi project 1h 6/R-012 - conference room
-
11:00 AM
→
1:00 PM
-
-
9:00 AM
→
9:30 AM
Network Epistemology in Practice 30m 40/S2-B01 - Salle Bohr
We will present our “Network Epistemology in Practice” project, an initiative to capture knowledge generation processes in large-scale science collaborations and to analyze them from philosophical, historical and sociological perspectives. The project is funded by the European Research Council and works under a Short Term Association with ATLAS. In the project, we will use network analysis and Large Language Models to see in what communication structures which topics are discussed and how preliminary insights are turned into publication-ready texts through the internal reviewing and commenting processes. Selecting and harvesting relevant material from ATLAS internal platforms and storing it in tailormade databases as well as the careful handling of personal data are other key elements of our project.
Speakers: Adrian Wuthrich, Arno Simons, Dalal El Youssoufi, Michael Zichert -
9:30 AM
→
10:10 AM
Agents Propose, Mediators Act: Safe Actions for Operations Agents 40m 40/S2-B01 - Salle Bohr
Agents are moving from answering questions about infrastructure to operating it, for example by restarting services or resubmitting jobs. This shift makes mistakes costly: once an agent runs commands itself, one wrong step can cause damage that is hard to undo, which an agent caught in a retry loop may repeat many times. Existing safeguards do not prevent this. Prompts and command allowlists restrict which commands an agent may issue, but not whether a permitted command is safe in the current state. We therefore argue for mediation: the agent decides what should be done, but it never does it. Instead, it proposes actions with declared effects to a small trusted mediator, which checks each proposal against the system state and policy before carrying it out or refusing it. As a result, an agent stuck in a loop can only keep proposing. More generally, safety then rests on the mediator, which is small and fixed, instead of on the model, which changes with every upgrade. We built this approach for automated recovery of cloud services (ARCS 2026) and explored it for systems code written by untrusted code generators (PACMI 2026). In our recovery experiments, mediation removed nearly all harm caused by the agent, at some cost in recovery time. These results leave two questions we would like to discuss: which operations can be described precisely enough to mediate, and where should humans step in?
Speaker: Laurent Bindschaedler (Max Planck Institute for Software Systems) -
10:10 AM
→
10:40 AM
-
10:40 AM
→
12:40 PM
-
10:40 AM
Workshop summary, reflections 40mSpeaker: Jason Oliver (Adelaide University (AU))
-
11:20 AM
Experiment operations: vision for the future 40mSpeaker: Christoph Paus (Massachusetts Inst. of Technology (US))
-
12:00 PM
Physics analysis: vision for the future 40mSpeaker: Lukas Alexander Heinrich (Technische Universitat Munchen (DE))
-
10:40 AM
-
9:00 AM
→
9:30 AM