Choose timezone
Your profile timezone:
WLCG and EGI accounting meeting
→
Europe/Zurich
Description
Attended:
Adrian and Tom (STFC), Gergely, Alessandro, Catalin, Matt, Renato, Andrea (EGI), Nikolaos, Maria, Julia (CERN)
Demo:
Nikolaos ran a demo of the current prototype. It already includes data for local submissions and classical Grid sites, although cloud data has not yet been integrated.
Cloud information follows a slightly different data structure and includes additional attributes such as the number of VMs, memory consumption, and related metrics. Support for cloud data is expected to become available in the prototype within the next couple of weeks.
All information — including cloud resources, local submissions, and classical Grid sites — will be integrated into a common schema and displayed through a unified UI, enabling comparisons across different resource types. A new “resource type” attribute will be introduced. The default view will show only metrics common to all resource types, while cloud-specific metrics will also be available, though not enabled by default.
At the bottom of the current page, there is an automatic validation section showing the results of consistency checks between the legacy EGI portal and the prototype. Full consistency has been confirmed, except for a few sites affected by double counting in the EGI portal; the prototype displays the correct values for those cases.
Comments from EGI colleagues:
EGI colleagues requested a table similar to the one available in the legacy EGI portal, showing monthly data metrics. Julia noted that navigation in the current portal is not particularly convenient, since users can only view a single metric at a time, making it difficult to obtain a comprehensive overview or combine multiple metrics. However, implementing the requested functionality would require an additional view (one per metric), while the default view would remain a summary table containing totals for different metrics.
Gergely asked whether user-level information would be available. Julia explained that this functionality will not be included in the summer deployment, as it raises privacy and authentication/authorization concerns. The request is expected to be addressed later by STFC as part of the system’s future evolution.
Catalin asked about the technologies used, deployment, and dependencies. Nikolaos explained that the current implementation relies on InfluxDB as the repository backend and Grafana for visualization. CERN plans to move away from InfluxDB in the near future. The system also includes a message queue architecture: APEL consumes raw data, aggregates it, and publishes monthly summaries to the message bus, from which the prototype retrieves the data.
Deployment is relatively straightforward. CERN uses Kubernetes, while STFC relies on virtual machines. Adrian confirmed that deployment activities at STFC are progressing well and that no major issues have been encountered. STFC is expected to expose the prototype for evaluation and feedback very soon.
Julia added that CERN and STFC agreed that the next high-priority task is to remove the dependency on the legacy APEL central service for WLCG data. Under the new approach, the prototype will consume raw data directly, without APEL aggregation. Since the raw data already contains monthly summaries from sites, only lightweight pre-processing should be required. This change is considered important because WLCG datasets may evolve significantly in the near future, and the system must remain highly flexible.
For classical Grid sites and local submissions, no aggregation is required. However, cloud information still requires aggregation across VMs, meaning the legacy APEL system will continue to be used for cloud data for a longer period. As a result, the system is expected to operate in hybrid mode starting early next year: using the legacy APEL central service for cloud data, but not for other resource types.
Julia asked to provide VOs to scientific domain mapping at least as a static list.
CHEP presentation:
Panos will share the accounting presentation slides in the coming days. Gergely raised concerns regarding the RI logos, and Julia forwarded this information to Panos after the meeting. Panos is travelling to CHEP today.
The next meeting will take place in two weeks.
There are minutes attached to this event.
Show them.