Login

WLCG Sustainability Forum Meeting #1

→ Europe/Zurich
Zoom (CERN)

Zoom

CERN
WLCG Sustainability Forum Meeting #1
Zoom Meeting ID
67888669522
Host
Markus Schulz
Alternative host
Caterina Doglioni
Useful links
Join via phone
Zoom URL
Hide

Meeting summary 

For Disclosure of Delegation to Generative AI (https://panbibliotekar.github.io/gaidet-declaration/), see end of minutes

Summary of Talks and Key Discussion Points:

  • Forum's Purpose and Scope (Caterina Doglioni, Dave Britton, Markus Schulz)

    • This forum, previously called "Sustainability Forum," is focused on environmental sustainability due to the broad nature of sustainability (17 sustainable development goals, not all environmental).

    • It was created following plenary workshops and meetings in December and May with the HSF, identifying a need for a dedicated space to address environmental aspects of computing.

    • Topics include facility efficiency (PUE, hardware scheduling, heat recovery), lifecycle management, location impact, CPU/GPU/storage impact, and software efficiency (generators, detector simulations, workflow optimization, data replicas, compact data formats, fast simulation, but also from possible new approaches like ML, interactive analysis, quantum computing).

    • The forum discusses and acts on the three recommendations of the 2024-2027 WLCG  strategy, which are also mentioned in the European Strategy Briefing Book.

    • Mandate: ensure progress on action items related to agreeing on metrics, providing a framework for collecting energy efficiency information, and facilitating energy-efficient hardware.

    • Overall goal: develop and promote a sustainability plan covering stakeholders in software, computing models, facilities, and hardware technologies.

  • Forum Operations:

    • Public discussion forum with a broad (but focused) audience for technical discussions.

    • Identify areas for progress, set goals, follow up, and disseminate outcomes to various bodies (WLCG Management Board, HEPIX, Open Technical Forum, experiments).

    • Liaise with WLCG Technical Coordination Board and LHC Experiments, and link with existing national and international environmental sustainability activities.

    • Coordinators: Caterina Doglioni, Dave Britton, Markus Schulz. Liaisons for experiments are being established.

    • Mailing list sign-up here: https://gms.web.cern.ch/group/wlcg-sustainability-forum/details 

    • Meetings: Monthly on the first Wednesday at 16:30 (with potential adjustments, e.g., December) → will change to avoid clash with HEPIX meeting to the 3rd or 4th Wednesday of the month - next meeting (29/10) will be announced via e-mail on the mailing list

      1. Focus on one topic with sufficient depth and discussion time. 

      2. Date chosen a month in advance to avoid clashes with HSF Seminars [link] and Compute&Accelerator Forum meetings [link] - WIP: create a single category

    • Google Doc available for topic ideas here

    • May set up a Mattermost channel if the need arises

  • WLCG Server Power Consumption Monitoring (Natalia Diana Szczepanek):

    • Objective: Enhance monitoring with power consumption metrics by leveraging existing benchmarking infrastructure (Hammer Cloud, Panda, OpenSearch).

    • Challenge: Power data (IPMI, Redfish) requires elevated privileges, posing a security risk.

    • Proposed Solution: Sites implement a privileged daemon to expose power measurements to an accessible file; a benchmark plugin queries this file.

    • Progress: Feasibility study with sites, aiming for a unified approach. Hub Power Collector Github repository created to consolidate solutions.

    • Current Status: Four collectors, currently two main solutions: DESY's (used by Freiburg, INFN) and Prometheus Node Exporter (used by CERN, Glasgow).

    • Data: Collecting measurements from six sites, ten queues. Preliminary observations show expected power behavior at high load but unexpected low power regions at given loads, requiring further investigation.

    • Next Steps: Standardize site collectors, define a common interface/output, and provide a unified, recommended solution with reference implementation and deployment instructions. Data analysis is ongoing.

  • Server Power Consumption Data Collection and Analysis (Domenico Giordano):

    • Methodology similar to Natalia's work, but this approach uses a dedicated server with sysadmins to saturate a full server and collect power data via IPMI rather than job slots.

    • Data is visualized in a public table including CPU models, server configuration (SMT, CPUs, sockets, RAM), HEPScore, average power consumption, standard deviation, and HEPScore per watt for ranking.

    • Need to calibrate IPMI values against PDU values; some sites (CERN) show significant discrepancies (up to 30%) that are being investigated.

    • Aim to build a database of power consumption and performance per server model, also to help sites that cannot contribute directly.

    • Crucial to include metadata like GPU presence, RAM, and disk as they affect power consumption.

  • Green Computing at Manchester and Glasgow (Michael Sparks):

    • Ongoing studies of examining embedded carbon in infrastructure and measuring energy used by clusters and software.

    • Focus on "explore and measure with a bit of monitor" stage, aiming to capture data relative to performance and throughput.

    • Goal in the long term: improve utilization, standardize processes, identify common issues, and integrate with HEPScore.

    • Data collection: Prometheus in Manchester (10-second granularity, 24/7), IPMI in Glasgow. Monitoring power plugs, but also RAPL with and CodeCarbon for in-process CPU power.

    • Early results: reproducibility studies, collation of energy evaluation tools, and a lifecycle perspective project.

    • Future directions: standardized monitoring, carbon/energy tracking in software, code analysis to predict load/carbon, and carbon intensity-based scheduling (not for WLCG).

  • Benchmarking and Performance Updates (Emanuele Simili):

    • Updated benchmarking plots show x86 surpassing ARM in efficiency (HEP score per power).

    • Linear correlation observed between average measured power and TDP, potentially useful for approximating power when direct measurement is unavailable.

    • Accounting attempts faced issues with large data gaps when multiple VOs were running, making Prometheus database inconsistent. This led to "garbage data."

    • Need to either drop or rework the accounting tool for a common, bug-free version.

    • Attended NetZero workshop, highlighting the need for increased awareness about computing's physical impact and a working group defining metrics for power measurement/efficiency.

    • Interested in embedded carbon studies; servers can be close to 1 ton of CO2 with large error bars, will contact authors of https://arxiv.org/abs/2506.14365 .

 

Transcript

Caterina Doglioni - intro & logistics

This forum was previously known as the Sustainability Forum, but we decided to use the term environmental sustainability. This is because there are many dimensions and many aspects of sustainability. There are 17 goals for sustainable development, and not all of them are related to the environment. We use environmental sustainability as the qualification because, as Eduardo from the HSF, who has kindly advertised and joined this meeting, knows, there are also other aspects to sustainability, even for software and computing.

 

Why are we doing this? We need a forum because we identified this as one of the next steps from the two plenary workshops and the meetings in December and May, respectively, that were joined with the HSF. We touched on a number of things: efficiency of facilities, such as PUE, hardware scheduling, and heat recovery. There are also aspects that we haven't really touched on, like lifecycle management and location, and things that we might not be able to control very much, but we want to at least estimate. We talked about the impact of different CPUs, GPUs, and storage technology. There has been a lot of work done on the benchmarking side of things. There is also the impact of software efficiency because that is something that comes into play. The longer you run some software, the more energy you consume. This has to do with the experiments and packages that are widely used, generators, and detector simulations. Experiments are also concerned with optimizing things, both for environmental reasons and because it is a need to maintain the number of resources within the budget. This includes optimizing workflows, reducing replicas of data, and minimizing the impact of sharing results and data. You can talk about compact data formats, data sharing, and fast simulation. There are new approaches as well coming in, such as machine learning, interactive analysis, and quantum computing.

 

There is an interest in environmental sustainability from other projects, so these things would not really fit into the existing events or meetings to follow up on the progress. I am talking about this from the perspective of an HSF person and a WLCG person. There is also the fact that a lot of really good students have been working with us on these topics, and we started actual hands-on work on open questions alongside the really good work that was presented at the previous meetings. We will see a snapshot of that today in the two intertalks.

 

This forum is going to be discussing and acting on the three recommendations of the strategy, 2024-2027. This also made it into, at least, the draft that I have seen of the European Strategy Briefing Book. So there is a paragraph on environmental sustainability of computing for future colliders. What are the practical things to do? We need to agree on metrics and provide a framework to collect information for energy efficiency. We need to facilitate energy-efficient hardware. This also depends on experiment software readiness and common libraries availability. This is also going to be the focus of the WLCG workshop in December. The overall overarching goal is to develop and promote a sustainability plan that covers many different aspects related to the numerous stakeholders of software, computing models, facilities, and hardware technologies in the lifecycle.

 

The mandate for the forum is on the following slide, and it has been circulated. We want to ensure that progress happens on the action items concerning these three points. How will we do it? We will provide a public discussion forum that is attended by a necessarily broad audience, although we will advertise the second meeting more broadly than this one. We are expecting people who genuinely want to work and listen to join, rather than it being a big-picture kind of meeting. We can have big-picture meetings, but these are more technical meetings. We want to identify areas where progress is needed, set goals, hopefully leading to activities and plans by the people working in it, then follow up on them, and then communicate and disseminate the outcomes to various bodies, such as the WLCG Management Board and HEPIX and the Open Technical Forum, but also potentially the experiments. We will also look for feedback from all of these. 

 

There is also the matter of liaising with the WLCG Technical Coordination Board and working closely with the LHC Experiments to follow up on their activities and feedback. An important part is also to link up with existing activities on environmental sustainability that are emerging, both national and international activities. There is a backup slide about that. The coordinators are me, Dave, and Marcus, and we intend to work closely with domain experts for all the topics above. There are also liaisons for the experiments. We do not have a full list, but we could probably put it out. We count on them to also help us propagate the news to the right people. For this first meeting, we did not really expect anything because we had not met with them yet, but this is a work-in-progress kind of thing. If you are not already there, you can sign up to the mailing list to receive announcements on when the meetings are, and contact us with the other mailing list. We can also set up a Mattermost channel, but we have not done it yet. If you think that we should have a Mattermost channel, maybe we can. It might be useful because some of us are already talking in mostly national Mattermost channels, so maybe we want to make that more public. We will see.

 

Today, we are having the first meeting, and we would like to stay within an hour. The slot is generally once a month on the first Wednesday at 16:30, unless we announce it otherwise. I think in December, we will have to move the meeting, both because there is the WLCG workshop and because it conflicts with another, I think it is an HSF seminar. So we want to avoid collisions with the HSF and compute an accelerator forum. We will focus on one topic and try to go into sufficient depth on that, and have enough time for discussions. We are not that set on the topics. There are a lot of things that we want to cover. There is a Google Doc that I put here with ideas that we have already collected, just themes that we have identified, but anyone can add here what they want to talk about and why. Then we can think about how to put it together so that we have an agenda that is scheduled well in advance, so people can contribute as well. That was it from me. I do not know if anyone else has questions about the introduction or the process. Dave is also here. Domenico, you have a raised hand.

 

[Domenico Giordano]: Yes, Caterina, thank you. I have a simple question about the time slot. The first Wednesday of the month is also the Wednesday when the benchmark working group meets at 3. We typically are not so good at saying one hour; there are a lot of topical discussions. 

 

[Caterina Doglioni, Dave Britton]: We will revisit the date (possibly keeping on a Wednesday). 

Natalia Diana Szczepanek - transcript 

 

[Natalia Diana Szczepanek]: Hi, I'm Natalia, part of the HEPIX benchmarking Working Group. Today, I wanted to give an update on WLCG Server Power Consumption Monitoring. This topic started at the WLCG workshops in Paris this year, where I presented on enhancing drug monitoring with power consumption metrics. I've included a few slides as a short recap.

 

We're using the existing submission infrastructure for benchmarking jobs to various sites. For ATLAS, for example, we use Hammer Cloud, which automatically submits jobs via Panda to different SKUs. Inside the jobs, we collect data into our OpenSearch database, giving us a good overview of site performance. Using this infrastructure, we also wanted to start collecting additional metrics, specifically power consumption. The idea was that we already have plugins in place to collect information from the system and hardware.

 

When we submit jobs to get insights within different containers, our plugins collect hardware information from the entire system, not just the running cores. This led us to believe it might be possible to collect power consumption data, but there are challenges. Power consumption data is typically retrieved using system-level tools like IPMI or Redfish, which require elevated privileges. This poses a security concern, as site administrators are reluctant to grant such privileges to job-submitting users.

 

Our proposed solution is to implement a privileged daemon to expose sensitive information. The power measurement output would then go into an accessible file, and we would only query what sites decide to share, without using additional commands. Then, with the benchmark suite and its plugins, in addition to collecting performance information like load, frequency, and memory data, we can add another plugin to query this file and collect power consumption data. We then graph the output from this file, creating nice time series, similar to our other plugins.

 

I presented the summer's progress and next steps during the WLCG workshop, and I'd like to briefly reiterate them. The proposed strategy involved a feasibility study with a few different sites. The goal was to identify a unified approach for metric availability at sites and to scale site adoption from three pilot sites to a larger number. We also aimed to coordinate and collaborate on central data collection and validation. These were the main points, and we always encourage participation in our meetings to discuss problems or ideas.

 

After Paris, a repository called Hub Power Collector Repository was created specifically for this purpose. It's a space to collect, compare, and consolidate solutions for measuring power consumption across different sites. On the right, you can see its structure: different collectors for solutions from various sites, a documents section for collectors and presentations, and a still-empty folder for a Unified Collector, which is our future direction. This is a continuation of the work I briefly recapped.

 

The main objective is to document and compare existing site-specific collector implementations. Currently, we have four different collectors. We want to define a common interface and output format and provide a unified, recommended solution for all sites to adopt. This includes a reference implementation, a standardized directory structure, an agreed data format, and deployment instructions to make it as easy as possible for new sites to adopt these solutions.

 

Currently, the solution proposed by DESY is being used by Freiburg and INFN, while CERN and Glasgow use a very similar approach, primarily utilizing the Prometheus Node Exporter. Here, you can see the adoption status. We've decided on CPU models that have already been benchmarked, and here's a full list of all CPU models from which we've collected data. I wanted to highlight which solution is being used. Daisy, INFN, and Freiburg use the exact same solution proposed by DESY. CERN uses four queues, and Glasgow uses two, so the solutions are slightly different, but both use Prometheus Node Exporter. We believe these two solutions should cover most needs from other sites, especially those already using Prometheus Node Exporter. The "pink" solution will likely be easier to implement, and the Daisy solution might be better if something similar isn't already in place.

 

In total, we have six sites and ten queues involved. For example, Freiburg has collected over 4,000 measurements, and Glasgow is very close to 4,000. Daisy has also collected a significant number. AGLT2 uses a slightly different approach but is still collecting many measurements. CERN was added recently, so we don't have many measurements yet, but it's great to have CERN on board.

 

I want to present some preliminary observations, as the work is still in progress. Here, I'm showing the kind of data we can see after our measurements. One data point represents one measurement, meaning a job ran for about 90 minutes on a specific CPU model at a given site, from which we collected the quantified power consumption. This is presented here as one for a given load per physical core, showing how power behaves in terms of machine load. Each data point has a different color corresponding to the median frequency, and you can distinguish different governors based on the shape.

 

We can see three regions here. The biggest one, which is "expected" in the sense that it reminds us of plotting performance versus load per physical core, is what we anticipate. For low load, we see that the frequency is relatively low for the on-demand governor and rises with increasing load up to fully loaded machines. However, we did not expect the other two regions, which show much lower power consumption for a given load. The analysis is ongoing, but I wanted to highlight that we are seeing some interesting things that require further investigation.

 

Going back to our objectives and next steps, the proposed strategy seems to be working, as we are now collecting many measurements from six sites and ten queues. We are in the process of identifying one unified approach, but for now, I believe we will end up with two solutions that cover most of the needs shared with us. Site adoption is increasing from three to six sites, and we hope to increase this number further. For coordination and collaboration on central data collection and validation, we have a Grafana dashboard presenting the data we collect, and OpenSearch includes all information. The data will also eventually be available in the benchmarking table.

 

I also want to take a minute to announce that during our benchmarking working group meetings, you can get a better understanding of the repository and the solutions I've described. Since Paris, we've had five or six presentations on this topic. I think it's very relevant work, and I invite you to join us on Wednesdays at 3 PM.

 

In summary, the repository has been created, and existing code from pilot sites and new sites like CERN has been added. The objective is to standardize site collectors by comparing existing implementations and to deliver a unified solution with a common interface, format, and deployment guidelines for new sites that wish to adopt it. We currently have two solutions in place that meet most needs: the service from Daisy and the Prometheus Node Exporter used by CERN and Glasgow. Data analysis has just begun, and we are starting to uncover interesting regions and patterns that require deeper understanding and detailed investigations. Hopefully, I can share more details on that at the next meeting. Thank you very much. Do you have any questions?

 

[Caterina Doglioni] Thank you very much. So, we open the floor for questions because, I guess, you have to leave after the presentation, so this is the time when we'll discuss this.

 

[Jose Flix Molina] So you rely on the sites to provide the consumption, via some file that you can read. Are you considering that, for example, since we are benchmarking different CPU types, and we have the power consumption of the different CPUs we have available, and this may be the most power consuming in a server. If I have the same CPU model on my side, and I can’t have power accounting information (e.g. IPMI) maybe you could estimate my power consumption based on the load and comparisons with other sites who release this information. 

 

[Emanuele Simili] I'm probably rephrasing what has been said already. There are different solutions to power accounting, and the Glasgow site is actually a mixture of what you say. We have a system that calls IPMI, and that report the things through node exporter. You need a script behind the node exporter, since the node exporter alone does not expose the power. We could propose a solution that works on worker nodes with default software, and have a repository with an example for this tool that reads the power consumption (IPMI or other solutions) and reports them in a default location, like /tmp or /var. So if we have an agreement among the grid sites we can easily have everyone’s power accounting, even if the individual sites will need root privilege to install any tool on the node. 

 

[Natalia Diana Szczepanek] So, I don't know if I understand the question correctly, but what… so what we are doing. Now we are relying on the site to collect the information in real time. By real time, I mean, they are querying the IPMI each minute, saving it in a file and reading it. There are some edge cases and we are discussing how to solve them. With the data and metadata we have, it should be possible to predict the power consumption of those who do not provide information. How the tool works in terms of the jobs that I'm submitting, there’s idling for 5 minutes, then we collect information for around 80-90 minutes, and then we are idling again. So, we have the information about the power consumption in idle state as well. For Emanuele’s question, of course we will need site admins on board to work with us, but that's the idea, exactly.

 

[Domenico Giordano] Yeah, if I can jump in… well, you saw already my presentation one hour ago. This is the second part of our contribution of today, so you got already the answer, but essentially, you are anticipating a point that I will cover in the next 5 minutes. 



[Andrea Sciaba] Hi, yes, uh… I was wondering if you also record idle? So the consumption, like, waiting for the workloads to launch, and you may want to subtract it from the full load power consumption. 

Another thing: it may or may not make a difference if you compare two different nodes which have exactly the same CPU, but that have very different amounts of memory, as that uses non-negligible amounts of power. So there is a danger that people will interpret this information like a property of the CPU, but actually there is a big component which is not coming from the CPU. 

 

Domenico Giordano - transcript

[Domenico Giordano] Natalia has already mentioned that we use this suite as a way to collect power measurement and collect metrics together during the execution of the benchmark. 

 

The only difference in this contribution with respect to what Natalia has shown is that we can also rely on sysadmins in running on a dedicated server. Exactly the same procedure, with the difference is that here we saturated a full server. On the contrary, on the approach that Natalia just described, we saturate only the job slot. But for the rest, whatever we get is the same. So, essentially, we submit the same configuration that wants to access information like power consumption from IPMI, and we get the same report in the same structure. 

 

I want to share that in addition to the information from the sites that have already executed this suite since April, when we released it, now we have data in our database, and we are starting to visualize it in a public table. The new version shows not only the CPU models, but also the configuration of the server, like SMT on or off, number of CPUs, sockets, all those things. You can see that we structure them at the level of number of numerals, level 2, level 3 cache, or other, Andrea was mentioning, different values or RAM as well. Now, in addition to the score that we collect, and the number of measurements, we have also the power consumption, the average power consumption across those measurements from those servers. The standard deviation, that is an indication of the accuracy of the multiple measurements. And then, clearly, having those two independent quantities, the HEPScore value and the power consumption, we can have the HEPScore per watt, and we can rank servers based on these data. 

 

At the link for the public table, you will find not only the public table, but also the CSV that, essentially, is the backbone of the table. And, currently we have just an small number of sites that contributed with measurements. With version 3, so you can see just those servers, with CPU models that are all AMDs. Emanuele is working to then contribute also with the ARM nodes that they have in Glasgow. 

 

The interesting part is, again, that if in the past we had only the HEPScore ranking, now we can have the electric power ranking, or even better, the absorbed water one. Okay, and this will open to many other studies. What is crucial is having all the metadata, including the GPU part, not only as Andrea was mentioning, RAM, disk, etc, but having the information if a GPU is there, even idle, is important, because this affects the power consumption.

 

So, I'll leave the slides as a reference, in particular the ones about how to run the suite, because if people are connected here would like to run they just have to follow the instructions here. 

 

What I want to highlight, again, is in a plot what Natalia showed, is the fact that we need to calibrate the IPMI values with respect to the PDU value, so what essentially is consumed at the plant. And we have done this in the past. What we see is that at least for certain sites there is a 30% discrepancy that I would consider quite high. So, for what concerns a single server in a chassis, this is fine. We are within few percent. For quads, we have a bit of an outlier at CERN with respect to other sites, like Freiburg, that was measuring 8%. We are investigating, we are in contact with the vendors, it may be that we need calibrations. 

 

So, now, the only final thing that I want to say is to reiterate what the previous discussion was mentioning. With the probe approach of Natalia, plus, this approach where we get data directly, proactively from sites, what we can do is to build a database of power consumption per server models, and performance per server model that can be useful also for sites that cannot, for some reason, contribute to the measurement. And I think this is the other value of the effort that we are doing. 

 

[Jose Flix Molina]  Yeah. I have a very quick question. How we are going to disentangle that the CPU that we are measuring or trying to estimate the power is sitting in a single server or in a quad server? 

 

[Domenico Giordano] Currently, the suite allows the inclusion of additional metadata information. The easiest would be that, at least for this approach that I'm describing. For the other one, the one that, essentially, Natalia is reporting of… So far, what we have seen is that if we have a similar server, say, multiple sites, then we can start also to infer if there are discrepancies, certainly because we see different trend lines on different sites. But the approach from Natalia, let's say, requires more data with respect to the one on the slides that, on the contrary, relies on the active action of a site expert, and the site expert can enrich data with this information. So, saying, for instance, this is a quad, and say even this is the correction factor that is needed. We have to think in terms of how to do that, but at least there is a way to add metadata information

 

Michael Sparks - transcript

 

[Michael Sparks] What we're doing is looking at green computing at Manchester using our local cluster, and working with Glasgow, who are using their cluster. Initially, we're looking at a number of different things. In Manchester, we've examined the embedded carbon in the infrastructure, and we're measuring energy used by the cluster (a proxy for carbon use, depending on energy source) and the energy used by the software running on the cluster.

 

If you consider measure, monitor, analyze, predict, and iterate, we are currently at the "explore and measure with a bit of monitor" stage. The aim is to capture what we can relative to performance and throughput. This isn't just about power usage, but also the actual throughput, and validating this information from both internal and external sources. The goal is to improve utilization, as running at slightly different power levels (like with different governors) might still yield the same throughput, especially considering CPU throttling. We also aim to standardize processes across both Manchester and Glasgow, identify common issues and requirements, standardize the scripting and data collection processes, and potentially integrate with HEPScore.

 

We have scripts to collect power consumption for different sites, and we're collating power information, CPU usage, RAM usage, CPU contention, disk usage, network usage, for all these elements. We're also considering upfront how to integrate these into existing accounting systems. Different sites have different requirements, and security perspectives on what software can be run are a driving factor.

 

In Manchester, we're capturing data using Prometheus. Glasgow is using IPMI, with things running directly on-site on machines. We're querying the power consumption and PDUs across the full cluster with 10-second granularity 24/7 in Manchester, with similar monitoring in Glasgow. We're monitoring things like the plugs, not just on the machines, but also outside, to see what's happening outside the physical machine as well as inside. We're using tools like RAPL and CodeCarbon to measure CPU power consumption inside the process, to get an indication of what's going on within the running software, as well as outside. A whole bunch of different scripts are used for this.

 

This data comes from running clusters with known jobs and real VOs, and specific workloads set up for reproducibility. We encounter real-world problems, and there are differences between tools and results, so this analysis is a work in progress.

 

We have some early results. A Master’s student looked at the reproducibility of data from both internal (Code Carbon) and external (power supplies) sources. A summer intern collated various energy evaluation tools, including Code Carbon and others. We also had a larger University of Manchester UMRI Pump Prime project that examined these issues from a lifecycle perspective, fitting into our broader picture.

 

Our aspiration is for standardized monitoring, and for building software with carbon or energy tracking built in, which is just starting. This could potentially form a basis for measuring and prediction. Can we analyze code to predict potential load and therefore predict carbon? And can/do we want to do any scheduling based on carbon intensity? This slide has the people who have been involved so far. 

Emanuele Simili - transcript

[Emanuele Simili] Since the last meeting, I've made some progress on benchmarking and performance. I have updated my benchmarking and performance plots. On the left side of the slides, the small plot shows the HEP score on the few machines that could run, and then I have the HEP score per power. As we see, x86 definitely surpassed ARM, making them the most efficient architecture in both performance and performance per watt. The latest machines I tested were AMD Turin ones. I didn't do the test myself, so the data was colored differently, but the last one I tested myself was the Turin. As you can see, x86 performs pretty well now, making it probably the best choice at the moment. As I told Domenico, I will run the latest suite so I can upload the results to the HEP score database. I also have a question for later: How easy is it nowadays to download the data that Natalia was showing before (performance and performance per watt)? That would be very useful for the next procurement phase. But yeah, we can talk about that later.

 

I also updated another plot where I fit the average power I measured with the TDP for each machine. We see that there is almost a linear correlation, which seems good enough. This indeed takes into account all components that are not the CPUs, or the memory, or the disk. I agree that every server will be very different, but all the range I tested are more or less along the lines, so they could be a useful way to approximate average power when you don't have direct measurement.

 

Regarding my attempted accounting, I did not present what I wanted at ACAT because the data was really bad. The lesson learned is that if you extract garbage data for a long time, you get a lot of garbage. As you can see on the plot on the right side, there is only one server group that was kind of okay, and it was kind of okay because it's an ARM. Those are our machines, and only ATLAS was running jobs on them. When there is only one VO, the accounting is pretty easy because the data extraction works flawlessly. In all other cases, when there are multiple VOs, the Prometheus database cannot keep up, and so I have large gaps in the data. We either set it to zero or interpolate between the previous and next value. In both cases, this doesn't work. I tried many combinations, like interpolating 10, 20 bins, and then going to zero, but it doesn't work. This is an example of good and bad nodes. The bad one is one of the Dell nodes on top. As you see, there is a big mismatch between the IPMI and the estimated power and accounting overview running on that server. For the good series, as you see, they were running very few jobs, and so there is almost a linear correlation between the two, which is why the error is low.

 

So, what's next? I think I need to either drop this tool or rework it completely. Actually, rather than developing many different branches, especially if one is buggy, I prefer that we work on a common version that does the job.

 

The other thing I did was attend the NetZero NetDRIVE workshop, which is an initiative in the UK to drive innovation in research infrastructure, particularly in approaching net zero. I got a bit lost there. From the NetZero workshop, it was kind of interesting, although it brings together many disciplines that don't really match each other. I see that some people don't have a clear idea about computing. For example, when we visited the data center, someone asked, "What is this noise?" Because when people submit jobs to the cloud, they think they really go to the cloud, so raising awareness about what happens with jobs is important. There were other branches, like ours. I don't really know why they need computers and why they need to reduce the consumption of their digital infrastructure, but that's not for me to judge.

 

I got in touch with a few people and even started participating in a working group that wants to define metrics for power measurement and efficiency. I think we are kind of okay with the HEP score, but the other interesting part is that someone is studying embedded carbon. We did some quick exercises, also with the help of ChatGPT, and servers are close to 1 ton of CO2, depending on the configuration, and there are very huge error bars on it. So that's again something very interesting that we can work on. I can try to learn something from that side and share it with this group. There is even someone in Glasgow that does lifecycle analysis of servers, which I should get in contact with. So, those are the insights I've been working on, and that's all for me.

 

Disclosure of Delegation to Generative AI (https://panbibliotekar.github.io/gaidet-declaration/)

The authors declare the use of generative AI (GAI) in the research and writing process. According to the GAIDeT taxonomy (2025), the following tasks were delegated to GAI tools under full human supervision:

  • Formatting
  • Summarising

The GAI tool used was: Gemini 2.5 Pro.

Responsibility for the final manuscript lies entirely with the authors.

GAI tools are not listed as authors and do not bear responsibility for the final outcomes.

Declaration submitted by: Caterina Doglioni

Additional note: started from raw transcript in next tab, asked Gemini Pro 2.5 to summarize in bullet points and action items without adding any information (power consumption of inference only: up to 1 Wh/query according to https://www.sustainabilitybynumbers.com/p/ai-footprint-august-2025, average 2 queries per talk), then in-person pass to correct misunderstandings and remove spurious action items (many)

 

There are minutes attached to this event. Show them.
Powered by Indico v3.3.9-pre