Resource calculation model

→ Europe/Zurich
31/S-028 (CERN)

31/S-028

CERN

30
Show room on map

Participants

Andrea Sciabà, David Lange, Andrea Sartirana, Jan Iven, Concezio Bozzi, Davide Costanzo, Pepe Flix

Discussion

We went through the questions in the google doc that we'd like to answer with a resource calculation model.

Agreed that the dependency on the average pileup is important to study, in particular for HL-LHC, as it's a parameter that might be tuned to control the resource needs.

About calculating the LAN bandwidth needed by a site, an important input is the workflow model that we are also working on.

Andrea Sa. describes Ken Bloom's framework (https://github.com/sartiran/resource-modeling/wiki).

Granularity is one year but it might be changed; it should be enough for our immediate purposes. There are two limitations: network needs are absent and there is no provision for different hardware types (e.g. GPUs).

About branching the code in the original repository or in a new one, we need to ask Ken if he plans to keep developing his framework.

About the network needs, probably a separate model for the site has to be developed: it cannot fit in the general resource calculation model. Will ask Markus and Dirk to see if there is some previous work done at CERN.

The results from such model, including e.g. things like the CPU efficiency, can then be used as input parameters in the general model. Andrea Sa. points out though that CPU, storage and network are not independent variables, so their dependency must be included in the model.

Concezio and Davide agree that Ken's framework seems a viable option also for LHCb and ATLAS. Some previous work exists in their experiments, but very preliminary.

Again about network needs, Andrea Sc. proposes to look at the real network traffic at sites, just as a way to extract the order of magnitude, not for precise measurements. Pepe will do it for PIC, and Andrea Sa. will do it for GRIF, with the added value that GRIF is a federated site and therefore also some WAN traffic is involved.

Pepe suggests to also take into account hardware obsolescence, replacement and improvement, which incidentally is the case for Ken's framework, and to do separately for CPU and storage. David argues that this is a task for the cost model and it's more complicated: it's not clear that the right place is the resource calculation model. Concezio warns against taking the flat budget model for granted: we know that resource extrapolations changed with time. In fact, Ken's framework is not limited to the flat budget model, there are several parameters that can be changed. David says that at the end for our purposes here we just need a multiplication factor to apply to the available resources.

Concezio asks if we can use the framework to calculate errors for the estimates: it should be easy, by just varying the input parameters within their errors.

Andrea Sc. describes his approach to model functions of time using internally pandas TimeSeries: the advantage is the ability to have arbitrary functions of time that take as an argument timestamps and to work with very fine granularities.  The conclusion is that for the moment we don't need such approach, but it might prove useful in the future.

We finally agree to start "playing" with Ken's framework, with ideally each experiment providing a simple toy model based on it.

The next meeting is scheduled for March 20 at 14:00 CET, unless we decide to have another one before (but we expect to be able to work via email).

 

There are minutes attached to this event. Show them.
    • 14:00 → 15:00
      Discussion 1h