EURO HPC

[Enrico Fermi Institute] 14:49:55
Do you want to share? Yeah.

[Andrej Filipcic] 14:49:56
maybe it's it's a screen. Can you hear me? Right?

[Andrej Filipcic] 14:49:59
Okay, that's Michelle didn't

[Enrico Fermi Institute] 14:50:02
Great. So we want to show a little bit what's going on the European side.

[Andrej Filipcic] 14:50:04
Just

[Enrico Fermi Institute] 14:50:08
Yeah, then we can just as a

[Andrej Filipcic] 14:50:09
Right? So just a bunch of slides. But let me know if you are interested in anything else.

[Enrico Fermi Institute] 14:50:12
Yeah.

[Andrej Filipcic] 14:50:18
Oh, on some specifics over here. So maybe it's a bit too generic.

[Andrej Filipcic] 14:50:21
So the Irish Pc. Joint to the taking his, let's say, a company of 31 States, which I call out here on the right side.

[Andrej Filipcic] 14:50:34
All the members apart. Basically all Europe, and Turkey. Apart from me, Okay, and Switzerland.

[Andrej Filipcic] 14:50:39
And in the first place, which ended last year, the Web, 8 machines funded.

[Enrico Fermi Institute] 14:50:43
Okay.

[Andrej Filipcic] 14:50:46
So 3 prixes scale machines in the range of 250 to 350 billion blops.

[Andrej Filipcic] 14:50:51
so those one Lumi in Finland, Leonardo, which will be inaugurated to November in Italy, and Marin Austin, which will be a bit later.

[Andrej Filipcic] 14:51:02
It goes to Pickerman just finished, but not much details on this machine or yet no, apart from the talk today, Will, he had a quite large Cpu Partition of 30 peasa flops.

[Andrej Filipcic] 14:51:14
Which is quite good. For, let's say so. The second phase is the 6 years up to 27, and the currently approved machines, the high range one the exa scale me which would be a Jupiter the the so the machine was just approved but the procurement, was

[Andrej Filipcic] 14:51:35
not yet done so all no details on this machine, just the the plans right? Basically there.

[Andrej Filipcic] 14:51:42
One to reach one, Maxa flop with some or okay, that's enough.

[Andrej Filipcic] 14:51:47
And so there will be 4 arrangements. So for Hpcs in so investments here between 20 and 40 million Europe rate per each, and those one will be in Greece.

[Andrej Filipcic] 14:52:02
Hungary, or on an island. I think also there'll be some collocated quantum computers.

[Andrej Filipcic] 14:52:12
So the first generation, and this will be approved probably next month.

[Andrej Filipcic] 14:52:18
I was skating So this is just a mission which you can read that day later on.

[Andrej Filipcic] 14:52:24
Basically your Hpc wants to support leadership, supercomputing, including quantum computing and all the data infrastructure around it.

[Andrej Filipcic] 14:52:35
Then they want to develop. They're on hardware, and they want to evolve industry a lot.

[Andrej Filipcic] 14:52:42
Let's say to bullet. So the budget. The budget is pretty 50% of from European Commission and 50% from the hosting states.

[Enrico Fermi Institute] 14:52:46
Okay.

[Andrej Filipcic] 14:52:55
So these are the countries that decide to build the Hpc.

[Andrej Filipcic] 14:52:58
although for the smaller machines you're European Commission only funds 35%.

[Andrej Filipcic] 14:53:04
So in the phase, one, the 3 S. 1 billion euros were spent for the face to 7 to 8 billion is actually foreseen on the on the plot on this table on the picture you have a detailed breakdown from the European Commission and Then there would be the same matching contribution from

[Enrico Fermi Institute] 14:53:11
Okay.

[Andrej Filipcic] 14:53:25
the all the Member States. Okay, and also 200 Me, let's say, 200 million is meant for hyperconnectivity.

[Andrej Filipcic] 14:53:33
So for Terabyte Network and 50% of the money spend for new product infrastructure.

[Andrej Filipcic] 14:53:42
There are many projects in the Tv activities going around it.

[Andrej Filipcic] 14:53:45
So, maybe one important one is you eurocc or European competence center which basically he's a very large project with 30 participants of participant.

[Andrej Filipcic] 14:54:01
State let's say so. Most of them, and the funding is about 1 million Europe or country per year.

[Andrej Filipcic] 14:54:07
the goals are basically to training and connection with the industry and collecting.

[Andrej Filipcic] 14:54:14
So the knowledge on Hpc. Whatever that means. There's also centers of excellence, for example, which are mostly dedicated to, let's say, support software development or scalability, extensions of particular groups.

[Andrej Filipcic] 14:54:28
They can be dedicated to a particular particular field of science, like chemistry, or molecular dynamics, or something like that, or they can be a bit wider in scope for specific.

[Andrej Filipcic] 14:54:38
Let's say data handling for access case something like that.

[Andrej Filipcic] 14:54:44
the about 10 they send us a tax sentence, initial funded between 6 to 8 meetings per project, and those calls would be continuing all the time.

[Andrej Filipcic] 14:54:53
So to to this period. There are 2 bodies. So research, generation, advisory group and infrastructure advisory group, which basically accepts form recommendations for the illusion in develop and so forth basically, for everything for the research calls for funding and for infrastructure deployment

[Andrej Filipcic] 14:55:16
another part of it is your Pm. Process initiative with the name to build European Cpu and Gpu.

[Andrej Filipcic] 14:55:24
Of course, maybe he'll be slightly written. That's later.

[Andrej Filipcic] 14:55:30
There's also your master for Hpc. Which is just a common university program.

[Andrej Filipcic] 14:55:36
So this is a project. The tries that's many countries and universities.

[Andrej Filipcic] 14:55:42
Let's say about 30 of them will try to put the Hpc.

[Andrej Filipcic] 14:55:47
Studies master status typically in sync and share. Let's say, students share lectures, and so on.

[Andrej Filipcic] 14:55:57
there are about 30 projects altogether, so the resource location access is only provided to you in typical users.

[Andrej Filipcic] 14:56:07
So basically to members of European Union, the extended one, actually the European Commission, we so share is very similarly managed as praise before.

[Andrej Filipcic] 14:56:19
So the place, like calls for publications, with some changes, The first one is developing batch parking.

[Andrej Filipcic] 14:56:26
with basically immediate access. So let's say, within a less than a month, maybe even, we think to mix 2 weeks And this is not negligible even in resources.

[Enrico Fermi Institute] 14:56:31
Okay, this.

[Andrej Filipcic] 14:56:37
So you can get something like up to half a 1 million Cpu hours.

[Andrej Filipcic] 14:56:43
for these access, and you get it for for up to a year.

[Andrej Filipcic] 14:56:48
Then the regular access, which is a couple of 10 million Cpu hours.

[Andrej Filipcic] 14:56:53
Cisp reviewed, and there are also there will be calls future on for industry in public sector.

[Andrej Filipcic] 14:57:00
This is not yet right, finalized. Yet, because of the funding issues.

[Andrej Filipcic] 14:57:05
And let's say, charging for the industry. So the hosting entity, share.

[Andrej Filipcic] 14:57:11
So the owner of the the other house of the Hpc.

[Andrej Filipcic] 14:57:14
The country, the policies there are completely regulated by country policies or decisions.

[Andrej Filipcic] 14:57:21
So each State can do whatever they want with their latch.

[Andrej Filipcic] 14:57:28
so overall the design is of some of Hbc.

[Andrej Filipcic] 14:57:37
Is quite classical, but not all of them are really classical.

[Andrej Filipcic] 14:57:40
Hbc anymore, as you know, Vega can. Slovenia can was designed to be strict mind of heavy duty data processing, and outbound connectivity which works actually pretty well.

[Andrej Filipcic] 14:57:54
For Atlas, where Vega contributes something right between 1340% of Cpu.

[Andrej Filipcic] 14:58:01
during the last year, Let's say, then, the second one, Lumi.

[Andrej Filipcic] 14:58:03
They have a very large dedicated partition for visualization and services, and they will provide only so they will provide set object storage for long term data preservation.

[Andrej Filipcic] 14:58:16
So on, and they want to provide all the more than tools modern nose.

[Enrico Fermi Institute] 14:58:19
Okay.

[Andrej Filipcic] 14:58:20
From 5 I mentioned it here. It was not been built, but they set that they will be have much larger cpu partition and open access, because the Government decided that the this machine needs to support Ac.

[Andrej Filipcic] 14:58:35
So this was great already about on overall, in the architecture, so most of these machines are Janet purpose.

[Andrej Filipcic] 14:58:46
Some maybe less general purpose than the others, but they basically all the all of them needs to adapt to the user needs.

[Andrej Filipcic] 14:58:54
So they are. It's a bit different. So they're not completely free to set the policies.

[Andrej Filipcic] 14:59:00
How these machines will be set up, and what services they can provide, because overall the European, your Hpc.

[Andrej Filipcic] 14:59:07
Governing Board, which is representative. These from States can say on what to do with these machines.

[Andrej Filipcic] 14:59:16
Right. And there are many countries that participate in these calls, but they don't have Hpc.

[Andrej Filipcic] 14:59:23
But they would like to to use it. And for basically, for all the science.

[Andrej Filipcic] 14:59:27
And so also interesting, does this stuff. So the current machines mixture of Cpu and Gpu partitions.

[Andrej Filipcic] 14:59:37
So Dcp is mostly Amd. Then some intel recently, for example, would be intel.

[Andrej Filipcic] 14:59:45
Then there is a one arm machine that will be in Portugal based on fujitsa and they have both Nvidia and and the but most have Nvidia Gpus, and some have like you only have Md So Hello, me.

[Enrico Fermi Institute] 15:00:03
Okay.

[Andrej Filipcic] 15:00:05
Is the same, Okay, what's the name of the Ocf.

[Andrej Filipcic] 15:00:08
Right so. But in any case most notes have gpus.

[Andrej Filipcic] 15:00:14
So most of the hardware is Gpu compromises between 60 to 80%.

[Andrej Filipcic] 15:00:20
It depends on the machine. Well, there's one small machine, Cpu only, but all the big machines have.

[Andrej Filipcic] 15:00:25
Let's say, 24% of Cpu notes, not not even Cpu power.

[Andrej Filipcic] 15:00:31
Right? Okay, computing power. So the storage is typically last with Seth, and some also provide some kind of yeahf.

[Andrej Filipcic] 15:00:40
So this one is less popular, and most of these machines, basically a apart from Lumi and Carolina, is in in in Czech Republic we're built up by artists during the future machines.

[Andrej Filipcic] 15:00:59
Will most definitely. So the large one next Xs K machine, which will be built in France, will be arm based, so it would be arm cpu plus gpu, as well.

[Andrej Filipcic] 15:01:10
So details are not clear yet the goal is to build it somewhere in 2425, and after that the next one.

[Andrej Filipcic] 15:01:20
So let's say them excess scale machine, whatever it would be.

[Andrej Filipcic] 15:01:24
They have strong, wishes. Let's see, for now it should be risk 5 based next slide.

[Andrej Filipcic] 15:01:33
So some thoughts, some observation. After 1 point, 5 years of operations of these machines, so each of them have of the order of something like 500 users, which might seem a bit little, for some but in actually most of these users, are completely newcomers, since many other users already have allocations on

[Andrej Filipcic] 15:01:57
the large existing machines, let's say, in Italy, Spain, Germany, or France.

[Andrej Filipcic] 15:02:02
So go. Ones that are part of price, and do these users have really a lot of different kinds of workloads, so that many node computes jobs We've done Cpu on Gpu And this is mostly carol th the the majority i'll say, is chemistry or material

[Andrej Filipcic] 15:02:21
science, although something like at least on Vega. There are something like 30 different applications that the user want to run a lot of users also do small notes or small core parameter scans on tons of independent jobs let's say, let's see like and many many users in the last

[Andrej Filipcic] 15:02:43
year start to use machine learning, even. That less users do analysis with machine learning, and this is rapidly actually growing.

[Andrej Filipcic] 15:02:51
Because let's say it's quite simple with Tensorflow and all the case.

[Andrej Filipcic] 15:02:55
So to locates the around and atar machine, at least in Vega we have a really big pressure on Gpus, so the next machine will buy. We'll have much larger Gpu Partition Oh, some users also do extreme data process processing.

[Enrico Fermi Institute] 15:03:08
Okay.

[Andrej Filipcic] 15:03:14
No I don't mean I let's see here.

[Andrej Filipcic] 15:03:17
But, for example, like something I cry micro microscopy or different stuff, where they produce, let's say, a couple of tens of terabytes per measurement that they want to process is same interactively some hpcs, allocate for notes, only but there are many that can run any type

[Andrej Filipcic] 15:03:32
of jobs, Also, observe, we have observed my experience, that many users are not quite happy with the default a data organization of the Hpcs, which basically more or less doesn't exist.

[Andrej Filipcic] 15:03:45
I would say, although we have other tools in my future. But let's say, within your your Hbc.

[Andrej Filipcic] 15:03:51
The data Migration is a movement. We're not yet discussed and many users stick to containers and some demand event.

[Andrej Filipcic] 15:04:01
But virtualization basically here for your Hpc. What user demands to use it basically should be provided to another later.

[Andrej Filipcic] 15:04:11
there are much more users on your Pc. At this point, and was ever in place.

[Andrej Filipcic] 15:04:16
So this number we probably wrote. So that's it. Cumulatively to 50,000. Pretty soon, on all the all on all the machines and there, are a lot of really a lot of newcomers due to simplicity or taxes, you basically just submit a proposal, not even a

[Andrej Filipcic] 15:04:33
Proposal. Application which quick description. And you will get an access within less than a month.

[Andrej Filipcic] 15:04:39
The usage for the interstate is rising a bit.

[Andrej Filipcic] 15:04:43
this is mostly small or medium enterprises, but this is still not extremely high.

[Andrej Filipcic] 15:04:50
Let's say more or less, the entire to use 20% of Hpcs by European law.

[Andrej Filipcic] 15:04:57
Let's say, by European funding regulations, but they're not yet at 20.

[Andrej Filipcic] 15:05:06
Far from 20% of usage. This point although some Hpcs like the one in Luxembourg, was built entirely to support the industry.

[Andrej Filipcic] 15:05:15
7 countries also decided to provide resources through your Hbc. For 4.

[Andrej Filipcic] 15:05:20
Let see, Slovenia for sure, because I know what's going on here.

[Andrej Filipcic] 15:05:25
You have the same message from Spain or the others be shared.

[Andrej Filipcic] 15:05:28
And lately, even in Germany. So I think that the German wants to only keep Daisy and Kit not not sure if this is official yet, but other countries will probably follow a similar way.

[Enrico Fermi Institute] 15:05:36
question.

[Andrej Filipcic] 15:05:40
So. The and let's see. European. Sorry. Yes, go ahead.

[Enrico Fermi Institute] 15:05:42
Questioning.

[Enrico Fermi Institute] 15:05:47
so you said. Several countries have already decided, you know, like Sylvania, with highly successful Vega.

[Enrico Fermi Institute] 15:05:53
What about the Vega design makes that so much easier to integrate in these would be some of the Us.

[Enrico Fermi Institute] 15:06:06
Snowflakes.

[Andrej Filipcic] 15:06:09
I'll show, because in Reggae we need the pressure to support civilian vessels.

[Andrej Filipcic] 15:06:14
So some some other more classical Hpcs are hesitant in this respect.

[Andrej Filipcic] 15:06:20
But let's say Vega is not so different in hardware.

[Andrej Filipcic] 15:06:23
Architecture than the others apart, that, apart from that that really required a large pipe which can at the moment this pipe can do 600 gigabits per second to one.

[Andrej Filipcic] 15:06:35
That's it to Jean. And this will increase in the future.

[Andrej Filipcic] 15:06:40
So it's mostly a matter of decision. What you are allowed allow users to do over there

[Enrico Fermi Institute] 15:06:49
Okay.

[Andrej Filipcic] 15:06:51
done. The network connectivity. Will will likely boost a lot in the next 2 years. On 2 to 3, as let's say, especially if there's zone.

[Andrej Filipcic] 15:07:01
One or a bit network is seen, don't There are some still open questions about the funding is came, and who can do the networking, and so on.

[Andrej Filipcic] 15:07:14
long term, long term, data story is not that part of the plans So it's a bit on a wild but there's a high pressure of many communities to use this as well. Right?

[Andrej Filipcic] 15:07:25
So the Hpcs. This point are not obliged to provide long-term storage.

[Andrej Filipcic] 15:07:29
Let's say when the Hbc. Is decommissioned, the storage is really likely to be the commission as well, and a new storage will be read, brought up in the new machine right?

[Andrej Filipcic] 15:07:39
But this this will need to change the future. One thing, one thing worth to stress is that some leadership your projects like destination, Earth?

[Andrej Filipcic] 15:07:49
Well, I'm not sure you know it, but destination out to basically as apples.

[Andrej Filipcic] 15:07:54
Ecmwf Weather Agency, and it you mets up.

[Andrej Filipcic] 15:07:59
But they, is to provide a digital between digital going to for Earth right?

[Enrico Fermi Institute] 15:07:59
Hmm.

[Andrej Filipcic] 15:08:06
which includes satellite imaging weather collection.

[Andrej Filipcic] 15:08:12
whether the data collection on weather forecasting, and so on, and basically do a global model of our team predictions And so on.

[Andrej Filipcic] 15:08:23
With, basically, it's a huge project. And this this organization already officially asked, join to the taking if they could use your Hpc on the production.

[Andrej Filipcic] 15:08:33
Level, I'm basically joined the deck and agreed, for now they can use 10% of the All the resources right.

[Andrej Filipcic] 15:08:43
European Commission. But up to 10% and more organization to follow this way, for example, destination Earth doesn't have enough funding or money to to do anything without your Hbc.

[Andrej Filipcic] 15:08:57
At this point. So more projects with this I like this will follow, and maybe even let's see Good.

[Andrej Filipcic] 15:09:07
But this was not discussed yet. I will skip the next slides, because I just a bit of an overview for computer as you can.

[Enrico Fermi Institute] 15:09:10
Okay.

[Andrej Filipcic] 15:09:15
You will see them later on when I upload them

[Andrej Filipcic] 15:09:20
Okay, That's it.

[Enrico Fermi Institute] 15:09:23
Great. Thank you. We have sub raised hands. So, Paulo, his handwriting for a while, go involved

[Paolo Calafiura (he)] 15:09:30
yeah, Do I remember? Oh, yeah, yeah, yeah, yeah, I saw one slide in which you mentioned that short term Let's say the next generation would be armor and the next next one generation may be the risk 5 And I'm wondering if you meant for a cpu replacement, and and therefore

[Andrej Filipcic] 15:09:46
Right.

[Paolo Calafiura (he)] 15:09:53
also, having accelerator. So are you just saying it will be Qr.

[Paolo Calafiura (he)] 15:09:57
More pure risk, file.

[Andrej Filipcic] 15:09:58
No, our movie has accelerators.

[Paolo Calafiura (he)] 15:10:00
Okay, okay, So like, something like.

[Andrej Filipcic] 15:10:05
Yeah, something. Yeah, it's not it will be Grace hopper, style, or separate chips, or whatever

[Paolo Calafiura (he)] 15:10:12
Okay, okay.

[Enrico Fermi Institute] 15:10:16
Okay, there's a hand up for you.

[Ian Fisk] 15:10:18
yeah, my question was I to actually one? Was the the project Earth?

[Ian Fisk] 15:10:24
Is that a strategic alliance between your Hbc.

[Ian Fisk] 15:10:26
And the project. And the is it multiple year? Does it different from the typical peer review?

[Ian Fisk] 15:10:31
Oh!

[Andrej Filipcic] 15:10:31
Yes, it's completely different, because this is a long term project for at least 10 years, And even so, so, it's exactly like So no.

[Andrej Filipcic] 15:10:41
Let me see, let's say.

[Ian Fisk] 15:10:42
Okay, so, but is that. But does that? Is the door open to other multi like things like that?

[Ian Fisk] 15:10:50
Lcg. Lhc. Negotiating such an arrangement

[Andrej Filipcic] 15:10:53
I think so. I mean, the the thing is that European Commission needs to find such projects in interest to support them. And actually those projects are typically listed in S Free Table, where, for example, high luminosity is right

[Ian Fisk] 15:11:10
Okay, was there. I I may have missed it.

[simonecampana] 15:11:13
I think

[Ian Fisk] 15:11:17
But is there a second ex scale machine in France someplace?

[Ian Fisk] 15:11:22
I I thought the only one was in Germany. Just understand.

[Andrej Filipcic] 15:11:23
David by the the official one that was accepted already. So they're going to proof is in Germany, Jupiter, Franz will likely come next year.

[Enrico Fermi Institute] 15:11:30
Okay.

[Ian Fisk] 15:11:32
Okay, nice okay.

[Andrej Filipcic] 15:11:33
I mean the call for for proposing

[Ian Fisk] 15:11:39
Thanks.

[Enrico Fermi Institute] 15:11:41
and from Maria

[Maria Girone] 15:11:43
maybe just I want to say that, hey, research infrastructure like, Okay, yay is, There's a lot of ongoing discussions.

[Maria Girone] 15:12:01
Sandra knows. Well, between let's say, the larger communities and your Hpc.

[Maria Girone] 15:12:09
In order to try to motivate further collaborations very much like those programs like this destination Earth, which indeed is a priority for European Commission.

[Maria Girone] 15:12:21
But we are also having a number of projects now that will allow us to do.

[Maria Girone] 15:12:28
Arindi. Some. I think we'll we, for instance.

[Maria Girone] 15:12:35
We will present tomorrow. Indie, So what we're doing with the d Eulich Super computer center for what concerns a development and use of a Gpu resources scale for distributed training the reason a pipeline in a also European project that will allow us to valuate open source

[Maria Girone] 15:13:00
solutions like risk 5 and sequel. So there are a number of opportunities and race side very well is very easy.

[Maria Girone] 15:13:09
Actually to work on the development side with your Hpc.

[Maria Girone] 15:13:13
And the we get granted the resources on for developers.

[Maria Girone] 15:13:20
So even within 5 days I mean, we didn't working week, so it's very, very, nice collaboration, at least at this, level, we need to build on this and go further, and that is less obvious.

[Maria Girone] 15:13:33
And we require your some common actions, let's say at least, when we are talking to your Hpc.

[Enrico Fermi Institute] 15:13:47
Samantha was first. Okay.

[simonecampana] 15:13:49
it's a follow-up on. Yes. Question. I think one of the requisites of for entering one of those special programs like I know.

[simonecampana] 15:14:02
I don't remember how they're called, but turn up, Grant based the more long term.

[simonecampana] 15:14:06
He's a first that you. You are an impactful science, and of course, so you know, it's really I'll be trying to define what is impactful.

[simonecampana] 15:14:17
But of course, it's the one who saves the health has a simpler way of demonstrating that impactful Also, if we want to apply for something like, this, I think it's important to make a a lot of progress on the software area.

[simonecampana] 15:14:32
because one of the other things one has to demonstrate is to use an Hbc.

[simonecampana] 15:14:38
For the value of an Hbc. And already don't use much of the interconnects. That An Hpc. Office.

[simonecampana] 15:14:45
So if we are also cheap on Gpus and the use of those architectures, then we become not such a great candidate for one of those one of those programs.

[simonecampana] 15:14:55
So. I think we have to build our story, and we have some technical improvements at the software level that now was to to build a better story, and my question is, if there is something like this in the Us.

[simonecampana] 15:15:10
because if there was, then we could try to build even a more coherent story across you.

[simonecampana] 15:15:16
Open the us.

[simonecampana] 15:15:22
Do you have a notion of sciences that get into a program once, and then there is a movie ear engagement with the Hbc facilities

[Ian Fisk] 15:15:31
Oh, I I think the dirt might be able to answer better, But I think this is one of the things that the Us.

[Ian Fisk] 15:15:37
There is a for definitely a push these days from the Us.

[Ian Fisk] 15:15:40
Funding agencies for science to make effect use of the large-scale computing facility.

[Ian Fisk] 15:15:48
And so, whether it it's not there's there's not a program like your Hbc.

[Ian Fisk] 15:15:53
Because it's only one country, but it does mean that there is a There is alert like if you look at where the national apps of main investments there.

[Ian Fisk] 15:16:03
A lot of the investments have been made in Central at facilities with the expectation that the calculations are done.

[Ian Fisk] 15:16:07
There.

[simonecampana] 15:16:08
Right.

[Enrico Fermi Institute] 15:16:09
Yeah, the the thing is, I mean at the moment they're very high level discussions going on that they because there's the push from the funding agencies that we should use more Hbc: And it's not just us it's in general because they pay for these facilities they want them

[Enrico Fermi Institute] 15:16:25
to be used, but that now is a pushback, and that's what the conversation is a very high level.

[Ian Fisk] 15:16:27
Good.

[Enrico Fermi Institute] 15:16:32
Lis mentioned something that there are groups talking about what they need to do in terms of changing their policies, to actually allow that because the the application process for the deed program for the lcc application process the inside application process is just not geared charles towards these used cases competitive poses that are unique that can only be done there which

[Enrico Fermi Institute] 15:16:56
is just not a good match, and that's it like this is above all pay scale.

[Enrico Fermi Institute] 15:17:01
Here. The these conversations are going on hopefully. Something comes out of it as we'll see

[Taylor Childers] 15:17:05
not sure that that is the case. I mean the so.

[Taylor Childers] 15:17:10
The insight program offers the opportunity to get up to 3 years of allocation through a competitive review process.

[Taylor Childers] 15:17:19
The challenge. Is laying out your case, and I would argue that the way you approach this for the leadership computing facility is you have to play to their mission right.

[Taylor Childers] 15:17:34
I mean, their mission is to provide the biggest computers, because people need them, not because Hi good

[Enrico Fermi Institute] 15:17:40
But Taylor is this is you basically have to sell it, and you have to sell it in there way that you basically dress it up as something that can.

[Enrico Fermi Institute] 15:17:48
You can only do there, and that's not what we want.

[Taylor Childers] 15:17:52
I agree, and but I would argue that you can easily make the case based on the fact that you are reaching great scenario, and if you don't get access to the machines, then you'll be able you'll be slower in your science, achievements, and I

[Enrico Fermi Institute] 15:17:52
Thanks.

[Taylor Childers] 15:18:14
think that's a viable, are you? I think the part where you guys have trouble in, especially an insight program proposal is the fact that you don't have enough of the workloads that take advantage Gpus, right?

[Enrico Fermi Institute] 15:18:31
Yeah, that's probably we're trying Lcc: right now.

[Taylor Childers] 15:18:31
I mean the challenges is

[Taylor Childers] 15:18:35
Yeah, for sure and

[Enrico Fermi Institute] 15:18:35
That's easier to justify. I think, if we ever get to the point that you could say, Okay, if we get like a huge insight proposal, we could, You can make the science use case if you can do something you couldn't otherwise, do because it basically adds, 50%, of your own capacity, or

[Enrico Fermi Institute] 15:18:50
whatever. But then a little bit kicks inside is still only you.

[Enrico Fermi Institute] 15:18:55
You do an allocation proposal, and you get the decision, and then you get it like 3 months later, a few months later, It's too short a time scale.

[Enrico Fermi Institute] 15:19:05
You would basically have to ask a year or 2 in advance to fit our planning process within the experiment, because you can't just drop that on top of Cms.

[Enrico Fermi Institute] 15:19:14
And expect that we basically throw our plans out the window.

[Enrico Fermi Institute] 15:19:18
And now effectively use

[Taylor Childers] 15:19:19
Yeah, no, there. There definitely needs to be more discussions above our pay rate.

[Taylor Childers] 15:19:25
I mean the challenge. There is to some extent you have to change how the leadership computing facilities are are reviewed so that we can accommodate stuff like that

[Enrico Fermi Institute] 15:19:44
you're so I'm sorry. The the you mentioned that obviously these machines are a mixed Cpu Gpu.

[Enrico Fermi Institute] 15:19:55
Next generation will they be more well, More of the flops and actual compute power power Use be in the accelerator realm, or will there be some machines where arm sort of provides the every lifting

[Andrej Filipcic] 15:20:14
well how does it say hard to predict? But that, in my opinion, there will be always machines built in that way, That's as many user communities can use them.

[Andrej Filipcic] 15:20:24
So, and several sites know that already. Right? So nobody will go to a complete dedicated machine, for example, even Jupiter, which is excel scale, It will be easier to build it up and reach the highest top 5 500 number only going only gpu right but they don't

[Andrej Filipcic] 15:20:45
want that I mean, nobody would actually want that. So on on cpus.

[Andrej Filipcic] 15:20:53
It depends right. But so, still quite many users are used in x 86, right so.

[Andrej Filipcic] 15:21:00
But arm is not so difficult in respect. If you use Cpu only part. When you have Gpu it will be slightly different, but arm will definitely be a larger players in the next couple of so something like that

[Enrico Fermi Institute] 15:21:14
but my take

[Enrico Fermi Institute] 15:21:14
But my takeaway from what you've just said is that at least for the next generation likely to have as a significant Cpu footprint, because they're sort of mandated to be as usable as possible to the communities the the broader comp broader, scientific and such communities

[Andrej Filipcic] 15:21:35
Right yup

[Enrico Fermi Institute] 15:21:36
right, Okay, thanks. Okay, let's let's move on So there's any other questions for hunger.

[Enrico Fermi Institute] 15:21:48
I think we should move on. Hey? Thank you.

[Andrej Filipcic] 15:21:50
Well, welcome!

M100

[Enrico Fermi Institute] 15:21:52
Okay, we have a couple of slides from some European Cms.

[Enrico Fermi Institute] 15:21:57
Efforts, recipes. Daniela isn't connected, I think, unless if he's here, you should speak up.

[Enrico Fermi Institute] 15:22:04
He told me he couldn't. So this is this.

[Enrico Fermi Institute] 15:22:08
Is integration basically at the Seneca, at the Canal Tier one.

[Enrico Fermi Institute] 15:22:13
So they have the co-locator visiting the same data center.

[Enrico Fermi Institute] 15:22:18
There is the Seneca Mcconnie, 100 Hbc.

[Enrico Fermi Institute] 15:22:21
Which is basically a clone in terms of system architecture to to summit.

[Enrico Fermi Institute] 15:22:26
So it's power plus and video, and they they integrated it as a subside of the Tia one.

[Enrico Fermi Institute] 15:22:32
So since they're co-located on the same data center, they have really fast network interconnect.

[Enrico Fermi Institute] 15:22:37
They tie it together. The Hbc. Can see basically the kind of T.

[Enrico Fermi Institute] 15:22:42
One storage system. You saw The services are provided by the data center, and they run it as a subset of the T one.

[Enrico Fermi Institute] 15:22:52
So the Cms operations only sees the T one, and then they can internally via some pilot customizations they can select, which parts of the workflow that are centered at Tijuana can run on the Hpc.

[Enrico Fermi Institute] 15:23:05
Side, And they basically where we are today, it says, almost complete.

[Enrico Fermi Institute] 15:23:09
I think it is complete now, because the announcement came out after the slide was sent to me.

[Enrico Fermi Institute] 15:23:15
You see, some slides how it's how it's integrated. So on.

[Enrico Fermi Institute] 15:23:18
Did you see it in the in the Monet? The sub-site concept has some unique challenges in how you monitor it.

[Enrico Fermi Institute] 15:23:27
Good.

[Paolo Calafiura (he)] 15:23:27
I'm sorry. Are we looking at lights because we these slides

[Enrico Fermi Institute] 15:23:31
I that I forgot to re share it. Oh, that is yeah, I mean, bring it back up the because in the U.

[Paolo Calafiura (he)] 15:23:33
Alright.

[Enrico Fermi Institute] 15:23:41
S. All the Hpc sites they're using.

[Enrico Fermi Institute] 15:23:44
We basically put the concept of it. See a 3 grid side on top of it, which makes the monitoring and accounting, and so on really easy, because everything is important and there's a unique sign if you have a subside, and is a little bit more difficult because everything is kind of hidden in the under the umbrella, of

[Enrico Fermi Institute] 15:24:02
the T one, and then you have to kind of dig into this like some subfields and identify us to, and there has been some work on going in the monitoring and the monitoring sites on Cms to to make that easier Doesn't.

[Enrico Fermi Institute] 15:24:16
This model make it easier to accommodate. It makes it makes perfect sense, I mean, for them.

[Enrico Fermi Institute] 15:24:25
It's great because they're I mean, they're co-located anyways.

[Enrico Fermi Institute] 15:24:29
It's it makes perfect sense for it's a bit more difficult if you're like geographically, is organizationally separate and entities.

[Enrico Fermi Institute] 15:24:40
So in the Us. It's kind of difficult, because the Hbc are usually stand alone, So So what is it that has changed between few years ago And now?

[Enrico Fermi Institute] 15:24:51
With regard to Cbm Fs. It seems like initially, people were very, very wary of it, You don't want to put this on our Hpc.

[Enrico Fermi Institute] 15:24:58
Because it'll crash everything or whatever I mean. Is it is it Technology has gotten better?

[Enrico Fermi Institute] 15:25:02
Or is that people have gotten less afraid of it? Maybe people have got less of for it just became familiar with Also, a lot of people are using it, not just us that.

[Enrico Fermi Institute] 15:25:12
Helps and then I don't worry about it anymore, because any recent machine with the recent Os no problem running Cdm: Fs: access Yeah.

[Enrico Fermi Institute] 15:25:24
It just built my own, and I mean from the ocean side we bring on new sites, because we only use see?

[Enrico Fermi Institute] 15:25:31
If you have a phone, Zack, and only if they directly ask can I please run?

[Enrico Fermi Institute] 15:25:37
Cvs or if they have any other problems, do we give them the option by Why, why have that conversation? Sure somebody's not hitched into it?

[Enrico Fermi Institute] 15:25:53
Okay, even on the Lcf. No problem. Be it worked on, Sayta out of the box.

[Enrico Fermi Institute] 15:25:58
Physically it worked on summit, out of the box so I didn't know the issues.

[Enrico Fermi Institute] 15:26:01
I have to go click at the squid there so I can actually write it on the batch node.

[Enrico Fermi Institute] 15:26:05
But it worked on the logarithm, which runs the same operating system

BSC update

[Enrico Fermi Institute] 15:26:10
And then, Antonio, are you connected? Okay, So Antonio can say a few words on what we're doing at Marinosa

[Antonio Perez-Calero Yzquierdo] 15:26:12
Hi! Yes, I am! Can you hear me?

[Antonio Perez-Calero Yzquierdo] 15:26:17
Yeah, okay, So yeah, I don't know. Zoom is for is the current supercomputer in?

[Antonio Perez-Calero Yzquierdo] 15:26:26
and this is the the largest Sbc center in Spain.

[Antonio Perez-Calero Yzquierdo] 15:26:29
I don't know through 5 days. Plan is actually in the procurement face as a explain before so we are accessing bsc and Madamos room as a project mediated by pick So that's the double CD.

[Antonio Perez-Calero Yzquierdo] 15:26:48
Spanish tier one, and fortunately, let's say, interestingly, the Lc.

[Antonio Perez-Calero Yzquierdo] 15:26:54
Computing has been designated as the strategic project in Vsc program.

[Antonio Perez-Calero Yzquierdo] 15:26:58
So this means that we basically well, we still have to request the allocation.

[Antonio Perez-Calero Yzquierdo] 15:27:02
But we are getting quarterly grants of about 6 or 7 million hours.

[Antonio Perez-Calero Yzquierdo] 15:27:09
A Yeah, available at this for for Cms. And I think it's about the same amount, for for Atlas.

[Antonio Perez-Calero Yzquierdo] 15:27:16
So we are getting these allocations. Let's say, regularly.

[Antonio Perez-Calero Yzquierdo] 15:27:19
Okay, however, the case is very difficult for for Cms.

[Antonio Perez-Calero Yzquierdo] 15:27:24
The environment is extremely challenging, because well, for security reasons, no incoming or outgoing connectivity is allowed in the compute notes.

[Antonio Perez-Calero Yzquierdo] 15:27:36
this means that well, everything that needs to happen for for the same, it will run a job like what I have in now, on the on the right hand side, accessing what being connected to the water management being able to to access the software of course conditions data and finally access to storage all these things

[Antonio Perez-Calero Yzquierdo] 15:27:54
are at a Yeah, a cat. Basically, all this connection, even, we have in recently discussing the possibility of having some added the it's services.

[Antonio Perez-Calero Yzquierdo] 15:28:07
And this is not the not not even this is is allowed.

[Antonio Perez-Calero Yzquierdo] 15:28:11
So of course, I shall stop at 4 for Cms, as tasks require.

[Antonio Perez-Calero Yzquierdo] 15:28:15
Stephen, services such as the ones I. That is correct.

[Antonio Perez-Calero Yzquierdo] 15:28:20
What we have is a login note which allows a site and a share file system mounted on on the execute notes.

[Antonio Perez-Calero Yzquierdo] 15:28:29
And And yeah, we can access this. This distributed file system. Be Sh: Fs: So what we are doing Well, he use these capabilities to to build the the model that you can see in the next slide which requires a sensible substantial amount of integration.

[Antonio Perez-Calero Yzquierdo] 15:28:49
Work, Yeah, So what the components that that we have, let's say in our favor to make this thing work is, first of all is the condor split Startup.

[Antonio Perez-Calero Yzquierdo] 15:28:57
So it uses the the share file system as a communication layer for the job.

[Enrico Fermi Institute] 15:29:02
Yes.

[Antonio Perez-Calero Yzquierdo] 15:29:02
Management. Well, you can see. Yes, Abc. And D.

[Antonio Perez-Calero Yzquierdo] 15:29:10
In the in the in the diagram below, where basically condor is kind of well, it's communicating between the study, and they actual starter where they were.

[Antonio Perez-Calero Yzquierdo] 15:29:19
The job run let's say, is communicating via passing files through the file system.

[Antonio Perez-Calero Yzquierdo] 15:29:23
Okay, then for software, what we do is basically replicate the Cbm Fs and repositories and Bsc: we, we we get what we need a peak.

[Antonio Perez-Calero Yzquierdo] 15:29:34
And then basically send the files and and be in the environment What are the Nbsp.

[Antonio Perez-Calero Yzquierdo] 15:29:40
For the conditions. Data is, we cannot access a databases, remote databases.

[Antonio Perez-Calero Yzquierdo] 15:29:46
We have to pre fetch those conditions, make them into files, pretty, place them into Bsc.

[Antonio Perez-Calero Yzquierdo] 15:29:51
And finally for storage concerns, we have developed our own service for input and output data transfers initially for output. Now for the stage out, let's say now, we are also commissioned this for for like.

[Antonio Perez-Calero Yzquierdo] 15:30:08
That so it's kind of white comboluted.

[Antonio Perez-Calero Yzquierdo] 15:30:12
The system you can see on the 2 2 extremes of on the diagram, Cern: Of course, the Cms water management system the storage, etc.

[Antonio Perez-Calero Yzquierdo] 15:30:20
And, on the other hand, the Bsc. And how we have to build all this intermediate layer at the up.

[Antonio Perez-Calero Yzquierdo] 15:30:27
Pick this bridge. Okay, next, please. Yeah, So what's the current status?

[Antonio Perez-Calero Yzquierdo] 15:30:35
Okay, system. The system works the services, and infrastructure that we have deployed.

[Antonio Perez-Calero Yzquierdo] 15:30:41
this as a allowed us already to to run a test, very reasonable scale.

[Antonio Perez-Calero Yzquierdo] 15:30:47
15,006 Cpu cores in in modern option 5.

[Antonio Perez-Calero Yzquierdo] 15:30:51
This is realistic Cms jobs, and this is an integrated.

[Antonio Perez-Calero Yzquierdo] 15:30:56
Well, I aggregate the output rate of 500 megawatts per second.

[Antonio Perez-Calero Yzquierdo] 15:31:00
Okay, So it's capable of sustaining societies.

[Enrico Fermi Institute] 15:31:03
Yeah.

[Antonio Perez-Calero Yzquierdo] 15:31:03
So the staging out works is commissioner and ready as I'm as I'm mentioning.

[Antonio Perez-Calero Yzquierdo] 15:31:12
Yup, probably. Let's say now, Okay, it's actually in discussing this Cms: workloads that can fit into this model.

[Antonio Perez-Calero Yzquierdo] 15:31:21
And with the constraint that I explained before. So what we, what would I call realistic Senior Cms workloads so far, this tests are Gen: same task change jobs.

[Antonio Perez-Calero Yzquierdo] 15:31:32
For example, in this case, minimum bias production. So it means there is no access.

[Antonio Perez-Calero Yzquierdo] 15:31:38
Cool, Okay, or there is no input, data. A full simulation, however, in the style that same as mostly performs, is in the form of a step chain.

[Antonio Perez-Calero Yzquierdo] 15:31:49
So it's a single single condor job running all the 4 stages since him did there Rigo, Where in the in this 2, stages they pile up libraries are access be enterprise.

[Antonio Perez-Calero Yzquierdo] 15:32:04
Okay, so we can have a triple A, So what we could do in order to be able to run this full step chain is to copy the premix data samples into the Ac.

[Antonio Perez-Calero Yzquierdo] 15:32:15
we have, let's say, ask about that this possibility.

[Antonio Perez-Calero Yzquierdo] 15:32:19
But but okay, copying data sets all the size about the of about the petabyte.

[Antonio Perez-Calero Yzquierdo] 15:32:28
It's not the currently allowed. There's no, there's no capacity in the Karate marinosum for that. Perhaps in modern option.

[Antonio Perez-Calero Yzquierdo] 15:32:35
5 dimension, but not that at present. Okay, So that rules out this type of phone simulation.

[Antonio Perez-Calero Yzquierdo] 15:32:41
What to look, let's say, and then what we are doing right now is commissioning this stage.

[Antonio Perez-Calero Yzquierdo] 15:32:48
The stage in right. So So this customize data transfer service in order to push files from pick a storage for simple, but it even we could get through triple a into peak and then They are into Bsc.

[Antonio Perez-Calero Yzquierdo] 15:33:02
In order to enable and running workflows which require input data.

[Antonio Perez-Calero Yzquierdo] 15:33:05
For example, we are thinking of participating or enabling broader reprocessing at the admiration.

[Antonio Perez-Calero Yzquierdo] 15:33:14
And this is the current situation It's it's not only okay.

[Antonio Perez-Calero Yzquierdo] 15:33:19
Let's say, in relation to many things that have been discussed so far.

[Antonio Perez-Calero Yzquierdo] 15:33:23
Yeah, in this workshop. It's not only the the the capabilities that that we are allowed, or that actually that we're not allowed to to to have a Dsc: together with how Cms operates for example, step chains are preferred over does change right So this already restricts

[Antonio Perez-Calero Yzquierdo] 15:33:45
very much what we can do in in Bfc. I think that's that's it.

[Enrico Fermi Institute] 15:33:53
I just wanted to have a call

[Enrico Fermi Institute] 15:33:55
I just wanted to have a comment. This was but Antonio showed the split, starter method that this HD Corner integration.

[Enrico Fermi Institute] 15:34:01
That's actually what we did. What we used for the Lcf.

[Enrico Fermi Institute] 15:34:06
Theta, integration, the prototype integration that we used during the the 2120, 21 Lcc.

[Antonio Perez-Calero Yzquierdo] 15:34:10
Yeah.

[Enrico Fermi Institute] 15:34:14
It worked that, too. It's it's a little simpler there even then, because you, since you do have edge services that you can call out from the edge.

[Enrico Fermi Institute] 15:34:22
So certain things are not quite as complicated as Pcs.

[Enrico Fermi Institute] 15:34:25
But we followed the same general integration, principle.

[Antonio Perez-Calero Yzquierdo] 15:34:28
yeah, that our case I don't know. I I would say it's particularly interesting because we are really being asked and enforce a right.

[Antonio Perez-Calero Yzquierdo] 15:34:37
We we have been asking false to use marinos room bye, from the funding agency point of view.

[Antonio Perez-Calero Yzquierdo] 15:34:44
Right, I mean, Oh, yeah, we have the the notion that Cpu is going to be got in in in further incoming request, let's say, funding requests for for our Lc computing projects how about on the other hand Bsc: is not very friendly in terms of allowing things that will

[Antonio Perez-Calero Yzquierdo] 15:35:06
make the integrate.

[Enrico Fermi Institute] 15:35:07
And at the funding agency have no no way to influence Pcs.

[Enrico Fermi Institute] 15:35:11
They can just say No, we don't

[Antonio Perez-Calero Yzquierdo] 15:35:13
It's okay. Yeah, it's like A, It's kind of I don't know.

[Antonio Perez-Calero Yzquierdo] 15:35:15
I see it as kind of paradoxical, because really we're kind of been trapped between the 2 forces squeezing us in the in the middle.

[Antonio Perez-Calero Yzquierdo] 15:35:23
Right? So yeah, it's it's making it quite a lengthy and and and at those project to integrate this.

[Antonio Perez-Calero Yzquierdo] 15:35:32
Well, we are advancing. We are trying actually to make it as universal as possible.

[Antonio Perez-Calero Yzquierdo] 15:35:37
Let's say, in in relation to Cms workflows, because otherwise it would not be able to.

[Antonio Perez-Calero Yzquierdo] 15:35:44
We will not be able to use the resource. But again, it's it's it's difficult

[Enrico Fermi Institute] 15:35:50
Okay, Any other questions, comments.

[Ian Fisk] 15:35:56
I I had one which which is sort of to Antonio, and sort of, I think, to the larger group which is, do we, hey, Chris?

[Ian Fisk] 15:36:04
We want to take advantage of sort of the Wlcp.

[Enrico Fermi Institute] 15:36:06
It's

[Ian Fisk] 15:36:09
And the sort of the larger organization structures that we have to basically say that network connectivity downside is some is is necessary to work.

[Ian Fisk] 15:36:21
I think it's it's really it's it's very impressive technical work to be able to go around this.

[Ian Fisk] 15:36:25
But this is something that we could sort of like. I wonder if there'd be any benefits sort of pushing from Mobile

[Antonio Perez-Calero Yzquierdo] 15:36:34
yeah, I'm not. I'm not usually involved in the in the political discussions.

[Antonio Perez-Calero Yzquierdo] 15:36:40
so I I couldn't tell myself. I I don't know if see money, for example, with the we provide

[Enrico Fermi Institute] 15:36:47
I mean, I mean the one thing, Antonio. If you said that they want to reduce your funding for great computing and replace it with Hbc.

[Enrico Fermi Institute] 15:36:55
I mean at that point they need at that point I think they expect that that Hbc allocate the capacity kind of counts as a replacement, and don't they need.

[Antonio Perez-Calero Yzquierdo] 15:36:55
Yeah.

[Enrico Fermi Institute] 15:37:07
Like Ws. G agreement at that point, but they actually consider this to be an equivalent replacement

[Antonio Perez-Calero Yzquierdo] 15:37:13
yeah, in principle, the idea is that for for Cpu intensive workloads estimated at about 50% of the Cpu requirement.

[Antonio Perez-Calero Yzquierdo] 15:37:24
No request, 50% would be provided by by the Yeah, by Cpc.

[Antonio Perez-Calero Yzquierdo] 15:37:30
And then we still would have some Cpu for data processing.

[Antonio Perez-Calero Yzquierdo] 15:37:34
Let's say, for the usual Oh, there'd be a one that's kind of the idea.

[Antonio Perez-Calero Yzquierdo] 15:37:40
But in order to do that, the yeah, like, I said, we, we we are being forced a kind of to transform this into a and universal resource, which is, which is not yeah, is very much not so

[simonecampana] 15:37:52
yeah, to commit to comment. Several people talk to to the funding agents, including myself, talk to the funding agency and the pick, and also to be a C.

[Enrico Fermi Institute] 15:38:02
Yeah.

[simonecampana] 15:38:05
But it seems to be a triangle that doesn't really understand each other.

[simonecampana] 15:38:11
So I think what Antonio is saying is correct. They're trying to push this on.

[simonecampana] 15:38:17
The throat, and of course we are trying to push back. Now.

[simonecampana] 15:38:21
Of course funding agents. Is not obliged. The pledge right?

[simonecampana] 15:38:27
I mean good the funding. It just says, Okay, this is the money we have. And you know, if you want X Tab, you can not okay to use this

[Ian Fisk] 15:38:35
Okay, I guess some money that point with, and my point was sort of like.

[Ian Fisk] 15:38:40
Did we want to? When we're writing the email we set in relatively strict criteria about, but services needed to be run, and what the expectations were in terms of quality of service, and availability but also in the development of the protocol and this occurs to me as a place where

[Ian Fisk] 15:38:58
like the Wsg. Could decide that one of the protocols that's necessary to be considered a site is this?

[Ian Fisk] 15:39:05
And it doesn't. It's not guaranteed to work.

[Ian Fisk] 15:39:07
But I think that in some sense exciting without it is almost guarantees that it will not change

[simonecampana] 15:39:14
Yeah, I mean, it would be useful if the I would say the peak management would make on these for my former request to Wcg: because a a reality what pick has done is to do a lot of diligent work to try to overcome the limitations.

[Ian Fisk] 15:39:32
Right.

[simonecampana] 15:39:34
hey? It would be good if this would be the other way around, and at some point they would say, cannot do so.

[simonecampana] 15:39:41
We can not offer tier one services with this piece of facility, and then we would have a discussion with the funding agency on on those basis at the moment those discussions they led, the not too much, to be honest I don't know if Antonio has more detail that's what I understand also from

[Enrico Fermi Institute] 15:39:54
Okay.

[simonecampana] 15:39:58
pip

[Enrico Fermi Institute] 15:39:59
Could be try to move on and maybe move that offline, because it's not yeah, it's interesting, but it's also it's internal Wsg Spanish funding agency.

[Antonio Perez-Calero Yzquierdo] 15:40:10
yeah, Thank you.

[Enrico Fermi Institute] 15:40:11
So that's not relevant to the I think we have one more presentation and then we still need to have the cost discussion.