Eastern Time

[Kenyi Paolo Hurtado Anampa] 11:05:44
Okay, So good morning. Everyone. Today, we will call queues.

[Kenyi Paolo Hurtado Anampa] 11:05:51
Okay, 2 different things. One is resources. This is going to be all of the morning session, and then in the afternoon we are going to talk mostly about networking and the assistant reference of what Hpc and Clouds and then R and D: So for the I would focus area, we

[Kenyi Paolo Hurtado Anampa] 11:06:12
will start with the just summarizing at the very high level.

[Kenyi Paolo Hurtado Anampa] 11:06:17
What Atlas and Cms have done, and the case of at last.

[Kenyi Paolo Hurtado Anampa] 11:06:24
Well, this is this has been okay. The the and shame they got a self contained your site, and this is they're they're linked to the Ukrainian, and they have their own screen Cdfs.

[Kenyi Paolo Hurtado Anampa] 11:06:42
We will This will be talking more detail in good next. Okay?

[Kenyi Paolo Hurtado Anampa] 11:06:47
And then for Pms: This is basically describing what it was done about 5 6 years ago.

[Kenyi Paolo Hurtado Anampa] 11:06:55
During the demo testing that Cms: did with production portfolios and the the the way this was done was by extending an existing Csi, and it more particularly the formula the resource with resources in the Cloud and This was done Via.

[Kenyi Paolo Hurtado Anampa] 11:07:14
He cloud again, this will be describing more detail in the next few sites, since this is since this was done this way, maybe in terms of production integration, we have the same reservations as Hpcs in terms of historic for work, on my chains which means that all data must be staged

[Kenyi Paolo Hurtado Anampa] 11:07:36
to existing sites.

[Kenyi Paolo Hurtado Anampa] 11:07:42
go on with the next slide, and this is for advice Fernando.

[Fernando Harald Barreiro Megino] 11:07:48
yeah, sure. Yeah, So this is the overview of what we are working on in Atlas.

[Fernando Harald Barreiro Megino] 11:07:54
So we have 2 main projects, the one on the left is on Amazon, and this comes through a Fresno from California State University.

[Fernando Harald Barreiro Megino] 11:08:04
And here we have basically, panda queue storage element, and also a squid, And those are always the 3 main cost components that we will have later with.

[Fernando Harald Barreiro Megino] 11:08:13
An it's also the the egos, and then the second part is Cook is the product that we have in.

[Fernando Harald Barreiro Megino] 11:08:23
Google it used to be. Us atlascentric But now, this year, since the middle of July, it became a worldwide, dotless project.

[Fernando Harald Barreiro Megino] 11:08:35
And so Atlas, is. A collaboration, is it's participating in the budget.

[Fernando Harald Barreiro Megino] 11:08:42
and here in this project, we do have a similar setup, us in Amazon with panda queue truth, storage element, and the squid.

[Fernando Harald Barreiro Megino] 11:08:50
But we also how we work like on a analysis facility prototype, with a 2 bitteran task.

[Fernando Harald Barreiro Megino] 11:08:59
so the integration of this of these cloud resources. We're done by the route team on the ponder team.

[Fernando Harald Barreiro Megino] 11:09:07
So we take a different approach done. If you are like trying to extend her side, and we we just generate our self-contained on cloud native side, in the case of truth, on the storage so it works in the way that to download the key for from Amazon or from

[Fernando Harald Barreiro Megino] 11:09:28
Google. And with that the key you can sign Url. And with the Url in the center Url, you say, you can upload a particular file until an hour from now, or you can download or Delete of and then this key needs to be put into ruthie and into fts so that they can generate

[Fernando Harald Barreiro Megino] 11:09:47
the assigned Url to with the downloads or the third party transfers for the compute path. It's all based on kubernetes and native integration.

[Fernando Harald Barreiro Megino] 11:10:00
There is particular. There is no nothing like a condor in the setup, and then we have Cvm Fs installed in the closer to our kubernetes planning That was one of the things that actually took most of the F to get at the very reliable

[Fernando Harald Barreiro Megino] 11:10:18
state, and then also the this quick part I mean, that's you can either run it in part as a part of the who could need this cluster in, Google for example, I just run load balance load balance the instance, great and the other thing that for the computer I always use is the outer

[Fernando Harald Barreiro Megino] 11:10:41
scaling. So when there are no jobs cute, for example, the panda compute part.

[Fernando Harald Barreiro Megino] 11:10:48
It shrinks to a minimum, and then, if you submit a lot of tops it, the the cluster grows up to, or the limit, or as much as needed for hosting all of the jobs yeah the the setup, is, it's not bound to any particular cloud provider

[Fernando Harald Barreiro Megino] 11:11:07
it's just Stanford protocols and technology.

[Fernando Harald Barreiro Megino] 11:11:10
So you can in principle use the same setup in other cloud.

[Fernando Harald Barreiro Megino] 11:11:13
Providers. For example, I tried out the dependent part one time in in Oracle Cloud, just to see that it works.

[Fernando Harald Barreiro Megino] 11:11:22
Yeah, then in the next slide, please.

[Fernando Harald Barreiro Megino] 11:11:27
So one of the things that you can exploit on all of these commercial clouds, is all the different types of architectures that they have and that you don't always have on on grid sites.

[Fernando Harald Barreiro Megino] 11:11:41
One particular example is on Amazon. We were doing some arm testing.

[Fernando Harald Barreiro Megino] 11:11:47
So for this case it was Johannes and the teenager team that were trying to build the Athena simulation software for arm 64.

[Fernando Harald Barreiro Megino] 11:11:58
They had done the building, and they wanted to do a small physics.

[Fernando Harald Barreiro Megino] 11:12:01
Validation, or run a of a whole task. With that.

[Fernando Harald Barreiro Megino] 11:12:04
But there was not really any volunteer, any available grid site with arm resources that could set that up.

[Fernando Harald Barreiro Megino] 11:12:12
So what we did is we set it up in in Amazon with the cravat on 2 notes in the in the right side diagrams.

[Fernando Harald Barreiro Megino] 11:12:25
It's just the first validation that you honest it with 10,000 events.

[Fernando Harald Barreiro Megino] 11:12:29
And he compared the X 86 that had been executed in track.

[Fernando Harald Barreiro Megino] 11:12:32
I believe, against the arm, 64 on arm as an end.

[Fernando Harald Barreiro Megino] 11:12:36
It was it was matching quite well. And then, some weeks later, we prepared the full physics, validation with a 1 million events, and that was fully signed off few weeks ago.

[Fernando Harald Barreiro Megino] 11:12:49
So in principle, simulate simulation could be executed.

[Fernando Harald Barreiro Megino] 11:12:57
like in standard. Production. Now and I mean, we don't do this in particular for the cloud, but we do it more like it was discussed yesterday in the Hpc.

[Fernando Harald Barreiro Megino] 11:13:06
Session, where most of the next generation Hpcs are going to come up with a with more on Cpus and X.

[Fernando Harald Barreiro Megino] 11:13:17
86 is going to be dominant so it's a preparation for that.

[Fernando Harald Barreiro Megino] 11:13:20
Other things, so other exotic architectures or resources that can be Houston. Huh!

[Fernando Harald Barreiro Megino] 11:13:27
Thanks that can be used in in the cloud. For example, there is a user that is doing some trigger studies for a filter and there he's using Fpgas on Amazon or Johann for building the software for He uses very large notes on on Amazon and

[Fernando Harald Barreiro Megino] 11:13:47
Google and also Cpu stuff. Next slide, please, And if anyone has a question or comment while I'm going through the slice, you can interrupt me.

[Fernando Harald Barreiro Megino] 11:14:01
Now we come on, Google, just running Google as a great site, you can see 2 different approaches on the right top plot.

[Fernando Harald Barreiro Megino] 11:14:14
You can see how we were doing like scalar test.

[Fernando Harald Barreiro Megino] 11:14:17
That was done during the previous funding around until we were trying to see how how far we can scale it in a single client region, and we were getting to a 100,000 course in Europe West one P.

[Fernando Harald Barreiro Megino] 11:14:35
Which is an one of the one of the European regions, and if you would want to scale this out even more, you could replicate the setup to to whatever, to us to multiple regions in Europe, and so on and reaching a very high number of costs what we are doing now in

[Fernando Harald Barreiro Megino] 11:14:56
since it's fully worldwide Atlas Project is, we're running at the moment, a fixed, dry, a fixed size grid side.

[Fernando Harald Barreiro Megino] 11:15:05
We started with 5,000 calls, and we moved it to 10,000 costs.

[Fernando Harald Barreiro Megino] 11:15:10
It's exactly a month ago, and we can run any type of production.

[Fernando Harald Barreiro Megino] 11:15:16
we are not running analysis at the moment, because we need to reorganize the the storage.

[Fernando Harald Barreiro Megino] 11:15:23
In particular, we need a data discontent, separate, stretch disk so that user outputs don't end up in the same storage element.

[Fernando Harald Barreiro Megino] 11:15:32
but the other one Is this: The discrete site has worked very well.

[Fernando Harald Barreiro Megino] 11:15:38
It's very reliable, and also a very low error rate.

[Fernando Harald Barreiro Megino] 11:15:41
And when the the errors are usually very focused on particular situations, like, for example, I great to machines with low disk, or one at the time, there were issues with the with the phone tasks and I had, to fix that and our goal, is to do like a mix of

[Fernando Harald Barreiro Megino] 11:16:02
both both versions like, mix the the on demand fast scale out with a fixed size.

[Fernando Harald Barreiro Megino] 11:16:12
So we plan to run more or less. A a flat queue with 5,000 cores, and then on top run a dynamic queue, which processes urgency requests.

[Fernando Harald Barreiro Megino] 11:16:24
Or we are going to do something that we call the full chain where all of your steps in a in a simulation in our production, chain I run inside the same resource and you don't export the you only export the final in order to reduce the egress, cost

[Fernando Harald Barreiro Megino] 11:16:48
yeah, and the next slide. Thanks, Kenny. The other thing that we tried out is this analysis facility prototype?

[Fernando Harald Barreiro Megino] 11:16:56
what we wanted to do is like task scaling evaluations.

[Fernando Harald Barreiro Megino] 11:17:03
So we installed Twitter and task on on Google.

[Fernando Harald Barreiro Megino] 11:17:08
We integrated it with the Atlas Am. So in anyone from Atlas can connect without needing to question in particular new account or anything.

[Fernando Harald Barreiro Megino] 11:17:20
And then we have a couple of different options that the user can select for tasks.

[Fernando Harald Barreiro Megino] 11:17:27
You use this first light version, but then we also have machine learning images.

[Fernando Harald Barreiro Megino] 11:17:31
so that other people use tensorflow, and all those libraries, and you can also, if you want to put notebook with a cpu, and that will take a little moment to to put you need to provision the the machine.

[Fernando Harald Barreiro Megino] 11:17:50
You need to install Cvmfs and load Cmfs, and then added to the to the cluster.

[Fernando Harald Barreiro Megino] 11:17:57
That takes a couple of minutes. But then you have a notebook with a Gpu just for yourself, and you can without as long as you need, and for the task part which is, in my opinion, a very good example, for great scalar, for cloud scalability the right lower plot was

[Fernando Harald Barreiro Megino] 11:18:20
and look at Signage, who was trying out running the same task, but with a different number of workers.

[Fernando Harald Barreiro Megino] 11:18:27
So he ran first with 100 workers, and it took 40 min.

[Fernando Harald Barreiro Megino] 11:18:30
Then he re rerun the same task with the 200 workers, and the duration was half and so we'll until the the last part where he uses 1,500 workers and the task is done within just a few minutes and the the thing about this is that the cost on the cloud is

[Fernando Harald Barreiro Megino] 11:18:51
roughly the same, except for maybe they're just scaling or scheduling overhead.

[Fernando Harald Barreiro Megino] 11:18:57
but the cost is roughly the same. If you run way very few workers, and if you run with a lot of workers and for the use himself, it makes a lot of difference if he gets the results in 1 h or in 5 min, and yeah, we also should consider the in the cost the

[Fernando Harald Barreiro Megino] 11:19:19
calculation with the salary of the of the user himself, since he's optimizing his time. A lot.

[Fernando Harald Barreiro Megino] 11:19:27
Yes, and that's it. Can you? For next night.

[Kenyi Paolo Hurtado Anampa] 11:19:32
it's from then Yes, and so then for Cms again.

[Kenyi Paolo Hurtado Anampa] 11:19:37
This is what it was done a few years ago, and again, as I mentioned before, we did this by integrating cloud resources in one of the their sites at the Fermi website.

[Kenyi Paolo Hurtado Anampa] 11:19:51
via head cloud. So do you basically have a workflow injected.

[Kenyi Paolo Hurtado Anampa] 11:19:55
which is the resource provisioning trigger.

[Kenyi Paolo Hurtado Anampa] 11:19:58
This enters the facility interface which talks to the authentication and authorization mechanisms.

[Kenyi Paolo Hurtado Anampa] 11:20:04
Then the decision. There is a decision engine and a facility pull.

[Kenyi Paolo Hurtado Anampa] 11:20:09
There. And the decision engine basically talks to a provisioner that will be talking to the Microsoft.

[Kenyi Paolo Hurtado Anampa] 11:20:16
The same day cloud. And so this is basically a diagram of the head cloud architecture.

[Kenyi Paolo Hurtado Anampa] 11:20:22
What you have from there is basically going to restart this in the local sources in the cloud.

[Kenyi Paolo Hurtado Anampa] 11:20:34
So you have connecting to the HD. Calendar schedulers in the gliding wms in procedure, And that's How everything.

[Kenyi Paolo Hurtado Anampa] 11:20:43
Is connected in this case

[Kenyi Paolo Hurtado Anampa] 11:20:53
Okay. And the next part is Lans. You know. I think they're talking.

[Dirk] 11:21:01
yes, so Lanceium was already mentioned yesterday. It's it's an interesting new.

[Dirk] 11:21:11
Does that new company, and they because they they're not like a your traditional full service cloud provider that sell you that basically operate worldwide and give you anything you want in terms of capabilities And instance, types And whatever there they really geared towards utilizing low cost renewable energy

[Dirk] 11:21:35
to provide cheap compute, basically. And they're almost like a part of the business model is almost like an energy utility.

[Dirk] 11:21:42
basically they get money for for being able to low chat, and and they're they.

[Dirk] 11:21:47
They construct They're constructing the data centers right now in in in areas with that very high on renewable wind energy, and we we did a test a few months back where we integrated them into production.

[Dirk] 11:21:59
We run a few small workflows. It was all on free cycles as a as a test.

[Dirk] 11:22:05
Basically they they have a bit different than aws. Google. They only support singularity containers, not vms.

[Dirk] 11:22:11
And what we did is we just ran a pilot job in the singularity container, and then then the pilot itself is just to stand that Cms pilot.

[Dirk] 11:22:21
So it runs our payloads in in in a nested singularity container, you know.

[Dirk] 11:22:26
Cvm. Fs and local squid were provided from Nancy.

[Dirk] 11:22:30
We we work with them on that. They currently don't have any local managed storage.

[Dirk] 11:22:34
Just so job scratch. So in, And we basically run these resources like we do Opportunistic was G.

[Dirk] 11:22:39
Or Hbc. Site set where we don't use manage to storage.

[Dirk] 11:22:42
We just used triple a reads to get the input and then stage out to formula.

[Dirk] 11:22:48
So that's that covers the runtime.

[Dirk] 11:22:50
The provisioning integration is another problematic area potentially for long term, because they have a custom api, which is not compatible with aws or Google I mean it already They're running singularity containers so you need some way to start up.

[Dirk] 11:23:05
A container, and what we're doing right now is, we're just using vacuum provisioning.

[Dirk] 11:23:10
So we just when we run to run a test, we just start up a container manually when needs.

[Dirk] 11:23:15
And that's relatively simple through the Api, because the Api is just.

[Dirk] 11:23:19
You can. You can run like some script that that call out to the Python Api.

[Dirk] 11:23:25
Call out to the Api and tell it how many containers are running.

[Dirk] 11:23:28
If it's less than 10, you bring it up to 10 So that's that's that's basically the level of integration that we have right now.

[Dirk] 11:23:36
So we think it's interesting enough. It will cry a little bit of work to to get really working to get it fully integrated.

[Dirk] 11:23:44
But we're working on with with Lansing pretty occurring A small number of cycles for more tests.

[Dirk] 11:23:51
Is it? The plan is maybe to to get some cycles there and then see if we can.

[Dirk] 11:23:57
When there's particular load from Cms specifically on Fermi app, that we can say, Okay, we bring up lensing resources and that freeze up resources at formula to do stuff that is most suited to a tier.

[Dirk] 11:24:10
One

[Enrico Fermi Institute] 11:24:28
just to get in there for a second in the case of all of this, this is very much oriented around production jobs. Do you think we could organize sometime in you know, the next year or something like that trying to interface this with either coffee Casa or the elastic analysis facility?

[Enrico Fermi Institute] 11:24:45
Effort to see if we can gain more flexibility for more bursty analysis jobs, much like what Atlas was doing with Google Cloud and whatnot

[Dirk] 11:24:56
we could try I mean the

[Enrico Fermi Institute] 11:24:58
The security is gonna be a nightmare at elastic analysis.

[Dirk] 11:25:03
Yeah, the the the thing is it? It really depends how well everything plays together with the provisioning integration.

[Enrico Fermi Institute] 11:25:10
Facility for sure

[Dirk] 11:25:10
I mean, they have a simple Api. They just pass it.

[Enrico Fermi Institute] 11:25:10
Yeah.

[Dirk] 11:25:13
You basically you need a token associated with your account, and then you have, a if you're a single python script, like a monolithic python script that they give you where you can tell it to start a container and bring up something so so it's it's

[Enrico Fermi Institute] 11:25:25
Okay.

[Dirk] 11:25:27
relatively simple, so sure, I mean, we can look at it It's it's a matter of Do you want to do it?

[Enrico Fermi Institute] 11:25:32
I mean

[Dirk] 11:25:35
Do you want to do tests, or you want to do it for real?

[Dirk] 11:25:37
Because when if you do it for real, then you actually need to have a paid for a number of cycles sitting there that you can use for tests, we can just go whenever

[Enrico Fermi Institute] 11:25:46
Yeah, I I think we would need to get the facilities at the or the at least the one affirmative action, as it is right now and then go for a more for real test with people's actual, analysis.

[Enrico Fermi Institute] 11:25:57
Jobs. Once we have that set up. I think that would be the better way to see how this actually works.

[Enrico Fermi Institute] 11:26:06
So this is like a year, time scale, or something like that.

[Enrico Fermi Institute] 11:26:09
Your analysis, facility. Do you have any implicit dependencies on shared file systems, or anything like that? Or does everybody because we're at or because we're a fermi, lab we're restricted from using shared shared file systems aside from like, X d And stuff, Okay, yeah, I was gonna

[Enrico Fermi Institute] 11:26:24
say that might be 1 One challenge. Is stretching out to right is, How do you structure file system out there?

[Dirk] 11:26:29
They.

[Enrico Fermi Institute] 11:26:29
Exactly, but thankfully. We've already been forced to solve that

[Dirk] 11:26:33
Maybe because Lindsay just mentioned a year, the the time horizon on that at the currently Lens Zoom.

[Dirk] 11:26:41
As I said, the young company need a starting up. They're kinda still building the data sent us like the the, the So they have a test data center.

[Enrico Fermi Institute] 11:26:45
Hmm.

[Dirk] 11:26:51
That's in Houston, which is not really using renewable energy.

[Dirk] 11:26:54
But where they basically just deploying the the the whole hardware software integration that they're working with, and that's what we've been testing, what they're building right now, and which is supposed to come online at some some point later, this year, or early next year, i'll leave really the big data centers

[Dirk] 11:27:08
which are co-located to like wind, energy, hotspots, and and and Texas.

[Dirk] 11:27:15
There's not much else there, but they're building a data center and they'll be the interesting.

[Dirk] 11:27:20
Ones basically because that's real, renewable in a jet.

[Dirk] 11:27:22
This lots and lots of basically power capacity there. And they they call more importantly, they're gonna connect them to a 100 gigabit to the Us.

[Dirk] 11:27:33
Net and everything else

[Enrico Fermi Institute] 11:27:34
Okay, they're actually going to appear to. They're going to actually can connect Peer with the snap for sure.

[Dirk] 11:27:41
That's what their plan is, because they're they are kind of pushing They're they're basically making the sales pitch hard to academic users.

[Dirk] 11:27:51
I mean, I've seen talks for them on Scc. Os was G.

[Dirk] 11:27:56
Did they basically travel around Europe Because for Europe it's it's like running compute on cheap powers and even bigger concern right now than in the Us.

[Dirk] 11:28:06
because power prices there traditionally have been much higher.

[Dirk] 11:28:09
And now I extremely much higher than than in the Us.

[Enrico Fermi Institute] 11:28:12
But they're gonna connect to Esnet versus Internet to

[Dirk] 11:28:17
Probably I mean they. They said. They're they're really. They.

[Dirk] 11:28:20
They basically point out, that's that's another of this selling points.

[Dirk] 11:28:24
They point out that they want to not charge, for egress.

[Dirk] 11:28:32
So like yeah, like not charging for egos in good network.

[Dirk] 11:28:39
Integration for academic workloads seems to be they're they're they're focusing on that, I mean, because I mean, you have to look at what the they are low quality of service.

[Dirk] 11:28:50
Somewhat by design. They're not like Amazon ready, Sell you to give you a Vm.

[Dirk] 11:28:55
And promise you 99 point whatever. David. Then Zoom tells you.

[Dirk] 11:28:59
If the if the room, there's no wind, we're gonna load chat like crazy.

[Dirk] 11:29:04
So we're gonna evacuate you, and that's that's fine.

[Dirk] 11:29:08
but that also means that they have to have other selling points, because and other target markets, because they're not gonna attract the like the financial sector or industry.

[Dirk] 11:29:18
That wants like high up time, compute service, sitting somewhere

[Ian Fisk] 11:29:22
But but the turkey thing was, is net not Internet too right?

[Dirk] 11:29:26
I'm I'm not exactly sure they They basically we had.

[Ian Fisk] 11:29:28
Oh sure!

[Dirk] 11:29:30
And and you have to remember the discussions we had with them.

[Ian Fisk] 11:29:32
Right.

[Dirk] 11:29:34
But most months before these data, they're still under construction, so I don't think the network is connected yet.

[Ian Fisk] 11:29:38
Right. The only reason I ask is that it is one of the things that yes, Net Charter is very is relatively strict, and it will allow you to connect formula or Bnl or cern to land team resources but for instance, it won't carry from a university so one of the

[Dirk] 11:29:39
Nope.

[Ian Fisk] 11:29:58
endpoints needs to be under the Es net charter, which limits we can be a little limiting

[Dirk] 11:30:04
I mean at the moment I would. Read It as a a statement of intent that they want to do everything they can on the network integration side to make it easy for us to to to use their facilities and then what's actually deployed on how things are connected I think we have to wait for for these data

[Ian Fisk] 11:30:08
Okay.

[Ian Fisk] 11:30:15
Right.

[Ian Fisk] 11:30:22
Right, and it could It's fine that that matches under the charter.