[Eastern Time]

let's let's see what we've been doing previously, and try to wait till maybe 5 after before we get started, and I think then we'll we'll jump right into rotation

[Enrico Fermi Institute] 14:01:53
we'll do it just a couple of minutes. Here.

[K-T Lim] 14:02:08
I only figured out how to log in discern

[K-T Lim] 14:02:22
So I can upload

[Enrico Fermi Institute] 14:02:35
Okay.

[Enrico Fermi Institute] 14:02:38
Yeah, Just let me know, Kt: when you're ready to to start sharing.

[Enrico Fermi Institute] 14:02:43
probably wait for people to into the room. I've hosted the sort of relevant charge question Here, What can us, Alice?

[Enrico Fermi Institute] 14:02:53
Us. The related international efforts, like

[K-T Lim] 14:02:59
Yup, so I'm ready to share any time, and I've uploaded a Pdf.

[K-T Lim] 14:03:08
Of my slides and presenter notes to the indicator page

[Enrico Fermi Institute] 14:03:11
Okay, awesome. Thank you. Yeah. In the morning session we had a few more people.

[Enrico Fermi Institute] 14:03:16
So let's see if we can We can get some of those folks back, maybe like 2 more minutes.

[K-T Lim] 14:03:17
Okay.

[Enrico Fermi Institute] 14:03:23
And then, yeah, we can serve

[K-T Lim] 14:03:23
Sure.

[K-T Lim] 14:03:29
It's a little early for lunch here, but

[Enrico Fermi Institute] 14:03:52
okay.

[Enrico Fermi Institute] 14:04:34
okay, Maybe let's go ahead and get started. I'm gonna stop sharing Kt: and then you can.

[Enrico Fermi Institute] 14:04:39
You can start sharing your slides

[K-T Lim] 14:04:39
okay, okay.

[K-T Lim] 14:04:52
Let me see

[K-T Lim] 14:04:53
Let me see! Oh, geez

[Enrico Fermi Institute] 14:05:03
Okay.

[K-T Lim] 14:05:07
Hold on need to grant access, because apparently I updated something

[K-T Lim] 14:05:07
hold on!

[K-T Lim] 14:05:25
okay, I will be back in a second

[Enrico Fermi Institute] 14:05:27
okay.

[K-T Lim] 14:06:01
hey? Hopefully, this works better. That looks a lot better.

[K-T Lim] 14:06:08
okay.

[Enrico Fermi Institute] 14:06:08
Okay, great.

[K-T Lim] 14:06:16
Oh, is that good

[Enrico Fermi Institute] 14:06:17
Yep.

[K-T Lim] 14:06:21
Okay? Well, that's good started then. Thank you very much for inviting me Happy to share a little bit about the ribbon observatories experienced with cloud computing.

[K-T Lim] 14:06:37
how we got to where we are, where we think we are, and where we're going.

[K-T Lim] 14:06:44
Let me just start by saying that usually clouds are very bad for a strong numbers.

[K-T Lim] 14:06:49
You can see some clouds on the horizon over Laserna, in the left, and then there are plenty of desk clouds in the Milky Way, and all of those block use of things that as tremors like to see but in this, case they're actually pretty good so we

[K-T Lim] 14:07:02
we like the way that the cloud is working out for us.

[K-T Lim] 14:07:07
What is the Ruben Observatory doing? The Ruin Observatory is being built on top of a mountain in Chile, in order to perform the legacy survey of space and Time the survey will scan the scary, taking 20 TB tonight of school 30 s images

[K-T Lim] 14:07:23
that'll cover the entire visible sky every few days.

[K-T Lim] 14:07:25
This is essentially a movie of the whole sky, or at least that we can of the sky that we can see in the we have several different data products that are produced on different cadences.

[K-T Lim] 14:07:36
So first of all, we have prompt data products that generate primarily alerts.

[K-T Lim] 14:07:43
these are indications that something has changed in the sky from what it used to be, And so we need to process the images from the telescope within 60 s to issue those alerts, So the other.

[K-T Lim] 14:07:57
Telescopes thing, then follow them up, and and and observe things that have changed our data, release Production executes once a year approximately, and it reprocesses all images that have been taken to date using a consistent set of algorithms and configurations and so

[K-T Lim] 14:08:16
that's obviously a data set that's growing each time, and the complexity of the analysis is likely to grow each time as well.

[K-T Lim] 14:08:24
so that that needs to to oh, go faster and faster as we're as we're progressed, saying, because we want to issue one day, release each year, and finally, definitely, not at least, we have the Ruben science platform which provides access to the data.

[K-T Lim] 14:08:42
Products and services for all science users, and project staff to do analysis and reprocessing of the data that has been taken, not shown on this slide, but also important is our internal stuff.

[K-T Lim] 14:08:56
Developers need to do both ad hoc and production style processing as well So that's another Sync: Okay, and storage.

[K-T Lim] 14:09:05
So a kind of architecture that we have to actually perform this as a data management system that looks like this.

[K-T Lim] 14:09:15
Here we have the telescope as kind of an input device off on the left hand side.

[Enrico Fermi Institute] 14:09:18
Okay.

[K-T Lim] 14:09:19
My colleagues who are working on actually building the thing that's pictured behind me would argue that they're doing a lot of the work.

[K-T Lim] 14:09:27
But we think that most of it is in the data management system over here.

[K-T Lim] 14:09:30
so we grab stuff at the summit on the left hand side of the Us. Data facility.

[K-T Lim] 14:09:35
We have the prom processing chain that's on running in near real time. And issuing alerts.

[K-T Lim] 14:09:41
After this hands community in the middle of the right hand side of this diagram we have offline processing that is, executing in sort of batch mode.

[K-T Lim] 14:09:51
It's high throughput computing, not high performance computing.

[K-T Lim] 14:09:54
and it's running across multiple sites. We have partners, and in France, at Cci and and in the Uk who will be executing large portions of the data release production.

[K-T Lim] 14:10:06
And and then finally at the bottom, and in the upper right.

[K-T Lim] 14:10:10
we have dedicated resources for this science user access and analysis on the Ruben Science platform.

[K-T Lim] 14:10:18
I'll talk about that more later

[K-T Lim] 14:10:23
We did a number of proof of concept engagements to try to determine how the cloud could work with us, and with this architecture.

[K-T Lim] 14:10:32
so we did 3 different engagements, with 2 separate cloud vendors, and they're documented in a bunch of data management, technical notes, which are all linked from this page, the first one in each series is the the goals of the engagement.

[K-T Lim] 14:10:49
What we set out to do, and then we have a report of what we actually manage to accomplish.

[K-T Lim] 14:10:54
So the first engagement we mostly leverage to get sort of cloud native experience, and how to deploy services.

[K-T Lim] 14:11:05
And systems in modern technologies to improve our deployment models, to get things containerized, etc.

[K-T Lim] 14:11:14
and not just have them running as shell scripts or things that an individual developer ran.

[K-T Lim] 14:11:20
we learned about potential bottlenecks, and how I been with delay product networks.

[K-T Lim] 14:11:25
Obviously, we're transmitting data from from Chile to the Us.

[K-T Lim] 14:11:29
that's over 200 ms, and it's a 100 gigabit network.

[K-T Lim] 14:11:35
So very high, bandwidth. So we need to get the data, And we need, we need to make that work efficiently.

[K-T Lim] 14:11:41
And so there were a number of bottlenecks.

[K-T Lim] 14:11:42
There that we worked through, And we learned about how to interact with the vendors, what mechanisms and and ways of working with them.

[K-T Lim] 14:11:53
worked well for us and for them. The second engagement was with a different vendor.

[K-T Lim] 14:12:00
We tested workflow execution, Middleware. This is some of our custom.

[K-T Lim] 14:12:05
middleware at a modest scale, up to about 1,200 virtual cpus, and we were able to make use of spot or preemptable instances, to run a lot of our processing It's easy to retry something a a particular quantum of processing if you feel for some reason if

[K-T Lim] 14:12:23
the if the processor went away, and that reduced system by a considerable amount.

[K-T Lim] 14:12:29
When you allow preemption that way, and the third engagement we tested improved workflow execution, middleware.

[Enrico Fermi Institute] 14:12:36
Yeah.

[K-T Lim] 14:12:37
so actually at a similar scale, up to 1,600 Vcpus.

[K-T Lim] 14:12:45
and here we we also did some transfers over the long call network again, and learned about the desirability of having http to persistent connections for uploading to object stores in particular sure so all of these taught us something about working with the cloud and some

[K-T Lim] 14:13:07
of the things that we learned that people don't necessarily talk about a lot, or that when we were able to work with a vendor who had relatively low bureauucracy high flexibility and a willingness to assist you know at our well to find point of

[K-T Lim] 14:13:26
contact and rapid internal processes that made things work much more smoothly.

[K-T Lim] 14:13:33
As we went through these engagements, and through a subsequent working with these vendors, deep engagement with the vendors and engineering teams, being able to talk to the actual product managers and even in in some cases engineers who are working on these products, also was useful and something that turned

[K-T Lim] 14:13:53
out that was quite unexpected is that consultants can also be very useful.

[K-T Lim] 14:13:59
So there are a number of consultants who are obviously fully trained and certified for building things on these vendors.

[K-T Lim] 14:14:08
Clouds, there. They don't know any more than the you know.

[K-T Lim] 14:14:14
People, at the vendors necessarily, but unlike the vendor engineers, they are allowed to work on your code The The vendors can't work on your code that that would cause.

[K-T Lim] 14:14:26
All kinds of problems, especially since in our case all of our code is open source.

[K-T Lim] 14:14:31
but the consultants can. They can actually modify things and and update your own code to work better in the cloud.

[K-T Lim] 14:14:40
And so that was that was something that turned out to be very interesting.

[K-T Lim] 14:14:47
we did a lot of cost modeling, so we have already had very complex internal spreadsheets to understand what our data sizing and compute sizing requirements would be.

[K-T Lim] 14:15:02
we adjusted them somewhat to fit the cloud storage models, and and how things will work there, and the vendor our vendors also produced spreadsheets that Then match.

[K-T Lim] 14:15:17
Those needs to the available technology cheese and their their quoted prices for them.

[K-T Lim] 14:15:24
so in our case, our compute costs compared with high energy.

[K-T Lim] 14:15:29
Physics are not that large, and we're talking only something in the millions of core hours, and that's only in year 10 of the survey.

[K-T Lim] 14:15:37
when we're doing the maximum memory processing of of the entire survey contents.

[K-T Lim] 14:15:42
So that's quite reasonable. The storage costs, on the other hand, for frequently access data turned out to be a major. Problem.

[K-T Lim] 14:15:50
we are expecting to have hundreds of petabytes of results that are both sort of in process that are that are the that are being developed.

[K-T Lim] 14:16:01
For the next data release as well as the results that are part of the previous data releases that are already public.

[K-T Lim] 14:16:08
So those force costs can be very large. And I we have had a number of you know.

[Enrico Fermi Institute] 14:16:13
Okay.

[K-T Lim] 14:16:19
It's kind of debates about why the on prem storage calls seem to be less than the in cloud storage costs.

[K-T Lim] 14:16:26
I mean, in some cases it's because the total cost of ownership is, it's somewhat different.

[K-T Lim] 14:16:31
sometimes things like people, like administrators, can be charged different accounts, and they don't actually fall under their projects.

[K-T Lim] 14:16:37
Budget, but I think a lot of it is also that in the cloud you're paying for more durability and performance.

[K-T Lim] 14:16:44
The then we often need in science, right? So in sense, we often have replicas of the data in other places, so we don't need, you know, 8 nine's or something like that worth of of during our Bill in one place and also we can schedule, when we're going to access

[K-T Lim] 14:17:02
data often, and so we don't need the kind of perfect that you might need for commercial workloads.

[K-T Lim] 14:17:09
Egress. Costs are often a problem, but there are mitigations that can exist, and of course, if you can keep most data, that's most data transfers, either inbound to the cloud or or entirely within.

[K-T Lim] 14:17:21
The cloud. Then there are no egress costs, and so that helps a lot.

[K-T Lim] 14:17:26
if we managed to do most of the data summarization and visualization within the cloud, and then only have the results.

[K-T Lim] 14:17:33
Exit that also limits the the egress quite a bit.

[K-T Lim] 14:17:38
the vendors tend to give credits for egress based on the total amount of spending that you're doing on all the other services that you're buying from them.

[K-T Lim] 14:17:47
And so those credit is, can also help minimize the egress costs.

[K-T Lim] 14:17:51
And finally we did look at, but have not yet moved on getting a dedicated interconnect.

[K-T Lim] 14:17:58
So with a dedicated intercourse, connect, you're not using the public Internet or or the the public egresses, and as a result, there can be substantial discounts on the egress, costs because it's a kind of paid for in a lump sum rather than on a per

[K-T Lim] 14:18:14
byte or per gigabyte basis. So the final decision that we made was to have a hybrid model.

[K-T Lim] 14:18:21
So we have most of the storage and the large scale compute.

[K-T Lim] 14:18:25
And I'll explain why. In a second on Prem, at the Us.

[K-T Lim] 14:18:29
Data facility which is located at slack national accelerator lab.

[K-T Lim] 14:18:33
so The users, however, will be supported in the cloud on a cloud.

[K-T Lim] 14:18:37
Data, facility, not that is, actually vendor agnostic, but we're anticipating that it will be at the on Google side platform for various reasons.

[K-T Lim] 14:18:47
And so it looks something like this: We have again the telescope sending data to the Us.

[K-T Lim] 14:18:52
Data facility and the data release processing and prom processing both occur there with the main archive storage.

[K-T Lim] 14:19:00
But in the cloud data facility, we have the ribbon Science platform services.

[K-T Lim] 14:19:05
we have a cash of data that's both, for for relatively small data sets that we can copy in their entirety and for partial storage of other data sets that are than are being used frequently and also per user storage would also be

[K-T Lim] 14:19:28
stored entirely in the cloud. This shows the user batch would be executed at the Us data facility so that it could run against the archive storage.

[K-T Lim] 14:19:37
And I'll talk about that, and where those dividing lines might be in a bit

[K-T Lim] 14:19:42
So. The ribbon Science platform What is it really? Again, It's for us.

[K-T Lim] 14:19:48
It's for our science users who are coming to use dedicated resources that are provided by the project to access our large data sets and use web based applications on them.

[K-T Lim] 14:20:01
So there's a core. Yeah, where it provides access and visualization and sort of structured expeditions through the through the data set with query generation tools as well as lots of visualization, including joint visualization visualization, of images and

[K-T Lim] 14:20:23
catalogs we then have Jupiter notebooks.

[K-T Lim] 14:20:29
it's actually quite common now. Was not that common?

[K-T Lim] 14:20:31
a few years ago, when we were starting out on this vision.

[K-T Lim] 14:20:37
But that's for more ad hoc analysis by users, and then we have web Api's.

[K-T Lim] 14:20:43
These are web services that are, have interfaces that are defined by the international Virtual Observatory Alliance for astronomy that provide access to both images, both raw and processed images as well as catalogs of things, seen on those images and so that

[K-T Lim] 14:21:04
provides. Excuse me, both remote access and and a little bit of processing and that we can do things like.

[K-T Lim] 14:21:11
Cut it out, sections of images or paste together images.

[K-T Lim] 14:21:15
So this is the the user experience. The users will have, And behind those 3 major aspects there are the data releases an alert filtering service user databases user files, all kinds of other infrastructure that's necessary to make that work.

[K-T Lim] 14:21:33
So our uses of cloud services. Obviously the primary one is going to be the revenue and science platform.

[K-T Lim] 14:21:41
the reasons for putting this in the cloud alright include these.

[K-T Lim] 14:21:46
So there's security by putting this in the cloud.

[K-T Lim] 14:21:48
we can use separately manage identities that have nothing to do with the identities at our on-prem facilities at slack.

[K-T Lim] 14:21:59
so all of our users do not need to get slack accounts.

[K-T Lim] 14:22:03
this is very important, because department of energy has a lot of restrictions and and it's not necessarily very rapid at generating accounts at labs.

[K-T Lim] 14:22:14
So, having being able to maintain our own accounts, makes things much more efficient, and allows us to integrate with things like federations that we couldn't otherwise necessarily do.

[K-T Lim] 14:22:26
it also means we have a good story for cyber security at the lab, because we have very relatively limited interfaces with the onframe facilities.

[K-T Lim] 14:22:34
There are dedicated, there are certain services that be queried from the cloud, and those can be listed and tracked, and understood.

[K-T Lim] 14:22:46
A huge benefit is elasticity right? So, especially after we have an annual data release, we're expecting the hordes of astronomers.

[K-T Lim] 14:22:55
Will descend on us and and want to look at what's new in that release.

[K-T Lim] 14:22:59
this might also happen. For example, around key conferences, when people are trying to do work So in the cloud we have essentially infinite elasticity.

[K-T Lim] 14:23:07
We can see up the rubber science platform by deploying more notebook servers, more api servers. And even more portal servers arbitrarily, and so you can hint we're expected to be able to handle those loads relatively easily the back end

[K-T Lim] 14:23:26
services. It could be an issue. But we can. We can also do that in a scalable manner, using object stores, and scalable files, scalable distributed file systems and a scalar distributed database in the back end potential advantage.

[K-T Lim] 14:23:47
we are looking to prove, but haven't quite yet.

[K-T Lim] 14:23:51
is that you could bring your own resources in the cloud.

[K-T Lim] 14:23:56
So if a science user had a grant or some other means of providing cloud resources on the same cloud, the cloud vendor that we're using, they confederate those resources with the ones that are already press events for the ribbon science platform and essentially expand their

[K-T Lim] 14:24:18
capabilities. Okay, kind of compared with trying to actually purchase hardware and slack lab or or send computers or something like that. This is much.

[K-T Lim] 14:24:29
Much much much much easier. And so it gives people the ability to use all the same facilities, software and user interfaces that they're familiar with at a larger scale, just by adding on to what's present And finally the cloud can also provide access to new technologies things like

[K-T Lim] 14:24:52
gpus Tpus or software technologies like sophisticated infrastructure services that are harder to deploy at a lab on premises.

[K-T Lim] 14:25:03
that's and again you don't need to buy them and keep them working.

[K-T Lim] 14:25:10
31. You can rent them when you need them, and then further away.

[K-T Lim] 14:25:15
So for the large scale compute we have executed fairly large production.

[K-T Lim] 14:25:25
This is our data preview 0 point 2, which is only for 5 years and only for a small portion of the sky, not not the full data release.

[K-T Lim] 14:25:35
Production but we did. We're able to actually done on larger numbers of nodes, 4,000 virtual cpus again.

[K-T Lim] 14:25:42
Not that much compared to high energy physics necessarily, but pretty large for what we're doing, and but we're not expecting to execute the main survey data release production.

[K-T Lim] 14:25:51
On this the cost of storing or addressing the large, their process products is too excessive to do that.

[K-T Lim] 14:25:58
we might be able to do user batch in the cloud.

[K-T Lim] 14:26:00
But it. It'll have some of the same drawbacks, and that works expecting user batch jobs to also want to process, large fractions of the the available data.

[K-T Lim] 14:26:11
And so transmitting all of those into the cloud, or and even storing them temporarily, can have some difficulties.

[K-T Lim] 14:26:20
But if we were able to do it, if we can get the caching and the sort of automated transfers working well, then there would be the advantages of having the again the security and technology kind of advantages that we would not have on premises right now

[K-T Lim] 14:26:43
we're going to require the users who wants to execute that large scale?

[K-T Lim] 14:26:49
Batch jobs get slack accounts, and that made may eventually become a problem

[K-T Lim] 14:26:55
we've found the cloud to be extremely useful for development. Testing.

[K-T Lim] 14:26:58
Again the elasticity being able to scale up at at will, and that technology advantages of being able to use new machines, large amounts of flash storage, for example, things, like that that are not easily purchased especially now with supply chain issues in an on-premise model

[K-T Lim] 14:27:21
has been very helpful for development, and we've also been able to do things like rapid prototyping with advanced services such as server lists, all kinds of deployments.

[K-T Lim] 14:27:31
There is a possible future I mentioned. We have a distributed, scalable database that runs on on-premises that will handle and serve a lot of the catalogs that are being generated for the stores and galaxies that we're detecting on these images that

[K-T Lim] 14:27:48
database has been customized for astronomy, and has a lot of advantages.

[K-T Lim] 14:27:53
one is that it does spherical geometry, which it's kind of difficult and a lot of in a lot of databases alright.

[K-T Lim] 14:28:02
It does share. It's what cost shared scans, where multiple queries that are touching the same tables.

[K-T Lim] 14:28:09
a do share ios essentially, and that makes things much more efficient and can provide, I guess, well understood maximum query times, while the minimum query times may increase the maximum query times for certain types of queries can be limited and so

[Enrico Fermi Institute] 14:28:31
Okay.

[K-T Lim] 14:28:32
we can, we can guarantee that your query will finish in a certain amount of time.

[K-T Lim] 14:28:38
we also have special indexes, especially spatial ones, that allow us to do astronomical types of queries much more efficiently so.

[K-T Lim] 14:28:49
A lot of these differentiators are kind of going away with cloud deployments, Spherical geometry is becoming more available through gis kinds of packages.

[K-T Lim] 14:28:59
The shared scan is still a win. But when everything is on Nvme flash the number of iops is so high, that you know you can do individual ios for each query without actually loosing a lot a special indexes that we have are still

[K-T Lim] 14:29:18
a bit of an issue are are still better in house than they are in the cloud, and retrofitting them to the cloud.

[K-T Lim] 14:29:25
Databases is difficult, and finally, storage costs, Course can still be an issue again, because week we can do this cheaper, and in-house rather than using the cloud storage, and then finally users, in the cloud for archival or tape replacement storage

[K-T Lim] 14:29:47
maybe comparable in terms of total cost of ownership.

[K-T Lim] 14:29:51
This is something we're still investigating, especially if you don't retrieve the data.

[K-T Lim] 14:29:56
If you do have to retrieve the data, then there are large egress costs again to get it out of the cloud and engineer on premises, storage and so that, becomes an issue.

[K-T Lim] 14:30:07
But if you're in that kind of a disaster situation, it may not be that bad

[K-T Lim] 14:30:15
one other aspect of the cloud that has been kind of important, I guess, is that is reliability.

[K-T Lim] 14:30:22
so, while I mentioned that in some cases the durability of storage might be overkill and other faces.

[K-T Lim] 14:30:29
Well, we do actually experience higher higher reliability and higher ability to deliver, to our end users by deploying on the cloud than on premises.

[Enrico Fermi Institute] 14:30:37
Okay.

[K-T Lim] 14:30:38
first of all, one of the the may be negatives is that we've seen that kubernetes upgrades will roll through Our clusters semi arbitrarily there are some controls that you can put on them.

[K-T Lim] 14:30:50
But the the vendors kind of want to update it when they want to update it.

[K-T Lim] 14:30:55
So we need to make sure we've designed services to deal with these kinds of rolling outages.

[K-T Lim] 14:31:01
not all of them are, but we can. We will adjust them over time again.

[K-T Lim] 14:31:06
The durability of storage is extremely high, maybe more than necessary service outages are quite rare and usually short compared with some of the outages that we've had on prem and 24 by 7 support for basic infrastructure and even for higher level services is often better than we have

[K-T Lim] 14:31:24
on prayer, where it may just be 8 by 5 essentially So while sometimes the reliability in the cloud is more than you need, and so you're paying for more, than you actually need.

[Enrico Fermi Institute] 14:31:33
And

[K-T Lim] 14:31:41
In other, cases, it's actually it can be an event so we're trying to wrap up here.

[K-T Lim] 14:31:47
conclusion and status and Plans The hybrid model seems to be suitable for our use.

[K-T Lim] 14:31:52
Cases. We are practicing today with an interim data facility on the Google Cloud platform, which hosts simulated data until the telescope is built.

[K-T Lim] 14:32:04
we're we're working with that. So, But to give scientists a chance to work with data that looks like the real thing.

[K-T Lim] 14:32:09
And and using all the tools that they will eventually have.

[K-T Lim] 14:32:13
We're building out our back end on prem infrastructure to practice the integration with the cloud and tune, the various caching parameters.

[K-T Lim] 14:32:23
About What gets sent to the cloud went, and we are obviously continuing to track development and cloud services and pricing.

[K-T Lim] 14:32:32
and I'm happy to answer any questions

[Enrico Fermi Institute] 14:32:41
Okay. So we have a couple of hand-raced Lindsey.

[Enrico Fermi Institute] 14:32:45
Why don't you go first

[Lindsey Gray] 14:32:48
yeah, sure, actually, just a quick operational one about the fact that they're rolling through kubernetes, upgrades kind of at that.

[Lindsey Gray] 14:32:57
Their own whim in particular, since Kubernetes is up updating the spec and the interface that you're talking to how much maintenance burden have you found that to be as the spec makes backwards and compatible changes

[K-T Lim] 14:33:16
the the upgrades of Kubernetes itself have typically not been too much of a problem.

[Enrico Fermi Institute] 14:33:17
See.

[K-T Lim] 14:33:26
we don't run on the the we, well, we run our development clusters on sort of the latest, more bleeding edge versions, and our production clusters on the more stable versions.

[K-T Lim] 14:33:41
So we've typically seen any problems already, either in the development clusters or even at the summit where we're probably a little bit more rapid to update Then then, even on the stable channels in the cloud.

[K-T Lim] 14:33:55
So we're been really prepared for any of these things that happen

[Lindsey Gray] 14:33:58
okay. So the there's nothing really nasty about the cadence of updates or upgrades on the cloud side of things.

[Lindsey Gray] 14:34:05
And you have. You feel like you have control over the situation by and large, alright, cool.

[Lindsey Gray] 14:34:10
Thank you.

[K-T Lim] 14:34:10
Adequately. Yes, you do have to have people that are dedicated to to so keeping this up to date and porting them How You I know that There are some people in science who like the idea of well we're going to install our service on the machine and then sort of wrap the

[K-T Lim] 14:34:26
whole thing in amber and just kind of leave it there and have it run.

[K-T Lim] 14:34:30
And it the model. It cannot be that way. You have to deal with Os upgrade updates and and service updates, and be on top of

[Lindsey Gray] 14:34:33
Right.

[Lindsey Gray] 14:34:40
Okay, Cool: Thank you.

[Enrico Fermi Institute] 14:34:44
Okay, Tony.

[Tony Wong] 14:34:46
yeah, Oh, hi! So I I I got a basically A, you know, a couple of questions in rolled up into one with respect to storage, because you kept mentioning, you know, concerns about to the cost of egress and storage So you know, when I looked at Google and and Amazon, I noticed that they

[Tony Wong] 14:35:05
have many, many different levels of storage reliability, you know responsiveness, you know, backups, you know, 24 by 7 availability, etc., and so forth.

[Enrico Fermi Institute] 14:35:16
Okay.

[Tony Wong] 14:35:16
Data, 0 do a study to determine which point in terms of the levels of of storage services that it needs where Cloud would be more, you know, will look more favorable, cost-wise compared to on-premise And you know and then if you if you look at the level

[Enrico Fermi Institute] 14:35:22
Yeah.

[Tony Wong] 14:35:40
of service, then the other opposite side of the question is also: Did we all do a study to optimize costs with regards to social media most of the storage is going to be on prem but some of the storage is going to be on the cloud.

[Tony Wong] 14:35:56
So at what point is there? Is there a tipping point where a big pace to have a cloud services?

[Tony Wong] 14:36:03
You know storage, and how much does it Really, you know, Is it 10% of your data?

[Tony Wong] 14:36:08
20% of your data. At what point does it really pay to go on the cloud

[K-T Lim] 14:36:14
Okay, So yeah. A couple of things. First of all, just something that that our communications people make me say we.

[K-T Lim] 14:36:23
We try not to use abbreviations for the name of the observatory.

[K-T Lim] 14:36:26
We prefer to just have a call. The Vera Rubin Observatory rather than vro Second of all, Yeah, we did extensive modeling of how frequently our data is going to be accessed, because the there are obviously a different levels.

[K-T Lim] 14:36:49
Of oh, of access and and different prices for those are all ranging all the way from sort of standard object store in which there's actually even above that, there's there's positive file systems which tend to be quite expensive object store, which is frequently accessed, where you don't get charged

[K-T Lim] 14:37:10
so much per operation, Are you gonna charge? A tiny bit per operation, but not providing.

[K-T Lim] 14:37:16
and then, and and then there's a large rental cost for bytes per month, all the way down to the archival storage.

[K-T Lim] 14:37:25
cold storage, where you get charged a lot for retrieval, but considerably less for the actual storage per month, and so we looked at for each of our data, sets how many accesses would, we be expecting to have and as a result.

[K-T Lim] 14:37:44
What category of stores could we use for them? Unfortunately, a lot of our data sets.

[K-T Lim] 14:37:51
we don't really know. We know that what kinds of kinds of access patterns we have for our own processing, for the data releases or for yeah, they're prom processing or even somewhere for our device, But the science users might do anything And they can look at any data.

[K-T Lim] 14:38:11
anytime, and we it's hard to say which pieces of data will be accessed more than others.

[K-T Lim] 14:38:18
So a lot of that ends up kind of being attributed to the the most expensive object store.

[K-T Lim] 14:38:27
As a result, and again, as I mentioned some of the the total cost of ownership concerns or total cost of ownership, that the cloud vendors are charging may not be fully charged to the projects when it's on premises and as a result our

[K-T Lim] 14:38:45
on-premise costs considerably lower for those kinds of things we did, I mean so we we have those numbers in terms of comparison of on-prem, and in the cloud they will differ, depending on your institution.

[K-T Lim] 14:38:59
And the cost of hardware. I think we've been getting good deals from the vendors when they can actually deliver.

[K-T Lim] 14:39:09
so the the we we have, I mean. We know at what dollar per month cost.

[K-T Lim] 14:39:19
It would make sense to switch and we're not anywhere in the ballpark right now.

[K-T Lim] 14:39:24
but, as I said, for the archives, storage, when you compare it with tapes and tape robots and tape drives, and things like that, it may become that may have crossed over in terms of being cheaper in the long run to stored in the cloud than it

[K-T Lim] 14:39:46
is to store it on print, but some of that depends on what you assume in terms of how often you're going to retrieve it.

[K-T Lim] 14:39:52
Originally we were going to write all the raw data to tape immediately, and then actually reread it every year to do the reprocessing which would both guarantee that it was actually readable as well as as make the the costs of of retrieval

[K-T Lim] 14:40:14
make, Mcdonald's of storage of that raw data lower.

[K-T Lim] 14:40:17
But it turns out that the raw data now is, as time is going on, and it's not too bad to actually store it spinning.

[K-T Lim] 14:40:26
There are other reasons to store it spinning, and so we will not be doing that.

[K-T Lim] 14:40:30
So the the actual number of retrieval from tape is hoped to be near 0,

[K-T Lim] 14:40:41
Does that answer Enough of your question? I'm sorry

[Tony Wong] 14:40:42
Yes, it does. I think it is very informative. Thank you.

[Enrico Fermi Institute] 14:40:47
Right, Dirk. You want to jump in

[Dirk Hufnagel] 14:40:50
yeah, I had to relate related questions, and they're both on cloud cost What I was curious about is how you do budget for like cloud what you spend on cloud throughout the year.

[Dirk Hufnagel] 14:41:06
If you, if you tell yourself like for this year, we want to spend.

[Dirk Hufnagel] 14:41:09
We have this budget for cloud or if you're a bit more flexible where you allocate funds for on-premise or cloud throughout the year, and then, related, independently, of how you set that target we actually control your cost especially in light of still keeping the ability to support these elastic

[Dirk Hufnagel] 14:41:26
use cases, because I mean at some point, if you account, if you available money goes to 0, you can't really be fully elastic anymore.

[Dirk Hufnagel] 14:41:32
So

[K-T Lim] 14:41:33
Yes, yeah, the way that our budgeting works. We have separated out the cloud and the on-prem costs.

[K-T Lim] 14:41:44
so the they are not one. I mean. They're they are stemming from an original budget.

[K-T Lim] 14:41:51
But that budget has been divided up relatively early, and we've actually one of the ways of getting a discount from the Cloud provider.

[K-T Lim] 14:42:03
In this case, Google Cloud Platform was to provide a commitment that we would spend a certain amount.

[K-T Lim] 14:42:08
So we've already kind of pre budgeted.

[K-T Lim] 14:42:12
That amounts for a number of years, in order to to be able to to get substantial discounts, We don't expect them to be problem.

[K-T Lim] 14:42:24
One of the nice things is that it is just fungible dollar amount, and we can spend that on any services.

[K-T Lim] 14:42:32
so if we decide we we don't want what we originally wanted.

[K-T Lim] 14:42:36
We want to change it to something else. That's no problem.

[K-T Lim] 14:42:40
it's all the same dollars the in terms of the elasticity and budgeting.

[K-T Lim] 14:42:48
It is true that we do have to put in to place quotas and throttles, so that our users can't just all chew up the entire budget in the first, week and so they're they're too, need to be controls like that that are imposed in

[K-T Lim] 14:43:05
the services.

[Enrico Fermi Institute] 14:43:10
Okay.

[K-T Lim] 14:43:11
But that would apply on Prem as well. It's also not infinitely adjustable

[Dirk Hufnagel] 14:43:13
but on premise at least, it's limited.

[Dirk Hufnagel] 14:43:17
You can't run more than what the deployed hardware allows you. Do.

[Dirk Hufnagel] 14:43:22
You have enough data to to do to say how well your controls work

[K-T Lim] 14:43:29
so far again, because most of the access has been by friendly users and staff.

[K-T Lim] 14:43:35
we have not had any problems. Our Our budget is is large enough to cover all the uses that people have been have been doing of it.

[Dirk Hufnagel] 14:43:38
Okay.

[K-T Lim] 14:43:46
but the we are tracking on a weekly basis.

[K-T Lim] 14:43:51
How much we have been spending, and if anything looks like it is getting out of sync, then we can investigate and try to try to control that in terms of actually implementing the the throttles in the services the user facing services.

[K-T Lim] 14:44:10
we only have a couple of them in place right now and again.

[K-T Lim] 14:44:15
They have not been triggered, so it's hard to say

[Enrico Fermi Institute] 14:44:18
Yeah.

[Dirk Hufnagel] 14:44:18
Okay, thanks.

[Enrico Fermi Institute] 14:44:22
Okay, Done

[Douglas Benjamin] 14:44:24
Yeah, thanks for the nice talk and the the details. Alright on slide 10.

[Douglas Benjamin] 14:44:31
You sort of list the reasons why you know the use of the Cloud services, and on also the advantage, the advantages, it said security.

[Douglas Benjamin] 14:44:41
But that really meant authorization, right in the sense of you have 2 different user communities to get access to on.

[K-T Lim] 14:44:41
Yeah.

[Douglas Benjamin] 14:44:52
Do we site Resources requires a different level of authorization even within the like host lab itself, and necessarily what is currently available will fit it.

[Douglas Benjamin] 14:45:06
Identity and the moves with federated identity across the Us.

[Douglas Benjamin] 14:45:13
Dewey. Complex. Change that calculation at all for you

[K-T Lim] 14:45:19
So there are. There are 2 things there

[Douglas Benjamin] 14:45:23
Because I can't imagine the cloud is more secure.

[Douglas Benjamin] 14:45:24
Given all the acts of what have happened with cloud vendors and the services on them

[Enrico Fermi Institute] 14:45:29
Okay.

[Douglas Benjamin] 14:45:32
Then you imply with your statement

[K-T Lim] 14:45:35
yeah, yes, to no, I mean so off, first of all.

[K-T Lim] 14:45:41
Okay, let me say that at least the the cloud vendors are, I mean.

[K-T Lim] 14:45:56
Well, I think they have the ability to be highly secure, whether they're delivered.

[K-T Lim] 14:46:02
commercial level services are highly secure or not, or another question, but I would, I mean, even though our lab has a great cyber security team, I would still say that the cloud vendors are devoting more resources to, security, than we are.

[K-T Lim] 14:46:18
and so the it would seem that it would be at least comparable.

[K-T Lim] 14:46:23
that's sort of on a overall holistic level.

[K-T Lim] 14:46:26
in terms of the authorizations. Yes, when I mean one of the things that we're that we're doing is that the the Ruben data data sets are open to all astronomers within the us and Chile as well as selected names Astronomers from international

[K-T Lim] 14:46:52
partner, institutions. And so, even if we had an authentication of federation for the Us.

[K-T Lim] 14:47:04
Which we are point we are using in common and planning to use equivalents in Chile and in Europe.

[K-T Lim] 14:47:12
the the authorization part of it is still sort of unique to the project, And it also may include international partners who have some difficulties with the Us.

[K-T Lim] 14:47:32
Government or with us, labs

[Douglas Benjamin] 14:47:33
Yeah. I Msn: from the the cut the country. She's non grata

[K-T Lim] 14:47:36
Yeah, and so those I mean, sometimes not not even countries per se.

[K-T Lim] 14:47:45
But citizens of those countries who are working at other institutions, so that that can be better.

[Douglas Benjamin] 14:47:49
Right.

[K-T Lim] 14:47:54
And then the final part. The final thing is that in that security bullet Well, I guess I can pull it up again.

[K-T Lim] 14:48:00
But

[K-T Lim] 14:48:04
Where to go.

[K-T Lim] 14:48:11
Here we go. So I mentioned the limited interfaces, so I think that even if we get, and we kind of expect to have our user facing services packed eventually, just because a lot of it is even even if it's built on vendor infrastructure it's still our code and

[K-T Lim] 14:48:33
we could still have security problems. So even if that's hacked the interfaces with the on-prem facilities are limited in terms of what they can do, what they can extract, and the what they're extracting is public data or essentially public data.

[K-T Lim] 14:48:51
Anyway, it's screwed data rights holders. And so the the exposure to the lab is considerably smaller than if people are actually logging into machines at the lab and have access to the networks in a more general way etc. does that make sense

[Douglas Benjamin] 14:49:07
Hey! How'll make sense? I I realize that one of the difference between Reuben and the root, the science community and Ruben is a little bit different than me mit experiments, because it's one overall arching organization more people have certain accounts than don't on the Lhc

[Douglas Benjamin] 14:49:29
experiments. For example, the last quest, quick question I had on this.

[K-T Lim] 14:49:31
Yeah.

[Douglas Benjamin] 14:49:35
Was you also implied that storage costs are one of those things that you're worried about eating budget.

[Douglas Benjamin] 14:49:40
So that's why you limited You essentially. Have a lot of the story.

[Douglas Benjamin] 14:49:47
Most of this

[Douglas Benjamin] 14:49:51
The size of the storage, and the cloud seem to be limited relative to what you have on Prem.

[K-T Lim] 14:49:58
Right.

[Douglas Benjamin] 14:50:00
And is that really? For did I get the sense that it is to sort of cost containment for storage

[Enrico Fermi Institute] 14:50:01
Okay.

[K-T Lim] 14:50:07
Yes, I mean, basically we. We have hundreds to even thousands of petabytes of data that we were expecting to store by the end of the survey, and that cost of storing all of that for months and months and months in the cloud would be excessive compared to our

[K-T Lim] 14:50:30
budget. So we are storing on the order of, you know, single digit percents of that in the cloud with the rest of it on prem, in order to fit within the available budget

[Douglas Benjamin] 14:50:50
Thank you very much

[Fernando Harald Barreiro Megino] 14:50:56
Yeah. Yeah. Simple question, Where did you choose? Google over other cloud?

[Fernando Harald Barreiro Megino] 14:51:04
Then there was just so the best deal, or anything else

[K-T Lim] 14:51:10
0 are a number of reasons, I forget whether we actually have.

[K-T Lim] 14:51:15
I don't think we have a a public document that actually states this.

[K-T Lim] 14:51:23
yeah. But I can see, I think, that there were a number of factors Pricing is only one of them, and as I mentioned the the ability to work with Google, and there are flexibility and ability to to work well, with our engineering teams, was

[Enrico Fermi Institute] 14:51:31
Hmm.

[K-T Lim] 14:51:58
actually a fairly major factor as well. The quality of their services.

[K-T Lim] 14:52:05
in some cases, I mean a lot, a lot of the things that we use.

[K-T Lim] 14:52:08
We try to be vendor, agnostic, and so the interfaces are all the same.

[K-T Lim] 14:52:15
But the performance underlying those interfaces can change from from vendor to vendor. And I guess it's something that I've actually kind of complained to Google about sometimes that that they often seem to offer a much better product.

[K-T Lim] 14:52:31
Than we need, but at a somewhat more expensive price. And so sometimes, though they're taking advantage of those, those improved performance formus capabilities can actually be useful.

[Enrico Fermi Institute] 14:52:38
Thank you.

[K-T Lim] 14:52:44
so for things like, yeah, I mean, just as an example of the the object store retrieval, even for the the sort of the coldest archive level of storage the latencies to actually retrieve the data can be almost the same as for normal object store on Google cloud whereas

[K-T Lim] 14:53:03
Amazon's glacier. There are much greater latencies, most of which we wouldn't mind, but it can be nice to have that capability to just grab something if you you need it every once in a while.

[K-T Lim] 14:53:15
So they're they're very a variety of reasons like that.

[K-T Lim] 14:53:20
I don't think I can go and say everything that in terms of the vendor comparisons at this time, just because I don't think it's all public

[Enrico Fermi Institute] 14:53:29
So. So it was very Ruben using the subscription model with Google

[K-T Lim] 14:53:37
subscription model in terms of what, where we have.

[K-T Lim] 14:53:42
a sort of it's not a prepaid kind of plan, but it?

[K-T Lim] 14:53:49
Is, there is a commitment to spend a certain amount over a certain amount of time, and and we get a number of discounts as a result of that

[Enrico Fermi Institute] 14:53:59
So so have you done that. Has that negotiation happened like one?

[Enrico Fermi Institute] 14:54:05
Or as it is it then, you know, have you gone back, and you know you committed to doing something for some amount of time, and then you went back and negotiated a new deal of you have you done anything like that, and if you have has it been have there been significant like deltas in the

[Enrico Fermi Institute] 14:54:21
pricing, or anything like that

[K-T Lim] 14:54:23
Right. So we have done that once for the interim data facility, which was for a period of 3 years, and that time period is kind of coming up soon.

[K-T Lim] 14:54:35
we are in the middle of working on purchasing a of doing a similar kind of purchase plan for the beginning of operations of the survey, which will, start in 24, and that negotiation I mean, it's it's actually a question there's there's some question

[K-T Lim] 14:54:58
about whether we can do it as a source agreement that's being thought about as well as what the pricing will be.

[K-T Lim] 14:55:07
We are in our discussion. So far, we expect that the person will be, if anything lower, then we're we're getting it before for a number of things.

[K-T Lim] 14:55:17
we are cognizant and worried about potential for vendor lock in.

[K-T Lim] 14:55:23
We've had cases with unnamed databases where licenses have radically escalated in price, and with other systems, where similarly, there can be unexpected costing, increases.

[K-T Lim] 14:55:38
we don't necessarily see that happening here for a number of reasons.

[K-T Lim] 14:55:41
Again, We're trying to use commodity interfaces and services that we could get from another vendor if necessary, or even deploy ourselves on premise We absolutely had to and and they're also ones that are very commonly used commercially, they're not sort of unique

[K-T Lim] 14:56:02
to science in any way, and so there are additional pressures to keep the cost down that way.

[Enrico Fermi Institute] 14:56:07
Okay, Thank you.

[Enrico Fermi Institute] 14:56:08
Okay, thank you. Doug. Did you have another comment?

[Douglas Benjamin] 14:56:10
yeah, it's 2 questions. They're slightly different. One is sort of high level on the science platform in the cloud.

[Douglas Benjamin] 14:56:19
How long do you envision, negotiating your contracts with the vendor?

[Douglas Benjamin] 14:56:25
For is it like a 5 year period, and then you renegotiate after 5?

[Douglas Benjamin] 14:56:29
Or is it for the sort of lifetime of the data taking?

[Douglas Benjamin] 14:56:34
Because I know the data has to be around for much longer than the actual bucks bear.

[Enrico Fermi Institute] 14:56:36
Hello!

[Douglas Benjamin] 14:56:40
The The telescope, runtime

[K-T Lim] 14:56:41
Right? So there are 2 things there. One is that the survey is scheduled to run for 10 years We'll be taking data for 10 years, and we are on the project budget.

[K-T Lim] 14:56:54
We are committed to providing data products and services to the science users.

[K-T Lim] 14:57:00
For 11 years after the shortest survey operations.

[K-T Lim] 14:57:04
so 10, plus actually sorry. No, I see correct that.

[K-T Lim] 14:57:10
I think it's been updated to 12, years. So it's all it's 10 years of data taking one year of processing the 10 years.

[K-T Lim] 14:57:16
And then one year delivery of that data release. So that's 12 years.

[K-T Lim] 14:57:21
We will. There There is a a plan for for archiving this data.

[K-T Lim] 14:57:28
And and and preserving it for indefinite periods.

[K-T Lim] 14:57:35
After the end of the project. But that plan is not funded by the project.

[K-T Lim] 14:57:39
It has to be funded by the Nsf. Separately, and so we don't know exactly how that's gonna happen. Of course it is.

[K-T Lim] 14:57:44
you know 14 years in the future. Now the negotiation for the purchase of the cloud, so services will be for a particular term, and it's likely to be something on the order of 3 to 5 years, and then adjusted thereafter some of that is on the vendor

[K-T Lim] 14:58:04
side, actually that they are not sure how much they want to commit to.

[K-T Lim] 14:58:09
For for a long period of time. But okay, I think it also benefits us in that we have the opportunity to change and necessary

[Douglas Benjamin] 14:58:19
Okay? Next. And then my orthogonal question, was you mentioned?

[Douglas Benjamin] 14:58:24
Bring your own for the the science platform. Does that mean? Bring your own?

[Enrico Fermi Institute] 14:58:28
Yeah.

[K-T Lim] 14:58:28
Yeah.

[Douglas Benjamin] 14:58:32
Since you said, Google and your good, the science platform will be in Google.

[Douglas Benjamin] 14:58:35
Does that mean? Bring your own to Google? Or does that mean I'm a researcher at a university, and my university is made available.

[Douglas Benjamin] 14:58:45
Some compute resources for me to use, and I want to stitch my university stuff

[K-T Lim] 14:58:52
Yeah, So stitching the university stuff with the cloud platform will be a little bit more difficult.

[K-T Lim] 14:59:02
We will have interfaces to be able to do sort of bulk.

[K-T Lim] 14:59:05
Downloads of chunks of data that you might want to process locally.

[K-T Lim] 14:59:10
but that's not It's expected to be used mostly by certain collaborations that will be working to get and using large scale facilities like nurse.

[K-T Lim] 14:59:25
the what I was referring to there specifically was for a smaller scale, You know, collaborations or individual investigators who might have, you know.

[K-T Lim] 14:59:40
Do you do what each you know? I don't wanna spend $10,000 on a Google account and purchase compute, using that, and then federating that with the science platform and being able to expand their capabilities using all the same, tools that they're already using but just

[K-T Lim] 14:59:59
now having increased resources to be able to work with it just by having a separate account

[Douglas Benjamin] 15:00:05
okay, yeah, And so you for the the out of the scale, the out out of perfect Google premises stuff you're really assuming a certain size before it becomes tenable and feasible.

[Douglas Benjamin] 15:00:20
What I mean is there's a there's a certain number of scientists buying bound to work to get collaborate together to Do a you know, provide a certain amount of resources?

[Douglas Benjamin] 15:00:30
To do some work? Then it's worth your time to bring stuff in.

[Douglas Benjamin] 15:00:34
If someone shows up with, you know, 5 computers in their data center, you still gonna do that.

[Douglas Benjamin] 15:00:42
You know, there must be be a threshold

[K-T Lim] 15:00:42
Yeah, so yeah, if if they again, so, if they have 5, computers, and they wanted to download data that fits on those 5 computers.

[K-T Lim] 15:00:52
And then do whatever they want with it. That's fine.

[K-T Lim] 15:00:55
and there's no problem with that. If there. If they have, you know 10,000 cores, and you know they want to download a 100 petabytes worth of data. Then we need to chat with them about how we do that.

[Douglas Benjamin] 15:01:08
Okay, thanks.

[K-T Lim] 15:01:13
But I mean, again, the the the advantage here is is really about the flexibility of using any resources that are available in the cloud to work on the same data, because all the data is already there

[Douglas Benjamin] 15:01:27
but it's in the cl in the same cloud. Provider: Okay, thanks.

[K-T Lim] 15:01:30
In in the same cloud. Yes.

[Enrico Fermi Institute] 15:01:33
Okay.

[K-T Lim] 15:01:34
Yeah, I was looking at

[K-T Lim] 15:01:40
it's very interesting, but I think zoom with Bill is only usable to bring data into aws as far as I could tell, exporting.

[K-T Lim] 15:01:54
It is not so easy

[Lindsey Gray] 15:01:57
that Dna of a day. It's a it's a truck You should be able to go.

[Lindsey Gray] 15:02:01
Both directions

[K-T Lim] 15:02:02
Yeah, they they have an E ink label on it, though they're automatically changes to point to the local Amazon data Center

[Enrico Fermi Institute] 15:02:13
Okay, other questions or comments for Kt

[Enrico Fermi Institute] 15:02:22
Okay, if not, Katie. Thank you very much. This was really informative, and I think everybody learned a lot from this

[K-T Lim] 15:02:27
Thank you. I think there will also be talks, maybe on very similar.

[K-T Lim] 15:02:32
So topics at software and computing atlas workshop, I think there's in October and they're even talking about.

[K-T Lim] 15:02:44
Maybe one of us going to Cern in the spring for similar kinds of conversations.

[K-T Lim] 15:02:49
So perhaps we can also talk then, anyway. Thank you very much for inviting me

[Enrico Fermi Institute] 15:02:52
Great. Yeah, Thank you.