Matthew Bellis: Okay we're recording.
Edgar Fernando Carrera Jarrin: All right, so please raise your hand if you're still working on the exercises I know many of you are because of the all of the issues with the.
Edgar Fernando Carrera Jarrin: With the clusters and that's fine, we can we can take the time of the session I mean we it was intentional that we booked this time for feedback and.
Edgar Fernando Carrera Jarrin: Discussions so we have an extra time.
Okay.
Edgar Fernando Carrera Jarrin: The rest of you are working right, the ones who are problems or are still in the middle.
Edgar Fernando Carrera Jarrin: yeah we can we can listen to you, we can get some feedback, how did you find the workshop, where are the.
Edgar Fernando Carrera Jarrin: obstacles that you found was a useful.
Dillon Fitzgerald: I have a quick question.
Dillon Fitzgerald: Yes, yes.
Dillon Fitzgerald: One spirits are doing is it scale for yourself and.
Dillon Fitzgerald: It would be on the user to pay for the cloud computing.
Dillon Fitzgerald: that's correct right.
Edgar Fernando Carrera Jarrin: that's that's right as as we mentioned yesterday that there is this free resources for $300 that Google offers you.
Edgar Fernando Carrera Jarrin: Apparently view use this this preemptive ball machines, you can make that.
Edgar Fernando Carrera Jarrin: amount last for for a very long time, of course, they just give you three months, and so there's a is a drawback there, but even if you put a little amount, you can you can work and test this for quite some time.
Edgar Fernando Carrera Jarrin: anchor should I just go through.
Matthew Bellis: Those those few slides that I put together just about the discuss cost for a second.
Edgar Fernando Carrera Jarrin: Just yes.
Edgar Fernando Carrera Jarrin: Please, please do please do.
Matthew Bellis: But the one i'm just gonna say.
yeah.
Edgar Fernando Carrera Jarrin: We forgot about that mad sorry.
Matthew Bellis: No, no, no, because I just today, I was I wasn't listening.
Matthew Bellis: No, no it's all good it's all good dylan i'm just going to flash up a few slides that I will also share with the group it's really just kind of touching on this idea of cost and it's not going to explain everything, but there are some links embedded in here that you can reference.
Matthew Bellis: So it's really just a you know there's.
Matthew Bellis: A really, really I back of the envelope calculation is probably too complimentary to these slides but you're absolutely right at some point, if you want to scale this up the cost falls upon you.
Matthew Bellis: And we've gone, you know we've used a lot of these terms kind of interchangeably there's Google cloud platform, and this has a whole suite of cloud computing services.
Matthew Bellis: And this, you know makes use of the same infrastructure that Google uses and then you know separately but also integrated into that is Google compute engine and the Google compute engine is where you are, you know now using all of these clusters and Cooper knees and this infrastructure.
Matthew Bellis: To run Google cloud could also be used just to store data right, you can upload you know, a whole bunch of data files to the Google buckets and never.
Matthew Bellis: interact with compute engine.
Matthew Bellis: And you could use compute engine if you wanted to without doing any sort of real storage, if you just wanted to calculate pi out to a trillion digits or something like that you know just arbitrarily where you're not storing data.
Matthew Bellis: And I mentioned that to start off because there are different costs associated with what you're doing i'm going to share.
Matthew Bellis: Again i'll share these slides i'll post them to this lesson, but there are links in here about both cloud storage costs for your data but also compute engine costs.
Matthew Bellis: And so, when you go to try to figure out how much this might cost you, you have to worry about you know just the basic storage and there's pricing examples here.
Matthew Bellis: And that storage really might be, you know your output histogram files or your output route files your output enter polls your output.
Matthew Bellis: csv files, if you want to write everything to text and then there's the engine Google engine costs for running your code and that basically goes by the time that you're running.
Matthew Bellis: And then, if you're doing something extra there are gpus as well as other accelerators that you can use.
Matthew Bellis: So there's quite a few little flags and switches do you want something with eight nodes you want something with 16 do you want something with four.
Matthew Bellis: And then there's also costs associated with your data if you're moving it in or out of storage so moving within Google cloud generally is pretty cheap, although it can be expensive if you're moving let's say from you know, North America to Europe.
Matthew Bellis: And then there are also costs associated, sometimes with moving off of Google cloud.
Matthew Bellis: If you want to download everything, and so this is far, far too simplified an estimate, but I provided a link to a calculator page.
Matthew Bellis: So if you were going to do one month of analysis and I think what i've done here is I overestimated the costs but suppose you were just running one single for node instance for 24 hours a day, seven days a week.
Matthew Bellis: Your cost would be $195 However, this is not preempt double and so, if you didn't mind using a preemptive node where maybe your analysis flow stops for a day or two, and you have to restart it.
Matthew Bellis: The cost can be considerably less I think it can be even like a quarter or fifth of what this cost is.
Matthew Bellis: You don't have to you know pay for the Stern certain storage of data and if everything is accessed through X Ray D.
Matthew Bellis: you're not paying for the storage of the you know petabyte of cms data, but maybe in the end you wind up producing four terabytes of data or maybe you're doing some intermediate processing and that would be at least in US dollars, you know, on the order of $360 per month.
Matthew Bellis: or sorry sorry sorry, here we go $163 per month.
Matthew Bellis: So, with the the access that you do now you're using extra D, so you know your your program goes out and reaches out to CERN storage, but in the interim, you may be producing some intervals.
Matthew Bellis: And so the the cost can range, you know anywhere from like you know $20 which is easily within this $300 that you're mentioned.
Matthew Bellis: To you know if you are somehow processing a huge amount of data, you have very loose cuts your just producing massive vegetables that you're going to like you know slim down you've got a 10 terabyte server you know at your home institution, it may be more.
Matthew Bellis: So, like I said I will share these slides.
Matthew Bellis: With everybody, but that's just a real real general general back of the envelope calculation Just to give you an order of magnitude of what some of this might cost.
Edgar Fernando Carrera Jarrin: Thanks matt as much as we can.
Dillon Fitzgerald: start going out, although I guess, I was just wondering about certain cases where you're looking at really.
Dillon Fitzgerald: Rare processes, where you can't even really get a sense of with your analysis come to properly tuned on a small subset of events.
Dillon Fitzgerald: Anyway, yeah.
Matthew Bellis: yeah no you're absolutely right, you might be looking for something like I want to you know, I have some rare process, this is going to produce 25 checks let's say and that's so rare, as you say you know you might you might not be able to find that in a very small sample.
Matthew Bellis: yeah and run over everything, and so you might not produce much output, but you might have to run for a long period of time to run over all of the Monte Carlo and all of the All of the data, the collision data.
Dillon Fitzgerald: yeah that's a good.
Dillon Fitzgerald: there's a balance between how much you're putting in.
Dillon Fitzgerald: Okay, thank you.
Edgar Fernando Carrera Jarrin: All right.
Edgar Fernando Carrera Jarrin: anyone else, want to comment complain.
about my desk job is running for the last 48 minutes.
Is it something with my nipple connection, because the browser the instruction in the browser indicates that it will be done just in like 26 seconds for.
Edgar Fernando Carrera Jarrin: That devious page, we are, we need to put the.
Put the IP address in the book in the browser.
It is still it, it says that the connection has time to change that it is something that is due to the slow network connection.
Edgar Fernando Carrera Jarrin: Probably is i'm not quite sure I don't know if someone has experience, are you getting a lot of this because you know when when you.
Edgar Fernando Carrera Jarrin: don't think that has to do with it, do you get a lot of disconnections from the terminal or.
No, no, no.
Edgar Fernando Carrera Jarrin: Okay.
Edgar Fernando Carrera Jarrin: i'm not exactly sure.
Edgar Fernando Carrera Jarrin: Maybe the.
Edgar Fernando Carrera Jarrin: Any planets have some comments.
Clemens Lange: No, no, we have to look at this in detail.
Clemens Lange: Yes, look at the cluster I cannot say no to like just like well.
Matthew Bellis: Now it's probably also a good time to mention that, in addition to any discussions that we have now or before the end of the workshop.
Matthew Bellis: Hopefully, by tomorrow fingers crossed, we will be sending out a survey to all participants, where you can collect your thoughts.
Matthew Bellis: provide feedback both positive and constructive criticism for anything that we can do to improve this workshop going forward.
Matthew Bellis: One of the questions is about people's interest in future workshops, whether they cover the same material or whether we expand on the material that's presented.
Matthew Bellis: So even if you nothing POPs into your head, right now, we will be more family collecting feedback from anybody who would like to provide it.
soon.
Dillon Fitzgerald: are part of the lessons that were sort of step through in this workshop publicly available on the open data portal or is that for different set of lessons.
Matthew Bellis: To everything the entire workshop all of the the material will remain live all of the So if you go to the from the indigo page, there was a link on the main indigo page, which was tutorial site.
Matthew Bellis: And that tutorial site has all of these exercises I imagine you know we collected some notes, there was a couple of maybe typos and stuff that we will try to fix, but there are no plans for for that website to go dark anytime.
Dillon Fitzgerald: Right okay so.
Edgar Fernando Carrera Jarrin: As long.
Dillon Fitzgerald: As it peep for people that have never attended the workshop but are just filtered through the open data for likely be following the same set of tutorials or or now.
Dillon Fitzgerald: Are these tutorials on open data for dollars.
Matthew Bellis: So anchor you can.
Edgar Fernando Carrera Jarrin: No, no, they are not linked directly from the open data portal so here's here's The thing that the open data portal is not really meant to be the main.
Edgar Fernando Carrera Jarrin: site for instructions it's just a place where we store the data that's what it's good for, so we are building though these cms open data guide.
Edgar Fernando Carrera Jarrin: Which is another website, which will serve what one says it's in the best a better shape than it is right now, because it's quite incomplete yet.
Edgar Fernando Carrera Jarrin: But this, this is what we will be advertising as the main hub for all the information we have, and this is lights probably this all this material will be linked, certainly from that page so stay tuned I mean we will make sure that it is possible.
Edgar Fernando Carrera Jarrin: But once it's ready to to to go, I mean you can access it already and help build it if you wish, because an open project only get help.
Edgar Fernando Carrera Jarrin: But yeah that that's the idea, so it will be, it will be there.
Julie Hogan: And in fact most of us, you know as we've been writing these lessons when you eventually see the open data guide a lot of the pages, particularly for physics objects will look almost like a mirror of the tutorial.
Julie Hogan: In many ways.
Dillon Fitzgerald: Okay, great thanks.
Edgar Fernando Carrera Jarrin: All right.
Edgar Fernando Carrera Jarrin: anyone else any comments.
Edgar Fernando Carrera Jarrin: So, how did you find the workshops to sit I.
Edgar Fernando Carrera Jarrin: know you were very excited about this.
Edgar Fernando Carrera Jarrin: material.
Edgar Fernando Carrera Jarrin: did your friend is useful boring.
yeah.
Cecilia Uribe Estrada: I very.
Cecilia Uribe Estrada: Useful actually I.
Cecilia Uribe Estrada: Think is really i'm very impressed with with all.
Cecilia Uribe Estrada: The people that put this workshop together and give the teaching, because I think is very good and Lisa is in my point of view I I think is.
Cecilia Uribe Estrada: kind of brought you know because I think you try to cover people, that is just starting, you know to learn these tools and but it's very interesting because even people.
Cecilia Uribe Estrada: Like me, that have some experience already I mean I found it really interesting because, obviously, we can learn about other things that are going on, and we have been not using, for example.
Cecilia Uribe Estrada: I knew about the cloud, but I was in I never run a project there so it's quite interesting to know how everything works there and.
Cecilia Uribe Estrada: The prices, for example, and everything, so I think he says, it was very, very useful and I really want to congratulate everybody, and so I think that the students were very active as well participating, so I think it's a success, so thank you.
Edgar Fernando Carrera Jarrin: Thank you sincerely.
Edgar Fernando Carrera Jarrin: yeah and, by the way, i'd like to thank everyone who participated in preparing all this material, it takes quite some effort, so thank you guys and not only.
Edgar Fernando Carrera Jarrin: You know the senior people like Julie my mad government's caddy but also the collaborators get some Stevens also testing things working through the examples and breathing exercises so thanks a lot now, so that people who.
Edgar Fernando Carrera Jarrin: helped as well with some tricks and other type of knowledge.
Edgar Fernando Carrera Jarrin: Thank you very much for attending everyone.
Edgar Fernando Carrera Jarrin: And, as I said, some someone's got to say it, thank you Edgar for all your work behind the scenes, I think we've.
Matthew Bellis: All seen how much work and you've been putting in and how much sleep you've been sacrificing to get stuff going so thank you from all of us as well for really being the point of contact for this effort for much of it.
Matthew Bellis: Oh.
Edgar Fernando Carrera Jarrin: My pleasure my pleasure.
Edgar Fernando Carrera Jarrin: anyway.
Edgar Fernando Carrera Jarrin: yeah so thank you very much, I think.
Edgar Fernando Carrera Jarrin: So clemens I think we will leave running this is this cluster for for the rest of the day, how does it work do they get killed already because they are preventable can people still.
Edgar Fernando Carrera Jarrin: You know, make some more tests for at least a few more hours, how does, how does this work.
Clemens Lange: yeah we leave things running for for a while, I mean until sometime tomorrow afternoon, so I mean not too late afternoon but sometime the afternoon and.
Clemens Lange: I mean, since these are preventable nodes it can happen that your cluster.
Clemens Lange: mean that individual nodes of the cluster basically get removed, but they should be auto created that's the beauty of coordinators that.
Clemens Lange: Sam so it's nothing for you to you know worry about how to take care of and might be that at some point, you see, in the cluster details that you know there's a note being recreated but that's basically.
Clemens Lange: Some point tomorrow.
Clemens Lange: I mean, will do you need the classes and we will probably kick you out of the tcp and project.
Edgar Fernando Carrera Jarrin: Okay.
Edgar Fernando Carrera Jarrin: yeah that sounds that sounds good.
Edgar Fernando Carrera Jarrin: So yeah you have a few hours are still have you you feel like you're testing a few more things learning and.
Edgar Fernando Carrera Jarrin: that's that's fine until tomorrow there's a question.
Edgar Fernando Carrera Jarrin: not related to the workshop.
Edgar Fernando Carrera Jarrin: Does anyone know here about the cms DAS when it is happening just an unrelated question that someone should me privately but.
Edgar Fernando Carrera Jarrin: I don't.
Edgar Fernando Carrera Jarrin: I don't actually.
Matthew Bellis: I only know about the January dates I think sometimes they have them in the summer, but then that next one, I know, will be in January at least the fermilab one.
Kati Lassila-Perini: Just a small reminder for the user to have this resources i'm not sure if you can create clusters, but if you can create clusters.
Kati Lassila-Perini: Please remember that, with the resources that we have the clusters are supposed to be in the European region so if you if you happen to follow some instructions that.
Kati Lassila-Perini: asked you to create clusters, well, I think you have your cluster so you should be working there, but if, by any means to create a cluster then take note that they are supposed to be created in your European European regions.
Cecilia Uribe Estrada: Thank you so again just to I don't know a suggestion, because I saw that.
Cecilia Uribe Estrada: Several students were having a lot of troubles to connect to the cloud and things like that.
Cecilia Uribe Estrada: And so, probably they actually couldn't really go on produce the route files and things like that so maybe it would be worth it just at least to mention, for example, right that when you change the I mean the first.
Cecilia Uribe Estrada: The first program that you produce as we are ready, so produces some empty histogram right and.
Cecilia Uribe Estrada: So, but it's worth it probably to mention that we, when you change the number of events to minus one, for example, you UK the very nice blow the PC later on, and so maybe it would be nice as well, to point out, in general, just how you run.
Cecilia Uribe Estrada: Our family five years, with a particular a like file absurd that you decided to run and just something like that in general for for the people who don't follow that Thank you.
Cecilia Uribe Estrada: Right.
Edgar Fernando Carrera Jarrin: Great for the for this very last works, though you mean right.
Cecilia Uribe Estrada: that's right.
Edgar Fernando Carrera Jarrin: All right, yes, I think we should this band just for the record I mean we didn't.
Edgar Fernando Carrera Jarrin: yeah it's a point i'm I want people to get engaged and ask questions and things like that, and if you put too much information in the page it goes the other way but yes you're right we didn't update that so it stays there for posterity yeah.
Kati Lassila-Perini: Very good yeah.
Kati Lassila-Perini: It is fair it's the last line of the instructions, so if you've read these instructions to the nt will find.
Kati Lassila-Perini: The number of events.
Edgar Fernando Carrera Jarrin: Or is it.
Cecilia Uribe Estrada: yeah yeah is there is there, but I was just thinking about an example with, for example, a particular file know for all the bands, especially because I mean in in during the hour all all other days we had.
Cecilia Uribe Estrada: Time to you know to to do things are a concern but today's the last day of the workshop and because, as I said, several students were having troubles just entering and trying to do something.
Cecilia Uribe Estrada: Probably they cannot ask the ask question because they didn't have the time actually go right to that point does the reason why, as well, I was thinking that probably a small example with that particular five would be useful for name but well that's just a comment, thank you.
Edgar Fernando Carrera Jarrin: All right, thank you.
Edgar Fernando Carrera Jarrin: Thank you everyone, so I think we can declare this workshop.
Edgar Fernando Carrera Jarrin: Over and I hope we can see you next time don't forget, if you wish to complete them the feedback form that.
Edgar Fernando Carrera Jarrin: will send out more.
Edgar Fernando Carrera Jarrin: Thanks a lot.
Chokri Ben Romdhane: Thank you all.
Edgar Fernando Carrera Jarrin: I hope I see you in the future.
Emanuele Bagnaschi: Yes.
Cecilia Uribe Estrada: Thank you very much.
Thank you.
Chokri Ben Romdhane: Very much have a.
Matthew Bellis: Great day.
Miguel Lopes: bye.