WEBVTT 1 00:00:02.820 --> 00:00:03.300 Edgar Fernando Carrera Jarrin: alright. 2 00:00:06.990 --> 00:00:08.189 Edgar Fernando Carrera Jarrin: You can hear me right. 3 00:00:10.830 --> 00:00:11.160 Yes. 4 00:00:13.080 --> 00:00:14.370 Cecilia Uribe Estrada: Okay, thanks. 5 00:00:17.699 --> 00:00:24.690 Edgar Fernando Carrera Jarrin: I apologize because, for some reason I I don't seem to be able to quickly foolish. 6 00:00:26.400 --> 00:00:29.550 Edgar Fernando Carrera Jarrin: Because I have forgotten the the link to this. 7 00:00:30.600 --> 00:00:33.180 Edgar Fernando Carrera Jarrin: To this demonstration, but notice and. 8 00:00:35.940 --> 00:00:52.260 Edgar Fernando Carrera Jarrin: So the link is broken, but the lesson is there, I hope you can access it from the prep work link, but if you click here, you will get an error which I certainly screwed something up but it doesn't matter you can. 9 00:00:52.260 --> 00:00:53.010 Edgar Fernando Carrera Jarrin: Access the. 10 00:00:54.300 --> 00:00:57.750 Edgar Fernando Carrera Jarrin: This link, you know click here and go to home. 11 00:00:58.020 --> 00:01:04.740 Edgar Fernando Carrera Jarrin: You can get the lesson, or you can access the lesson from the indigo agenda, I think that also should work. 12 00:01:06.510 --> 00:01:09.510 Edgar Fernando Carrera Jarrin: poses for that to me, you know things going on. 13 00:01:10.200 --> 00:01:10.560 Cecilia Uribe Estrada: I don't. 14 00:01:10.830 --> 00:01:14.580 Cecilia Uribe Estrada: read that Okay, so it is time for the demonstration. 15 00:01:16.020 --> 00:01:22.470 Edgar Fernando Carrera Jarrin: This time, you can you can take a break from working on the terminal and. 16 00:01:23.760 --> 00:01:31.110 Edgar Fernando Carrera Jarrin: As far as I know you can just watch and please ask questions and also, I like to. 17 00:01:32.340 --> 00:01:38.430 Edgar Fernando Carrera Jarrin: To to ask the the rest of the organizers especially clements and catty if they're around. 18 00:01:39.120 --> 00:01:58.170 Edgar Fernando Carrera Jarrin: To just jump in and clarify something that I might be saying wrong or if they want to add something that that's also welcome OK, so the idea of this demonstration so as all my demonstrations they really demonstration, so you can just watch, unfortunately, you have to listen to me. 19 00:02:00.300 --> 00:02:07.170 Edgar Fernando Carrera Jarrin: But this is really cool stuff So if you haven't heard that before sure you, you will find it interesting. 20 00:02:08.610 --> 00:02:16.530 Edgar Fernando Carrera Jarrin: So the basic idea is that you can use a commercial clouds available. 21 00:02:17.970 --> 00:02:31.860 Edgar Fernando Carrera Jarrin: In order to run the very analysis that we just did, for instance, especially the first part is given part which takes a lot of resources, you can run this on this commercial cloud. 22 00:02:32.730 --> 00:02:44.490 Edgar Fernando Carrera Jarrin: Okay, so, so what the way that i'm going to proceed is i'm going to go through basic concepts, by the way, is just as a disclaimer i'm not an expert on this. 23 00:02:44.970 --> 00:02:57.270 Edgar Fernando Carrera Jarrin: i'm still learning, and so I will give you my version of the these different concepts somehow very simplified and the idea is that you can. 24 00:02:58.290 --> 00:03:01.890 Edgar Fernando Carrera Jarrin: Try to Google more information and learn a little bit more. 25 00:03:02.940 --> 00:03:19.800 Edgar Fernando Carrera Jarrin: Okay So the first thing is the Google cloud platform i'm sure you have heard it and i'll just go with the flow and try to search for it in my in my Google and the first link that it comes out is this one right. 26 00:03:21.180 --> 00:03:26.790 Edgar Fernando Carrera Jarrin: And then you know if you're logged in with the with a Google account like like I am. 27 00:03:28.410 --> 00:03:45.720 Edgar Fernando Carrera Jarrin: With my gmail account, then it will prompt you to accept $300 of free credits so essentially you can test this for free, you get three months $200 for three months, and you can play around and whatever. 28 00:03:46.800 --> 00:04:04.350 Edgar Fernando Carrera Jarrin: stuff you want to do, and whatever resources, you can get with those $300 and Okay, but I have used up actually all my my money so it's not free for me anymore, but it doesn't matter, because what we're going to do is going to be really. 29 00:04:04.860 --> 00:04:09.600 Edgar Fernando Carrera Jarrin: A couple of cents, at most, so i'm going to go to the console directly. 30 00:04:12.360 --> 00:04:28.170 Edgar Fernando Carrera Jarrin: And if you are following the directions, the tutorial that's my second concept right, so the Google cloud platform has its console so we're calling this the console and. 31 00:04:29.100 --> 00:04:40.080 Edgar Fernando Carrera Jarrin: And Essentially, this is, this is the place on the web, where you can find all the different services that Google provides so you can do amazing things if you're like, especially if you're a computing person. 32 00:04:41.160 --> 00:04:53.340 Edgar Fernando Carrera Jarrin: You can find a lot of goodies here, but the most important part is this compute for us right because you have here to compute engine area where you can, for instance. 33 00:04:54.240 --> 00:05:06.660 Edgar Fernando Carrera Jarrin: start a fire up a virtual machine if you don't happen to have a good powerful machine, you can you can use to try to rent one of those machines here. 34 00:05:07.290 --> 00:05:19.980 Edgar Fernando Carrera Jarrin: You can even set up storage, you know if you did a test on the cloud, you can set up your your own version of disk things like that Okay, and then you also have this Cooper daddy's engine. 35 00:05:21.270 --> 00:05:40.830 Edgar Fernando Carrera Jarrin: Which brings us to the next concept, I guess you were you went through this some of the Cooper nettie basic concepts yesterday during the prep work, but essentially the burnett is is a software that allows you to manage the. 36 00:05:42.000 --> 00:05:45.450 Edgar Fernando Carrera Jarrin: The containers containers in general okay so. 37 00:05:46.980 --> 00:05:52.650 Edgar Fernando Carrera Jarrin: So let's let's look at this, I presented you the console, of course. 38 00:05:53.940 --> 00:06:04.170 Edgar Fernando Carrera Jarrin: The keyboard medicine engine is one of the services for Google and but the T burnett is itself is a software which orchestrates. 39 00:06:05.220 --> 00:06:09.870 Edgar Fernando Carrera Jarrin: containers in a computer cluster so as I was trying to explain during the. 40 00:06:11.160 --> 00:06:12.420 Edgar Fernando Carrera Jarrin: The prep work session. 41 00:06:13.950 --> 00:06:22.740 Edgar Fernando Carrera Jarrin: You know, it will be fantastic if you can throw in those containers that you have been playing with you can throw them into a very large or larger cluster. 42 00:06:23.220 --> 00:06:33.960 Edgar Fernando Carrera Jarrin: of computers and then you can use those computers to execute all the jobs that we have been working with and have the output much, much faster so that's the idea. 43 00:06:34.530 --> 00:06:47.670 Edgar Fernando Carrera Jarrin: And this is a nice picture that I found of the way the the architecture works essentially what would you when you sign in or where were you when you access this web web platform. 44 00:06:48.300 --> 00:07:01.470 Edgar Fernando Carrera Jarrin: You interact with an API of the Google super net is engine, essentially, given the key burnett is engineering Google is a place where you can create clusters with Cooper nettie is. 45 00:07:02.520 --> 00:07:10.080 Edgar Fernando Carrera Jarrin: Okay, and do all this kind of the deployments of your containers, so they can perform analysis, for instance. 46 00:07:11.340 --> 00:07:12.540 Edgar Fernando Carrera Jarrin: And the. 47 00:07:13.860 --> 00:07:30.480 Edgar Fernando Carrera Jarrin: The basic concepts are the master node as you saw in their worker nodes so the master nodes notes they're different master nodes the the Google infrastructure and they take care of instructing the worker nodes. 48 00:07:31.980 --> 00:07:39.090 Edgar Fernando Carrera Jarrin: On the kind of parts that they need to run and essentially apart in the. 49 00:07:40.290 --> 00:07:51.750 Edgar Fernando Carrera Jarrin: Google keyword net is abstraction is pricing would be just a docker container so you can think that a part is just a docker container you can put many containers in a pack. 50 00:07:53.310 --> 00:07:54.930 Edgar Fernando Carrera Jarrin: Everything will be doing that. 51 00:07:56.130 --> 00:08:04.170 Edgar Fernando Carrera Jarrin: And then you can run this and in parallel Okay, and on top of that, you can put some services or. 52 00:08:05.220 --> 00:08:22.050 Edgar Fernando Carrera Jarrin: Those kind of things, for instance, if you're running container with a with a web server you can create a load balancer service or you clients can access that web server things like that so that's that's a key burnett is cluster. 53 00:08:24.150 --> 00:08:35.310 Edgar Fernando Carrera Jarrin: But we are interested in running workflows essentially right like we did we just did we were interested in running something like this using our containers. 54 00:08:35.790 --> 00:08:53.700 Edgar Fernando Carrera Jarrin: We are interested in skimming then merging perhaps our files, you know after you skim you Maybe you can maybe run like a 111 root file for each container and you run a bunch of these containers and you get a bunch of this route files and then you can merge them and then. 55 00:08:55.080 --> 00:09:05.490 Edgar Fernando Carrera Jarrin: This merge files files will be would be the ones that that we were working with just recently, you know the ones that I asked you to copy from the certain certain box. 56 00:09:06.150 --> 00:09:16.530 Edgar Fernando Carrera Jarrin: And then we can perform an analysis over them and then we can applied histograms I keep I keep making mistakes in my typing. 57 00:09:17.940 --> 00:09:22.530 Edgar Fernando Carrera Jarrin: So so yeah that's the really nice thing about this keyboard matt is. 58 00:09:23.790 --> 00:09:26.460 Edgar Fernando Carrera Jarrin: Engine clusters, we can. 59 00:09:27.660 --> 00:09:49.170 Edgar Fernando Carrera Jarrin: execute workflows, and so this could be very automatically done and usually, when we work on workflows we we talk about yama files, so all these all these workflows are configured using this famous yammer files Okay, so we talked about parts. 60 00:09:50.550 --> 00:10:03.780 Edgar Fernando Carrera Jarrin: But, in essence, when you interact with Cooper nettie cluster you don't really create parts Okay, when you create is something called deployment. 61 00:10:04.350 --> 00:10:18.600 Edgar Fernando Carrera Jarrin: And deployment is one of the difference abstraction layers in Cuba natives, but they are above parts and so, in practice, you always create deployments in Cooper natives not not parts right. 62 00:10:19.170 --> 00:10:33.840 Edgar Fernando Carrera Jarrin: And the other concept that I like to mention is ARGO ARGO is software and it really wraps around the Cooper nettie his main commands and such as I understand it. 63 00:10:35.070 --> 00:10:53.070 Edgar Fernando Carrera Jarrin: So it's you can see it as having the keyboard net is the commands on asteroids because it allows you to handle those workflows that we need to do in a much easier way so that's our goal, and will, we will be using this and that's why I mentioned this. 64 00:10:54.360 --> 00:10:55.140 Edgar Fernando Carrera Jarrin: So okay. 65 00:10:56.670 --> 00:11:03.750 Edgar Fernando Carrera Jarrin: If you're a regular Joe like myself and want to do some particle physics analysis. 66 00:11:05.790 --> 00:11:18.960 Edgar Fernando Carrera Jarrin: You can you can go to the Google cloud platform Okay, let me put my my own instructions in my other computer, so I can remember them, and neither skip anything. 67 00:11:19.860 --> 00:11:27.540 Edgar Fernando Carrera Jarrin: Alright, so yeah you can you can just go even right now I don't I don't want you to go right now but. 68 00:11:28.020 --> 00:11:37.980 Edgar Fernando Carrera Jarrin: Later, you can you can check this out by yourself and so you will arrive to this console and you can check out all the different products, but we are interested in this Cooper nettie Ascension. 69 00:11:39.570 --> 00:11:57.330 Edgar Fernando Carrera Jarrin: indigo platform, you can have different projects, right now, I have two projects this cms open data we we will be accessing this project tomorrow okay today, I just wanted to use my personal I call it two. 70 00:11:58.800 --> 00:12:10.170 Edgar Fernando Carrera Jarrin: Years of killed because that's my institution, and I was using this for teaching for, for example, and so I created this project term and I used that one is already checked so that that's fine. 71 00:12:11.430 --> 00:12:18.240 Edgar Fernando Carrera Jarrin: So you can differentiate if you're doing different different analysis Maybe you can create different projects, for instance. 72 00:12:19.980 --> 00:12:30.090 Edgar Fernando Carrera Jarrin: And then we can create a Cooper natives cluster of machines right, so we go here keeper net is we click on clusters. 73 00:12:32.010 --> 00:12:40.770 Edgar Fernando Carrera Jarrin: and the first thing that appears is okay, do you want to grade our Cooper natives cluster, and I say yes, create one, and you have other options. 74 00:12:42.720 --> 00:12:57.330 Edgar Fernando Carrera Jarrin: that the good thing about this environment is that at all stages, it gives you the ability to learn more so you can all you do is we'll be seeing links with the learn more stuff. 75 00:12:58.410 --> 00:13:15.540 Edgar Fernando Carrera Jarrin: Okay let's just go to create and I would like to create an extender to Cuba natives cluster not really a a predefined or autopilot because I want to show you a few things so i'll go for this one okay. 76 00:13:16.680 --> 00:13:32.640 Edgar Fernando Carrera Jarrin: And you know, even if you have to if you're out of your free credits your $300 credit don't be afraid to just create your clusters, because at the end we're gonna destroy everything so it's not going to cost you anything. 77 00:13:33.960 --> 00:13:34.800 Edgar Fernando Carrera Jarrin: Unless you try. 78 00:13:37.920 --> 00:13:46.470 Edgar Fernando Carrera Jarrin: Alright, so you can give it a name for instance I can I can put my my own name and create a cluster. 79 00:13:47.340 --> 00:14:01.200 Edgar Fernando Carrera Jarrin: You can see that you can choose different selves for the workshop tomorrow for a hands on part we first you won't have to create your your cluster by yourself. 80 00:14:01.770 --> 00:14:15.120 Edgar Fernando Carrera Jarrin: Okay, it will be created for you're ready, because all these steps may take some time and some sometimes there are some errors and such so we don't want to spend time doing that that's why I set up this demonstration. 81 00:14:15.990 --> 00:14:32.520 Edgar Fernando Carrera Jarrin: But you can choose your your your cluster you can choose it to be created in different regions, tomorrow the cluster that you will be using we will be created in Europe okay so we'll take care of that automatically automatically hopefully. 82 00:14:34.050 --> 00:14:37.590 Edgar Fernando Carrera Jarrin: And all right, but i'll just stick to this US central one. 83 00:14:38.880 --> 00:14:47.520 Edgar Fernando Carrera Jarrin: Okay, and there are many ways of creating this, you can see, the different options here but okay i'll go with the standard or. 84 00:14:48.660 --> 00:14:53.160 Edgar Fernando Carrera Jarrin: not really a standard way of creating anything but maybe. 85 00:14:54.330 --> 00:15:12.840 Edgar Fernando Carrera Jarrin: A recommended way of creating this So there are a few things that we can change, for instance, we can click here for the default pool, and it will ask us how many nodes we want to have in our cluster remember when. 86 00:15:13.920 --> 00:15:15.270 Edgar Fernando Carrera Jarrin: We saw this. 87 00:15:16.500 --> 00:15:27.030 Edgar Fernando Carrera Jarrin: picture here, so these are the worker nodes, and these are the guys maybe the physical machines or virtual machines that will harbor the parts, so they can work in them. 88 00:15:27.840 --> 00:15:40.140 Edgar Fernando Carrera Jarrin: So the more nodes you have maybe the more jobs, you can you can you can put in them, but for now we can just click on one node with the ability to. 89 00:15:41.910 --> 00:16:02.820 Edgar Fernando Carrera Jarrin: have some auto scaling, meaning that Google will decide whether it needs more machines or not, but we can give it a range so from zero to four machines so because you can put 100 maybe, but it will cost you a lot of money so for machines work for us, for now, and there is also. 90 00:16:05.040 --> 00:16:06.420 Edgar Fernando Carrera Jarrin: an option to. 91 00:16:08.820 --> 00:16:12.690 Edgar Fernando Carrera Jarrin: To use preventable what is called preventable machines which. 92 00:16:15.990 --> 00:16:24.750 Edgar Fernando Carrera Jarrin: Really reduces the Code, the costs, I don't remember exactly where was that if clemency you can remind me. 93 00:16:26.100 --> 00:16:28.650 Edgar Fernando Carrera Jarrin: I probably missed that somewhere. 94 00:16:38.610 --> 00:16:38.910 Kati Lassila-Perini: You. 95 00:16:42.690 --> 00:16:46.200 Kati Lassila-Perini: can go to notes in the to the left. 96 00:16:49.980 --> 00:16:50.370 Edgar Fernando Carrera Jarrin: Many. 97 00:16:51.540 --> 00:16:53.130 Edgar Fernando Carrera Jarrin: indications that you have here. 98 00:16:54.270 --> 00:16:56.820 Kati Lassila-Perini: In the Left notes and they're done. 99 00:16:59.100 --> 00:17:00.690 Edgar Fernando Carrera Jarrin: So we're in. 100 00:17:01.860 --> 00:17:04.350 Edgar Fernando Carrera Jarrin: These options here so. 101 00:17:04.530 --> 00:17:06.330 Edgar Fernando Carrera Jarrin: see that you have this. 102 00:17:07.320 --> 00:17:17.670 Edgar Fernando Carrera Jarrin: This option of enable preventable notes, so the claim is that if you use these preventable notes, which you get the description there. 103 00:17:19.110 --> 00:17:34.350 Edgar Fernando Carrera Jarrin: These are instances that allows up to 24 hours Okay, and the price is really lowered, so if you want to test something for 24 hours, maybe it is best to to check this so you can you can reduce your costs. 104 00:17:35.370 --> 00:17:41.610 Edgar Fernando Carrera Jarrin: Okay, so all right, I have chosen for machines, so they can be scaled also. 105 00:17:43.980 --> 00:17:49.410 Edgar Fernando Carrera Jarrin: And the notes in the notes that you can choose what kind of machine, you want to use. 106 00:17:50.850 --> 00:18:04.410 Edgar Fernando Carrera Jarrin: So, by default, you will get a machine with tcp years and four gigabytes of memory, but it turns out that each cms w job takes about two gigabytes of memory, excluding all the overhead due to the different. 107 00:18:05.280 --> 00:18:27.390 Edgar Fernando Carrera Jarrin: servers and things that we will see that will be running in parallel with in them in the machine so let's choose, perhaps the standard for with four cpus and 16 gigabytes of memory just for our check and once we have done that, and you, if you want, you can you can click on this preventable. 108 00:18:28.890 --> 00:18:32.670 Edgar Fernando Carrera Jarrin: We can create around our cluster. 109 00:18:34.200 --> 00:18:44.040 Edgar Fernando Carrera Jarrin: it's that simple it's just a few clicks away don't be afraid of creating this it will take a moment see the wheel here spinning. 110 00:18:45.180 --> 00:18:48.480 Edgar Fernando Carrera Jarrin: It will take a moment, but just a few minutes. 111 00:18:49.560 --> 00:18:56.040 Edgar Fernando Carrera Jarrin: And while we wait for that I, like you, I like to introduce you to the cloud shell. 112 00:18:57.570 --> 00:19:08.730 Edgar Fernando Carrera Jarrin: So let's go I always like to work with image of so so you're here right and you're trying to interact with the whole cluster and this. 113 00:19:09.480 --> 00:19:24.120 Edgar Fernando Carrera Jarrin: This part right here I mean you have to interact through some machine and through some terminology through some console that you, you can type things on and so Google provides you with such a computer. 114 00:19:26.250 --> 00:19:35.760 Edgar Fernando Carrera Jarrin: And you can access that it's gone the cloud Shell, you can access that through this little icon here, which is just a prompt I can. 115 00:19:36.900 --> 00:19:40.890 Edgar Fernando Carrera Jarrin: Let me know if it is too small, the letters are too small. 116 00:19:42.360 --> 00:19:48.750 Edgar Fernando Carrera Jarrin: So you can collect you can click here well even well the what the cluster is building. 117 00:19:49.560 --> 00:20:07.560 Edgar Fernando Carrera Jarrin: Okay, and this will be your gateway not not only to the Cooper natives cluster but to to anything you want to do in the Google cloud platform, you know if you're an APP developer, you can you can have another project and and have a different type of resources and such. 118 00:20:09.540 --> 00:20:24.750 Edgar Fernando Carrera Jarrin: But this terminal is essentially a machine that that Google provides you with that can be used to to interact with whatever service, you are you're building inside that Google cloud platform. 119 00:20:26.700 --> 00:20:43.080 Edgar Fernando Carrera Jarrin: Okay, and the one command that is used for interacting with all this Google cloud platform services is the Google cloud okay and i'm sorry big cloud, so there is a cloud command. 120 00:20:44.340 --> 00:21:02.040 Edgar Fernando Carrera Jarrin: That rules everything here the documentation is very extensive, but you can do, for instance, help and I think that's the right and yeah get a description of what you can do with this and want to make this a little bit bigger. 121 00:21:03.840 --> 00:21:23.310 Edgar Fernando Carrera Jarrin: Right it's a very, very powerful command that in the idea is that with this G cloud, you can control or you can get access to the different services Okay, and so, at some point, what we want to do is to connect to the cluster that we just we just made. 122 00:21:24.360 --> 00:21:30.450 Edgar Fernando Carrera Jarrin: We haven't connected yet because it's not done is still spinning and we're waiting for that. 123 00:21:31.860 --> 00:21:46.440 Edgar Fernando Carrera Jarrin: But before that you have to so I probably don't have to do this again but i'll just run this command and, just in case you have to get some sort of authentication and the idea is that you will run this G cloud. 124 00:21:47.640 --> 00:21:55.980 Edgar Fernando Carrera Jarrin: Application login and it will prompt you to some clarification and you can copy this thing or click on this thing. 125 00:22:00.630 --> 00:22:18.360 Edgar Fernando Carrera Jarrin: You don't really have to copy, this is, I can click on it, it will ask you right it's just like a normal authentication I will authenticate myself with the with my gmail account again, although I thought I think I didn't need it, and it will provide you with a. 126 00:22:19.470 --> 00:22:24.240 Edgar Fernando Carrera Jarrin: With a kind of password or key that's the right. 127 00:22:26.430 --> 00:22:44.370 Edgar Fernando Carrera Jarrin: term for it and you have to enter your verification key right here Okay, and once you have done that you're authenticated and you're going to start interacting with the with the all the resources in the Google cloud platform, so all right, our cluster is already made. 128 00:22:45.900 --> 00:22:50.550 Edgar Fernando Carrera Jarrin: Against again I know because there is this Green checkmark. 129 00:22:51.990 --> 00:23:00.030 Edgar Fernando Carrera Jarrin: Right, and now I need to connect to it if you go to this three dots here, you see the connection. 130 00:23:00.960 --> 00:23:10.890 Edgar Fernando Carrera Jarrin: button and if you click on it, it will give you a command, you know, a Google G cloud command with which you can connect to your cluster. 131 00:23:11.340 --> 00:23:23.760 Edgar Fernando Carrera Jarrin: And it makes sense right because it says OK, Google cloud container clusters get credentials for the Edgar cluster that I just created in this zone and under the projects have yourself. 132 00:23:25.980 --> 00:23:33.630 Edgar Fernando Carrera Jarrin: I can choose to copy this directly into the cloud Shell by myself or ask the thing to run it in the cloud. 133 00:23:34.680 --> 00:23:38.250 Edgar Fernando Carrera Jarrin: And if I present that it will. 134 00:23:39.630 --> 00:23:52.590 Edgar Fernando Carrera Jarrin: It will copy the command, for me, and I can just executed and it asked me if I authorized this connection I say yes, and once I doubt that I have done that. 135 00:23:53.040 --> 00:24:00.270 Edgar Fernando Carrera Jarrin: I have connection to my burnett is cluster at this moment I can interact with my cluster. 136 00:24:01.020 --> 00:24:17.190 Edgar Fernando Carrera Jarrin: i'm still outside, in a sense, you know it's not like I login to the cluster does not the way that you should think about it, because sometimes you know when you when you want to use some computing resources and some institution, you have to log in to the institution. 137 00:24:18.450 --> 00:24:29.460 Edgar Fernando Carrera Jarrin: I don't think this is something like that you are always working in this machine that Google provides you to interact with the cluster but you've never logged into the cluster that's the difference. 138 00:24:30.720 --> 00:24:44.880 Edgar Fernando Carrera Jarrin: And in order to interact with a cluster we have this very powerful command again so that completes actually my creation of the, by the way my creation of the cluster so that's the first. 139 00:24:46.020 --> 00:24:48.570 Edgar Fernando Carrera Jarrin: DEMO, but if you click. 140 00:24:49.980 --> 00:24:51.600 Edgar Fernando Carrera Jarrin: And you see if you click. 141 00:24:53.760 --> 00:24:56.250 Edgar Fernando Carrera Jarrin: On the arrow you can go to the next. 142 00:24:58.560 --> 00:25:06.150 Edgar Fernando Carrera Jarrin: part and then, so the next part is about the very powerful command just as G cloud. 143 00:25:07.200 --> 00:25:18.570 Edgar Fernando Carrera Jarrin: We have the cube cpl command or some some people call it to God oh I don't know why, but equally chip Carl I like to call it gypsy do it is to pronounce for me. 144 00:25:21.060 --> 00:25:29.430 Edgar Fernando Carrera Jarrin: Alright, so i'll be i'll be teaching you about this command and how to use it to to interact with the cluster okay. 145 00:25:32.370 --> 00:25:33.390 Edgar Fernando Carrera Jarrin: Alright, so. 146 00:25:35.430 --> 00:25:36.690 Edgar Fernando Carrera Jarrin: So first off. 147 00:25:38.070 --> 00:25:40.740 Edgar Fernando Carrera Jarrin: I put a link in my instructions. 148 00:25:41.940 --> 00:25:55.560 Edgar Fernando Carrera Jarrin: And i'm going to copy here and there is ample instructions on this command, you know G cloud and cpl are the commands they are not just any command, they are the commands to control widget cloud. 149 00:25:56.250 --> 00:26:05.280 Edgar Fernando Carrera Jarrin: You can access any resource in the Google cloud platform with gypsy DL you can control your your cluster to cluster that you just created. 150 00:26:06.570 --> 00:26:17.790 Edgar Fernando Carrera Jarrin: And I like to from time to time I check out this cheat sheet is the official cheat sheet from capabilities, you have many, many commands with a description of what they do. 151 00:26:18.570 --> 00:26:32.910 Edgar Fernando Carrera Jarrin: You know the kfc DL options of the different things that you can do with it it's not really a short cheat sheet and it's it's really large so just for fun we're going to run a few of these commands. 152 00:26:35.040 --> 00:26:38.310 Edgar Fernando Carrera Jarrin: On the terminal because we have connection already. 153 00:26:39.600 --> 00:26:39.990 Edgar Fernando Carrera Jarrin: and 154 00:26:41.310 --> 00:26:52.050 Edgar Fernando Carrera Jarrin: For instance, if you want to get your information about your notes, you can execute this command and, as a matter of fact, what you get is. 155 00:26:52.890 --> 00:27:04.140 Edgar Fernando Carrera Jarrin: Essentially, the configuration of your notes, if you if you were to install to burnett is in your own machines your own institution. 156 00:27:04.440 --> 00:27:14.130 Edgar Fernando Carrera Jarrin: You could see actually the display of the machines here, but in this case it's just the configuration we use through the Web to do it because it's rather automatic. 157 00:27:17.040 --> 00:27:20.220 Edgar Fernando Carrera Jarrin: The other thing that you could run is something. 158 00:27:21.450 --> 00:27:29.850 Edgar Fernando Carrera Jarrin: Like you can ask for the information of your cluster so there's another commands, for instance, that is called cluster info. 159 00:27:30.360 --> 00:27:44.010 Edgar Fernando Carrera Jarrin: And it gives you a little bit more information of the control plane, you were you were you were learning about this in Europe prep work the control plane, this is the master machine, the main machine. 160 00:27:44.970 --> 00:27:52.860 Edgar Fernando Carrera Jarrin: That is a given all the instructions to the different worker notes and some other stuff as well. 161 00:27:54.090 --> 00:28:05.520 Edgar Fernando Carrera Jarrin: And there is also command if you have if you had some bugs running you could get the information from the parts of there is this tipsy do get part command but. 162 00:28:07.140 --> 00:28:22.500 Edgar Fernando Carrera Jarrin: Of course we don't have anything right now there's there's no resources, no resources and in our namespace the namespace is kind of like a like an area like a like a subsection of the cluster you can reserve main spaces for different. 163 00:28:23.520 --> 00:28:32.460 Edgar Fernando Carrera Jarrin: purposes, for instance, I was teaching some high energy physics class analysis class and I had a few students, so I gave. 164 00:28:33.480 --> 00:28:48.840 Edgar Fernando Carrera Jarrin: A namespace for each group of students, so they don't interfere with each other, but it was the same cluster Okay, but I can distribute the the the resources through this namespace that's another feature of cybernetics. 165 00:28:51.300 --> 00:29:04.080 Edgar Fernando Carrera Jarrin: If we were on in some services like the load balancer I was talking about, if you had a web server, we can also extract information about the status of the of the services with this command. 166 00:29:06.690 --> 00:29:07.170 Edgar Fernando Carrera Jarrin: and 167 00:29:08.670 --> 00:29:11.910 Edgar Fernando Carrera Jarrin: Maybe the the most one of the most important commands. 168 00:29:13.140 --> 00:29:14.250 Edgar Fernando Carrera Jarrin: Is the. 169 00:29:15.360 --> 00:29:21.780 Edgar Fernando Carrera Jarrin: tipsy to create command, so we can actually get some information about this command. 170 00:29:28.170 --> 00:29:30.630 Edgar Fernando Carrera Jarrin: See you can you can run a late this. 171 00:29:33.390 --> 00:29:36.150 Edgar Fernando Carrera Jarrin: or embraceable you can run it like. 172 00:29:37.680 --> 00:29:38.700 Edgar Fernando Carrera Jarrin: Where is this thing. 173 00:29:41.250 --> 00:29:54.240 Edgar Fernando Carrera Jarrin: Well, there are many ways of running this card for me to read everything right now but, but this is the command, you know that creates elements in the in the qpr nerys Cooper natives cluster. 174 00:29:55.440 --> 00:29:58.440 Edgar Fernando Carrera Jarrin: So we're going to try to run a few of these things. 175 00:30:00.120 --> 00:30:11.910 Edgar Fernando Carrera Jarrin: time flies I don't think i'm going to be able to to finish this demonstration, but i'd like to give you an idea of what you could do, and you can follow along if you are tempted to try this by yourself. 176 00:30:13.020 --> 00:30:17.370 Edgar Fernando Carrera Jarrin: Anyway, let's let's imagine, for instance, that we want to create. 177 00:30:18.660 --> 00:30:31.140 Edgar Fernando Carrera Jarrin: A web server you know, this is the classical example and engine X, is one of the most famous web servers out there and what they have done, is that they have. 178 00:30:31.590 --> 00:30:47.310 Edgar Fernando Carrera Jarrin: A containerized application of this software, so it lives somewhere in docker hub right, you can go to docker hub, and where all these containers live, you can you can actually create your own container and put it in docker hub. 179 00:30:53.340 --> 00:30:57.360 Edgar Fernando Carrera Jarrin: And we can search for engine X, and this is a very official. 180 00:30:58.830 --> 00:31:15.060 Edgar Fernando Carrera Jarrin: repository for engine X and you can just docker both this engine X and have your your containerized web server so we'd like to we'd like to deploy this into our cluster we weren't to have one instance of this. 181 00:31:16.560 --> 00:31:17.760 Edgar Fernando Carrera Jarrin: Application running. 182 00:31:19.320 --> 00:31:31.500 Edgar Fernando Carrera Jarrin: So, tomorrow we're gonna have an instance of or several instances, maybe of our application running in in our cms is w container running so this is just an example to show you. 183 00:31:31.980 --> 00:31:44.760 Edgar Fernando Carrera Jarrin: very easily, and so the way that you would do something like this right gypsy to create actually Internet is you don't interact with parts you interact with this layer called deployment. 184 00:31:45.480 --> 00:31:57.960 Edgar Fernando Carrera Jarrin: And that deployment will make the appropriate number of parts in this case, this very simple command creates just one so i'm gonna run that and the deployment is created. 185 00:32:00.510 --> 00:32:14.040 Edgar Fernando Carrera Jarrin: And you can check the deployment with this command called tipsy to get and you can see that it's ready i'm ready, you know valuable it's running because it's server so it's running. 186 00:32:15.420 --> 00:32:30.630 Edgar Fernando Carrera Jarrin: And it will be running there and actually the Nice feature for developers is that if the instance does instance of this web server guys, for some reason this part because it will be uploaded I can show you that it's a pod. 187 00:32:32.130 --> 00:32:45.450 Edgar Fernando Carrera Jarrin: Right, I can get I can get the part the permission of the part and the status is that is running and it will be running until for some reason, if pressures, you know software in the crash crash. 188 00:32:47.010 --> 00:32:56.400 Edgar Fernando Carrera Jarrin: The Internet is will take care of that we're instantiate the depart and deeper running all the time, so you don't have to intervene, mostly. 189 00:32:57.450 --> 00:33:06.540 Edgar Fernando Carrera Jarrin: Okay, this is the minimalistic way of running or creating a new element Internet is so we already have something they're going on. 190 00:33:08.940 --> 00:33:18.390 Edgar Fernando Carrera Jarrin: But there is a more advanced way of creating elements in the kinky burnett is and that's through yama files. 191 00:33:18.990 --> 00:33:36.720 Edgar Fernando Carrera Jarrin: So i'm going to go ahead and delete my current deployment, so there are commands to do that right, you see the elderly deployment at some point, if you do this often you start seeing some logic into the arrangement of this words. 192 00:33:39.180 --> 00:33:43.110 Edgar Fernando Carrera Jarrin: And now we're gonna we're going to apply the. 193 00:33:45.600 --> 00:33:54.420 Edgar Fernando Carrera Jarrin: we're going to create this engine X server but through a yammer file, so this is more powerful right because. 194 00:33:55.740 --> 00:34:06.570 Edgar Fernando Carrera Jarrin: If you look at the yama file I have expanded it here in the lesson, if you look at the yama file it, it gives like a blueprint for to burnett is to. 195 00:34:07.170 --> 00:34:12.840 Edgar Fernando Carrera Jarrin: deploy the correct image Oh, by the way, I forgot to I forgot to mention that. 196 00:34:13.230 --> 00:34:30.750 Edgar Fernando Carrera Jarrin: When we were deployed, and when we were creating this deployment what we did was to call the image from docker you know this this day image, just like damaged you download it from from docker from for cms w and it just give it a name okay. 197 00:34:32.340 --> 00:34:40.980 Edgar Fernando Carrera Jarrin: But this is very minimal you can expand this configuration with a yama file, and for that you could do something like this. 198 00:34:43.710 --> 00:34:51.660 Edgar Fernando Carrera Jarrin: Right and, for instance, you can choose the diversion of the image and you can choose to have two replicas of the same part. 199 00:34:52.590 --> 00:34:59.400 Edgar Fernando Carrera Jarrin: Things like that you know we will go through these configuration for our own purposes later tomorrow. 200 00:35:00.390 --> 00:35:18.990 Edgar Fernando Carrera Jarrin: But that's possible and so that's the there another way of creating elements in the Cooper daddy's cluster and you just do to burn a CD i'll apply with the file that's what the F stands for and it's always a jamar five, and you can create things. 201 00:35:20.340 --> 00:35:24.480 Edgar Fernando Carrera Jarrin: I have to, I have to get the file, first, I have a. 202 00:35:29.550 --> 00:35:33.720 Edgar Fernando Carrera Jarrin: I have to get the file, of course, you have a few commands. 203 00:35:35.280 --> 00:35:40.920 Edgar Fernando Carrera Jarrin: that are available, including w debt so that's what's Nice, and then I can run my. 204 00:35:43.200 --> 00:35:45.840 Edgar Fernando Carrera Jarrin: My deployment, so it is created. 205 00:35:47.580 --> 00:35:59.340 Edgar Fernando Carrera Jarrin: We can check that it is actually created yeah and now we have two parts Okay, they are, they are being created, but at some point, it will be running yeah they're running already. 206 00:36:00.480 --> 00:36:23.850 Edgar Fernando Carrera Jarrin: This is the way the basic way or the the essence of how you create elements and Cooper nettie you can do this with the deployment of applications, you can do this with services like load balancers of or http services or nfl services, whatever you want that's the way you do it. 207 00:36:25.050 --> 00:36:38.310 Edgar Fernando Carrera Jarrin: Okay, and for the rest of the tutorial I was hoping that I would have time, but time is around through the limited I was hoping to go through the rest of the of the lesson. 208 00:36:39.390 --> 00:36:55.320 Edgar Fernando Carrera Jarrin: which included installing this ARGO workflow capabilities and I have some instructions that I tested that they work, so I hope that if you feel tempted or inspired You can check this by yourself. 209 00:36:56.340 --> 00:36:56.760 Edgar Fernando Carrera Jarrin: and 210 00:36:58.080 --> 00:37:07.050 Edgar Fernando Carrera Jarrin: And then I was, I was also having instructions on how to set up some disk space and run some nfl server. 211 00:37:08.370 --> 00:37:17.910 Edgar Fernando Carrera Jarrin: and to be able to access this this this space and get the but I didn't do that, but also an http. 212 00:37:18.450 --> 00:37:28.800 Edgar Fernando Carrera Jarrin: server so you can access the files that you could produce from outside, all those things can be done, they take time they have certain logic. 213 00:37:29.400 --> 00:37:37.650 Edgar Fernando Carrera Jarrin: I hope it is clear instructions that follow in case you want to test them, but all of these things that we have gone through. 214 00:37:38.310 --> 00:37:47.430 Edgar Fernando Carrera Jarrin: In this tutorial in this demonstration will be automatically created for you tomorrow, hopefully, if not, if not. 215 00:37:48.390 --> 00:38:01.170 Edgar Fernando Carrera Jarrin: We might have to go through all these steps again and, at the beginning, maybe, since we run out of time, I can explain, perhaps a little bit more about this ARGO and how it works. 216 00:38:01.740 --> 00:38:14.610 Edgar Fernando Carrera Jarrin: And how it will help us run our workflows I don't I don't know if this is clear, or if there are questions i'll be happy to to try to answer them. 217 00:38:27.900 --> 00:38:29.250 Matthew Bellis: drive a quick question. 218 00:38:31.470 --> 00:38:32.130 Matthew Bellis: Can you hear me. 219 00:38:44.370 --> 00:38:45.120 Matthew Bellis: hey girl. 220 00:38:46.680 --> 00:38:48.150 Matthew Bellis: But not sure if you're hearing. 221 00:38:49.500 --> 00:38:55.020 Matthew Bellis: Now i'm talking now can anyone else hear me or is it just. 222 00:38:57.570 --> 00:38:57.870 Matthew Bellis: OK. 223 00:38:57.960 --> 00:38:59.130 Edgar Fernando Carrera Jarrin: I can hear you now yes. 224 00:38:59.190 --> 00:38:59.790 Matthew Bellis: Okay, great. 225 00:39:00.120 --> 00:39:08.640 Cecilia Uribe Estrada: yeah I think it was a problem because during the during the talk, I think I couldn't hear us very well yeah. 226 00:39:09.270 --> 00:39:10.350 Edgar Fernando Carrera Jarrin: Oh sorry about that. 227 00:39:10.710 --> 00:39:11.940 Edgar Fernando Carrera Jarrin: No worries on and on. 228 00:39:12.600 --> 00:39:16.170 Matthew Bellis: Edgar Is there something we should do to gracefully shut down our. 229 00:39:17.280 --> 00:39:24.060 Matthew Bellis: Our instance here so that will be able to fire it up tomorrow what's the what which What should we do with our environment. 230 00:39:26.310 --> 00:39:27.930 Edgar Fernando Carrera Jarrin: Oh, you were following great. 231 00:39:30.480 --> 00:39:32.280 Edgar Fernando Carrera Jarrin: This is my favorite part of the workshop. 232 00:39:34.560 --> 00:39:35.010 Edgar Fernando Carrera Jarrin: Right. 233 00:39:37.050 --> 00:39:48.720 Edgar Fernando Carrera Jarrin: hi I would I would actually shut everything down just because so actually i'll go ahead and do that, so you can click here and in. 234 00:39:48.840 --> 00:39:49.530 Edgar Fernando Carrera Jarrin: A way you're not. 235 00:39:49.560 --> 00:39:50.460 Matthew Bellis: you're not sharing. 236 00:39:51.510 --> 00:39:52.680 Edgar Fernando Carrera Jarrin: Sorry that's okay. 237 00:39:55.290 --> 00:40:00.720 Matthew Bellis: I don't want it right, because we don't want to delete this we just want to shut it down is that right or no. 238 00:40:03.000 --> 00:40:03.540 Edgar Fernando Carrera Jarrin: well. 239 00:40:05.190 --> 00:40:05.850 Matthew Bellis: First of the same. 240 00:40:05.880 --> 00:40:20.130 Edgar Fernando Carrera Jarrin: I don't think there is a way to shut it down, you will have to delete this machines, if you don't want to get charged, I certainly don't want to get charged, so I will delete it a. 241 00:40:20.820 --> 00:40:34.950 Edgar Fernando Carrera Jarrin: Once you're familiar with this it's very easy to go again through the procedure and just start the cluster again, but of course it's kind of painful and that's the reason why you, you may want to do to other monetize this. 242 00:40:37.980 --> 00:40:50.010 Edgar Fernando Carrera Jarrin: And and use some tools like Tara form to to just run it from the to create the cluster from from from any script that's also possible instead of clicking. 243 00:40:51.240 --> 00:41:05.820 Edgar Fernando Carrera Jarrin: So so yeah I I will delete this that's the way to do it, I will delete this so I don't get charged that much and, if needed, I can I can recreate it tomorrow, but as I said. 244 00:41:07.650 --> 00:41:10.800 Edgar Fernando Carrera Jarrin: everything will be done automatically hopefully tomorrow. 245 00:41:11.460 --> 00:41:27.780 Edgar Fernando Carrera Jarrin: And, and you will, and we will concentrate on on senate setting it up, so we can run the workflow or at least an example of the workflow that we could be doing in order to perform all the analysis flow that we have gone through today. 246 00:41:30.390 --> 00:41:32.760 Matthew Bellis: awesome Okay, thank you just wanted to make sure thanks. 247 00:41:34.230 --> 00:41:34.860 Edgar Fernando Carrera Jarrin: yeah sorry. 248 00:41:35.250 --> 00:41:40.050 Edgar Fernando Carrera Jarrin: So far, about my scanner weird the configuration, sometimes of my headphones. 249 00:41:44.550 --> 00:41:45.540 Edgar Fernando Carrera Jarrin: Alright, thanks so. 250 00:41:48.570 --> 00:41:49.290 Edgar Fernando Carrera Jarrin: i'll see you tomorrow. 251 00:41:51.210 --> 00:41:52.110 Matthew Bellis: bye everybody. 252 00:41:52.140 --> 00:41:54.240 Matthew Bellis: bye bye i'm going to stop the recording.