Mario Lassnig: Recording perfect okay great welcome everyone to this first workshop on the data challenge monitoring for high lumen.
Mario Lassnig: We have a pretty packed agenda.
Mario Lassnig: But from what I can see some of this is partially overlapping, so we might not even need the full time, I guess, we will dynamically decided to make a break or not, in the middle.
Mario Lassnig: If there are no questions I would hand it off to Alexandra and Sean for any additional introductory words.
Shawn P McKee: Nothing, for me, and I think sondra has the first presentation right.
yeah.
Mario Lassnig: Okay perfect then thanks everyone for joining and let's go.
Alessandra Forti: Okay.
Alessandra Forti: So i'm going to do a morning touch on some of the stuff a.
Alessandra Forti: Quite a few of you might have heard already.
Alessandra Forti: And I my own sentence introducing a meter data challenges, because here we are talking about monitoring and we have.
Alessandra Forti: I don't know if, for example, the young people have a connected, but we have.
Alessandra Forti: also done and I would like to specify disease for on the.
Alessandra Forti: wsj dot challenge.
Alessandra Forti: Is not fail, I know that all monitoring for transfers.
Alessandra Forti: Okay, so as part of the planning for a challenge see there are these challenges that have been.
Alessandra Forti: documented there from my wsj Management Board and the challenges that should be every two years from easier on to the start of a child, can he.
Alessandra Forti: He panis, of course, it to increase the transfer rates and from my.
Alessandra Forti: Car I.
Alessandra Forti: The transfer rates until we reach the full bandwidth that is needed for the channel that you see.
Alessandra Forti: This year will be only 10% I didn't put the bandwidth here and I didn't put a discount of numbers because today's about the monitoring and not transfers of themselves.
Alessandra Forti: So you're literally my.
Alessandra Forti: Of these challenges, though, is to prepare and infrastructure.
Alessandra Forti: And the most hated it, we can use these bandwidth.
Alessandra Forti: And the data challenges.
Alessandra Forti: Someone has floors.
Alessandra Forti: Are not drive the expansion of the network past so they're really important detail the the whole thing.
Alessandra Forti: For a peasant the infrastructure is going to go to approach me, step by step, but you should develop a testament to follow you and be a long walk is already there blonde from the evolution and not have grant funds and then.
Alessandra Forti: Having to back off.
Alessandra Forti: So all love for experimentation participate in parallel and all sides should use the experiments productions research for.
Alessandra Forti: These you should go in parallel with the experiments activities so.
Alessandra Forti: If you have a bandwidth and experiments activity will use a fraction and the other faction will have to be filled with these.
Alessandra Forti: Challenges transpires.
Alessandra Forti: We needed to discriminate, of course, in the monitoring and this is one of the monitoring is so important.
Alessandra Forti: Now, the thing that is important, in my view is that who we should be on the common infrastructure for these.
Alessandra Forti: We have already requested resources under the umbrella.
Alessandra Forti: to install those services around the challenges the idea here is.
Alessandra Forti: To be a new, something that I found to magically goes on for, for example, a week.
Alessandra Forti: We started with our class and cms because it's Well, we know biased about that, but my obey is that, then we can integrate their own soil to be under the stairs seamlessly.
Alessandra Forti: And, of course, the monitoring so, which is what today is about.
Alessandra Forti: For me, for me it is the most important part is to get right because.
Alessandra Forti: Of course, if you can monitor what you're doing you don't know what is really happening.
Alessandra Forti: Also, what we are doing now, it.
Alessandra Forti: needs to be considered the baseline for future challenges, so it means that, as I say, the step by step that will incrementally build a monitoring infrastructure, as we go along.
Alessandra Forti: There is of course a larger monitoring different start central infrastructure server and the experiments that use it to various degrees and the blank, we are today from them is to be a dog on beta and not some finger of they're not in.
Alessandra Forti: Our warehouses say the experiments to use these environments degrees, so there is a lot of work to do, because.
Alessandra Forti: They have their own dashboards that might or might not be integrated in the central structure and.
Alessandra Forti: Even when they disintegrated that they use is an event summer use it more than others, and even when.
Alessandra Forti: He is not in the US on Jesus seminar.
Alessandra Forti: using the same tools, it might not be the same information included, so all these needs to be to be sorted out.
Alessandra Forti: and getting.
Alessandra Forti: more uniform monitoring is the work that needs to be done.
Alessandra Forti: So that you don't have to do what.
Alessandra Forti: What might have been in the past that you do have the information in different places and then you teach it manually.
Alessandra Forti: In in some way.
Alessandra Forti: Even if we are not going to do these all at once, I find personally acceptable, he said, some things are left behind, for example, not all experiments now not old information or hey at any rate, and not yet.
Alessandra Forti: it's all fine and put a thing is that we start to be something that can be expanded and one once.
Alessandra Forti: The invoice the parties are ready to do it.
Alessandra Forti: So here is an example on what we have started to look at.
Alessandra Forti: Of course say you have a central infrastructure, but to the sources are.
Alessandra Forti: Different, so there are different databases or different.
Alessandra Forti: places where the D dost order and here is a nice day on water.
Alessandra Forti: resource as it started before I.
Alessandra Forti: have lost the cms and alleged to be hopefully we can get more.
Alessandra Forti: Of these, in these Mr from today.
Alessandra Forti: For example, maybe a nice.
Alessandra Forti: can be added to.
Alessandra Forti: and
Alessandra Forti: Of course, indeed sample you see even attempted to win initial panted to have.
Alessandra Forti: To create the block.
Alessandra Forti: So so some I may say yes, the mother is if it is already blah blah blah it's true it's true.
Alessandra Forti: Bob said naked sts is not the whole story, you cannot reach the data and the work that needs to be done here.
Alessandra Forti: is to discover how much we can reach.
Alessandra Forti: The other side of the coin is, of course, for the site monitoring and then networking it so inbound and outbound speaker and bandwidth of a sudden things and then encouraging people may come up and when the future in the future, I like a pocket marking wonder what is there no worries.
Alessandra Forti: No like cannot all experiments monitoring is equal and opposite sides are equal.
Alessandra Forti: And there are some signs of that.
Alessandra Forti: Dynamic monitoring about easy done.
Alessandra Forti: There are.
Alessandra Forti: weather maps for the regions that.
Alessandra Forti: could be dynamic a others that can be static.
Alessandra Forti: Everyone have their own link so.
Alessandra Forti: That they can refer to, but again, a host or in this case, they are, they are different sources and we don't have a one place, I want to look at.
Alessandra Forti: These.
Alessandra Forti: is really important that we integrate can we integrate into the world to do with.
Alessandra Forti: My be bigger.
Alessandra Forti: Because we will have to extract from different sources but maybe if we have an API in the central monitoring and we can extract and push the data right important these my mates make things easier if they are, they are, we will have a.
Alessandra Forti: couple of talks from my two sides to that subtle ui to class and the T zero later on these.
Alessandra Forti: So, and the disease again and my usual the words of combination to this is no going anywhere you.
Alessandra Forti: know everyone to participate.
Alessandra Forti: And it is by step by step, so.
Alessandra Forti: There is some pressure on.
Alessandra Forti: At least on our clothes and the cms to do it the way I do realize that the other experience to my time it may have difficulties because he may use different setups and they have less manpower.
Alessandra Forti: But, at any rate, hopefully today we're going to have a better view of what we need them what is possible.
Alessandra Forti: and
Alessandra Forti: I want to reiterate that that might be future workshops and tracking progress and because.
Alessandra Forti: We cannot discuss I cannot blame everybody, today you force so.
Alessandra Forti: let's eat so Here are a couple of links.
Mario Lassnig: Thanks a lot to listen to have any immediate question, yes, I see lucha.
Mario Lassnig: used to make.
Latchezar Betev: I was muted i'm sorry about that, so thank you, the standard that was very comprehensive, I have a remark or salty which you said on your slide three, I believe.
Latchezar Betev: That is not necessarily to the monitoring, but to the general three number three to the general running of this data challenges What you said is that it has to happen in parallel with the experiment activities.
Yes.
Latchezar Betev: I have a I have a bit of a problem with that, in the following sense that we especially for the.
Latchezar Betev: let's say for the special cases like the tier one step infrastructure, which of course is going to be also used during these data challenges to make sure that this is it can handle the load.
Latchezar Betev: Is that the interference between data taking activities and the data challenge should be as minimal as possible, so, for example, this need to discriminate you, that means that the data charges, going to be on a much lower level.
Latchezar Betev: or outside of specific data dating activities that has to be that is something which I think we should consider.
Alessandra Forti: Yes, okay.
Latchezar Betev: Like, for example, the just to give an example that this year's data challenge was planned for the last week of September.
Alessandra Forti: Yes, I was meaning shall.
Latchezar Betev: That, that is, during when the some beans are expected from entity, of course, not colliding beings with everybody's going to start taking data so it's not just a question of only have the resources which probably are not going to be a lot, but also the involvement of people so that is.
Latchezar Betev: Taking so that was my remark okay that can be resources devoted to the data challenges being a data challenge it should absolutely not interfere with with the activities and experiment.
Alessandra Forti: yeah, particularly for the manpower, I will get our because this is why we want to be able to add services to run the data challenges, because we have two people who come around them.
Latchezar Betev: Yes, no, no, I agree, I mean i'm just say that this is one additional parameter, which has.
Latchezar Betev: To be taken into.
Latchezar Betev: Sure, I don't disagree with anything you said so just just a clarification.
Latchezar Betev: Thank you.
Alessandra Forti: Thank you.
Mario Lassnig: Thanks Sandra.
Alessandro Di Girolamo: I think, just to say is that is that I completely get on an empowered is the critical but in terms of the I think that, instead of writing in parallel, we should have said on top.
Alessandro Di Girolamo: Because what I think we would like to do here is to really understand how much more we can get so our the sea dream about these challenges is that were able to put stuff.
Alessandro Di Girolamo: On top we understand how much on top, we are putting and we can really put and remove more.
Alessandro Di Girolamo: I think this is the plan more than important, so I completely agree with you, within the hour, but in terms of not interfering.
Alessandro Di Girolamo: Sure, if there is something important in that moment experiments need to say we need to say in such a way we don't do it, but actually the idea would be that there is somewhere as REACH.
Alessandro Di Girolamo: A level that to get it and we crank up the.
Alessandro Di Girolamo: In the to putting these and that that whenever and in case that a problem we can switch it off.
Alessandra Forti: yeah yeah.
Alessandra Forti: decentralization of these.
Alessandra Forti: Exactly to do that.
Alessandro Di Girolamo: yeah okay thanks, so I do them in power button on the rest is really on top so.
Alessandra Forti: yeah yeah my main database, if I make a parallel my my integration same on top, I mean I they are stealing he have to still to run them and why the experiments are doing there.
Alessandra Forti: And, and if you can face the experiments activity actually feel all the bandwidth or the better, I mean we don't need to do this.
Alessandra Forti: So I mean.
Alessandra Forti: Again, these about that we can guys which office and say.
Mario Lassnig: I think the important part series that we have a comprehensive monitoring that shows us which part comes from the regular experiment activity and which is a little part that we have to put in top to feel the pipes.
Alessandra Forti: So that we can we can see.
Alessandra Forti: Exactly what is happening.
Alessandra Forti: And the fact a for the activities where I know the bandwidth and that will be perfect and we could have demonstrated.
Alessandra Forti: and got he continues.
Alessandra Forti: Use rj already have the infrastructure become much more than can use a full time something I need.
Alessandra Forti: Something that he run like a block.
Mario Lassnig: Okay, thank you to have any more questions, I see no more resistance.
Mario Lassnig: If not, I would suggest we move to the experiments dashboards and experiments input, the first one I have on my list is Ellis.
Maarten: Okay, can you hear me.
Mario Lassnig: Yes, very well.
Maarten: I will try and share my screen.
Maarten: I get some problem.
Maarten: Computer.
Mario Lassnig: Give me one second I can show you my tissue.
Maarten: yeah maybe it's faster because I get sucked into some new dialogue that i've not seen before.
Maarten: Okay excellent.
Maarten: So this is the presentation let's move to the next page page two.
Maarten: So, as you probably know, all Ellis monitoring is a Mona Lisa link is there in the title, this is the welcome page of Mona Lisa and also a menu expanded on the left hand side that you can.
Maarten: can get when you click on the greater greater sign that entrance into a smaller smaller sign it, if you want to close it and then for these data challenges.
Maarten: The the most interesting plots are the ones under STD transfers, you will see them at the bottom of the left hand menu, there are also a few other sub menus opened that have other interesting potentially interesting plots.
Maarten: But they are yeah so, for example, to see the total land and when traffic volumes without any separation so between activities okay let's move on to page three.
Maarten: here's an example.
Maarten: Transfer rates plot you see in the left hand menu let's say that they are at school transfers speed and it's highlighted so if you click on that you get a page like this.
Maarten: let's move on to page four.
Maarten: So he you see transfer rates in the last month this taken a few days ago, and you see on the right hand side of the plot that there was quite a bit of unusual activity exactly for the kind of exercise that we that we have in mind for the data challenges in.
Maarten: In the autumn of this year, so the case detail one.
Maarten: have now deployed a 20 gigabit opm link and they wanted to test it so a a data challenge was i've run for number of days it is, it has completed by now.
Maarten: And an effect from the Ellis perspective, the monitoring information was sufficient so So what we have in Mona Lisa is sufficient, however, a.
Maarten: It looks like there, there is a CAP somewhere that prevents reaching 20 gigabits per second it's rather capped at 10 gigabit per second somewhere on the network network path, and this is.
Maarten: Currently, not resolved, so this also demonstrate and eat very nicely right that's the sort of end user tools are not enough to to to understand what is happening exactly.
Maarten: Alright, so far, the test has been successful because, basically, the data rates that Alice needs have been demonstrated, but of course it would have been much better if actually the pipe could have been filled to capacity.
Maarten: let's move on to page five so here's the other plot transfer states, so if you click on that, then you get a plot like this.
Maarten: on any Mona Lisa page you you're always have a lot of options around near the top of the page.
Maarten: So typically called series options and particularly alternative views so if you were to click on that if it wasn't opened already on this page, then you.
Maarten: see that there's a possibility select on the finish states of these transfers done versus failed so let's see some examples on page six.
Maarten: So these are successful transfers in the last month, so various details of course and and and things brought it in in the same in a single picture towards the.
Maarten: The right hand side of the plot you see actually the activity, corresponding to these.
Maarten: kiss the transfers.
Maarten: let's move on to bates seven, for example, failures, so it looks very similar, but if you compare it with the scale of the previous blood, you will see it's like two orders of magnitude smaller than than that, so this is another example, and there were no failures actually.
Maarten: When no recent failures when the plot was made a few days ago.
Maarten: let's move on to page eight that's the last page already so we've seen that say that Alice have tooling to to see what's what's going on and, of course, also to drive these challenges and for Alice.
Maarten: These are the paths let's say that should be tested.
Maarten: And the expected rates, so what you see here is the data to zero is to go to take 70% of the data and 30% will be distributed over the tier one sides of Ellis.
Maarten: And each has a particular fraction that you can see in the in the table.
Maarten: that's it.
Mario Lassnig: Thanks a lot.
Mario Lassnig: of questions.
Alessandra Forti: yeah so The obvious question is where do you keep the data.
Maarten: Well, the data, of course, on the other side in in let's say databases used by by Mona Lisa.
Alessandra Forti: In principle.
Maarten: could be export it to some extent, but it's that's not going to be a trivial amount of work right and let's say, as usual, the the the benefits know the perceived benefits fight will have to be weighed against the cost and available effort so so anything is possible right but but.
Maarten: You have to be Be careful let's say with expectations, so there.
yeah yeah yeah.
Mario Lassnig: Nick.
Nick Smith: I saw on this one of the first slides there was.
Nick Smith: A little folder for extra D, I was kind of curious if you could expand a bit on what your storage element monitoring for streaming transfers is if any.
Maarten: yeah so Alice have since forever very detailed monitoring on every disk server that serves data to to Alice so the so Alice makes makes available a package to have interesting.
Maarten: The usual interesting quantities matrix.
Maarten: forward into into Mona Lisa and then can be made available plotter to the history is captor and all that so feel free let's say to play with this.
Maarten: Afterwards, but I don't think there's any of the other experiments have debt precise monitoring of all all its disk servers.
Nick Smith: So you've actually made a some sort of monitoring plugin for each kind of storage helmet technology that your site's run so.
Maarten: yeah but it's basically only X rudy.
Okay.
Nick Smith: And do you have any comment on, because I know as JI has been working for some time on improving the extra D UDP monitoring.
Nick Smith: Software because I guess, there was some.
Nick Smith: There was some report or otherwise worried that it was incomplete.
Nick Smith: Are you following that or is that related to this.
Maarten: I am not aware let's say have have have that discussion right, I mean maybe lectures, I can comment, but I think that say that, from the perspective let's say.
Maarten: We we have let's say what is what what is desirable let's say for Alice operations right, and of course it good the software can serve as an example.
Maarten: For for for for other monitoring packages Mona Lisa, by the way, is open source it so so it's it's it could be considered, because it has been it has proven itself over almost 20 years.
Nick Smith: Thanks.
Latchezar Betev: Maybe what I can comment about that.
Latchezar Betev: There are two ways to two distinct paths which we are monitoring the information and then it is cross checked, one of them is the low level information from every server, and that is.
Latchezar Betev: Monitoring on the network interface, or in principle, since it is already packaged together with us and with extra data comes for free if it's configured.
Latchezar Betev: And the second one is that we are monitoring all our clients and that there we are extracting the information directly from the extra the API.
Latchezar Betev: And this is also plotted, for example, this network traffic which marketing was Martin was talking about, we know exactly which clients and local which are remote and helmet state is read by every single class, and that is aggregated so we know this, we can have this information.
Latchezar Betev: But since we don't use anything else but X would be for all our data transfers, including.
Latchezar Betev: point to point third party copy transfers than we do here in the percent of all the bikes read and read by written by by our clients in Malaysia.
Mario Lassnig: On this actually have an operational question in your in your day to day.
Mario Lassnig: Data management operations life, do you have people actually looking at the success and failures between the actual storage storage copy and the.
Mario Lassnig: The job success rate when they need to stream it from.
Mario Lassnig: far away, or is it basically they go to this one, Elisa page and look at it once a day or how does it work.
Latchezar Betev: How does it work okay so, for example, this test.
Latchezar Betev: which you were talking about food Christie that was followed day by day by several people just make sure that everything is fine, because.
Latchezar Betev: It is a fairly important test for the rest of them again Martin has chosen a very good example Steve Martin, you can go back to the failure rate plot, I think it was.
Latchezar Betev: This one here okay so here, you see that there is a week what we usually what you usually see on this, this is all third party copies, and this is effectively.
Latchezar Betev: Copying have more storage to another, for example, evacuation or something like that, and you see that there are certain amount of failures, and these are followed the failures are followed.
Latchezar Betev: So why did why didn't you were not able to copy corrupted data or fail this course something like that that is something which is followed on whenever whenever this kind of transfers.
Latchezar Betev: From user client bases and that is typically something which affects the jobs.
Latchezar Betev: This is also integrate that with marissa and then the users typically say well listen, I mean i've run my job by that wanted to read hundred thousand thousand.
Latchezar Betev: And 5000 of those who are not accessible, because that is visible from the errors in the jobs.
Latchezar Betev: And that is something which will also follow because that may not be necessarily that the entire storage has failed, but they have a failed part of the storage we don't know about, and that is the way we discovered thing I don't know whether that doesn't your questions completely.
Mario Lassnig: yeah that's good thanks all right.
Mario Lassnig: Do we have any more questions.
Mario Lassnig: Yes.
Alessandro Di Girolamo: I have another one that related to the date.
Alessandro Di Girolamo: And do you simulate monitoring water for the diabetes, including the Mona Lisa.
Latchezar Betev: yeah the second.
Alessandro Di Girolamo: angle in this country.
Latchezar Betev: That is there, however, however, what we do not see the take itself, we see what goes to the tape buffer.
Alessandro Di Girolamo: yeah sure that's the same for others.
Latchezar Betev: Yes, we have we have exactly the same for.
Latchezar Betev: You so, for example, this thing which you see here is a tape sort of this is the, this is the tapeless custodial storage it's a database, but it is exactly the same for for all the paper cups which we have around the world.
Alessandro Di Girolamo: Okay, so so anyway, let me say 21 is for the dismantling one is for the disk storage.
Latchezar Betev: Know they're they're all the same, they are not we don't discriminate I mean they are discriminated that you know which one is which but, at the end of the day, you have exactly the same monitoring information for you to go.
Maarten: yeah but but elizondo is right so so so let's say.
Maarten: To date, storage elements are distinguished from the disk storage elements.
Latchezar Betev: They are, but by name and functionality it's not by monitoring.
monitoring.
Maarten: At first glance, you couldn't sit down from the monitoring, which is which.
Alessandro Di Girolamo: that's Okay, I just wanted to know, because, of course, you might be different than we might want to have the stress test separated from this that wasn't.
Latchezar Betev: No, no you're no you're right you're right, I mean, as a matter of fact, all the past which we have shown on the last, but these are.
Alessandro Di Girolamo: tape archives we.
Latchezar Betev: don't we don't have any.
Latchezar Betev: Interest in stressing the disk storage is they're used as they are used in the tapes are a lot more kind of finicky and they require a lot more attention.
Latchezar Betev: than the disk storage is in terms of.
Latchezar Betev: How, how do you put the data in what is the right there because you don't want to overload it, and all this kind of stuff and that goes back to my first command and during the meeting is that, especially for this kind of starts, you have to very.
Latchezar Betev: Much follow up is the what is the experimental activities before you schedule any data challenges on a date based storage.
Mario Lassnig: Okay.
Great.
Mario Lassnig: Thanks a lot, so I will stop the share now.
Mario Lassnig: um let's see next one on my list.
Mario Lassnig: I have alleged to be.
Mario Lassnig: As stuff will you.
Christophe Haen: hear me.
Yes.
Christophe Haen: Alright, can you see the slides.
Mario Lassnig: Yes.
Christophe Haen: Okay, great so I basically tried to keep it extremely short answer the very specific question, who asked which is basically the same as that is so we.
Christophe Haen: did not put a lot of snapshots from the monitoring, but just a summary of what we have and how it works.
Christophe Haen: So all our monitoring data is stored in our direct aid cpr portal and most of the actual contents is not in minute, but it's in some private database to already answered the question, although we have a wish to one day export everything to money rather than this private database.
Christophe Haen: There is no effort in that direction for seen at least for the.
Christophe Haen: For the coming data challenge now in terms of information we have, there we of course have.
Christophe Haen: plots for the direct access and so from the jobs from the command line and that kind of stuff and as well as for the fts three transfers.
Christophe Haen: So the information we have for fts three is roughly equivalent what is in the in the dashboards provided by fts.
Christophe Haen: And we have very similar information for the direct access so proportion of failures or success with which protocol with the different endpoints and throughput although also for tape but, similarly to Alice we don't work there is only be distinguished by names and function.
Christophe Haen: So all those information are there, but as I said on the in this in this portal and we'll ask if we have network plots and no we don't have and we do not intend to have because, from our point of view, this is purely an infrastructure.
Christophe Haen: issue, so we really don't see fit for for a middleware monitoring that say.
Christophe Haen: So that's all I have, I think it answers already quite a bit.
Mario Lassnig: Of.
Mario Lassnig: questions.
Nick Smith: You for the direct storage access does that mean you have some sort of damon at the storage on three sides collecting information about local access.
Christophe Haen: Know it's when we use so when we use the direct client to access the data, so if.
Christophe Haen: Obviously, someone is using directly D file commander etc CP command to we won't see that, however, whatever goes through the Iraqis is reported to Derek.
Nick Smith: Okay, so say client side collection.
yeah.
Brian Bockelman: So I certainly understand the the the thoughts around having network being an infrastructure issue, but it also strikes me that that is a.
Brian Bockelman: pleasant state of being that if.
Brian Bockelman: That might be related to not pushing the network's past their limits, do you as you look at run three and and scaling up a village CB are you increasingly worried about hitting bandwidth limits, so do you think that will they make you more interested in network side so.
Christophe Haen: Unless I can only command for myself right, but clearly network is one of the worry, I personally have but, at the global scale, simply because, as far as I know.
Christophe Haen: network is the only aspect of the infrastructure, which is not Plato doesn't have any fair share or whatever.
Christophe Haen: For the rest, you know that at least gets a certain fraction and is get another fraction and knowledge knowledge balances for the network is the Far West.
Christophe Haen: And we've seen it happen in the recent past that an experiment is doing some tests and the other experiments collapse basically so it's from this aspect that i'm sort of florid discussing with the sites.
Christophe Haen: Every now and then they tell us that.
Christophe Haen: They are planning an increasing network and this and that, so it should not be a problem and it's true that if you look out there and network becomes cheap and that kind of things, but it still remains.
Christophe Haen: An infrastructure concerns for us in the sense that we're not monitoring the fair share of a given site right so.
Christophe Haen: And we would know anywhere what to do with all those diagnosis we're not a network experts we're just customer of the infrastructure, so in that aspect monitoring it on our side does not make any sense we monitor the outcome, the transfers.
Christophe Haen: But not the infrastructure itself now, I completely agree with you that there should be a fair and decent monitoring for for the network.
I hope I answer your question.
Alessandra Forti: I think that part of what we're doing here is convenient also.
Alessandra Forti: an infrastructure that allows if not forced to get the first shares to to help them with the problem and it just highlighting, which is that a fair and experimented does some tests, they know that experiment collapses, I mean this is.
Alessandra Forti: One of the things that we are, we are supposed to so.
Christophe Haen: yeah that's why I understand that there is a need for network monitoring i'm just saying that we don't collect it on our side that's what i'm.
Alessandra Forti: saying so, no, I understand that I was just saying that part of what we want to know is, is is.
Alessandra Forti: And that's good.
Mario Lassnig: So, Christopher i'm considering that a global network monitoring is maybe not on the menu for the next couple of months.
Mario Lassnig: If we would be able to put some manpower into grabbing the data from the LGBT Fuck database and shuttled it into monitor, I think that would be a good first step know.
Christophe Haen: Possibly now.
Christophe Haen: This so, the truth is that I think we've asked for a technical students for that so Certainly, this would happily blend in, but I am not the one who asked to the technical student, I can tell you what exactly the project scope is, but it certainly is we'll be happy to discuss it for sure.
Mario Lassnig: Okay, that sounds good thanks.
Mario Lassnig: Any more questions.
Mario Lassnig: I don't see no more raise hands Okay, so thank you.
Mario Lassnig: Next, one on the list is a shared slide deck your cms and atlas.
Mario Lassnig: um, so I would suggest, since the first few slides or SMS that actually Nick starts.
Nick Smith: yeah sure can go to the first one.
Nick Smith: So basically only.
Nick Smith: Put one when when Mario asked for some slides I only put forth basically this one dashboard that collects basically summarizes the.
Nick Smith: The metrics collected through the fts servers we do have some others that monitor things like ratio cubes and things like that, but I thought those.
Nick Smith: might be outside the scope for this purpose we really only care about you know the throughput and fts state because, for the most part of us feel immediately offloads it's Q into fts and so as long as fts can push the files, you know that's that's the real.
Nick Smith: that's the sort of.
Nick Smith: let's say separation of concern right.
Nick Smith: So this one just basically allows you to group by the source and destination rtc name, you know what mts server is being used because we do occasionally use other fts servers but Besides, there are certain instance, we have one fermilab and rel and occasionally also wanted MIT running.
Nick Smith: And then we grew up, of course, by activity at the different.
Nick Smith: Reasons for the transfers and yeah that's that's what you see here, you have the throughput success failure efficiency doesn't isn't the prettiest part right now, because we're grouping by.
Nick Smith: keys with rather low cardinality so it's kind of a mess, but if you summarize it using CRC it would look better next slide.
Nick Smith: Just more of the same dashboard actually we have this efficiency matrix which just you know source and destination calm.
Nick Smith: Our columns and rows and it just summarizes the efficiency of of the of the links between those two and it's a nice it's a nice way to visualize quickly you know any particular site is misbehaving because you know if all columns or.
Nick Smith: or read for one row, then you know it's it's definitely an issue with the row, and if it all.
Nick Smith: grows or read from one calling, you know it's it's an issue for one of the columns.
Nick Smith: i'll just make a public service announcement that this is actually using a different, this is just an influx db data source, rather than the elastic search data source from on it.
Nick Smith: Because apparently the elastic search data source, you cannot create such a table in groupon for technical reasons, so I think there's an issue on it, and I do hope it gets results, so that we can move off of influx.
Nick Smith: Q statistics basically was thrown in here at one point, because we saw that we.
Nick Smith: We had gotten into a situation once where so many.
Nick Smith: Transfers were submitted to fts that they all got stuck in the sort of.
Nick Smith: In one particular state and so.
Nick Smith: What we do now, as we monitor state transitions and fts server to make sure that at least it's doing something at any given time.
Nick Smith: And the next page.
Nick Smith: is just more summary of the volume transferred, you know by whatever thing you've grouped by and files and Bytes and air details and that's basically the extent of our our transfer monitoring.
Nick Smith: Police the summary portion of it, of course, the details, you would go into coupon or the mts server to see the the actual error messages, but this provides a nice summary to see what's going on in your clients.
Nick Smith: One thing that's not shown here is AAA, which is our extra D federation, we do do a lot of streaming transfers and cms.
Nick Smith: I would say the extent to which they are monitored is not not ideal.
Nick Smith: We do have some client side monitoring, but we have a very inhuman in homogenous set of clients, we have both the same as software, where we have people's custom.
Nick Smith: analyzers that are independent of cms software and we have many different access protocols.
Nick Smith: Many jobs land on sites may actually use a direct storage access through a publix mouth, without even going through things like kickstarter D or or.
Nick Smith: or the other.
Nick Smith: Protocols so it's rather difficult for us at the moment to to capture all of our streaming just because of the diverse nature of the access methods and clients, but we hope to fix it.
Nick Smith: I think that's all.
Mario Lassnig: Okay, thanks, Nick do you have any immediate questions for cms otherwise I would continue with the first part, which is roughly similar.
Mario Lassnig: Okay, I don't see anything okay let's jump to the next slide so in atlas we do basically exactly the same thing.
Mario Lassnig: We have built this life page where we collected different to transfer monitoring select the the different fruits for cloud productivity, the rates and everything and what you see here are essentially.
Mario Lassnig: was his money generated to PNG that at an export it and shown here, so this is what the shift or slip like the shifters don't don't go to the actual monitoring service itself they look at this aggregation.
Mario Lassnig: But then when you go directly to monitor you see here that it's practically the same as cms with a little bit more selection.
Mario Lassnig: buttons on the top, this is because we're not using the fcs events directly but we're using the Russa events which take the fcs events enrich them a bit and then, in addition to enriched by the monitor infrastructure on top, so we get a little bit more.
Mario Lassnig: selection.
Mario Lassnig: per view you can see this from the different matrices transfer tables staging deletion, everything is in there, so.
Mario Lassnig: If we open up the transfer clause, for example, you see that this is practically the same.
Mario Lassnig: So.
Mario Lassnig: Then we also have the live monitoring for Russa next to the transfers, this is the internal state of how we see the traffic on the network from the receiver point of view, so this is the cues.
Mario Lassnig: The request the successful submissions how many of those awaiting homing submitted to fts, so this is what Nick previously mentioned as the as the monitoring that they also have internally.
Mario Lassnig: So if we zoom in here we have we have the view basically at the actual resolution level that we're using the system so.
Mario Lassnig: Everything is available in.
Mario Lassnig: I think this is in flux right i'm pretty sure i'll be able to correctly.
Mario Lassnig: And last because in Russia it's possible to kick off a petabyte scale transfer with a single rule we also have what we call this rule monitoring system.
Mario Lassnig: That basically shows you the progress of the rule over time, so there you see exactly how many files are in there, where they're going to what the what the rule points to and also.
Mario Lassnig: we're on the dashboard you.
Mario Lassnig: could look at it.
Mario Lassnig: and which fts job is taking care of that, so this is this is for this very large transfers that we might want to do in the data challenge, where this this will practice continues to follow the progress of.
Mario Lassnig: That was it.
Mario Lassnig: I have to see questions again questions.
Mario Lassnig: Richard.
Rizart Dona: Yes, hello, so I just wanted to make a comment about Nick so since a couple of months ago, it is possible, in reference to create this matrices with elasticsearch just that the SOS.
Rizart Dona: And we've done this in escape So if you if you need access, I can give you access, so you can take a look it's basically this new transforms.
Rizart Dona: functionality, that you can use.
Rizart Dona: But it is not possible and that's what I want to say.
Nick Smith: Oh that's great news i'm glad I complained because I get an answer.
Rizart Dona: yeah So if you like, Ping me, and I can give you access to from take actually a look up on how to compete with.
Mario Lassnig: allison.
Dewhurst, Alastair (STFC,RAL,SC): hi.
Dewhurst, Alastair (STFC,RAL,SC): And so I was just interested in the ratio rule monitoring, you had at the end there because something that is always.
Dewhurst, Alastair (STFC,RAL,SC): Or can be a problem is when you see a slow network speed, or whatever you like well, is it just because there's any you know small number of transfers left to do.
Dewhurst, Alastair (STFC,RAL,SC): Or is there a genuine bottom that they're all we just in the tail of events and that's why I did it started to drop slowly so.
Dewhurst, Alastair (STFC,RAL,SC): Is there then away from this because I mean it shows locks but in terms of having a monitoring where it's like yes atlases scheduled, you know petabytes of data to be transferred to your site.
Dewhurst, Alastair (STFC,RAL,SC): would be an interesting thing to know compared to you know there's you know 10 terabytes left to do because you know that if it's if it's a lot, then you would expect the network to basically be.
Dewhurst, Alastair (STFC,RAL,SC): And what it to be full, whatever the whatever the definition of color in that state So is there this monitoring, you know.
Dewhurst, Alastair (STFC,RAL,SC): And can't be expanded for cms to.
Dewhurst, Alastair (STFC,RAL,SC): And in order to get it, because this particular page doesn't show it, but then, if you do if you know the rule surely it's not difficult to calculate how much there is to do.
Mario Lassnig: yeah so there's a subscription monitoring and a rule backlog.
Dewhurst, Alastair (STFC,RAL,SC): monitor.
Mario Lassnig: I don't know Martin Thomas if you want to comment on this because I don't particularly know how the field.
Nick Smith: I would venture to say that the same information could in theory be extracted from fts on because pretty much immediately the entire set of applications that need to happen to satisfy the real issue will be.
Nick Smith: resolved into a transfer costs incentive to us and then it's a matter of monitoring, they have to ask you to see forgiven source destination pair of what the queue is.
Mario Lassnig: Right, but in this case, you want to know the progress of the rule.
Mario Lassnig: And you can query for that, through the fcs.
Nick Smith: Well, I mean, I think.
Nick Smith: The Chi I assume the comment was coming from a site admin.
Nick Smith: perspective, where they don't particularly care about the rule, but they do care about their their sites Q.
Nick Smith: I mean, I agree, you know for a user the real monitoring is more important than this IQ but for seven minutes it's the opposite.
Nick Smith: I guess, then the question is data because.
Dewhurst, Alastair (STFC,RAL,SC): I was under the impression, certainly with things like data carousel that often there was a lot held back in ratio and not submitting immediately to fts maybe maybe i'm I mean correct and in 19 99% of the time if it's in the FDA is that shows the kind of the backlog the do knows about.
Dewhurst, Alastair (STFC,RAL,SC): Can anybody column a is, is it the case that it should I don't know documents and things that the fps can can hold.
Mario Lassnig: I don't know by heart.
Mario Lassnig: Do we have anyone who knows.
Dewhurst, Alastair (STFC,RAL,SC): Okay, they may.
Nick Smith: have seen numbers, up to a million.
Dewhurst, Alastair (STFC,RAL,SC): Okay, so maybe that is the thing that is yeah just from the inside.
Dewhurst, Alastair (STFC,RAL,SC): So, having point of view, actually it's it's an fps monitoring thing.
Mihai Patrascoiu: I can comment on deaf ears backlog, and unfortunately we've seen even up to more than 2 million, but it's not actually something desirable.
Mihai Patrascoiu: But if it can happen.
Mihai Patrascoiu: It has happened.
Mario Lassnig: Thanks.
Mario Lassnig: To have more questions.
Mario Lassnig: If not.
Mario Lassnig: We have a look at the agenda.
Mario Lassnig: So next up, we have the central monitor team so who will give the presentation Pedro is, are you.
Pedro Andrade: Hello yeah it'll be me.
Mario Lassnig: Great thanks.
Pedro Andrade: Okay.
Pedro Andrade: So let me share my slides.
Pedro Andrade: So, can you see.
Pedro Andrade: Yes.
Pedro Andrade: All right, thank you very much, so let's.
Pedro Andrade: start this presentation.
Pedro Andrade: So, first of all thank you for for inviting us to present the the infrastructure in this workshop and today.
Pedro Andrade: we're presenting a little bit of which they do we have in wanting, and how we can use the 94 structure.
Pedro Andrade: Basically, the presentation will be will be split into the parts and the first one, I will try to give an overview.
Pedro Andrade: of which data, you can find today in monitor common data, like the velocity experiments specific data or even other data sets that might be relevant in the scope or for the scope of this workshop.
Pedro Andrade: And then it's the second phase, one of the details of how you can use the Multi cluster to actually maybe integrate new data sets or access those or existing data that is already today.
Pedro Andrade: Starting with the wc date in Monaco, and so I think it was quite good that we have already seen before my presentation, a few very concrete examples of dashboards that are out there.
Pedro Andrade: Today, so today I won't be actually giving many details on exactly dashboards that are available it's more of an overview of the type of data that is available, providing links, so you can actually go and explore by herself, the data that is there.
Pedro Andrade: And from the wcc point of view, there are basically two main categories that we provide.
Pedro Andrade: And this is kind of all the data that is managing is managed by the money team and is provided in a dedicated fund organization.
Pedro Andrade: The first set is about transfers, and this is monitoring of sight transfers, as reported bias, yes, the an extra fee.
Pedro Andrade: And then we offer dashboards one which is a consultant of a global file transfers of view we're putting together fts an extra fee, and then, then a dedicated dashboards one for fts and another one for for absolutely.
Pedro Andrade: From the point of view of sight monitoring and here we are talking about the data that we manage and we store and we provide dashboards as reported by ETF.
Pedro Andrade: So these are the best executed by the experiment test framework.
Pedro Andrade: And in addition to that, we also have since ETF for only the only test atlas cms and relatively we also integrate I dedicate the others producer of test results which will put together in order to combine to create these combined view.
Pedro Andrade: And in terms of dashboards, then what is provided is basically one historical view of the profile publication, where we compute availability and reliability.
Pedro Andrade: For the different sites and services and then another view which provides the direct access to the role test results from the site, my point of view.
Pedro Andrade: For this particular case, we also generate the most of the reports for wg which basically just clear create a monthly view from off the availability and reliability of the different sites in a PDF format.
Pedro Andrade: These two main categories of data, which is a common tool, the wsj experiments and have this dashboard that I just mentioned.
Pedro Andrade: stored and available from the wl CG get funding organization, so this is a dedicated organization that puts together everything which is why did you use specific.
Pedro Andrade: You can find the link for this organization, the top right corner there and then you can actually see the different dashboards that they were just talking about before basically the ones on the left side of the screen.
Pedro Andrade: So this is what we offer today, which is the common to all the wi fi experiments if now we go.
Pedro Andrade: experiment by experiment on which they do we can find today in wanted starting first bye bye atlas.
Pedro Andrade: That that are kind of.
Pedro Andrade: different topics from different types of data from different titles activities things like john monitoring, where we take data from Panda and the cirque de Su at the base.
Pedro Andrade: As we saw in the previous presentation did em transfer data from Russia events and tracy's there's also the DM account information which is.
Pedro Andrade: Lucia data which is only admin account of is available in each of your first and then he's integrated in wanted.
Pedro Andrade: And then there are other types of data, like coordinate the solution promiscuous metrics or even the logs or kinds of chemicals.
Pedro Andrade: That are metrics which are more from the perspective of a site staples board that.
Pedro Andrade: put together some of the metrics mentioned before, plus if you add a specific metrics that are higher level service availability metrics and then other data sets which are more specific, to a certain after services like the event index or even service.
Pedro Andrade: Also, all these different data sets they had available like four days of the velocity in one specific organization, and this is the homepage of that organization, where you have the links for all the different production dashboard available.
Pedro Andrade: And this is the starting point to explore the database.
Pedro Andrade: and moving on to cms and what is cms specific in one for the case of cms is it, similar to what I just mentioned, for others.
Pedro Andrade: That are data sets the job monitoring from the metrics.
Pedro Andrade: metrics about a and Russian similar to the ssd data in adwords for the case of cms, there is the ssd dashboard which provides site related information.
Pedro Andrade: And then there are other kind of more fine grained the building and cms metrics like a digital medics double agent, who gave and if you are able to reach this is not the full list.
Pedro Andrade: And we also need to do anything the gaze of cms several logs from different cms services like submission frustrated factory cms web or the new cms web version running on committees or even cut out for example.
Pedro Andrade: Also, similar to the blacklist from the cms we also integrate in with cms metrics related to the availability of their services.
Pedro Andrade: Once again, for the case of cms there is also one cms specific organization in one graph on the link is also present in this.
Pedro Andrade: Here, and that it can see very well organized the homepage where you can find all those different cms services and the different dashboards for Jerusalem for different purposes.
Pedro Andrade: Finally, looking a bit outside of things which are common to all the experiments with the w oC G view or things which are specific to the to each different experiments, and that is of course also other data sets in monitor.
Pedro Andrade: Maybe one thing, important to say is some data which is related to relates to certain under the velocity network here, what do we have today in of metrics available through our llc one and the eligible bn metrics as well as some as well as some persona matrix.
Pedro Andrade: And then, also from the certain data Center point of view, specifically for the Center the Center that are English and, obviously, you can also find information about the disability Center post metrics and the labs.
Pedro Andrade: Starting at the hardware level os independent service level sort of selfies specific metrics about services at the running the Senate, the Center like much cloud.
Pedro Andrade: Plus and this type of this type of services in mind there's also information about service availabilities kpis and their lives for most of the of the IP services that run individually Center.
Pedro Andrade: And in addition there's also says log data gathered from all the data Center notes, plus some logs which are service specific.
Pedro Andrade: In this case, this data is offered through one dedicated organization called money.
Pedro Andrade: This is the default organization that provides users access to the data sets I was talking about before you can see here dashboards like host metrics service availability and then one on the right side SL a's and kpis specific.
Pedro Andrade: And I said this was more of a of an overview of the data which is available today in monitor i'm not I don't have actually particular.
Pedro Andrade: screenshots of dashboard so you have seen them some of them before, but I hope that, with the links above and with this items, you can then have an idea of the record is there and then export a bit more details.
Pedro Andrade: Moving on to to the second part of the presentation, I would like, maybe, not to say a few words about.
Pedro Andrade: how you can use the modern infrastructure, and this is already the case for many of us, since several years now.
Pedro Andrade: And started, of course, with the way to ingest the data in the monitor infrastructure, this is a well established procedure and we are integrating.
Pedro Andrade: A lot of data over the last two years, through these kind of very standard endpoints so basically you can send to the modern infrastructure or any type of wondering, will it will events such as a metric and align with the log we offer several months, so you can figure data via http.
Pedro Andrade: Data can also be sand using collect the plugin but this particular option is mostly used for things running into certain data Center.
Pedro Andrade: And you can also expose your data through a promiscuous service that then we will scope or data can be put in a messaging profile, which will then we will we will gladly and get the data from there, and actually in done if data has to be sent from outside CERN.
Pedro Andrade: Only the view option is to.
Pedro Andrade: win but, in general, we believe that the combination of these four different options should be enough to sweet, most of the use cases.
Pedro Andrade: Also, also thinking that when you are free, for most of these cases to choose which clients, they want to choose which one to which client you want to use in order to send your that.
Pedro Andrade: This can be from your own developer tooling bison or using some existing tool that can talk to one of these different and.
Pedro Andrade: Once the data is there.
Pedro Andrade: In the process of bias for the existing data sets of four new the tip of the new data sets that may be added in the future and data can be accessed in different ways.
Pedro Andrade: The macedo more standard options thinking on a user that wants to plot, the data or wants to create a dashboard.
Pedro Andrade: That I want me know commendations to use good I found a ui, this is what actually I was showing the previous.
Pedro Andrade: In the in the previous slides with all this home pages that have links to many different documents, so all of that is going to find the ui, and that is where you can find different Watson dashboards and says, I would recommend that to to create this type of.
Pedro Andrade: coverage of dashboards if you want to do service discovery and one with option is to use the good news of the foods, which is quite good for for that type of work.
Pedro Andrade: And for programmatic access our recommendation is to use either the funding https or http proxy.
Pedro Andrade: Which is something that comes together with your funnel and allows this type of programmatic inquiries or you can go and some other ways to go do just one and have my notebook or the other way to to get the data and come and process it.
Pedro Andrade: One thing which is important to say is that kibana and swan and those are two services which are managed by other items and not the monitoring still this this integration is there and they can also be used.
Pedro Andrade: And one thing that I would like to mention, which is a question that comes very often these in terms of data retention and the different types of studies that we use.
Pedro Andrade: So basically what we what we have in terms of storage is the technologies that are you touch that are showing me this slide.
Pedro Andrade: We use Kafka for for transport transport so as a buffer of the data, and this is why the raw data that is kept only for for three days.
Pedro Andrade: And then the data can be stored in different back ends.
Pedro Andrade: In some cases, that it is holding our six search and that is great, except for something between one and three months maximum, even if the default is one month, so there are some exceptions where did they just take a bit longer for three months.
Pedro Andrade: And then I can get that data and can be started there something between five to 10 years and we have already several use cases like, for example, the velocity transfers, what we have many years of.
Pedro Andrade: updating it starting our search another tool that we use for some use cases and easy folks to be that only aggregate the data is kept using influx db continuous grading functionality that allows to aggregate the data continuously and then they can also be gap between five to 10 years.
Pedro Andrade: And finally, data can also be started the hdfs Of course this is not for blocking for discovery, this is more for archival or for offline analytics of the data.
Pedro Andrade: And then we are keeping the raw data for 13 months and also be compliant with it policies for the for later for data protection, and so, at the very Skype for three months, but it can be kept for longer if anonymized.
Pedro Andrade: final word.
Pedro Andrade: about the infrastructure is this capability that infrastructure offers for doing the processing of the data and, as I was saying before all the data into it in mind, it is buffered through cash.
Pedro Andrade: And because of that we have a processing infrastructure that allows that allows us to read from Kafka this data coming.
Pedro Andrade: From all these different data producers and process this data industry mean we and.
Pedro Andrade: We also have the data in hdfs the same developments Defense which also allows us to use the processing perspective on your match rate.
Pedro Andrade: But basically, what we do in terms of processing these we apply we run different types of jobs that will do different types of activities like they think Richmond.
Pedro Andrade: Where, for example, the most used or the most common example he is doing original date that we topology formation and we also have.
Pedro Andrade: Jobs running this process infrastructure that are doing the aggregation of the data so having receiving the raw data in a very.
Pedro Andrade: Fine grain with verifying that resolution would run jobs that aggregated data in something like one hour beans also jobs can also be run for doing standard video processing.
Pedro Andrade: Most of our jobs, they are spark ways, and they also, as I was saying before they kind of running swimming in batch mode, but we do have a few logstash jobs also running there that actually take care of more simple use cases.
Pedro Andrade: One thing which is important to say is that, in this processing infrastructure and today the monitoring runs.
Pedro Andrade: A few dozens of jobs for the wg use cases and for large experiment use cases.
Pedro Andrade: But users can also have the jobs being run in this infrastructure so as a user, you can also send your data want infrastructure and then also run your job here in order to process your data so it's something that we're always happy to discuss those those use cases and provide a solution.
Pedro Andrade: Then that's the basic what they have to say, for now, I like to thank you and that and, of course, to say that we are happy to discuss with you.
Pedro Andrade: Any any follow up in any other collaboration that the has to be put in place for the the scope of this work.
Mario Lassnig: Thanks a lot.
Mario Lassnig: questions.
Mario Lassnig: Pretty incredible because I have enough, then, let me start with mine.
Mario Lassnig: Right now, we have the ws EG organization that I saw on the monitor dashboard.
Mario Lassnig: overview right, so if we were to start with a.
Mario Lassnig: What is right now, this fts transverse dashboard and we want to enrich it with the information that we have from allege CB cms and atlas.
Mario Lassnig: To get a little bit more.
Mario Lassnig: selectivity like we have in the Atlas dashboard How would we go about this.
Pedro Andrade: talk today the MDS data that you see in this dashboard is already been enriched so we get the fcs documents directly from from the FDA is.
Pedro Andrade: demon from the FDA so servers this day is done already enriched with topology so that's the current enrichment that do that today we are doing got information outside squads revelations that apology related things and we already have a.
Pedro Andrade: spark job running this processing infrastructure that there was talking about before doing it, and if there is the need to provide further enrichment with additional metadata.
Pedro Andrade: That for sure is something that can be done and we had we were happy to discuss in that case, what we have to understand is what is the source of this excellent Meta data that has to be reached.
Pedro Andrade: What are the common keys that we have to do in order to do the efficient the nourishment correctly, so I think once, that is, that is clear, and we have access to this new funding sources it's been put in place.
Mario Lassnig: So in principle you could also operate on data that you read yourself it's not like, we have to push everything you could also read directly from the tables to exist from the systems right.
Pedro Andrade: Well, also the Meta data that is used for enrichment should also have been created wanted so Ideally, we have the documents.
Pedro Andrade: In wanted from us, yes, which is a good, we have the topology Meta data already in wanted to do, and this is what we are doing we're doing digital.
Pedro Andrade: If the additional modalities needed, it has to be integrated in one and then from our side, it will be a matter of extending the REACH much job to also take in consideration detrimental.
Pedro Andrade: So yes, you will definitely work much better if that metadata which is required is push the different experiments into one.
Mario Lassnig: Thanks.
Alessandra Forti: So one of the problems scorsese inside the which data we need.
Alessandra Forti: For, for these are we are, we are a.
Alessandra Forti: Time to understand a we're trying to do with a with a test.
Alessandra Forti: For them, rather than discussing discussed on the ticket.
Alessandra Forti: Starting from the current.
Alessandra Forti: dashboard.
Pedro Andrade: yeah I think that that approach makes sense, I think it's definitely a good idea to start from this dashboard.
Pedro Andrade: For seven minutes, then from there understand the which fields which additional fields will be needed will be meaningful and then understand how that information can be, if not in one product information can be integrated in one.
Mario Lassnig: Is there anything.
Mario Lassnig: But just from the top of your head, that would be really difficult to do.
Pedro Andrade: Well, as usual, the complexity is more on the virginity of the data and being sure that the different sources of information coming from the different experiments.
Pedro Andrade: and sent to monitor data coming from these different sources in it in a way that then can actually manage it all together so that's usually our biggest difficulty trying to build this type of common dashboards.
Pedro Andrade: But I think what was just being said before, if instead we know better, from the dashboard point of view, what are the common things that would like to lock into show that simplifies that exercise in that discussion.
Pedro Andrade: But I say from the rest in terms of inserting data from the technical point of view of inserting data processing data, making it available having dashboards.
Pedro Andrade: I think, from my point of view, the infrastructure is Danny stable, and this is being run for several years and I don't see that there's, the main challenge main challenge is usually ability that the link itself at the content at the schema level.
Mario Lassnig: Okay sounds very good to me um any more questions.
I.
Mario Lassnig: Is Nick.
comment.
Nick Smith: I think, generally, you know.
Nick Smith: Including additional metadata in the FDA is transfer.
Nick Smith: In some senses, a bit easier than joining it later right and maybe we should keep that in mind, the, for example, for cms case once we added the source and destination RC labels which were India fts metadata i'm ready in the aggregated information that basically solved.
Nick Smith: All of our enrichment issues, because the rsc name.
Nick Smith: As a pretty strict encoding that tells you the country and things like that.
Nick Smith: So there's point out that you know there's some advantage to including extra metadata and the transfer.
Job itself.
Pedro Andrade: Actually, I fully agree with me it's a very good example of how to how to how to solve a problem in the right way, and yet sometimes going on complicated processing processes.
Pedro Andrade: And enrichment processes these and you need some time lot more efficient one, and if the problem can be fixed at the source, with the raw data that's definitely a better way to go.
Pedro Andrade: But I think there's also analyzed from the perspective of which.
Pedro Andrade: Which dashboards want to be created which type of tax to be there and how we want to do the buys and these type of things that's what kind of shoes do should guide the discussion on on which doesn't need it.
ilija: First question.
Mario Lassnig: yeah go ahead.
ilija: Sorry, I joined a little bit late but.
ilija: For example.
ilija: Extra the transfers this data is collected, currently, this is not really usable.
ilija: Is there somebody who, who is committed to work on to get it posted it works and actually collects the proper data from both cms and atlas I mean I i'm afraid there's too much.
ilija: Too much time spent on sts when, for example, cms transfers large fraction of their data using extra.
ilija: So, could we is there, somebody or, should I can't go extra X rudy meeting some first days.
ilija: And you know we discuss this together since people hope to organize to fix it.
David Mason: What does not usable me.
David Mason: And I guess, I mean you, you know extra needs a protocol that actually some of our sites used with F fts So do you mean streaming or I think some of that has to be laid down to renew.
ilija: Now, so that.
ilija: So everything that doesn't go with this be direct access copies to scratch, or whatever.
ilija: If it's not really usable because even that I am checking how it actually ports fuji streams and summary streams and even extra doesn't report it well and most places don't report to us EG because collectors at them.
ilija: And then support nobody's putting so you are maybe getting.
ilija: 5% of actual transfers.
ilija: And, and even these 5% and not really believable.
ilija: Because it really depends, which version of.
Alessandra Forti: Extra.
ilija: He wasn't there was a lot of changes in the meanwhile, in.
ilija: In the meaning of what is reported.
ilija: So that all of that has to be completely overhauled.
ilija: and
ilija: We, it would be great if somebody that does come to these meetings to you know.
ilija: organize that work.
Pedro Andrade: For us.
Pedro Andrade: From the front of money team, so what what we can say that, from the terms of the extra the dashboard that we provide.
Pedro Andrade: You that this data is data that we are collecting the same using the same architecture and model, which has been put in place and many years ago.
Pedro Andrade: it's based like like you're saying on this collectors that the run and get data from different excellent the servers that, of course, in the different sites.
Pedro Andrade: That is definitely I think the situation so both collectors are always up and running, so I don't think there are there are breaks, it may be more the question of which extra deployments that are reporting and don't have problems reporting.
Pedro Andrade: I think the situation is different from experiment experiment.
Pedro Andrade: And yet I think I agree that maybe a more detailed careful analysis on the situation that something that will will be nice to have of course coordinator, without oxygen.
Mario Lassnig: I see a recent from lechuza.
Latchezar Betev: On this subject, I just wanted to add that, in our experience the proper excellent the reporting is only possible if you.
Latchezar Betev: Do it through the clients, that means that the fans have to report what they do to the extra day basically were using the hooks which are provided the next ready to do that, but.
Latchezar Betev: If you're looking for the server reporting, yes, you may have been, either under reporting or, for example, if you have several videos you can not segregate which Vo was making the request.
Latchezar Betev: that's very difficult if not impossible at some point, it was even considered to start kind of a showstopper because if you have a shared extra the server, and that is serving several videos you're going to get a global view, but you have no idea who is actually generating the traffic.
ilija: I don't think we have this series share the movie as.
Latchezar Betev: Well, they are, they are many of those as well yeah.
Alessandra Forti: I think you're talking about two different continents.
Latchezar Betev: yeah Of course I mean we're talking about what Okay, I mean we have fewer in the US than us and cms but in Europe, for example, and not even next video that did that also goes for dpm.
Latchezar Betev: And that also goes for the cash, which in in Europe, but to also to prevalent technologies, and that is something which would not be discarded because, as far as I know enough to say cms dpm is still widely used technology.
Latchezar Betev: And the cash is definitely many, for example, we know the Nordic size, the cash is what is used for tape access as well, so there you have it.
Paul Millar (DESY): Just just a clarification that multi VOD captions the sizzle we do have those but also we we know which Vo a user is.
Paul Millar (DESY): A member of so the monetary information service, I want to train nation is, at least in principle, available, which says, which.
Paul Millar (DESY): So if there are specific technical problems, then, then those that we could look into but certainly the authentication that's done in D cash identifies and knows which Vo.
Paul Millar (DESY): uses a member of.
Latchezar Betev: Right, but these, that is, that also some copper updated to the extra access to cash.
Paul Millar (DESY): I don't understand the questions.
Latchezar Betev: Well, the point is the point is that, for example, if you access it to X rudy yeah okay yeah, you have the authentication I just know that that was that was not a good question, the point is that.
Latchezar Betev: I mean our experience again I just repeat, is that the best way to do it is through client aggregation.
Mario Lassnig: QC and from Brian.
Brian Bockelman: As somebody who's worked with xp monitoring for for a long time and it's it's really not not even specific to X rudy but i'd point out.
Brian Bockelman: What what you're asking for here is, can we either monitor uniformly all the servers.
Brian Bockelman: Across the wc G that site setup or the flip side can can you monitor all the clients that would ever authenticate from a given Vo and you know from my experience I would.
Brian Bockelman: heartily agree with latches are here that you are going to have a better time trying to instrument, the the clients, because the videos tend to.
Brian Bockelman: manage a larger percentage not 100% cut the you know reels tend to manage about you know somewhere in the 70 80% of the clients range versus trying to validate each individual sites.
Brian Bockelman: Monitoring make sure that the sites aren't making changes to monitoring, you know, this is a level of management that.
Brian Bockelman: Quite jealous ELISE has but.
Brian Bockelman: Particularly cms an atlas don't have.
Brian Bockelman: A a close control of the sites and what they're doing.
Brian Bockelman: So I think, in the end, you really have to think about what you want to see you know, is it really are our business to see every transfer that occurs.
Brian Bockelman: Do we you know, do we care just about bulk data transfers between sites and storage elements and these sorts of things, because I think we can spend an.
Brian Bockelman: enormous amount of effort, trying to instrument every single transfer.
And I think.
Brian Bockelman: People have kind of voted with their feet on try, you know what they see the the payoff is for trying to do everything versus trying to to set.
Brian Bockelman: Some more modest goals like just looking at fts or.
mtd.
ilija: I mean I don't mind if we monitor 90%.
Alessandra Forti: Of the things.
ilija: i'm afraid we on mentoring.
ilija: 60%.
ilija: would be fun if we are monitoring night, we know that that's how much we do that that's fine, we can extrapolate autocratic percent, but if we actually monitor.
ilija: 50% or maybe we think we are monitoring 90% that we actually mentoring 50% that that's that's kind of cool.
Alessandra Forti: So.
Alessandra Forti: Apart from extra.
Alessandra Forti: d agree with you yeah, then we should not tied to my daughter everything we can, he was a bit worrying when they say that.
Alessandra Forti: They own the streaming stuff.
Alessandra Forti: All the activity from the worker nodes is not monitor the tone.
Alessandra Forti: I don't mind the local one that but, when it is remote is like the clans.
Mario Lassnig: yeah I think there is a point to be made that for those streaming accesses.
Mario Lassnig: There should be a natural CAP right So if you believe you're monitoring 90% of your client transfers across the wide area network and that somehow relates to the deployed network capacity.
Mario Lassnig: You could be somehow confident that it's not only 30%.
Mario Lassnig: But yeah I know I know it's it's a probabilistic monitoring so to see.
Mario Lassnig: Mathias.
Mattias Wadenstein: Yes, I was just going to suggest that.
Mattias Wadenstein: A good way of seeing how much is covered would be to correlate it for the network monitoring, especially on the places where we have dedicated links that are easily.
Mattias Wadenstein: assignable to sites, like the other co PA.
Mattias Wadenstein: Very good estimate if it's.
Mattias Wadenstein: Covering 90 or 9% of the transfer transfer data.
Mattias Wadenstein: yep.
Mario Lassnig: Okay.
Mario Lassnig: I was a lively discussion, I would suggest we make a little break we already one hour 30 in how about we continue at.
Mario Lassnig: 1645.
Mario Lassnig: So everybody has done.
Alessandra Forti: To.
Alessandra Forti: have a pretty good coffee or tea yes.
Mario Lassnig: Okay cool so See you in 15 minutes.
Mario Lassnig: Okay welcome back everyone.
Mario Lassnig: Next up on the list is.
Mario Lassnig: mckay will you.
Mihai Patrascoiu: Give the yeah Hello.
Mihai Patrascoiu: share the screen.
Mihai Patrascoiu: And let's see I think this okay can fundraise it.
yeah.
Mihai Patrascoiu: Okay, so I want to give a short monotonic overview.
Mihai Patrascoiu: For the fps project Okay, so I started with the architecture, this is a server instance, which is composed of multiple demands there's the killers DEMO and what we call them your server is actually the past with them.
Mihai Patrascoiu: And that does that does for them and connect to our server this has been optimizing the schedule and I create.
Mihai Patrascoiu: transfers which are actually being done by another component, called the task for later, this is a separate process the process name is called fts URL copy and being a separate process it communicates with the main sell well, we are their cue that actually Q.
Mihai Patrascoiu: And this, this is yours, for the one way communication between the stats for the agent to send information back to the server.
Mihai Patrascoiu: Now, both the curiosity man and password demon they also use their kill for a question coverage many times for doing states population.
Mihai Patrascoiu: involved bigger database and in case, our classes over that we keep a directory cue to be able to recover this is also the same thing is also available in case somebody start of the server.
Mihai Patrascoiu: And this is without monitoring messages, as you can see in the title now what happens when we put monitoring messages.
Mihai Patrascoiu: We get a new component, which is the FDA is messaging.
Mihai Patrascoiu: And this is responsible for collecting data from the server and sending it externally in this case to an active mq, which is actually the sun money active mq.
Mihai Patrascoiu: architecture of course doesn't change we only add an extra layer, which means we have an additional directory queue and this is dedicated only to monitor messaging.
Mihai Patrascoiu: And this is being populated by the killers demon by the task one demon.
Mihai Patrascoiu: and also by the optimizer within the last few months and it's also being populated by each of these transfer agents.
Mihai Patrascoiu: And we can see Iraq to market the diagram down for type of this thing of this thing messages that are being sent to the monitoring kill there's a transfer style complete.
Mihai Patrascoiu: there's the transfer state change and then there's the optimizer update.
Mihai Patrascoiu: So, as some summary as the takeaway message, all of these two architectural slides.
Mihai Patrascoiu: I, I want to sell the custom message that fps is is director accuse internally, this is for recovery and inter process communication.
Mihai Patrascoiu: And when you enable monitoring there's an additional component which stars handles only monitoring and sending the data to active mq there's a new Q, the monitoring tool which is being handled by this new component.
Mihai Patrascoiu: And there are four types of messages in the skill there's the does first start the task or complete tasks for state change and optimize it up there.
Mihai Patrascoiu: And this is how they met with the active mq topics, the stock goes to the monitor fps monitoring stuff that complete costs money on incomplete and so on.
Mihai Patrascoiu: and very important about the starting complete these are created only by the task for legion so you will not get a task or status for completing case of staging got archiving you only get them when when a task really started and the data is being moved.
Mihai Patrascoiu: there's a test for state, and this is created by multiple places in the US and asked for demos, and these are just ways to market with a task for state machine.
Mihai Patrascoiu: Every time file changes, a file state, according to the possible transitions between the state machine.
Mihai Patrascoiu: By newton's for state messages being created and in case monitoring this enabled will be sent to to the active mq broker and then there's the optimize that every time we can optimize on a given pair.
Mihai Patrascoiu: We create an optimizer Q message and we send it to death, yes, monitoring your state topic.
Mihai Patrascoiu: I don't think there's any point to put exactly the format of these messages they are in json so this means it's easy to extend them and actually they are quite comprehensive and they have a lot of data in them.
Mihai Patrascoiu: And we send all of it to active mq where it's being by Center monitors being aggregated in various ways.
Mihai Patrascoiu: And for a complete format of the of this messaging what fields, they have and so on i've put the link to the documentation and this would conclude the general overview of how monotonic works with the next of how this monitoring messages work between fts.
Mihai Patrascoiu: And the good thing is, it would be easy to even add a new message, but at the same time, I think the main actions are being covered and in case of periods are missing it's very easy to buy them.
Mihai Patrascoiu: And the last part of the presentation, I want to showcase a bit what we call the FDA is internal dashboard.
Mihai Patrascoiu: These are actually based on a fan, and they use Center money data so there's nothing really.
Mihai Patrascoiu: let's say hidden about them the same can be achieved by anybody, but this is the dashboards for us for our internal monitoring God debugging and so on, this is a picture, with the Atlas instance and i've only put the casper plots here there's multiple.
Mihai Patrascoiu: Sub sections.
Mihai Patrascoiu: And here's another picture we the Q times.
Mihai Patrascoiu: Thank you very much, in case that any questions i'd be happy to answer them.
Mario Lassnig: Any questions.
Maarten: Sound is very bad Peter.
This is.
Maarten: unworkable.
Mihai Patrascoiu: yeah sorry I was freaking out here to it's breaking up.
Mario Lassnig: Can you, maybe type the question and we come back to it later in the meantime we have any different questions.
Alessandra Forti: He said he was he was writing it I can you go back to slide seven.
Mihai Patrascoiu: Yes, this one.
yeah.
Alessandra Forti: I mean.
Alessandra Forti: quite an interesting.
Alessandra Forti: The r&d is quite interesting quantities to integrate them, maybe in a in a common dashboard.
Mihai Patrascoiu: The queuing times.
yeah.
Alessandra Forti: example like to understand a.
Alessandra Forti: transference a take a long time because of the sts queuing or.
Mihai Patrascoiu: The problem with these third parties that it's a be.
Mihai Patrascoiu: It can be misleading because we seen this picture to the left.
Mihai Patrascoiu: yeah, but we have a huge screen time for an SRM through, and you will see, for example, queuing.
Mihai Patrascoiu: queuing from tape for staging they usually have a very a pattern like this, where they accumulate a lot, and then they then they they are.
Alessandra Forti: Right.
Mihai Patrascoiu: But I think, as it stands franzia they longer for a transfer the king, time is that means either there was a problem or if the server cannot go to the backlog, so the backup is increasing, which of course we later time.
Mihai Patrascoiu: I think we can integrate it the data is already money.
Alessandra Forti: yeah.
Alessandra Forti: So it should be easy to add the stuffings we want bloat and also you i've already the better Vo.
Alessandra Forti: So, like torso he's a he he.
Alessandra Forti: is to me comes from different yes okay so that's easy.
Mihai Patrascoiu: We filter by vr because actually in a transfer in one of these messages that we send I think we are supposed to Vo However, we can also filter by the FDA cerebral because each each message has an fps endpoint name.
Mihai Patrascoiu: And in the case of fat loss it's a bit easier because most of the traffic is generated on by that, as we are.
Mihai Patrascoiu: With public it gets my testing.
Mario Lassnig: Some.
Mihai Patrascoiu: I was on to.
Mihai Patrascoiu: The pure it actually when when a task for fails, still a transfer a complete mess is generated.
Mihai Patrascoiu: because some of the fields in the transfer complete they involve, for example, transfer files state, which will be fair, the error type of federal, such as recover or not recoverable and so on.
Mihai Patrascoiu: And size.
Mihai Patrascoiu: i'm not sure the size goes into the the size goes in the transfer state message.
Mihai Patrascoiu: But i'm not sure it goes in the transfer complete message as well.
Mihai Patrascoiu: If, for example, we stop halfway until the halfway size that will be status updates and you will be able to find them by job ID but i'm not sure that as a complete message will have the.
Mihai Patrascoiu: The final side I literally have to check this.
Mihai Patrascoiu: Even if it is included.
Mario Lassnig: OK, for the in the previous noted throughput tests or one thing I was always looking at was the optimizer values for the given links, so this is part of the messages I saw on your.
Mihai Patrascoiu: slide for us yep we have optimize your messages.
Mario Lassnig: Okay, so it has done the delta for the actives over time, so this is definitely something we need to include because this shows us if we're actually pushing a link as much as we could.
Mihai Patrascoiu: When you don't have the data they give a snapshot, but if you compute two consecutive messages for the same better can gather data yep.
Mario Lassnig: Okay, so this will be very good to have.
Mihai Patrascoiu: This is something where I was investigating because we want to push the optimizer a bit more.
Mihai Patrascoiu: And yeah if you want to team up will be around.
Very good.
Mario Lassnig: Any more questions.
Alessandra Forti: yeah going back to one particular was asking, I think that what is behind the tempo discussed it briefly is the upside down and bandwidth of failed transfers.
Alessandra Forti: which we don't seem to be able to monitor at the moment, at least in our class, we have the number of fail the class first, but that is not feeling reporting.
Alessandra Forti: or connecting it to the occupied family, because if you partially transfer something and then these phases to buy bandwidth and i'd say type of waste.
Alessandra Forti: We might want to monitor at some point.
Mihai Patrascoiu: that's the way to have that would be exactly in this transfer complete which reports, the file state and Eric died there was support that task transferred bite.
yeah.
Mihai Patrascoiu: And this would show up, but what we have in this picture in the hospital, to put it would also show up, even if we have many the task force to put from this dashboard includes both hands for success and password failures.
Mihai Patrascoiu: But there's no way to map them.
Alessandra Forti: yeah that's the point.
Mihai Patrascoiu: yeah understand.
Mario Lassnig: Okay, thanks again we say.
Mihai Patrascoiu: Okay, thank you very much.
Mario Lassnig: We move on to the next topic.
Mihai Patrascoiu: We have.
Mario Lassnig: duck who will present message go ahead.
Doug Southworth: All right, thank you very much, start here.
Doug Southworth: Okay um so I just want to take a few minutes, this morning, or afternoon, depending on where you are and talk about net sage and how I think that.
Doug Southworth: Net sage might be able to fit in with some of the measuring measuring and monitoring that you're trying to accomplish.
Doug Southworth: So net sage is nsf funded and basically was designed around the idea of having a high level view of what traffic patterns are and to understand where flows are heading, we can also digest SNP information and really just be able to wait basically a relational view.
Doug Southworth: or a way to look at transfers over entire end to end paths and to have an idea of where transfers are happening.
Doug Southworth: A lot of times, we can see that you know specific numbers are happening specific flows are happening, but we don't necessarily know what kind of information is entering or leaving a particular system and so nets, he was designed to give a broad view of that.
Doug Southworth: So the the couple main installations out there, right now, we have several others as well, but this one portal that nets age is for the nsf international.
Doug Southworth: links that indiana University has a hand in as the international networks team and then probably more of interest this group is there isn't a na installation as well, which covers the nine links that are going across the Atlantic and linking us to your Europe.
Doug Southworth: And for that we currently don't have flow information, but we are collecting SNP data both current and historical so you can look back over periods of time, and you know transfers that have occurred and we'll take a quick look at that later on in terms of what that looks like.
Doug Southworth: So what made sage again we talked about this a little bit what it really does is give human readable summaries and patterns that's the the takeaway that I want you to get out of this because a lot of times it's very hard to drill down through.
Doug Southworth: Data we're we're oftentimes very good at creating large amounts of data, but trying to see correlations between.
Doug Southworth: different sites and understand what those large data sets mean when you're trying to understand the nature of transfers and what networks are actually doing versus what we think they're doing.
Doug Southworth: nets he gives us that kind of broad overview and makes the data simple enough that you can find.
Doug Southworth: anomalies quickly and then, once you find those anomalies those things that you're looking for that you can then drill down into those specific time frames to figure out what had happened during that particular transfer.
Doug Southworth: So the current data sources that nets age is able to digest and the ones we're using SNP, of course, we can take active person our data as well flow data.
Doug Southworth: flow is a big part of what net say digests and we also have a way to look at archives using T Stat basically take traffic analysis from large data archives.
Doug Southworth: One of the other pieces of nets ages, the science registry so most of the flows are tagged with who is information public publicly available, you know DNS look ups.
Doug Southworth: The science registry is a specific registry that we've created that is all voluntary information.
Doug Southworth: So if there is a specific project and you want those flows really identified well in that stage, you can then enter exactly.
Doug Southworth: What the project is what IP address it's originating from what science domain, it applies to and then this basically this gets processed before we go out and just do a general who is type of look up.
Doug Southworth: So that way we can ensure the integrity of the information in that stage when it comes to specific projects.
Doug Southworth: But within that sage we do identify the flows, down to the last octet we basically remove the last octet so nobody can pinpoint a specific machine or IP address this allows us to identify those flows, while still.
Doug Southworth: Maintaining that D identification.
Doug Southworth: So a little bit more about that we don't collect any pii there is never any information in that sage that is again tied to a specific person or project.
Doug Southworth: We only keep data currently on flows that are over 10 megabytes in size for data archives we go a little bit smaller and we monitor flows that are only one megabyte in size anything smaller than that gets discarded.
Doug Southworth: Because now you're down into emails and other types of small data that doesn't really give us the the picture we're looking for in that stage, and it also helps us again.
Doug Southworth: maintain that privacy when we're not looking at small flow is like chats and emails and such so the privacy policies available online, if you want to look at that.
Doug Southworth: And how we do our data retention, and you know how that's handled as well, so the basic pipeline for nets age is we're gathering s&p and or flow data.
Doug Southworth: We match that with information from the net say science registry and or who is data depending goes into the pipeline and you can see the flow here basically we throw out everything over 10 megs we stitch together flow eight.
Doug Southworth: flows that are longer than five minutes we do our tagging and then we, the last step is we do identify it so by the time it hits the nets age archive over here.
Doug Southworth: There is no personally identifiable information and then all of that data ends up on a graph on the dashboard which we're going to take a quick look at as well.
Doug Southworth: So just some quick takeaways from this, the entire idea behind nets ages, we want to help support science and research and give people visibility into what flows are happening what as an NP data is available.
Doug Southworth: and make it human readable and a very easy to understand very searchable format, because we can't find problems if we don't know that there are problems, so we have to be able to measure we have to be able to see.
Doug Southworth: And we have a lot of capacity to grow in that regard there's a lot more than that sage can do basically we're only limited by the data that gets put into it, so I wanted to take quick seconds, let me drop my share for just a second.
Doug Southworth: And i'm going to reshare.
Doug Southworth: This.
Doug Southworth: So again, this is the basic dashboard for the international circuits that we actively monitor.
Doug Southworth: we're going to take more of a look over here, though at the DNA dashboard.
Doug Southworth: So we are already collecting SNP data again, you can look at current data or you can look at historical data going back for as long as the net stage project has been around so approximately three years of data they're.
Doug Southworth: hovering over these links just gives you Min Max and average for the specified time period, we do have SNP graphs and totals based on the time frame for all the DNA circuits, as they stand now.
Doug Southworth: So, again kind of makes it easy to identify peaks and valleys and that's really all we're collecting for the DNA right now again it's just SNP data, so we can't the unlike a flow there's there's nothing specifically identifiable we can tell you, other than just totals.
Doug Southworth: there's also a way here in the current incarnation.
Doug Southworth: To look at specific bandwidth patterns over a specific time to see where there's hotspots again darker areas indicate that there are more transfers happening during those specific time periods and This again is scalable based on how far back you want to go.
Doug Southworth: Now, where this can get interesting for the data challenge that you're doing is, on the other, dashboard that I started to show you the one that works for the links that we have an interest in specifically as international networks at indiana university.
Doug Southworth: We have developed far more dashboards because we can work with flow data as well.
Doug Southworth: So one of them is this project dashboard this ties together everything that I was telling you about in terms of SNP data flow data, the science registry.
Doug Southworth: So what this dashboard does is it goes through the science registry and finds flows that match specific projects and then allows us to see.
Doug Southworth: What institutions are participating in that specific project, so in this case we're looking at atlas.
Doug Southworth: Now this isn't the totality of atlas This is just endpoints that we know specifically are participating in atlas there's likely other ones out there that we don't have in the science registry yet.
Doug Southworth: But we can see volume totals for the specified time period, we can see organizational pairs and how much data they've transferred during that time period, again we have heat maps that show us when the project is busier and when it's not transferring as much data.
Doug Southworth: Individual organizations and how much they've contributed to that project over the specified time period both both as a source and destination.
Doug Southworth: The specific sensors that have seen that flow data and what specifically, you know how many flows how fast the flow is where their fastest flow and the total volume and then we also have a map that maps out which endpoints are involved and.
Doug Southworth: This can be animated as well, I won't do it here, but you can also select individual specific pieces of that puzzle and see what endpoints we have mapped there.
Doug Southworth: So a couple other things I want to show you real quick and then i'll tie this all together.
Doug Southworth: We have the individual, the ability to look at individual flows between two specific endpoints or one endpoint and every endpoint they've transferred with again over a specified timeframe.
Doug Southworth: So these are all individual flows that we've been able to look at here and we can even drill down even farther if I click on any one of these links over here.
Doug Southworth: we've actually got specific flow information, where you can see, the protocol, the duration, the endpoints.
Doug Southworth: The sub nets involved because, again we do anonymize down to the last are not optimized but D identify down to that last subnet and then this is information that we have tagged specifically from the science registry that's been pulled in there as well.
Doug Southworth: One last thing is, we do have a flow analysis dashboard that ties together as an MP data and the related flow data during that specific time, so if you're looking at a time frame, such as this is the last week over the near circuit from New York to London.
Doug Southworth: If you see a peak that's of interest to you, you can grab that specific peak it's going to zoom in on that time frame and then the related flow data that occurred during that same specific time frame is.
Doug Southworth: displayed both in top flow pairs of sources and destinations by volume and rate.
Doug Southworth: And then, as you drill down a little bit further down here, we also show the individual flows during that selected timeframe of that SNP data and then Lastly, what protocols source and destination points were involved.
Doug Southworth: So.
Doug Southworth: What i'm what I think probably is of use when we're talking about net sage to this project is the fact that we can customize that dashboards to show the data that we're collecting for this specific project.
Doug Southworth: And if you want to see relational data in terms of s&m P among several sites and how that relates to flow is during those specific times we can tailor these kind of us to be something that would be of specific use to this project.
Doug Southworth: And while this may not be as accurate in certain senses, as the other tools that we've already looked at today.
Doug Southworth: I think what next age gives you is the ability to look at the broader picture and spot anomalies and spot traffic Spikes and see how each individual site and each individual link is reacting.
Doug Southworth: In the larger picture of this experiment and then, once you can see that you can then start asking questions about why is there a spike here.
Doug Southworth: Why is there more traffic here that we expected there to be this can kind of be the big picture tool that then allows you to see.
Doug Southworth: Where you need to zoom in and maybe use some of the other tools that you already have.
Doug Southworth: To ask deeper questions about was there a bunch of failed transfers or what the disk you know throughput look like on this particular node at this time.
Doug Southworth: So i'll go ahead and stop there, have you had a pretty brief window to talk about this, but just wanted to give an overview of nets age, and I think what it might be able to contribute to the project.
Mario Lassnig: Thanks a lot already see the first time it's allison.
Alessandra Forti: And he looks very interesting and.
Alessandra Forti: The fact that you use s&p data I seen this some of the UK universities or how do you get him on this map.
Alessandra Forti: Is it automatically because you have an agreement with the network providers who.
Doug Southworth: So the s&p data any SNP data we have is because we have a specific agreements.
Doug Southworth: With a network provider for, for instance jr is giving us SNP data that we can then use to display in nets age, the flow data, the only flow data we have is flow data that traverses the specific sensors we have.
Doug Southworth: We have funding for, for instance, we have sensors in New York, we have sensors in Seattle Hong Kong.
Doug Southworth: And those centers there when flows to reverse those we can then collect that flow data, and you know, provide a source and destination.
Doug Southworth: But we're not collecting flow data directly from an endpoint, for instance, if we see something from disk.
Doug Southworth: Or we see something from one of the other European sites it's not that we're pulling flow data directly from that endpoint it's just happens to be flow data that is traversing one of the links that we actively monitor.
Alessandra Forti: Thanks.
Mario Lassnig: Any more questions.
Mario Lassnig: Otherwise, I have to um.
Mario Lassnig: Oh yeah Paul good.
Paul Millar (DESY): Sorry, I couldn't get there fast enough to leverage ahead, and so one of the things I think sean's maybe even talk about next is is about.
Paul Millar (DESY): Using ipv6 to identify the purpose of the data transfers, so it looks like you're identifying the the the flows by.
Paul Millar (DESY): By IP address subnet.
Paul Millar (DESY): And that's not always enough because I mean, as we had already.
Paul Millar (DESY): discussed the this there are perhaps not so much in the US, but then in Europe it's it's more common to have.
Paul Millar (DESY): facilities that are shared between different research groups.
Paul Millar (DESY): So different CEOs.
Paul Millar (DESY): And in that case, they would see a common set of you know, common sub net but for different different user communities, so I was wondering if you had any thoughts about that.
Paul Millar (DESY): For example, identifying.
Paul Millar (DESY): The usage, rather than by subnet but by some other mechanism.
Doug Southworth: I actually I completely agree with you that's something i've thought about quite a bit over the past week is i've been preparing this so two things.
Doug Southworth: If we if we actually identify the flows, down to the IP address as opposed to the sub net but we only display information, down to the sub net so by the time we have the information.
Doug Southworth: By the time it's gone through the pipeline it's been de identified to we basically have to make sure that we're we're taking care of people's privacy concerns when we display information.
Doug Southworth: Inside of nets age, but we actually have the information in terms of identification of the endpoint we do have it down to its identified at the IP address level before it's I de identified so.
Doug Southworth: Two things if it's if it's in the science registry, we can at least get it down to the IP address level, so we know a little more specifically about the endpoints and what project that's associated with.
Doug Southworth: But there are some cases to where you may have a data transfer node that is serving multiple projects, so your points still valid, that even at the IP address level we may not be 100% able to.
Doug Southworth: say that this traffic burst is specifically part of the the Atlas project or something along those lines.
Doug Southworth: So right now we're not sage stands I think there's still a lot of value in being able to spot traffic Spikes, especially when we're talking about a network testing project where we're pushing more bandwidth to see how the network reacts.
Doug Southworth: And that's where collecting the flow data if we can form relationships to collect flow data, it can be valuable as well.
Doug Southworth: Because when you see an s&m piece traffic spike if you can see the associated flow data during that time you have a better idea about which traffic is which and which traffic you're interested in and the traffic that you can ignore the other side of it is.
Doug Southworth: talked about Sean mckee and the packet work the packet marketing working group that is working with ipv6 to identify flows.
Doug Southworth: we're actually really excited about that in the net sage project, because we believe that.
Doug Southworth: As that work moves forward will actually be able to have the science registry digest those packet headers and then we can absolutely with certainty say that this this specific flow.
Doug Southworth: was part of this specific project because I had this packet header associated with it, so as that packet marking.
Doug Southworth: work proceeds nets ages just going to become more and more accurate, as we have that kind of information available.
Mario Lassnig: Okay, any more questions.
Paul Millar (DESY): have another one, but.
Paul Millar (DESY): Mary are you.
Mario Lassnig: Also, Paul, we need to take care of it about.
Paul Millar (DESY): Timing so okay yeah yeah that's another good point so I push my question.
Doug Southworth: If you if you want to throw it in chat if it's something I can answer there, then I can continue on after we move to the next segment.
Paul Millar (DESY): Okay.
Mario Lassnig: So just quick i've seen that you have all of this available in an elk stack how difficult, would it be too.
Mario Lassnig: Much just information with what we already have in our monitor tech.
Mario Lassnig: I guess maybe there's more question for pivotal.
Doug Southworth: yeah i'd have to defer that one and probably get some of our backend.
Doug Southworth: engineers from the global knock involved in that i'm.
Doug Southworth: i'm sure it's a doable thing, but I don't have a good answer for you right off the cuff.
Mario Lassnig: Okay Oh, we should definitely dig into that um great so thanks again and let's move on to the next topic which is time.
Mario Lassnig: it's Sean.
Shawn P McKee: Okay.
Shawn P McKee: To find the mute button, so I wanted to give a quick overview on network monitoring our end, and specifically the data challenge needs in that context.
Shawn P McKee: So.
Shawn P McKee: Quick overview of course the meeting today is really to discuss what we want for our data challenges in terms of monitoring.
Shawn P McKee: And I think this meeting is really the the beginning of a what's going to be a deep and technical discussion about what we need, and how we get it, and of course the corresponding timescale.
Shawn P McKee: Where i'm starting from is that I believe the data challenges should identify the shortcomings that we have in our existing infrastructures it's not just the network of course it's always an end to end problem.
Shawn P McKee: And the goal should be once we identify those things that are problematic we make changes we fix them we incrementally work our way to the hl elledge Sierra and this talk i'm going to focus in particular on the network aspects of the data challenges.
Shawn P McKee: Most of you know that we have a fairly comprehensive proof sonar network monitoring infrastructure.
Shawn P McKee: And that provides never diagnostics problem identification and localization but it's only part of what we need.
Shawn P McKee: For the data challenge we really have to understand how we're using our networks, what we're able to achieve in terms of sustained throughput and perhaps most importantly, where we have bottlenecks, whatever the cause.
Shawn P McKee: So I wanted to sort of highlight what I expect will be network related issues that we will see as we do the data challenge.
Shawn P McKee: I think firewalls are continuing problem we've had long a long term relationship with firewalls and their impact on our ability to move data to have services work correctly, etc.
Shawn P McKee: There are always miss configurations at insights or in the network could be related to the empty you size to stack tuning the host tuning routing or how the applications themselves are built or tuned.
Shawn P McKee: And all of these can severely impact what we actually get out of the network of course many of our sites also have very old equipment, in some cases switches routers and hosts.
Shawn P McKee: And many of those are unable to exploit the available network capacity or they're unable to utilize newer features that we might want to invoke to try and improve our ability to use our distributed infrastructure.
Shawn P McKee: We also have not old equipment, but equipment that may have problems either.
Shawn P McKee: Requiring upgrades in os or firmware or tuning to fix bugs to improve performance or provide improve robustness or extra features that we want to utilize.
Shawn P McKee: And the last main area where I think we will see issues is whether or architectural problems either in the local area networks or in the wide area networks.
Shawn P McKee: A critical feature what we're talking about with the data challenges, is that we will simultaneously use the shared infrastructure amongst all the experiments and that's great because that gives us the ability to identify potential bottlenecks when we're jointly stressing the networks.
Shawn P McKee: Also, the existing design of many campus and site networks may not really be appropriate for their current and especially future use cases and it's really important to be able to identify those.
Shawn P McKee: I wanted to see a little bit about site networking I think there's a big opportunity here to try and improve what we have at our sites, so that we can meet the H hl HC requirements.
Shawn P McKee: To be successful we're going to need a large fraction of our sites participating as we, as we move forward and helping to develop this next generation of infrastructure and the associated networking, we need.
Shawn P McKee: So I have a few sort of off the cuff suggestions for consideration for site networking.
Shawn P McKee: Please, update and replace your purse on our hardware, in many cases sites have very old instances, and we are asking those instances to do a lot of measurements can be a challenge to get the measurements and get the data.
Shawn P McKee: there's, of course, hopefully opportunities for sites to also replace their local area networking or their connectivity to their wider in networking.
Shawn P McKee: And we, we want to have sites that are capable of participating in what we hope is a future infrastructure so as we refresh.
Shawn P McKee: hardware, we tend to get newer chipsets newer capabilities just built into those devices and it opens up some opportunities for exploring.
Shawn P McKee: New features that might help us better use utilize our infrastructure, I would remind sites, not to forget about security perhaps participating in the w oC G security operations Center working group is a good starting point.
Shawn P McKee: There is some additional hardware and services that you need to consider if you're participating with the w oC G suck.
Shawn P McKee: That could be a part of your refresh for your your networking and when equipment.
Shawn P McKee: I would also suggest that planning to automate your network provisioning so that you can get a consistent agile site configuration would be beneficial, especially as we explore future opportunities.
Shawn P McKee: And of course last and certainly not least, identify and publisher site network monitoring links, hopefully, we will have a location soon and quick to put that link.
Shawn P McKee: A little bit more about that I really think that one of the important pieces of monitoring that we need is to understand how the sites are actually using the network, both in and out.
Shawn P McKee: So I would suggest that maybe we could create a webpage that had at least three elements of first section with links to real time network monitoring.
Shawn P McKee: That at a minimum gives the site ingress and egress network rate, how many megabytes per second are you seeing in and out.
Shawn P McKee: Of course, more detailed monitoring is welcome and encouraged we've seen a lot of examples so far today and presentations.
Shawn P McKee: adding some additional description of the monitoring technology and characteristics would also be strongly recommended in this first section.
Shawn P McKee: The second section that I would suggest, would be a top level summary of the site network, you know what equipment and vendor and model, do you use.
Shawn P McKee: How how have you, you know connected your devices on your local area network, how do you connect to the wide area network doesn't have to be long, but it should give.
Shawn P McKee: sort of a summary of what's there, the third section, maybe optional would provide detailed site network diagrams so that we could help remotely understanding debug potential bottlenecks and maybe make recommendations for improvement and share best practices.
Shawn P McKee: So the other area, I want to talk about is the work in preparing for the future, we have this research networking technical collaboration that arose from the.
Shawn P McKee: meeting in January 2020 and in that meeting we had identified our Community identified three areas that they thought was important.
Shawn P McKee: First, was making our network use visible the packet marking that Doug just mentioned and i'll talk more about the next slide.
Shawn P McKee: shaping our data flows, so that we can more efficiently utilize the bandwidth we have available so traffic shaping or pacing and then overall a longer term goal to orchestrate the network, so that we can enable multi site infrastructures.
Shawn P McKee: So the Charter for the working group is at this URL.
Shawn P McKee: we're currently focused on a few practical and technical issues to try and improve how high energy physics and others are utilizing their networks.
Shawn P McKee: The network packet marketing is the current focus of this effort and it's really to provide visibility accounting and future network optimizations.
Shawn P McKee: there's a few links to different presentations and the Charter we're working on trying to implement this we've already got it actually in persona so that we can do transfers with mark beckett's.
Shawn P McKee: And we suitably mark them, given the mesh and then we get the packets, all the way through and and that's all good, but what we really need to do.
Shawn P McKee: is to mark the bulk of our traffic we don't have to get 100%, of course, the more we can mark the better for everyone.
Shawn P McKee: Really, the important thing here is that at any point in the network, you can look at flows, and you can understand what's actually happening, who wants to traffic what the purpose of the traffic is.
Shawn P McKee: I think this would be especially useful for understanding or flows during the data challenges, it would be great if we could make some significant progress.
Shawn P McKee: Even if we only get a few percent mark at least to start on this path, not only the marketing of course, but then having the ability to consume.
Shawn P McKee: The information that's in those headers so that net sage or the army networks could provide us with visibility into what's happening across our networks.
Shawn P McKee: So I started earlier with some expected network related issues, and I wanted to sort of briefly indicate roughly given what I talked about how we might identify those issues.
Shawn P McKee: So, in terms of firewalls interfering both personas are measuring and any sort of network monitoring that we can cobble together could both be used to identify where firewalls are impacting.
Shawn P McKee: Our infrastructure in terms of miss configurations, the same two sets of areas could be used to help us find miss configured are on.
Shawn P McKee: badly tuned or non functional areas in our infrastructure.
Shawn P McKee: In terms of old equipment, a lot of that would have to come from the site monitoring web interface that I mentioned, where there are descriptions of what the actual equipment is.
Shawn P McKee: And just seeing the fact that there is underperforming or poorly performing sites or areas in the network that could also come from the monitoring and it might be attributable to the old equipment.
Shawn P McKee: In the case of equipment in need of upgrades or tuning really it all relies on monitoring to really find those things, I guess, in terms of standardized tests from perhaps on or you could but.
Shawn P McKee: Basically, I think that the primary method would be having comprehensive enough monitoring for our sites and networks and then in terms of architectural problems that primarily would come from the site web that I was talking about.
Shawn P McKee: But in addition, of course, seeing that there are problems in again under performing sites during, especially during the data challenge on through the monitoring.
Shawn P McKee: So there's a number of areas where we can do work and I just made a quick list I don't know that it's comprehensive but it's sort of.
Shawn P McKee: They were the things that that occurred to me thinking about preparing for data challenges.
Shawn P McKee: Of course I believe very strongly that we want to get on the network monitoring pages as useful as possible, so creating a good complete example that other sites could use as a template.
Shawn P McKee: And an example I think that's a very beneficial thing to do, we have a lot of data from the person or monitoring, we know of lots of issues we haven't aggressively pursued.
Shawn P McKee: Getting those resolved, of course, it takes a lot of effort, a lot of potentially ticketing and interaction with the sites.
Shawn P McKee: But I think we're getting to the point where it's probably useful, especially before the first day of challenge to see if we can get at least some of the most degree just cases resolved or handled.
Shawn P McKee: A tool that might be useful, would be a simple script that could run on storage elements and maybe check the current networking stack configuration and system settings and perhaps recommend possible changes.
Shawn P McKee: there's, of course, working with all the sites that don't already have appropriate network monitoring to get something in place.
Shawn P McKee: Not just, of course, creating the webpage that documents it but also maybe helping them identify a method to at least get the ingress and egress for their site visible to the collaborations.
Shawn P McKee: there's work with the army networks to identify suitable monitoring resources.
Shawn P McKee: centralizing all the data that we eventually identify from both sites in the army network monitoring can we get all that and to one place to make it much more usable.
Shawn P McKee: The packet marketing has already been discussed what level could we get something in place for this fall's challenge.
Shawn P McKee: longer term, I think the traffic shaping and deployment is going to be an interesting topic, one that could benefit us, but I think this is more midterm likely a focus on the 2023 timescale.
Shawn P McKee: even longer term potentially is the implementation of the network orchestration capabilities.
Shawn P McKee: Harvey Newman leads an interesting group the GA D is that is discussing all sorts of options and opportunities, and they should continue to push ahead on that.
Shawn P McKee: At some point, integrating that will make sense, as part of our data challenges to see what kind of additional benefits we can achieve, and you know, maybe one other interesting area to work would be creating a best practices document regarding network monitoring tuning and architecture.
Shawn P McKee: Just to get broad input from a bunch of different sites and share that broadly amongst all of w oC g.
Shawn P McKee: So this is almost my last slide to summarize the data challenges are really to help expedite getting the right monitoring in place and to identify how we're using our infrastructure.
Shawn P McKee: And, especially where bottlenecks exist there's a large amount of work in various areas that we need attention even just within the networking aspects of monitoring.
Shawn P McKee: We need a plan to incrementally improve things over time, I think we have to cherry pick what we think are the most either viable or important components that we need for this fall.
Shawn P McKee: But there's also longer term items in my list and my list probably wasn't comprehensive they're probably many other things, to add to that list.
Shawn P McKee: But on the time scale of high luminosity i'll HC there are things that we're going to want to have in place to make sure we're able to see how we're using our infrastructure and to react to issues or problems.
Shawn P McKee: I know that finding effort is going to be challenging but we also need to consider that effort is often not shareable or redirect double if we don't push ahead on multiple fronts, we may lose time or effort that we might otherwise have.
Shawn P McKee: So I wanted to acknowledge funding related to a lot of this work and stop there and see if there are questions or comments thanks.
Mario Lassnig: FLEX lotion.
Mario Lassnig: I see no response yet now alyssa.
Dewhurst, Alastair (STFC,RAL,SC): Right and so.
Dewhurst, Alastair (STFC,RAL,SC): john, I guess, I mean, I guess, I agree with many of your what I do agree with your summary and and but it's like yeah that there's a lot to do so, I guess, my kind of question was.
Dewhurst, Alastair (STFC,RAL,SC): What do you see, as you said, the things we go to prioritize what's your personal priority list of things we should we should put at the top, I mean, is it.
Dewhurst, Alastair (STFC,RAL,SC): About gathering and you know, do you think that we're actually definitely missing stuff was it about organizing the stuff that we've got and yeah I mean, where do you see the homeless cuz.
Shawn P McKee: I think my my highest priority would be to make sure that we have the site sort of near real time network monitoring data just at the level of ingress and egress, so I think we prioritize by the largest sites and work our way down the stack.
Shawn P McKee: But we're missing a lot of that data, you know again persona or has some scheduled measurements to try and understand how the infrastructure is behaving.
Shawn P McKee: What really is going to matter for the data challenges how we're simultaneously utilizing all our sites after the fact we'd like to be able to understand where those bottlenecks were and why.
Shawn P McKee: I mean there's one kind of bottleneck that can arise because for this this week long period we're all going to be all the different experiments stressing our.
Shawn P McKee: source destination pairs and there may be hotspots that we don't currently see because we're not all simultaneously trying to do something.
Shawn P McKee: And so, identifying those kind of bottlenecks are important but also there are probably sites that have.
Shawn P McKee: Any one of the things that I listed they have old equipment miss configured equipment on updated equipment bad architectures.
Shawn P McKee: That will show up when we start comparing what we asked the sites to do in the data challenge and what they were able to achieve.
Shawn P McKee: And the easiest lowest courses measure of that is just how much data can you get in and out as a function of time during the week and then of course we correlate it with a thing so so in my mind.
Shawn P McKee: first priority is making sure we have sufficient course information about how our sites are able to use the network.
Shawn P McKee: it's also kind of hard right because many of these.
Shawn P McKee: Hidden things that are accessible.
Dewhurst, Alastair (STFC,RAL,SC): Yes, I would agree with that, I mean you know, trying to get, I guess, we need what 40 sites to be reporting, even if it's just one number or two numbers.
Dewhurst, Alastair (STFC,RAL,SC): The nowadays are in and out of their site and that's that's not a small it's a small thing to accomplish if you want to get that by by the full anyway, and thank you for the answer.
Shawn P McKee: Right and then you run up with a web page maybe we have 40 or 80 links, but that's also not that useful unless you can somehow capture it and do some analysis after the fact, I mean I think one of the challenges we have.
Shawn P McKee: Is, we need to think of what kind of questions.
Shawn P McKee: We want to answer after we run the data challenge and then work backwards and say what information do I need to make at least a partial answer for those questions.
Next is Martin.
Maarten: hi can you hear me.
Mario Lassnig: Yes.
Maarten: Okay, well, is just to sort of online was already said but previous to worse speakers.
Maarten: I would serve on the advice to focus on the low hanging fruits and for any initiative let's say try and do a cost benefit analysis and particularly watch out for integrating stuff into bigger systems.
Maarten: Earlier this afternoon we have we've already heard let's say that in the past, this was tried with extra D monitoring.
Maarten: which then ran into trouble and is now let's say, to a large extent, more trustful so the more complex that say we make the system, the more complex, it also is to to maintain and end to end and and remain understandable actually so that you that you can sort of.
Maarten: estimate yourself let's say whether the plots you're seeing can physically be correct, yes or no so so that is just generic advice right and then and then.
Maarten: We shall certainly see on the back end of the first challenge where basically we have a black hole in our in our understanding right, and then we can focus on those.
Shawn P McKee: Martin, I fully agree it's.
Shawn P McKee: If we can have very complicated monitoring that we try to achieve even if we achieve it, how do we utilize it, how does it help us was it worth the effort we had to put in.
Mario Lassnig: nexus you.
Eli Dart - ESnet (he/him): Know i'm.
Eli Dart - ESnet (he/him): So just an idea.
Eli Dart - ESnet (he/him): it's it seems like there are one could at least divide the the type of information we're talking about here into two categories right one, one is to help us understand.
Eli Dart - ESnet (he/him): The behavior of the overall system and the other is to help us troubleshoot something and figure out.
Eli Dart - ESnet (he/him): You know where if something's not working right right where's where's the problem right because problem you can't fix something until you've localized it and identified it right.
Eli Dart - ESnet (he/him): And so I think that might help with some of the complexity that we're talking about in terms of not trying to integrate everything.
Eli Dart - ESnet (he/him): But, but be able to get to things that can help us troubleshoot the stuff that we need to troubleshoot and then keep the the monitoring for for overall system behavior ideally separate from that just just a thought.
Eli Dart - ESnet (he/him): yeah I think that's a good suggestion.
Shawn P McKee: We do have different use cases here that are kind of implicit in in what was laid out in my presentation.
Mario Lassnig: Okay, thanks, a lot we move on to.
Mario Lassnig: monitors about the T zero Network.
Mantas Stankevicius: Hello.
Mario Lassnig: Let me share my.
Mantas Stankevicius: screen all right.
Mantas Stankevicius: hello, my name is month us from network communication engineering section and i'll show you how we monitor networks at mistakes at certain.
Mantas Stankevicius: Agenda is as follows, current and that's that system network statistics for short.
Mantas Stankevicius: Data volume, we are dealing with, right now, and some plans and what challenges that brings and overall motivation for the modernization.
Mantas Stankevicius: Currently, hundreds of.
Mantas Stankevicius: Network devices, we monitor mainly routers and.
Mantas Stankevicius: For monitoring.
Mantas Stankevicius: We use commercial software, called the spectrum and we use a login of spectrum called as a slugger to.
Mantas Stankevicius: query network devices using SMB to get the network traffic information, all this information is stored in the text files in the file system and.
Mantas Stankevicius: there's another process which aggregates and feeds are the demon which itself stores data into already files.
Mantas Stankevicius: For storage, as you see, we use are the tools and.
Mantas Stankevicius: The front end is php based.
Mantas Stankevicius: Data volume right now is, as we monitor only traffic router mainly we have roughly 260 devices and.
Mantas Stankevicius: We sampled 25,000 of our monitoring metrics every five minutes and monitoring metrics is as follows, for the device itself we monitor temperature cpu memory and for the interfaces we monitor bandwidth error drops discards basically we can monitor everything what's available via SMB.
Mantas Stankevicius: and
Mantas Stankevicius: In order to have finer granularity we plan to monitor not only router traffic but also switch traffic and better roughly increases volume by by 20 times.
Mantas Stankevicius: And yeah it's quite a challenge for us.
Mantas Stankevicius: Specifically challenge for a back end as each monitored item is stored in the file system as separate or the file and it's fine for hundreds of devices, but not half a million items which health impacts io.
Mantas Stankevicius: Also, a front end is not designed to support the income zillions of monitor items, so we have challenges welcome back end and front end.
Mantas Stankevicius: Huge motivation to use central your support the tools for both storage and virtualization in this case is monitor obviously it gives lots of benefits main benefit is ability to correlate data and network traffic statistics with our sources of information.
Mantas Stankevicius: And as we started this data centrally we allow users to use this data and the very last example is.
Mantas Stankevicius: Ongoing project called noted where technical student use the network traffic information for traffic analysis with some machine learning methods.
Mantas Stankevicius: And yeah great source of information for for studies.
Mantas Stankevicius: and few obvious things it's it's rough on the dashboard is much more user friendly compared to our digital and.
Mantas Stankevicius: yeah adding to.
Mantas Stankevicius: two or more banner side by side, which are the same X axis time access it's very easy we've got a file, which I couldn't tell about our digital.
Mantas Stankevicius: For the modernization, the first part stays the same for the time being, but instead of storing data locally.
Mantas Stankevicius: We send this data using message queue we sent them on it and forget.
Mantas Stankevicius: They take care of storage and virtualization tools which we benefit a lot.
Mantas Stankevicius: here's example have already.
Mantas Stankevicius: generated the blood and what marathon, I guess, so there is no data loss look is fairly similar with additional functionality provided by Griffin.
Mantas Stankevicius: And they're already several dashboards in open monitor.
Mantas Stankevicius: workspace.
Mantas Stankevicius: organization, where you can see, which the opm is the one etc and.
Mantas Stankevicius: Even more and.
Mantas Stankevicius: Some plans as we.
Mantas Stankevicius: plan to monitor all switches at CERN instead of just adding all switches like a waterfall.
Mantas Stankevicius: We want to exercise with a smaller amount and we offer possibility for a project, the monitor network traffic.
Mantas Stankevicius: between their devices and.
Mantas Stankevicius: Maybe we could benefit.
Mantas Stankevicius: Both sides, both ourselves so we could exercise and and you could get some I know some insights.
Mantas Stankevicius: and
Mantas Stankevicius: This is open offers So if you are interested feel free to contact me.
Mantas Stankevicius: And for a long term, we are looking for alternatives Open Source alternatives to replace commercial software which should be as good as it is as a commercial one.
Mantas Stankevicius: Preferably more user friendly work with as an MP NIBs.
Mantas Stankevicius: and
Mantas Stankevicius: yeah this is everything I have for you today.
Mario Lassnig: Thanks, a lot more.
Mario Lassnig: questions.
Mario Lassnig: I see no Ray sense.
Mario Lassnig: So I guess for the selection of the devices that were interested in this is the the storage farm in the data Center, is it possible for you to discriminate between experiences from were selected between the different experiments.
Mario Lassnig: This is something.
Mario Lassnig: I cannot answer right now, this, I think this a single storage might even serve multiple experiments i'm not sure I don't know.
Mantas Stankevicius: it'd be my supervisor cousin.
Veronique: Yes, we can we know where each server is connected to so we have the information Monday, yes.
Mario Lassnig: Okay.
Edoardo Martelli: If a server is serving more experiments, this cannot be discriminated.
Edoardo Martelli: So I think you apprentice PR eos us heartless is different set of servers from the US cms, for example, but US public is everything is mixed, then that would not be possible.
Mario Lassnig: Okay got it.
Thanks.
Mario Lassnig: If I don't see any more, hence I think we can move on.
Mario Lassnig: Next, one on the list is.
Mario Lassnig: use.
Mario Lassnig: Yes, now we can.
Laurent Duflot: So.
Laurent Duflot: it's going to be very quick it's a summary of what's currently available for the French site.
Laurent Duflot: So this is what we have, we have the cold weather map of the lh one which is provided by or no one another.
Laurent Duflot: And that's why also that one of the site is not put on this map.
Laurent Duflot: see here the lpp to to is directly connected to the one CC is to be three.
Laurent Duflot: And it's not connected to a network so it's not shown here.
Laurent Duflot: So on each of this link, you can over.
Laurent Duflot: The mouse and get the daily blocks.
Laurent Duflot: continuously updating and you should click on it to you have the weekly, monthly and yearly box.
Laurent Duflot: i'll explain a bit what the what it is.
Laurent Duflot: Which all of this is, but you can see that we have to connection to God and most of the sites are connected to both for fail over, but the other side it's.
Laurent Duflot: Just as an aside, this is the monitoring for the link you're going to the tier one so with also the lapd to to behind.
Laurent Duflot: This the top is for this last week and the bottom.
Laurent Duflot: Last month, sorry and the bottom before last year last 12 months.
Laurent Duflot: So, as an aside for the exact size of the challenge we already with the collection, that we are are close to.
Laurent Duflot: 880 to 90 G regularly for several hours and several times a week and all over the over the years, so it's not something exceptional.
Laurent Duflot: So, with the current of flash infrastructure, we can already from something like.
Laurent Duflot: And, with no major problems in the data Center.
Laurent Duflot: So he sets up a little bit the the size of the problem if we want to push all to ones, and only ones, have the same kind of capacity is already a lot of aggregated the language.
Laurent Duflot: A little bit more explanation on those on those on the map that I showed so, as I said, we have 200 gigabit connection to jail.
Laurent Duflot: So eloquent only so I don't talk about a champion, because the monitoring the available exam, so this is only he one we have the north connection say in a nice area and the South connection is Leo where the the the pacer he.
Laurent Duflot: Left labs are collected so called grief dissolved solids and also the one from Cosmo is connected.
Laurent Duflot: And the other sites, including the children, I correctly to the fast, so the the South network is of course us a lot more children there.
Laurent Duflot: And this is the assignment.
Laurent Duflot: Which is not completely static if there are any.
Laurent Duflot: I tried clicking on the link of course you've seen that a lot of the sites on the south of direct connection or not.
Laurent Duflot: They can be switched over and we can also switch them over if the garbage link South gets saturated.
Laurent Duflot: hello, so we have no direct many public monitoring called a PPT have to which is a an atlas new to so it's it's not it's one of the largest yet from France, so that's something we would like to have but it's currently not not available.
Laurent Duflot: Also, as an aside and currently some of the smaller side that other smoke connection other level to vpn sharing general Internet and he one, and also the bandwidth to share with universities, etc, and currently since we have in various States of confinement and remote teaching.
Laurent Duflot: Some of the bandwidth which used to be allocated to each one has been reduced to make room for videos.
Laurent Duflot: And also carry out that some of the levels in on this map needs to be updated, because, for example, one of the site is.
Laurent Duflot: So here the site here is not a 28 but hundred gig and those will also go to engage with this one also.
Laurent Duflot: But Okay, they are the religion of dating it's exactly the details.
Laurent Duflot: And as a the, this is the monitoring for for anaheim and we have so many drinks or the individual voters, they are collected at the children, but the afternoon tea behind the firewall so we have to find a solution if we really want to get access to them.
Laurent Duflot: In practice, the would be nice to be able to separate the traffic forming the lapd to and and the john.
Laurent Duflot: Okay that's all my.
Mario Lassnig: questions.
Alessandra Forti: Let me.
Mario Lassnig: Yes.
Alessandra Forti: Okay, so, as we were discussing the Friday.
Alessandra Forti: feeling good there is already a lot of information in that weather map.
Alessandra Forti: And it is clearly exporting the my decides, there must be an agreement with the wind.
Alessandra Forti: With your network.
Alessandra Forti: within your network fault 12th night and my if there is an agreement that I would it be possible exported on somewhere else, like more need for example and push the data, maybe tomorrow need.
Laurent Duflot: don't really know exactly what's involved with the we can.
Laurent Duflot: discuss it yeah.
yeah.
Alessandra Forti: Because.
Alessandra Forti: The the.
Alessandra Forti: inbound and outbound traffic from sizes, the first port of call like to understand the network traffic, in my view.
Alessandra Forti: It will be will be nice to have something like that and.
Alessandra Forti: Also, from other countries about France a is the first one that I see that as a sign of such a nice.
Alessandra Forti: display of all the site.
Mario Lassnig: To have more questions.
Mario Lassnig: If not we're.
Mario Lassnig: down to the last talk, I think.
Mario Lassnig: alyssa for all.
Mario Lassnig: Right, can you hear me.
Yes, right and is that.
Laurent Duflot: being shared properly.
Mario Lassnig: Yes, it's full screen.
Dewhurst, Alastair (STFC,RAL,SC): Okay, so as you.
Mario Lassnig: said on the last of rushing that so it get.
Dewhurst, Alastair (STFC,RAL,SC): A move on and so yeah I would, I would just comment about the differential vizio and I saw and lauren slides earlier about sharing the Nice.
Dewhurst, Alastair (STFC,RAL,SC): connection between the French sites, I did think that would be a very good thing for the UK to have an idea check and we don't have that available anyway we've got the.
Dewhurst, Alastair (STFC,RAL,SC): Customer persona machines, but nothing can met monitoring the connectivity between sites.
Dewhurst, Alastair (STFC,RAL,SC): And so my talk is just actually very quickly just covering what we've got available at the route to one as that we do have a weather map of our internal connection we use lever nms.
Dewhurst, Alastair (STFC,RAL,SC): to monitor the internal network traffic and that I think is the Open Source spin off of cacti, which is what we're using before and said, is a real time.
Dewhurst, Alastair (STFC,RAL,SC): We can hover over the links to find out the transfers and then, if you click on them there's a lot more detail which i've kind of done a partial screenshot on the right of just.
Dewhurst, Alastair (STFC,RAL,SC): A particular bought between some of our switches, and so we can drill down, we can see all the all the settings there and as to understanding internally if there's a problem or not between with our network flows.
Dewhurst, Alastair (STFC,RAL,SC): And then, in addition to that, and we've got a little bit about other monitoring, so we use nvidia's.
Dewhurst, Alastair (STFC,RAL,SC): what used to be humorous isn't that Cuba nvidia board, then as that's a commercial product and that's to ensure that the individual and network ports and.
Dewhurst, Alastair (STFC,RAL,SC): routers So this is the entire site and beyond just the tier one tier one is itself just a network or.
Dewhurst, Alastair (STFC,RAL,SC): are connected to the soup spine, so the the core infrastructure linking all the different parts together correctly.
Dewhurst, Alastair (STFC,RAL,SC): We don't see got persona and then we do have metrics of our individual machines or services which we can collect and present through our influx db and provide a service.
Dewhurst, Alastair (STFC,RAL,SC): And, and then, in addition to that there's also some bespoke monitoring prop plots that are created by the there are the contacts.
Dewhurst, Alastair (STFC,RAL,SC): Often they are just and either utilizing video infrastructure monitoring so there's just dedicated plots that would pull out monitor.
Dewhurst, Alastair (STFC,RAL,SC): which is specific to route for for certain things or they're using say bits of the Atlas or cms software structure, and then we make our own plot but we're basically running some jobs or some test transfers.
Dewhurst, Alastair (STFC,RAL,SC): By their infrastructure say they're just some examples here grab very quick screenshots I get to work a node and.
Dewhurst, Alastair (STFC,RAL,SC): profile honoring their general statistics it's got some network usage and packets dropped, etc.
Dewhurst, Alastair (STFC,RAL,SC): On there and then we've got a little bespoke cms AAA monitoring, so we can see how much is going going out to other sites for our triple A service.
Dewhurst, Alastair (STFC,RAL,SC): And then the last one is just some atlas download speeds that so James walder periodically runs this downloads files from other sites and then we can plot them and we can kind of see how.
Dewhurst, Alastair (STFC,RAL,SC): Our connectivity is to other sites and we found that that useful because it goes to our work and it's rather than to ICO cost of storage is.
Dewhurst, Alastair (STFC,RAL,SC): And then that's that's the that's the network monitoring we've got available at route.
questions.
Mario Lassnig: Thanks alyssa.
Alessandra Forti: think the same goes.
Alessandra Forti: For you, I mean the ui right the.
Alessandra Forti: First picture that is similar to with the influx db and pharma so.
Alessandra Forti: You don't even have to pull this stuff from this, which is themselves.
Dewhurst, Alastair (STFC,RAL,SC): And I think I mean, I think the thing is that I would get I mean.
Dewhurst, Alastair (STFC,RAL,SC): Well yeah good getting the information that we have got an influx db to some central certain thing I didn't think would be a problem at all.
Dewhurst, Alastair (STFC,RAL,SC): and getting stuff that's sitting in libra in a mess because it's got quite a lot of you know configuration details in as well, I think there would be a concern that we might.
Dewhurst, Alastair (STFC,RAL,SC): Over share and so that might be, you know, from a procedural point of view, a lot harder to share that and stuff an influx db is often you know it's it's you know already, but it doesn't have the detail in that might that might cause a security concern that.
Alessandra Forti: Some of the network monitoring we've we've got does have.
Thanks.
Dewhurst, Alastair (STFC,RAL,SC): I see I see long skirts and.
up.
Laurent Duflot: yeah so it's more.
Laurent Duflot: Just singular stoker remind me of something.
Laurent Duflot: my presentation, I only showed the HQ one monitoring, because this is what we have what if we have in a challenging.
Laurent Duflot: sites which are not on each one, and then the monitoring actual will not help a lot and we don't have a lot of bandwidth.
Laurent Duflot: On generic and and get it down to that that's local in what.
Oh yeah.
Mario Lassnig: i'm not sure I think it was she was unmuted for a second.
Mario Lassnig: Unless you're finished then it's a.
Bruno Hoeft: Different monitoring.
Bruno Hoeft: Data together would be maybe an idea.
Bruno Hoeft: Maybe eli can can say bird about what would be possible to or what could be possible for us not to share on monitoring data and one other point to look at would be drowned dre on half of EPA has a verizon net where it shows all the all the European connected endurance.
Bruno Hoeft: To lsc one, this would be on our starting point bear.
Bruno Hoeft: where information is available and.
Bruno Hoeft: But this would have been necessary to talk to Jay on and what they are able.
Bruno Hoeft: To to share in the central monitoring system.
Bruno Hoeft: Welcome
Mario Lassnig: Thanks for yes.
Eli Dart - ESnet (he/him): hey thanks for now um so um for as nat at the moment we we really have two primary things one is our persona or information and we can certainly.
Eli Dart - ESnet (he/him): We will add any of the relevant links for that to to whatever I really liked sean's idea of having.
Eli Dart - ESnet (he/him): At least just a central webpage somewhere where everybody can put all the links to their stuff.
Eli Dart - ESnet (he/him): And we can work through how those get integrated later, but at least having one place where you can go and find things like that, so we, we will certainly put those links to our persona information there The other thing that will put in is the link to the my yes net.
Eli Dart - ESnet (he/him): portal, which is, which has weather map like functions as well as some others at the moment that portal is undergoing a significant engineering rework our software development group has done some pretty amazing work.
In.
Eli Dart - ESnet (he/him): In adding a bunch of new features, as well as integrating a new flow system but that's not quite ready the timings interesting right it's not quite ready to share with the broader world just yet, and so, in the coming weeks we'll have more information there.
Eli Dart - ESnet (he/him): In the longer term we're hoping to be able to show some statistics, based on the packet marking work in the network technical working group um but but we don't have a concrete timeline there yet we're we're we're working on it we're just not quite there yet.
Mario Lassnig: sexy.
rick.
Rick Havern: rick hovering head of network engineering at J on it's kind of timely everybody talking about switching to influx and griff Ana because well So did we.
Rick Havern: Just this week, we released on the chant tools portal, the first iteration of the exact same sort of infrastructure, so please feel free to make your way to our tools portal and have a look at it.
Mario Lassnig: As I.
Alessandra Forti: was saying hey I might not know all.
Alessandra Forti: The stuff and maybe we should call like my first.
Alessandra Forti: Maybe maybe you know it.
Alessandra Forti: Probably.
Alessandra Forti: Call line find it.
Alessandra Forti: But they would be yeah.
Mario Lassnig: Okay, are there any more questions, we still have a discussion blocking the agenda for topics we have not touched with the presentations.
Mario Lassnig: So this is your time to speak up.
Mario Lassnig: No hands yet.
Mario Lassnig: There is one shot.
Shawn P McKee: I just wanted to say, I think you know.
Shawn P McKee: For this first meeting our goal was to understand what we wanted in monitoring what people currently have in monitoring.
Shawn P McKee: To open the discussion, but I think we can all probably see from everything that was shown today we're going to have to have quite a bit of detail technical follow up to actually figure out where we.
Shawn P McKee: want to put effort in, and you know what are the important things we want to achieve before the end of September.
Shawn P McKee: When we run the data challenge.
Alessandra Forti: So I expect.
Shawn P McKee: You know we'll have interesting follow on meetings that are going to focus much more technically in a much more detailed way trying to chart a path forward.
Shawn P McKee: This meeting is a an opportunity for everyone to sort of say what they think we should be monitoring what kind of questions, we should be able to answer.
Shawn P McKee: You know where we want to go, but there will need to be I expect quite a few follow on meetings actually get things done in place and usable.
Alessandra Forti: yeah absolutely I mean these is now one of the is a big head butts.
Alessandra Forti: In meeting a to give an overview to where to start from and what we have and they were not to reinvent the wheel and try to.
Alessandra Forti: You know.
Alessandra Forti: use what is there already.
Alessandra Forti: yeah and the starter and Charter, the path forward, as you say.
Mario Lassnig: You have a comment from me like.
Eli Dart - ESnet (he/him): This yeah um so just in the.
Eli Dart - ESnet (he/him): along the lines of what of what others have been saying and serving and let's let's begin where we are.
Eli Dart - ESnet (he/him): And then, and then chart a path forward i'd like to.
Eli Dart - ESnet (he/him): To echo something that Sean said in his talk, which is that we lots of the sites already have persona or infrastructure and that that's something that we I think it's pretty clear that we're going to need it.
Eli Dart - ESnet (he/him): And so, a good near term task, at least, would be to see to the care and feeding upgrades if necessary hardware refreshes should get planned on if they if they're needed.
Eli Dart - ESnet (he/him): That that's something that I think we know we're going to need and that we, we can go ahead and get started on, even if the configuration has to change later.
Eli Dart - ESnet (he/him): We know we're going to need that stuff so that's that's a near term place to start, I think.
Mario Lassnig: Yes, I agree with this as well.
Mario Lassnig: Okay, so I think we're nearing the end of the workshop.
Mario Lassnig: For organization, I took a couple of notes so also with the question and answers we have throughout the workshop, I will compile them into a slide deck already for tomorrow's to my general meeting I will also circulated for comments.
and
Mario Lassnig: With that I think we're at the end of the workshop.
Mario Lassnig: So Sean you want to add anything on top.
Alessandra Forti: Thank you.
Alessandra Forti: To everyone for joining, I mean we had a peek at people, and more so it's clear that this is an interesting topic for everyone.
Shawn P McKee: yeah thanks again everyone, this has been really beneficial, I think, good.
Shawn P McKee: And look for following.
Shawn P McKee: Meeting announcements and hopefully we can get significant participation.
Mario Lassnig: Absolutely great so let's officially close it thanks a lot everyone enjoy the rest of your day and see you soon bye bye.
Paul Millar (DESY): bye everyone.
Bruno Hoeft: bye bye.