is realizing natural is just not on that guy there's not been published by any of my bsm colleagues such that you experimentalists cannot search for it.
The concern of course is that what this model may be such that it shows up in no possible away okay it doesn't show up at all and he did that mean gifts to heavy particles or two families are acting particles.
at night. We will not sleep so well at night instead if there was the possibility that these model shows up in the Elysee data. And so, if he does, would we still see, okay, the failure of the standard model that is due to this underlying new physics
that we have no idea of what really is. Okay, so would we still see the Standard Model failing to describe the data, and that the our answer is that, that is high probability that we will not.
If we don't have to put in place some indeed the model independent strategy.
So if we just follow the regular path of modern dependent packages. So, and the reason being that the youngest under modern physics is not going to be, it is not we know it radical deformation of the Standard Model distributions is not that the deal he
did are gonna look because of new physics, globally, extremely different vastly different from what is predicted by the standard model and these we knew for political reasons, even before turning on the Elysee hey we know that there are plenty of low
energy phenomena that is too weak V's expressed that left but we knew that the vast majority of delicious data is going to be. We're going to be normal and today we know even more, that this is even more true.
So, new faces which show up as a tiny departure from the Standard Model distribution, or maybe a large departure in the distribution itself but localized in a very tiny probability region you may have a restaurant, in which case you will have much more
events than expected but this will be limited to a few was more low probability region. In terms of the entire landscape of data that we go back that the Elysee.
And also this new physics is most likely going to affect the few observers. Okay, unknown, because we don't know what these new physics is over the infinitely many observed bubbles that we can measure, making this comparison okay of the data with the
Standard Model without reference to anything else, quite complicated, business, and indeed that we say we proceed that ok searching for new physics in the standard way by more than dependence searches for new physics, which means that we selected so bubbles
that are sensitive to one, Peter deeper opposed to later the new physics model.
And of course, then we select these observable because they are sensitive to this model but genetically the same observed bubbles are not sensitive to another model of new faces, of course, the more modern dependent searches we do, the more we may we
be lucky, and then go into the signature of this right we as a model, even if it's not precisely the one you're searching for but we would like to be more than, than likely we would like if possible to design and approach, which instead aims at detecting
say genetic departures of the data from the standard model, which is not designed was designed is not based on a specific bsm more than and that's what we call model independent or model agnostic searches.
So, I think that that for literally, the way in which we can define this notion of model in Independence and just by your position with the regular model dependent case that I will have model dependent case you do struct your strategy visit on one postulated
or a set of postulated in physics scenarios here, you will not like to do that. And in the next slide that I will tell you precisely what model independent means in the specific framework that we are constructed, and that the.
And also, so the idea essentially is that we have a model that is the standard model. And that's the only physics model, we will be speaking ever about.
So, the standard model will be our only reference model, we are going to call it the reference model in the rest of this talk. Very often, and so the standard model does reference model will have to be competitive with the data without having an additional
reference that comes from an alternative new physics model from an from an alternative as formulated in terms of a specific Lagrangian or new physics model scenario says in regular model dependent searches.
So, the, the starting point is the same one of the entire statistical practice of the Elysee which is the maximum likelihood, which, which will apply to data as collected by Alicia for example that are independent identically distributed measurements
of a feature vector x. Okay. In general, this will be multivariate clearly variable that for example, we can imagine being the reconstructed momentum of a certain set, or multiple sets of particles that you may want to study simultaneously in the region
of interest at the moment of two laptops or four laptops and let this project and saw here, you already see that what we are aiming to do is not fully model independent, because we still have to select the finance data seems to select the region of interest.
While one may argue that the only fully model independent approach would be to take the entire outcome of CMS or Atlas before any the construction and anything can give it to the machine we believe that this simply not going to be possible to be done.
done. And so we consider say restricted school but, whichever is going to produce hopefully something which is much more model independent that then search for a specific new physics candidate model in the same region of interest and on the same I level,
a high level. I'm going to speak in my phone was going to be these little enter the tower distribution so so probability density is normalized to the total number of expected events, and that will add but the distribution of the features.
x, in, in, in the reference. So in the standard model are, and then to construct the maximum like really bothers his death, I need an alternative distribution, which is the end of XVVNW, and this wr the three parameters of this, of this postulated alternative
distribution. So this distribution is a composite demand statistical sense, the statistic that one implies in the maximum likelihood approach is the maximum over this w parameters the alternative depends on on the log ratio.
This is the extended the likelihood that you recognize denominator here and needed eliminate or denominator, you have the one for the alternative hypothesis or alternative distribution center in the denominator, you have the Standard Model, one.
So, when we really want to have a model dependence strategy.
Essentially what we would like to do is to take this, this as better the distribution to be equal to the one that is predicted by some alternative physical models that here is called the NP new physics model.
The three parameters w in this case for example on the signal strength of your restaurants just to say something, or in general the three parameters of these alternative models.
We we we typically do not use this formula directly as it is, we find approximates of this likely will send off the like liberation, but constructed again with this always with this idea that the, the, the, the alternative but should be should be restricted
to the space of alternatives that is produced by one possible new physics Lagrangian pre specified in people's too late.
In all the independent strategies that you would like to use an alternative but in some phenomenal logical parameters form.
For example, you can take it as that the reference distribution itself. Locally the scale the feature space by some positive function that I'm taking to be an exponential have some F.
Okay, f of would be a local function in X, depending automatically on this w parameters and the, there are the requirements, the central requirement you would like to do to add on this f is that it should be flexible function approximation because since
we don't know what we have the spec for new physics, we will like this dysfunction to be capable to adapt itself. By choosing the parameters to the largest possible variety of through new physics underlying the true distribution of our data so we would
like this maximum to go as up as possible. But thanks to the fact that that this distribution can adapt itself to what is really being observed as much as possible, and we would like to do so, while at the same time not having infinite flexibility that
will correspond to each individual data point here in the set is a, is a completely independent and can be Parker characterized in a completely independent way so we need to sort of compromise between flexibility and smoothness of the function approximate
set that we put here in this, in this exponent.
So, it is nice to notice that if you take f, to be the beach Weiss constant function. Okay. So essentially, to be an Instagram with WBP parameters okay also the Instagram, and you apply this formula you end up with the likelihood that is the starting
point for the right in the regular state model independent strategies that we do have already applied in different experiments for example a CMS it's called music, and essentially it's the been histogram test.
So in this case, F is the most simple and standard, the functional proxy man, that which is easter eggs. Okay.
If you said you take FB a neural network, you find that literally our proposal. Okay, so what we actually what our proposal literally is, if you can form a sample for music or see me I think you use to replace histograms with neural networks, being fully
literal sense. And I think that I don't need to convince this audience that this is highly motivated the attempt the neural network being effectively function approximate approximate flexible but at the same time containing emotional smoothness.
In the actually if you look in textbooks open neural networks that indeed introduced as convenient alternative to histograms to fit distributions in particular characterized by a better scaling of the performances with dimensionality.
So I think it's worth looking at what these.
These idea.
Brings. OK, so the the strategy that we follow.
I'm not gonna give too many mathematical details, but we want to evaluate this test statistic. Okay, which I'm rewriting here, and the, this will be done through a supervised training.
And if you use those specifically the same the last that we call maximum likelihood loss.
Well, this will be essentially an exact application of the exact of in the maximization of this thing can be turned into a supervisor training using is specifically designed loss.
So, the data.
The class one for the supervisor training, are the data. OK, so the data could be observed instances of the feature x. Okay, the ones that are really observed in experiment or of course, boy, two instances of the data some of these diseases that that
you may want to do to generate. And so in our proposal, the data are used for training, okay, each time you see the data you have to train the.
The second class for the supervisor training is a reference sample and reference sample is in collection. Okay, it's a sample of synthetic instances. Okay, of the features, accept that follow the distribution that is expected in the reference model that
is in the standard model.
Without again, going into the math that brings to this scheme of training that you can understand that some reference sample is absolutely needed for the strategy to make sense that because it gives it provides you the notion of normality.
Okay, so it tells you what that is mean that the data are normal so they will be to the reference model distribution, okay this distribution here that of course we do not have in laws that form all what we have are the Monte Carlo data or data driven
estimate of the, of the distribution of the feature in the reference. I bought. Okay, so this is the second class for for training, and that the origin of this segment of the second class for our supervisor training is not important, as I say that they
can come from Monte Carlo simulation they can be weighted the data sets, or they may have a data driven origin, you may have a control region and want to transfer from using again Monte Carlo or using some simulation that is based on the knowledge of
the reference sample you may want to generate these sample in a data driven fashion. Okay, they decided you works in exact same way.
There is nothing different absolutely nothing different in this reference sample in its composition natural and the properties test to have but then what we typically call a background sample in regular searches model dependence searches or measurement
or whatever other activity indeed and we want to do at the LSE. And again, as in regular case, more or less, we will like to, to have as abundant as possible reference sample.
Okay, in particular, a statistics of the reference data should be much larger than the absurd data set. Okay, or if you want to then the expected number of events that in the reference hypothesis, are and again this is similar to regular cases where you
would like to have Monte Carlo federal law and so this will need you will need the larger background samples.
So this is a schematic view of the way the algorithm works. You are as input. So they input, as the data, which are given in a completed the ambient form and this is a big storm just for illustration, then you have this reference sample someone else you
you go through the supervisor training, which provides in the first place, the brain that neural network, F, which is the thing that will go in here to the exponent of this thing.
so it's essentially the strain and neural network is essentially the maximum likelihood best feed to the log of the true distribution. Okay, the one that truly underlies the generation of the data, divided by the reference distribution.
And so this f gives the opportunity to perform a some investigations of people prefer potential tension with the data that you may have observed.
It may offer some first handle for a first characterization of the access for example for this specific data set where there is an adaptation of events around 0.8, the network realizes that that is this speak here.
So, by casting on this Bible one may discover, for example in the simple example that the tension, okay that one, someone sees by computing statistics is due to the events following you around in detail of the distribution around 0.8.
And then, of course, that the test statistic D, the one that we really want to use for our hypothesis test is obtainable by the milestone. Time to the minimum of the loss, using this specific specific loss, maximum likelihood loss at the end of training.
Okay.
And so this is the schema by which are given the data we compute the statistics, but also this function that is the regression to the, to the distribution ratio blog distribution ratio that can be used for first got the addition of the access, if you
observe but one.
Let me start from a simple illustration of performances that we do in a, in a one dimensional example with the exponentially. So it's one feature exponentially distributed in the reference model so the red the here.
It's indeed the reference or distribution, the total number of expected event, which I call background here in the reference hypothesis is 2000s, and I'm going to also consider in as a possible just for illustration of the performances I'm also going
to consider in alternative signal the presence of a signal which is an ocean localized that 0.8, which I call peak in detail of this distribution we didn't expect that of 10 of 10, and then purpose on fluctuating events.
So, the.
This, this first line refers to data set that which are thrown according to the reference distribution according to the exponential. And here you see that that.
And here you see like the what the neural network learns to be the distribution of this data, you see that the neural network output is correlated, of course, with the statistical fluctuation of the data for example this point that one here makes sense
and then the netbook go a little bit up with respect to the reference distribution which is the direction like here. Here for example the structure around 0.8 corresponds to a structure in the networker right here the thing seems more normal and indeed
the network goes much more straight and correspondingly the value of that that statistics here is six while here is to album, owing to the fact that there is some fluctuation that the network gets seen as a potential.
Say, non standard that manifestation in physics.
Then are you ever in the bottom.
The bottom histograms, you have is that the three data sets which have been generated instead that according to this new physics one picking the data distribution.
And so you see that when they do open realize the network realizes that there is something up and down 0.8, the more or less precisely. Okay, depending on how many events up and to follow, really, in this 0.8 the region so for example this last in this
case, the bit fewer fewer events, less localized. And so the networker here just makes a little shoulder, doesn't mean it doesn't manage to make a big like this other cases, and you also see that the value of the of the statistics is larger, okay it's
smaller.
Sorry. In this last case, but it's actually because the network has not been able to see that there was a resonant beacon because the events, the access the signal events in this, in this particular sample are less abundant.
We want to get to, of course, the more quantitative assessment of the performances which we can do by gaining the simple one dimensional example. So, our season hypothesis test.
So, as any other hypothesis testing is, first of all, to.
We need first of all to compute the distribution of the test statistics by the ability in the reference hypothesis. And this calculation of this P of t given out and will give us the cap of the ability to compute to evaluate the p value.
Once we observed the data we will have a value of the and the apathy of this be of the given our distribution we give us the feedback we can build this by by experiments this or we repeat training as a plane that each time we see each time is needed that
we have to train and so we repeated.
You have but
what is your.
What is your.
Okay.
So we would have been did the training on certain number of police up the data, and we make this histogram is orange juice Dora, of the, of the distribution.
the chi square with 13 degrees of freedom. Now 13 appears to be the total number of free parameters of this very simple network that we are using the word for neurons that we are using in to distribute discuss this one dimensional accent.
Okay, so these signals agreement that we certify synthetic formula that I will tell you more about later in law here you ever happens again.
In terms of statistical terms. So throwing toys. This time, according to this NP one new physics, the one interview slide with the peak at 0.8. And, and you see the distribution of goods is different from the other one, showing that the that the network
that, on average, its attention. With respect to the reference model, due to the presence of new physics. The disease the distribution of tea in over the stores, can be converted into P value by using the deal to our distribution here, and then finally
we need to have that score with the regular former.
And so what I'm plotting here, in particular on the battle axes. Okay, is a scatter plot that is the score. So the significance that we do, we do get by throwing toys according to empty one and and running our, our strategy on the x axis on the example
axis instead that you have something that we call here that ideal will be called the Zed reference okay the reference significant so the idea significance in in the in the rest of the story.
Now, this is something that we always do in our blog so this axis is supposed to be to represent that in absolute independent measure of the discrepancy of the data set.
So I'll match, some I want to find out much new physics, that is to see either in the specific data set, data set that time, attend, or in immediate terms, which is this, this red line here.
So in this case, the idea is that score on the reference score. The score is provided by the optimal test for new physics one model, essentially is the dead score of the optimum non patient test that we can hear perform exactly because we know the distribution
exactly in this one dimensional example. So, essentially, this axis corresponds to the sensitivity.
That will have a fully model dependent search that is fully optimized to see, in this case and be one. So to see the new physics that is being considered for testing instead on the other axis, the model, independent, I agree.
So, clearly this is not that it's clearly by consistency. The significance that we get in the model independent way must be less than the one that we get on these other axis okay because this is a model dependent optimized search, that is necessarily
optimizing to DVD with respect to any other strategy to the specific model attend ok so the purpose here is not to try to make our significant better than the one is other axis because that, that will be wrong.
If we had found this, what we see is that there's that there is a correlation. Okay, so there is a deduction for example in demo media and you have 4.8 ID and versus tip we do that we find in the model independent with our strategy.
there is some degree of correlation between how much is significant so much in physics, it won't be evident, easy to see, and they'll match we do see it. And also notice that if I now change.
Okay, new physics model so does NP do now, is an access in detail as you see here in the in the block that we do see find a simulate blocker. A similar correlation so they already went into physics three for some reason you do see the same thing type
of correlation.
And, and also you see that the median idea now is a bit lower than we what used to be, and the median of the neural network is also is also lower.
And finally, we can also consider another type of new physics, which is a big in the Balkans is very small peak here which you can see because there is quite a lot of statistics, and we observe a similar similar blocked.
A similar block. So, this is to, to say that as we seen the simple example but also confirm the moral structure at the kg that of course we have been dealing with in the last few years, there is a correlation between how much tension we do see on the
data and how much potential that is to see. Okay, so as quantified if you want by this correlation, it means that idea, and Zed and the the correlation.
Okay, as you see by scrolling through these different Plaza is not very is weekly sensitive to the type of new physics so this is a very different new physics type than this one than that one because of the different natural re which shows up.
Okay. Right. So, now let me go to the, to the, to the sky discussion of discussion topic formula that is going to be important that we will see for two reasons and in particular in preparation for the discussion of the inclusion of new sensor parameters.
So, we know that weeks will the theorem ensures that the maximum likely test statistics is the. That is what we're computing is we be distributed as a square with as many degrees of freedom, as parameters that are two feet.
So the number of W parameters here, we fixed the number of degrees of freedom of the chi square, square distribution for these statistics in the reference hypotheses.
And that well as far as the number of degrees of freedom so the number of parameters that we are feeding is literally the number of free parameters weights and biases of the neural network.
large enough large enough for people to the complexity of the model that is being feted. This is a. If you have a very simple model like a straight line.
Okay, then. Can you answer the beginning made a face to make this exotic formula hold. But if you have a very complex model, then even if you have millions of events.
These models can be complex enough for to adjust itself to fluctuations so that occur in a very real region of this data set, so can be sensitive to very low statistics portion of the data set.
So if you're familiar know thousands of millions of events, you may not find you will not find compatibility with a week's worth the theory.
For us actually it's more important to view the statement that the other way around. Given that the statistics of the data for us this is fix it because it's the one that corresponds to the luminosity that you're willing to study.
And so the, you can say that the theorem also provided the feet model is simple enough for the given for a given problem and for a given the statistics of the deep.
So what we're gonna we're gonna explode, this as a topic for tomorrow.
As I said, the bed is very heavily when we will be discussing the nuisance parliament and inclusion, but also at this stage we employ this empathy for more for model selection.
Indeed, that I say that I want to use a neural network, but I still have to tell you what is the architecture of these neural network and set all the regurgitation parameters for example, with clipping that we employ for as a maximum bound on the sides
of each weight, and by us, that we employ to realize the neural network. But in general, all David parameters of the algorithm should be optimized. According to some criteria, and we use this chi square compatibility as the criteria we the idea that overfeeding
literally means this sensitivity to a low statistics portion of the data set, but very much like I described the above here is the model is too complex that then low statistics day for example of the distribution, can, can, can be overstated.
Right. And so the value of the statistics individually and can be very much influenced by this low statistics portion of the data set and so you are not interesting topic regime.
So I think we can move so we can draw this equality that revelation of esoteric formula would mean that we are sensitive below statistics portion of data set, which would mean overstating that of course we want to avoid in our, in our implementation.
In particular, this criterion, as we see is using the in the following way one can select in architecture, and somebody certain number of neurons and layers.
Then one will have to introduce some of the additional parameters that for us is to say this a weekly finger, and one can select the value of the wiki, in the, in order to ensure compatibility with the chi square.
In order to ensure compatibility with a chi square. This can be done for each different for many different architecture, and the idea of course is that you would like to get the biggest possible architecture.
So you may extend the architecture in terms of number of layers or in terms of number of neurons and and try and find out to the largest. Okay, architecture, for which it is possible to identify a value of the wiki being that ensures sketchpad company
be okay. And that, by the way, can be treated in, in terms of available computational resources. Okay, so this will produce at the end is election on the, on the possible architectures, as because not all architectures, when they become too big and too
flexible. There is no way in which biweekly being, we can ensure compatibility. So at the very end. This notion, plus the notion that we would like the natural to have to be as expressive as possible to have as many parameters as possible because we want
to be ready to feed the, the most possible complex manifestation of new physics will provide the ladder, if not unique, at least a very narrow down very much the models that we should really consider for our.
So this is an illustration of how they were keeping selection works again in the simple one dimensional example, if the weekly being kept up your cat is very lose you see some I see some violations violation of the SMS magic formula of the statistics
of the distribution, which comes from what I told you before so sensitivity to sparse data points, which make the fact that they are on the right, if of course the weightlifting is just too strict, then clearly you have the vibration that comes from a
thing come from the boundaries in the parameters that not obviously, of course violate the which was the theater in between that that is a region where the compatibility with chi square is pretty good. And then you can optimize this compatibility for the nice case value of name, as you see here pro
of nine, as you see here pro provides nice compatibility, which we can also monitor in across during training by competing, the person deals of the empirical distribution that we see with the person feel expected for the case where we, again, 13 degrees
of freedom which is a number of free parameters of this net and the blog also gives you an opportunity to see that we do need that quite longer trainings.
Okay.
In order to make to be fully sure that these curves are flooding down.
You know, especially in, in, In a. So in this case, where we have compatibility. So, this, this large number of training, of course, is makes trading guessing execution, relatively long also the counter that we make this blotter, and of course to compute
the distribution of so this is we have to run multiple choice and so Run, run, trading, more and more times.
Okay, Mike. So, the, the, the last part of the talk is about the imperfections that the effect that our machine learning basic strategy for physics.
For model independent in physics searches. The.
Clearly, the predictions that we can make for the distribution of the features in the standard model or in the reference model are affected by the imperfect knowledge of, I don't know, part of the submission function or the total lunacy people like that
the the detector, the modeling of the detector that is in perfect. And these things are regularly described as the new sensor parameters that we're going to imagine, we know that the be constrained by auxiliary measurements, even if we all know that the
the weather cases like purely theoretical use of parameters of purely theoretical origin, the picture is actually not this one but we all, we still treat them in the same way.
So we are going to have a set of auxiliary measurements that are orthogonal to the data of interest that blow up reduce the likelihood for instance parameters, and the nuisance parameters, no define a reference hypothesis that is now composite, that is
now parameter dependent. so the main difference. So, what I was doing previously, is that now that efforts are okay is now we're family in market here in green of new dependent.
Simple hypotheses that define a composite.
The maximum likelihood the principal works in a very similar way. So now we have a denominator I bought this is that gets maximizes maximize the over the values of know.
And of course the likelihood needs to be the complete likelihood so this, the likelihood of the given the data of interested will be the extended person likely similar to the one that was written in a few slides the time so the likelihood of the noise
comes from dictionary measurements, these are sometimes called the, the, the prior for the nuisance and these calorie here means precisely in the nominator. Instead, what you have to advertise your alternative hypothesis, maximizing the little possible
parameters, W and the new, the alternative hypothesis is constructed. We constructed, very much like in the new nuisance case, that is to say we perform a local individual space, escape a features based, the scaling of that the reference distribution
that effort and distribution now depends by automatically or no okay precisely because now we're dealing with new sensor and the this rescaling is performing with exponential again of a neural network precisely like in the previous case.
So, in the alternative distribution or the effects that can be accounted for by the standard model are inside here. Okay, I correspond to feed to to feeding into some value this new parameters.
While all work cannot be accounted for from a standard model if you want this in this explanation and so the neural network is ready to see effects that go beyond the beyond the reference that movie on this.
I also like to like to go a little bit fast here with the math that, but there is a couple of items that it's important to understand. So, the way in which we proceed the to the evaluation of the statistics in the presence of noise sensor goes through.
Selecting at one point in this getting space of reference hypothesis that they are not that corresponds to the standard model the reference model with the nuisance Parameter Set today central value.
Typically we set the central we choose coordinates in engine space such that the simple value is zero. Okay, this, this value here okay this point here corresponds to the central value reference.
I bought this is that we use. Okay, by dividing and multiplying here in this formula, and then we split this log into the difference between two terms.
Okay, that are dow, and that.
So that's that is the second part here.
And this delta term knows nothing about new physics knows nothing about the DF neural network. Okay, is simply the likelihood ratio, maximizing over the nonsense, is the likely ratio in the standard model in the reference model itself that you see depends
on know, and that I'm willing to maximize over know, this is that, but this delta this object here is actually the interest for any statistical analysis that you may want to perform for the, for the specific data center that you're locked on to the center
they're looking at.
Essentially, for example the maximum value of this nuisance into the best feet of the nuisance given observation of the data and of course also, even though facility management is something that you would like to know, just to know if there is some cool,
pool of contention of the noise on set the record was the data with respect to the prior on the noise of the day that you used to have.
And so normally one approaches the calculation of this delta. And so the maximization of that is so the argument of this maximization, for example by beaming data.
And then, studying the dependent stuff because section in each being on the news as parameters then constructing their own personal likelihood and then maximizing, in principle, we could proceed in exact same way this is the exact same object or wherever
we find the more effective and successful so more and more easy to compute the effect on reasons parameters on the likelihood So, and then computer that computer, the argument of these maximization which is then going to be going to be maximized.
So we just computer directly the Amina the likelihood ratio between the new dependent reference hypothesis and the central value reference I bought this is computed.
the likelihood locally in a minute fashion fashion on the features space. We will do so by learning, the effect of the Muses parameters. This is a nice little rally by a regression on the distribution ratio between the distribution independent distribution
normalize to the central bank.
So this can be done by another station of other machine learning techniques that a method like you'd free in finance, for example, the mother minor project.
And that I will not that time to explain, you probably many of you are familiar with this would require a dedicated seminar.
But the essential idea is that one can parameter eyes is like you the ratio, if the effect of the recent on the distribution is no one can decide to tail expand the log of this ratio that's why here there is an exponential stopping at some finite order,
for example first or second order, which is that the analog of what one does. When one wants to infer effect of nuisance to on cross sections in the mean for example if you expand that linear order for example in essence or higher order, and then you
your livelihood using these expanded news as dependent procession here is more or less the same thing. Only that that this this ratio is learned locally in the future space, and it's learned through a machine learning methodology by which one can train
some neural networks that bad, one that I, too, and so on. to.
Dominique, to be similar to approximate this through delta functions that come from the dealer response. Okay.
So it says he just to be to be aware that that the quality of this reconstruction that this is a are well this deltas really correspond with the true coefficient functions so I well we have access to this locally to this ratio can play a crucial role
for the for our approach. Also be aware that these type of techniques are being developed independently from from our application, and also notice that the early.
Among the elements that underlie successful reconstruction of is that the terms that is the availability of samples of data in the reference I bought this is with non central values for the new instance, again, like you know regular.
So, this is just either just, you know, the Bluford the dependence on the new instance of your cross section you will need non central value nonsense, nonsense.
Louise answer.
Background samples, which we, in this case use for training that this.
These networks that aim at learning defect of losers parameters of distribution and once this is done that this is just the maximization that we built for the delta term.
I'm not going to go through the math, but the term is the one that really involves possible new physics. And it's constructed by training, again, there is a supervised training a using data as one plus zero and central value reference class the one or
other way around. And the important thing is that the second training sample that we have to employ by this construction, by which we split the, we divide the problem in the calculation of power and data. Now, the reference sample is only one guy that
is a unique reference sample that that is generated according to the central value nuisance hypothesis. So, this doesn't know anything about the presence of noise sense and this is taking care from bye bye bye data.
And this of course is essential because if we had to employ multiple reference samples with multiple values have been with us will make the Albert, very demanding.
The until I was saying that that the strategy is again a supervisor training that there is a slightly modified the maximum likelihood loss and also.
The other thing you have to be thinking but count is that to evaluate this, the man Pearson, the maximum likely the 36 you have to maximize over both wn, so there is also a step involving the maximization over these new parameters which which enter through
the otter issue.
Okay, but okay so technically apart. If we do all these things right if you compute right now, and delta, then what we're going to get by the weeks, was the theorem, we are going to get to the distribution of the delta t equals 2000 miles that is again
square with a number of degrees of freedom, which is equal to the number of parameters of the net. This is again an application of the weeks was the theorem that we will that we come out automatically if we do all right. Without need have, of course,
optimizing any algorithm, any any parameter either parameter of the algorithm. Okay, so we say that we selected the weekly thing in order to ensure compatibility with the chi square. Yeah, but this is a different problem is different problem the compatibility with a square to come out dramatically with the same value of the wiki thing that you already
optimized. And here is a really important aspect. Okay.
That is essential for any frequently hypothesis test. That is the fact that the distribution of your statistics must not depend solely on the nuisance on the value of the Muslims parameters.
Okay, so we like so much to do maximum likelihood, because by the weeks was direct, we have is I think that the formula, which in particular mean that no matter what is the value of new okay mama no matter in which point you are just getting reference
composite hypotheses, you will always get the same distribution. So that is the grief I think the dependence on the distribution the true value of the noise, because if it was not like this, then you should try all the possible values of the noise sensor
compute to your distributions, as a function of that. and then the only frequent this notion of a valid p value would be the maximum value scanning or possible value so know which is something that you cannot.
So this explains why in data analysis saying that he context, especially this maximum likelihood is so much more widely used, and these are the same, but also for for us as we can verify here, you have been green, the distribution of the Tao of the Dow
part of the statistics that you have in particular in this right, plus here, you have plotted that when you have nuisance parameters. These are tools as parameters correspond to scale in the PT, or daughter normalization.
When you set the true value of business and parameters one sigma in different directions, a way from the central value of zero, you see that the green distribution is quite different from the target chi square which is described here.
While y, if you then subtract to the delta term computer does explain to you. This engineers the cancellation. Okay, there is a correlation with the intelligent data such that T.
Instead, is the is the blue histogram, that follows very closely, okay that recovers very closely that's important formula. And so this plus really show the independence of the distribution on demo instance parameters to throw the validity of the topic
for.
We want and also check what happens if we have a new physics in so you can test the performances of sensitivity to the algorithm as it did before, to this and be be one, two and three models.
Okay.
And this serves to illustrate that strategy, as we understanding it is not only robust to non Muslims, it's also effective. That is to say, we don't lose completely sensitivity because of the noise answer, we lose essentially as much sense as much sensitivity
as we should, but example new physics, to which is in northern Arizona and the fact that insensitive to scale variation to scale and 70 so much more than that so big and this is through both with our model independent sensitivity, million sensitivity
that you see reported here, but also with a idea reference sensitivity the one of the regular test optimizer.
As I was describing describing before for each signal independently, you see the ratio of our median significance over the reference one, and you see that he's lucky be more stable, then they said bar itself, showing once again that there is a correlation
how much there is the seat to see and how much we see, and that correlation is not destroyed by the presence of by the presence of noise.
Okay, so I realized that I'm going a little bit longer.
So, let me.
So, let me move to this last part, which is the illustration of the application of the strategy to a problem that we consider nearly realistic.
So, it is an emulation of two particle final state at the Elysee, we say we say it's almost realistic because it does a reasonably high dimensionality of the feature space, which is five.
Because we include systematic uncertainties on escape at scale and normalization at the realistic level with three scenarios that correspond to the various small uncertainties we have for nuance the larger for electrons and then for electrons.
And also because we restrict ourselves to deal with a conservative upper limit on the number of reference data points. The idea is that may be that that goes simulations are too demanding to generate a very large reference sample, or maybe the statistics
is limited for data driven backgrounds so and so we made this conservative actually choice I mean, this choice that illustrates the, the additional difficulties that there are we do that we dealing with the problem in this, in this case that that to do
the unanswered since the DVD with the weekly finger okay that corresponds to the largest that is the information that we now have coming from the reference sample and also the fact that the effect of these loosens parameters that we have learned by this
like you three like techniques is more challenging just because we have listed.
This is that Assad. So model selection provides three maximally complex viable architectures. So we have, like the one layer case the player case and three layer case and with more than three layers we don't find any viable architecture in this case we
just try to make them bigger and bigger. The compatible with the chi square, and we end up but then we three possible models and this is one of them.
And all of them can be successfully validated for all the uncertainty scenario. This is it, similar to the one that you've already seen just in a more realistic case you see the green distribution tower which is very different from the chi square, it's
back to a chi square like form by this combination by the subtraction of this delta that apparently we engineered well enough to make this happen. And then we have sensitivity to type of rice on the new physics non resident one.
And again, you can check that this is the DVD, you can check it by comparing it with the model dependent strategies to search for the same that prime or if the scenario that the CCTV law works just as much as it should.
And so if you want to do a show of sensitivities that we do have been the one that the model dependent search has his states, lovely constant guard at least the bed is much less than individual.
Okay. So, to conclude that we think that the strategy now is fully defined the conceptually and methodological Lee, and that real progress requires studying a full fledged implementation in a realistic LSD fantasy context.
There are the actions that are currently being investigated.
We may target to last and final stage for a lifetime. So maybe more exotic things. And this is essentially in order to really outline the real implementation challenges, which are here we can only anticipate and they have to do with the, what I like to
call a statistically accurate enough reference sample, which would mean that we are unable to produce the Monte Carlo, to get from control region, a reference sample that is just large enough, or that is smart enough.
Because what really matters is that the sample produces an effective accurate description of the distribution in the reference a policy so for example using a way that sample, we found that it's very beneficial without the sample needs to be needing to
be large can just be mad generation of director of distributed toys since we have to run toys all over the place of course we need to generate these mysteries and again these me like why big samples, and also all the middle dollars associated with learning
the fact of movies and parameters on the like you could do play at all and they can be as optimized as possible. And finally training execution time which is considerable, which is considerable in view of the fact of these many ebooks that we seem to
need that the more we go out and dimensionality we seem that we need more ebooks and the more, the model is complex, as well. This for ballot are never going to be absolute obstruction to the applicability of our strategy is just a limit on the luminosity we
can handle. Indeed.
If you have smaller luminosity you you you may have to be happy with a smaller different sample toys duration, less most costly is less costly and of course the de facto systematic uncertainty is reduced, just because we have more statistical uncertainty
and then you may not need to learn as accurately. The new sounds like you've done of course also the training execution time. So, the bottleneck here is that you may be obliged to take the entire luminosity off on to this you can always do but then you
will have to divide it in little chunks that correspond to smaller data set, and then around the algorithm for each of them, and then provide, and then combine in some way the values which of course would be less effective in terms of Cincy DVD, then
inside the taking the entire process and treating it that.
I think it's also interesting that this potential doing additional challenges are very much connected with a generic needs of a Lucy and Lucy. Lucy analysis program.
I don't think I need to convince anybody that the study of way that the samples of generative models for me, Pastor Monte Carlo simulation, and also add things like the conceptual way to proceed in order to make toys and next week and go with what you
All these things are of general interest and the of course will will help our me to this well. They already mentioned the interplay to nuisance learning with the library and influencer, and here I'm referring to the FD studies as implemented but neither
which a Weber do also and do need actually deal to deal with the learning they affect on those parameters precisely as we do. So, this will be the deployment of these techniques could be a good reason to stimulate the developments in these areas and also
exploiting the developments that are happening. And finally, well, to improve concerning the training execution time we have not yet explored the think several aspects.
solution which will be to replace neural network with some key and cannot meet on the, and there is a paper that is going to be out them relatively soon a base it on this Falcon.
That will essentially put to zero. Okay, these, these execution time will make the training, essentially sometimes.
And I also want to mention before concluding that this development of this search this model independent searches, can also be good for other problems where you do have but two samples and you want to see if they are thrown or not from the same distribution.
So for example data validation would be one example, or even if you just want to compare to Monte Carlo generators whether they are giving the same or compatible things are not that one could employ similar techniques.
One could employ similar techniques. Of course we will. The purpose is to make real data analysis searching for real new physics or for new physics at the Elysee, and when and if these techniques, basically stopped being applied.
I suspect that the first outcome will be to find plenty of wrong with the godless or missing background so well as often happens I think when you do data analysis, but maybe this will also enable us to discover new physics, as well.
Some type of new physics that that we have not yet hypothesized that we are not searching for. But maybe is the right, new physics model. And I conclude and thank you for your attention.
Thanks a lot. I think it was very interesting. I feel they're already raised then gonna follow the order I see that because I'm over it then first.
So the first question, question is by throwing a porch.
Okay, so thank you. In fact, I have two basic question. First of all, do you understand what ethics is a more, the difference that you see between the optimal significance and the ones that you get.
This is a first question. And the second question is related to your remarks that you just made at the end of the talk. Do you think that this kind of strategy could be very useful in the case of GaNS, when people try to better.
Generate quick Monte Carlo's suffers.
Right, so I think we'll start on the second part of the question. I think that yes, GaNS, for example, in the past but I couldn't the simulation of the data response clearly fall in this category that I had at the beginning of things that I am of things
that that may may help. Okay.
Just because they will help getting more monte-carlo samples that will be using for the generation of toys or for the reference. On the other hand, that that is also somehow the other way around.
Maybe this is a if you design a new generator. Okay, for example, in general, the model visible again and you want to check if you use the same performances as they will generate or you can do it, in the sense that this is one possible application if
that of what we are doing on the development of this type of this type of strategies these can work for example if you want to check if you're ready wait a month ago that's a similar thing, gives you the same as you're known as your catalog that you generate
directly without waiting, while you will run our ticket and see if it sees a discrepancy or not, but yet the plain answer to your question is that this type of development, could be very useful in order to in order to improve but like the possibility
that we may have but then, but add these ladies are more on the case by case case so there are, there will be four suitcases in which will be limited by statistics of the reference and so this type of fast generator, will be greatly.
Concerning this instead the first question.
I don't think I have a brilliant answer so all what I can tell you is that the.
So what I have to add is that this model dependent strategy must be more effective than mine. That's all think as I know, there is this this boundary right and this comes from the fact that the model development strategy is optimal in, in particular case,
case, this is optimal really from the viewpoint of the morality in that sense of peace.
So, the, what we observe like spending mentally, is that the gap in performance is so the ratio between, let's say this, and this does depend does depend on the problem, not on the new physics, or not, not so much as I was saying on the new physics but
it may depend on the real problem that you're studying, and in particular seems to depend on the number of features. Okay, so that the more and which is also makes sense, right, you are agnostic, so if you know that you are looking to the right observer
but when there is new physics You are one dimension is only there, then it's going to be easier with respect to the case in which you are more and more like more agnostic and so you think that there are two features.
Okay, which you are searching for or three or four. So, the idea. I mean we seem to be that that the gap between this and that comes precisely from this one being very model dependent, this one the model independent and the more you make this model, independent,
the more the gap gets gets wider. Okay, so maybe I can just precise a little bit more point because in fact you discussed this evolution as a number of the new murder, but in fact for the same number of government tells you can adapt very different neural
network functions. The mothers, in fact, you, you, you could put Converse convolution networks or whatever happens inside your your design with the same number of parameters Do you have an idea of this X on this loss of sensitivity.
Okay okay now. Yeah, it's a question and it's part of the things that we have not yet explored until now our thinking, our thinking has been very simplistic, so we have so poor fit for what the fully connected the models, we have to adjust the number
Okay. And we explore this space with this idea of making the network contain the maximum possible number number of parameters, given number of layers and this defined set some boundaries, in which we can we can leave within these networks, we don't observe
a significant performance difference. So at least for the example we have studied this is not that model three is much better than model one or vice versa.
Okay. But, but we haven't investigated that enough. And, and then there could be space for promotional prices and good ideas to really select the top two yeah to exploit, right, the many more possibility that you have to define an architecture with different
layers of defense attitudes Yes, it's a good point, which.
This is the status is is relatively premium.
Thank you. We live to be determined, which I hope I'm pronouncing well, so.
I hope I pronounced it well. So, hello. Thank you for this very nice thoughts, it's very clear so it was very nice to listen. Okay, so I have a question.
If you want to add more conditions. So typically, when you do this optimization.
You often because we now study really the difficult parameter space in your physics so you always go often you converge, with the selection that has that are that are very tight.
So, you have very small number of signal and background events.
So, and if you want to add another condition so you would want to have more, so you don't want to. So you want to add another condition that if you have more choices you would select the, the, the, the selection, that, that has more left events signal
So how would you add this extra conditions so I see that you added the nuisance parameters so maybe another term to the test statistics. So, yeah, how would this go.
Okay.
I'm not sure I understand what you mean by condition. So, I would want. So if you want. So this is like kind of multiple multi targets optimization so if you want.
So you want to maximize the significance, but you also want more events at the end and you want control regions which are signal deplete. All this at the same time.
So and but I don't know how to do it.
No no I mean, the way we are thinking is, is, is very mature. So, the way I'm thinking is very much like a regular search. This is a, you do have to decide.
Okay, that there is a region of interest. This normally you do decide what is the region of interest based on the BSN model. Okay, here you don't. So you have to say like, and of course then since you don't have this, this, this, this knowledge, then
you will like this region of interest to be as broad as possible, so you will not like to go just I don't know in the extreme tale of the distribution of the laptop. So the idea is that you would like to make this lady is that on a broad selection final
final state. Okay, on this on this region of course if you need to add data live in background, then you should also be to do to worry of where you take us back and forth so decision cannot be infinitely large because you may you may not.
So, the, the, at the stage that is nothing more than this is it. So, the stage I would like to do it like like three knowledge, you would have to do it.
Yeah, yeah, yeah so but okay to make to make sense then this this dis dis senior is such region should be very broad, because you cannot stop saying okay let me search for Amanda dv, under the gv to 110 and then I do another one from what I understand
what you I cannot talk, the only criteria that we have in mind for selecting this individual is to say something broad, I don't know, to jets and matter, for example, I know something good, but not go straight into a particularly special region, because,
because the other point it will not make sense to be model independent on something that only shows up in this particular region on the face.
And for the first part for if you want to have. So you want to also select, not to have a tight selection at the end, so that your optimal selection is not the pipe.
In the optimization, or how should we think we want to think of that we do, so we do find something that is supposed to be, we are constructing the strategy that is supposed to be finding some things everywhere so we never got.
Okay, we can get to the end once we have once we have a success or or attention. But the nice thing of course of this is that well and the hope of course is that if I think is better but it will also the sensitivity will also not depend that match on
the gap, there's another study that sometimes we do for example we take the simple example and we start seeing Jesus in the DVD, instead of looking to the entire region you look, the region above a certain threshold.
And, for example in this case one can see that if one doesn't go really affords on the beacon and then loses and to DVD. We We We are so sensitive.
With with this includes inclusive selection then with the more restricted selection so the purpose is really not to select events is really that the machine will tell us where the events, do not match with us on that model.
Okay. Yeah, thank you.
So thanks for the question we move to an excellent Tony Robin.
Yeah. Can you hear me now. Yeah. Very good. Thanks for the talk, can you go back to you can you can you go back to your outlook.
53.
Yeah, okay.
So I have two questions. I mean, the first one when you say realistic electricity final state, maybe use that I missed it.
But, you know, of course the NFC Olympics with this large data samples and so on so it might be interesting just as a common to use your method or try to use a method on these samples.
And the reason I say this is that then you have also comparison with other people who took exactly the same samples right so this would be a kind of direct comparison of different strategies on this.
But yeah, definitely. One second. I think of this would really be done.
Inside this pediment on the real data, because these will fail.
If we fail it to account for the reality, right. So, I was here I was advocating my book in.
And then there is work starting a knowing this new direction, and really look at the concrete the final state because you see, if I say that I'm his mother the noise answer.
I forgot the music. So, yeah, that's what I had in mind but please be like, Yeah, okay. Okay, I mean I think it would still be interesting to compare different methods on similar topics.
But if you already have convinced experimental oh no I don't want to work with anybody I'm just saying that, make sure that they can I can I say something.
so I think the words and exercise some going in this direction, but I would like to stress the fact that the setup of deletes the Olympics.
It's more. It's narrower than the celebrities would cover.
Okay, because that's really considering a very specific case where basically you are after a bump and and what some normal news that is tested in Olympics is the nature of the jets that build the object on which you do the bump and by what we're what
his method is really trying to do is to replace the maximum like a fleet, that in the case could be like a abandoned West, something which is more generic well you don't assume the launch date.
So I think that that both.
Both the matters that contribute to the listening because in this method of trying to relax the signal I bought this is, but in different places. Because what angellist Olympics is doing is trying to replace an event selection of course once you start
so who stops with something and start start getting boosted I don't know what. But then once you add that you actually run a bump and so you're trying to basically announcing the signal selection, the contribution of your signal.
Here is the opposite thing you don't want to talk to this election instead you try to have as loose as possible selection and you act at the level of the my philosophy, without assuming the line shape of the signal.
So it could be a bump would be something else. So in some sense atheism complementarity there which doesn't mean that what you say is wrong, I mean asked me to knock I mean, these metal could be used there and we should do it.
I think, yeah.
Sorry, go ahead. Sorry.
No, but what I'm saying is that certainly this is an interesting exercise I think at some point someone in parliament was trying to know what, but just let's keep in mind though that compliment.
But then, are you not swamped completely I mean now if I think about PSM I have super small signal on a very large background, I mean I am not swamped by the background.
And the point is that we would like not to be just one dimensional example that I show you the signal to bag around the region was, this was 10 over 2000s.
So, yeah. Clearly, the, so that the thing is that that thought should be smart enough, such that, in the highly populated region, it would somehow be precisely on top of the standard model.
So, normally we try to be precisely on top of this, this mood function that you see for example, these are the example here that I have. So the network has no problem in seeing z the network is at the same time, motor and, and flexible.
So for example, please. This one is this little bump up over 2000, this one is this one is 10 over 2000s it's more easy because it's in the india india india.
But again, we think we're not and, and also. So this, think of comparing with India significance precisely means this, so you have for example here that you can see at three sigma, that we can see DTC something.
So, this one exactly so just understand that. So, for example, these ideas that score here essentially means okay in practical terms that you go and cut on in this region which is the most effective way to see this big.
Okay, and you throw out all the rest because you knew a priori that the peak was going to see here. Right.
Yeah, yeah. So of course we do lose some sensitivity if instead we consider the entire the Gnostic way but not that much.
Okay. And sorry just briefly a second question, second slide of your
54 Can you just comment a bit more what you mean with this comparison between Monte Carlo.
What example suppose you you you you generate, I mean, you an example at this carrier you told you right I mean, suppose you for example want to validate whether the mother referee waiting is is working for that particular problem, or I suppose that you
have a new tuning of PD our implementation of PDF or suppose that you want to make. I don't know some gang, that is that is that is generating Sam the generating sample that will be like as good as the one that come from the from the deco simulation.
These things you can test because our sir can be used as a two sample comparison. If you get the so if you just call the references sample one and the data sample to, then this is a strategy to see if the two samples are thrown are not from the same distribution.
And did you try this.
We indirectly Yes.
When we pulled out for example that there was a little change in the central value of the matter what was that was it. I mean, yeah, yeah. Something that ugly we did, because sometimes with the Monte Carlo is not.
If we don't do the Monte Carlo index that same way which we have done in the previous paper then for the video sorry for the training then you do see.
Yes, in some sense, we did, but never systematic No, not yet. Okay, thanks.
So, Next one is Bob causing.
Bye everybody.
Thanks, as well for me.
I'm not a slide seven.
So, you didn't. I don't think I heard the word Goodness of Fit Test in your, in your talk, but of course as you know, this was defined at the bottom of this slide is precisely the definition of a Goodness of Fit Test.
When you want to test a given hypothesis against all an unspecified alternative hypothesis. And there's a whole world of literature on. Goodness of Fit tests.
Some parametric some non parametric, but that basically your answers to Bertrand for example which of course I agree with were basically.
In the introduction of a book on goodness of fit testing. And if you're trying to have an omnibus goodness of fit test you. You're going to be worse against any specific alternative than than something custom designed for that.
So I recall this. The first fist at meeting that was actually called fist that was a slack in 2003, which were Jerry Freeman gave a talk, the jury Friedman of machine learning, authorship fame.
That's of course, several generations back in the machine learning revolution. Before deep learning etc. And he, but I just put in I'm going to put in the chat a link to this, because I think the deep theory behind this, as you mentioned, is the same
as two samples test his, his talk to us it's to sample test as a reference for what he was what he was talking about.
And the, the, the main point of his conclusion was indeed that that the power of the, of the machine learning.
Goodness of Fit Test
against different alternatives dependent strongly on the machine, the quote I'll put in the chat mentions that boosted decision trees are good at finding particular alternatives the.
Some nearest neighbors were better against other alternatives.
So I think
I just mentioned.
I think it'd be useful to to mention that this is in the goodness of fit world in which there's a huge amount of literature, precisely on the point that Tanya, and we're trying we're asking about in terms of.
If you don't know what you're optimizing against, how do you how do you how do you try to go about it that's that's the that's the, Ill post problem of the omnibus Goodness of Fit Test which doesn't stop people from looking for, powerful on the most benefit
as you pointed out, it's a really useful thing to do, but it's a way to talk about the theory of the limitations. Thanks. Thanks again.
Okay, I look forward to seeing is a Aquila.
It is, I think, Yeah, I think it's not.
that we're interested in, in a wide enough of course search region, where there is not so many tools so I don't think it's a I mean is not at all something that we know very standard we have to do.
And as far as it goes at four. I think that the concern of which mall that you have to use, it's a legitimate concern and then think that we are not yet, addressing this discussion at the level of political deepness I would like to, but I think our thoughts
are quite consistent with the idea that you can control the smoothness of the model that the neural network is spreading mentally a good, good way to be flexible but at the same time.
And that essentially what we do is, compromise on the several aspects.
Yeah, but I mean of course, if one compares problems which are compatible and in a way which is a proper, then, then so then one can we had some comparison also in our paper with other.
The second paper in particular with other strategies that you can look at.
I think the the closest parametric version to what you were doing in turn the goodness of fit world to me is something like naman smooth goodness of fit tests of those, that's a that's a parametric attempt going back to name into to try to have a smooth
distribution. The true testing against for.
Anyway, I think, I don't, I didn't hear anything you said that, that, that was a variance with with what I'm familiar with in the goodness of fit worlds because it's very nice.
Thank you very much.
So it was all done.
And I think this is the last question.
Yes Hi, thank you for presenting this idea.
So I was thinking in the context of this slide 21, where you show the, the test statistic distribution for the reference hypothesis.
And I was thinking in the spirit of using data driven simple to model the reference distribution and here.
So one way to kind of sample from this is to first determine the generative model, and then kind of sample from this generative model.
But I was thinking, would be enough good approximation to just use bootstrapping, so we can have just bootstrapped, the same from the same data driven reference institution samples and then retrain the think and then we produce this temple, would be good
Does understood that that they the answer.
The answer is problem the problem depends, to some extent, so much accurately you want to this reference to be and in statistical terms, this really depends on the problem so I don't think I can answer in general to this.
Okay, thank you.
So I think I think we're done right.
I still see Bob and raised but I think it's from the previous one.
So, yeah.
So okay, so I think we close here. Thank you again for for the, for the interesting seminar, and also everyone for joining.
And we will see you next year, gonna have a seminar in January. On the SP computing system, and you will be notified in advance. But, of course, when it's going to be the most likely because I don't think this efficient is going to change soon.
Thanks everyone for joining.
By. Thank you. You have an invitation again Bye bye.
Bye.