WH11NE- Sunrise: Okay. So good afternoon. Good morning. Good night. Wherever you happen to be. It's very, very nice to be back at the lpc, and well, 1st of all, can I confirm that you can hear what I'm saying? Those of you who are far removed from the Lpc.
WH11NE- Sunrise: Can you hear me? Okay.
Andrew Askew: I can hear you.
WH11NE- Sunrise: Okay, fantastic.
WH11NE- Sunrise: Okay, so so my goal today is to tell you about 3
WH11NE- Sunrise: machine learning methods, I think might be of interest to you. But before I do that, I just want to make a few
WH11NE- Sunrise: editorial comments. It's a good habit. There is often to go to this website. The review, the thing that's interesting here is that if you look at this plot you'll see that there's really an exponential rise in the number of papers on machine learning.
WH11NE- Sunrise: And actually, a lot of them are quite interesting, right? I mean, some of them are very interesting. In fact, just a couple of days ago there was this really nice talk by Daniel Whiteson that showed that you can actually now do amazing things. Things that we thought just 10 years ago were impossible, can now be done.
WH11NE- Sunrise: and in fact, it's so overwhelming that you might come to the conclusion. Well, hey, I'm not expert. There's nothing I can do that's useful. So let's just bail. But actually, no, there's still lots of interesting things that can be done with relatively simple ideas. And that's what I want to talk about today.
WH11NE- Sunrise: Now, one thing about machine learning.
WH11NE- Sunrise: as you know, if you read the papers is that you cannot be on the 18 without a highly contrived acronym for your message. Right? It's really important. And you know, if you can only afford, you know, if you're going to get as far as building, you know, abbreviations. Iqn. Sorry that doesn't cut it. It has to be an acronym right? And so I'm gonna talk about 3 acronyms.
WH11NE- Sunrise: Iqn PIN and Rtde. So you can see clearly. I'm not quite part of the a team, because Iqn is not a word, nor is Rtde. But I'm getting there right PIN. None of these are my acronyms.
WH11NE- Sunrise: So this is what I'm going to talk about. And I'm particularly taking these 3, because I think they are particularly interesting and relatively well.
WH11NE- Sunrise: so unlike Daniel's talk. Right. My slides are true mathematics. The point of that is that when you're done with this talk, and you want to learn a bit more. You can sort of look at them and do a big, deep dive so your head might hurt a bit more than it did with Daniel's talk. I apologize at the time.
WH11NE- Sunrise: so if you simply just go to pet Ml, and just look at a few things that it's it just
WH11NE- Sunrise: filled with things. Some of these are from Ml. Some things are from other places. So I'm talking about. You know, Iqn. There's this amazing paper that came out, I think, last year on diffusion models. There's this extraordinary paper on physics for neural networks, for solving the dyson-schringer equation of all things
WH11NE- Sunrise: and and so on. So there are lots of these things, and I'm going to talk about each of these, not these particular papers. Well, I will talk about the 1st one. But I'm going to give it
WH11NE- Sunrise: simple examples of of these things that I think will be of interest.
WH11NE- Sunrise: So the 1st thing we we shall talk about is IQ. But, by the way, please stop at any time. This is supposed to be relatively informal, and and I would really like questions as we go along, to make sure that we're all on the same page.
WH11NE- Sunrise: So let me begin. So one of the great things about being an old fogey is that you know lots of people, lots of young people from around the world who do all the work and all you have to do is stand up and talk about their work. So this work is done by actually a former student of mine, Michelle Conchira, who's now college.
WH11NE- Sunrise: Braden is at University of Maryland, former college students, and much into Ali. So this is really great. And actually, this
WH11NE- Sunrise: summer, for example, I'll be working with a student from India, along with Sergey and Pushpa, on some projects which is just marvelous.
WH11NE- Sunrise: So a very, very common task, right is to model these kind of density, the probability of y given x and additional density.
WH11NE- Sunrise: Y can be anything. X can be anything. X could be parameters, etc. So here are a couple of examples of things that you may want to do. You may want to model, for example, the direct response function. If you're writing, for example, a parameterizer simulation where you actually have to code something and code response functions. Then one of the things that you want to be able to do is to map, for example, you know the
WH11NE- Sunrise: 4 momentum at the generator level to the 4 momentum at the level of reconstruction, for example.
WH11NE- Sunrise: or you might want to model a statistical distribution where Theta in this case are the parameters that you care about, which may contain uses parameters as well as parameters of interest, otherwise, a bunch of data. So if you want to model that and so on. So there are really a huge number of things of this nature that you would like to be able to model.
WH11NE- Sunrise: And of course, as you know, this has been done in many, many ways. But we're going to do it. Using this thing called Iqn. This is quantile networks. So what are these things? Well, 1st of all, let's begin with something very simple, a cumulative distribution function.
WH11NE- Sunrise: right? And just to give you the definitions to remind you, it's simply the probability that some random number is less than some fixed number y.
WH11NE- Sunrise: and it could be conditions on something. So this one is conditioned on X,
WH11NE- Sunrise: okay, so this is the Cdf.
WH11NE- Sunrise: not the one that's used to be over there. But the Cdf, and so
WH11NE- Sunrise: that's something that's important to to know.
WH11NE- Sunrise: So what is a quantile
WH11NE- Sunrise: quantile? Well, if you actually take the inverse of this inverse function you can go from. Instead of going from Y to Tau. You can go from Tau to Y, and that is called the Quantile function.
WH11NE- Sunrise: I can see immediately why that could be useful. Because if you could somehow model this quantile function, then you could sample Tau uniformly.
WH11NE- Sunrise: but between 0 and one. And immediately you have an event generator. Right? You can generate. And so this is a useful thing to be able to actually model. And it's very in principle, very simple.
WH11NE- Sunrise: So by definition, an implicit quantile network is just any neural network whatsoever
WH11NE- Sunrise: that is able to approximate this quantile function
WH11NE- Sunrise: and typically this, this function that you want to approximate is conditioned on both the Tau number that goes between 0 and one, as well as the conditioning number X, which could be multidimensional. And Theta are just a bunch of parameters that you're trying to fit.
WH11NE- Sunrise: Okay, so that's that's what the 9 Qa is.
WH11NE- Sunrise: Nothing more than that. So any new network, your favorite new network doesn't matter what it is. If it's disabled, possibly to come to our function, then it is an Iqn by definition.
Sam Bein: Harrison.
WH11NE- Sunrise: Yes.
Sam Bein: Oh, yeah.
Sam Bein: can I ask? Hi, have you? It doesn't seem like you have assumed anything about the dimensionality of X yet. And I assume that you are not making statements only about when X is a 1 dimensional.
WH11NE- Sunrise: True.
Sam Bein: Wanna confirm that that was the case. Okay.
WH11NE- Sunrise: That is the case. Yes, this is very, very general, and in fact, these things are most useful precisely when the problem is multi-dimensional.
Sam Bein: By the way, Sam, where are you?
Sam Bein: I'm in Saint Johnny. I'm in a Belgian office.
WH11NE- Sunrise: Okay? All right. Yeah.
WH11NE- Sunrise: So the point is is that this model that once you've built it is both generative, that is, you can actually sample, use it to sample like this. You generate a bunch of towels from uniform distribution, and then you plug in these tiles into your into your function, whatever it is, and that then
WH11NE- Sunrise: produce for you a bunch of y's. And so this is a generative model.
WH11NE- Sunrise: But and this is also what's very interesting is that
WH11NE- Sunrise: like things like normalizing flows. For example, it's also a regression model, because you can then take the derivative
WH11NE- Sunrise: right. Take a derivative of your function. F. With respect to Tau, to the inverse, and lo and behold, it actually gives you an estimate of the actual density.
WH11NE- Sunrise: So if you, if you are really able to model
WH11NE- Sunrise: this quantum function with sufficient precision. You both have a way to generate data as well as to actually get a smooth approximation to the conditional density.
WH11NE- Sunrise: So this immediately shows, you know, that you could actually generate likelihood functions for this right? What would you do? You basically generate a whole bunch of faces. That is the X's. For every X. You simulate a bunch of data, you collect all this data. These are your trading samples. And you simply construct a quantile neural network. And and then, having done so, you use auto diff autograph.
WH11NE- Sunrise: Take the derivative respect to Tau, and lo and behold, you have your approximation to a likelihood function.
WH11NE- Sunrise: and so on.
WH11NE- Sunrise: So now for a bit of mathematics, because now the reason I the reason I tend to give talks where there's mathematics is because I I
WH11NE- Sunrise: today, we have these amazing tools that allows essentially anyone to do very, very sophisticated calculations, right? Machine learning calculations.
WH11NE- Sunrise: However, I still think it is useful to know and understand the underlying mathematics behind these things for 2 reasons, one. So that you're not fooled into thinking that this is magic.
WH11NE- Sunrise: And 2, if you understand the underlying mathematics, you can actually do amazing things with the thing that you've just trained. Right? That's that's what's important.
WH11NE- Sunrise: So this is the loss function that is used. And let me just quickly go through it. It has 2. It has 2 pieces.
WH11NE- Sunrise: So if it should turn out that y is larger than the the function. F,
WH11NE- Sunrise: then then this is the loss. Otherwise this is the loss.
WH11NE- Sunrise: And and if you use this loss function, and you train in a little way, lo and behold your model, whatever it is, it could be the most sophisticated transformer in the universe, or a lowly single, you know one layer neural network. It will approximate the ontel function.
WH11NE- Sunrise: Now.
Claire David: Harrison.
WH11NE- Sunrise: Yes.
Claire David: Hello on the slide? 10, you say. Sufficient capacity. Where will that sufficient capacity be embodied in the math? Formalism.
WH11NE- Sunrise: The the sufficient capacity is embedded is, is is important in allowing
WH11NE- Sunrise: your minimizer to arrive at a point in the parameter space, such that the model F can approximate the quantile function accurately. So if you have, if let's say, for example, you have a just a 1 day neural network without too many nodes. Then you'll find that the approximation, even though even though a low, level, neural network and a transformer, will both model the same thing.
WH11NE- Sunrise: but it may well be the transformer, because it has more flexibility. I more capacity, will do a better job of approximating that same quantile net quantile function. So that's where the capacity comes in.
WH11NE- Sunrise: And of course, the problem is that we do not know a priori how big a function is required
WH11NE- Sunrise: to model, a particular particular target function, right? And that's where the artistry comes into play.
WH11NE- Sunrise: So let me just so. I just showed you on the last previous slide the the quantized loss.
WH11NE- Sunrise: So I just wanna make one comment.
WH11NE- Sunrise: And I made that comment because it shows
WH11NE- Sunrise: something that's very important that if you understand what your minimization is actually achieving, you can do some very interesting things. So it turns out that if you actually just go through the mathematics of minimization of these functions. Whatever they are, they don't have to be neural networks.
WH11NE- Sunrise: There is a thing. And here it says it's called the the risk Functional. It's just a, it's just an integral over the data. And where where this P, here is the problem, distribution of the data. L is whatever loss function that you've chosen.
WH11NE- Sunrise: You can show that if you minimize this thing, you get this expression in blue. Sorry in yellow.
WH11NE- Sunrise: blue, yellow, yellow, yes,
WH11NE- Sunrise: this is what happens when you finish preparing a talk 2 h before you're supposed to give it. Although I'm improving, I am improving.
WH11NE- Sunrise: So here's a problem for the students
WH11NE- Sunrise: use this formula in the yellow box
WH11NE- Sunrise: right to show that indeed, the
WH11NE- Sunrise: function, the last function given on the previous slide does actually need to
WH11NE- Sunrise: an approximation of the Quantile function. And what is really again, very important, that conclusion is absolutely independent of the nature of the function that you're using the function class, you can use deep neural network of various kinds you can use, you know, polynomials, whatever it doesn't matter. This conclusion is still still holds. It's very, very general.
WH11NE- Sunrise: And, in fact, a version of this conclusion was 1st published way back in 1919. So it's actually quite an old idea.
WH11NE- Sunrise: anyway, that's all I'm gonna say about minimization.
WH11NE- Sunrise: So there's a particular problem that was that was looked at by, you know, my colleagues listed down here in this paper. And that is what if you wanted to actually
WH11NE- Sunrise: model this jet response function? So let's suppose the X is the 4 vector of
WH11NE- Sunrise: of a jet at the generator level, that is to say, before the jet enters the detector. And let's say the P. Rico is the momentum after you know reconstruction, etc.
WH11NE- Sunrise: Now, going back to Sam's question,
WH11NE- Sunrise: any n dimensional density can always be factored
WH11NE- Sunrise: in this way. Right? Of course, this is not unique. Right? There are many different ways of splitting this function into 4 pieces. But what's important to note
WH11NE- Sunrise: is that is that every one of these is actually a 1 dimensional distribution
WH11NE- Sunrise: in the thing that's to the left of the bar, right? Because it's conditioned on the thing to the right. And so these are all one dimensional densities. And so what was done in this paper by my my colleagues here is, that is, that
WH11NE- Sunrise: is, to take 2 approaches, one approach which is actually analogous to one of the things that Daniel talked about a couple of days ago is that
WH11NE- Sunrise: each one of these functions is approximated with its own neural network, using the quantit loss.
WH11NE- Sunrise: So now you have these 4 functions, you slap them together, and you have approximation to the individual density
WH11NE- Sunrise: that you can be that can be used for sampling. The other approach that was that was looked at was, you have single network that's able to basically output the 4 quantile functions in some way.
WH11NE- Sunrise: and that what was done. And and here are some results from this paper. The
WH11NE- Sunrise: the the dots in red are predicted from simulation, and the the
WH11NE- Sunrise: dots in Rico or the histogram black histogram are the reconstruction. So sorry? Yeah, read the prediction, and blank is what we expect. So it does a very good job on this population problem
WH11NE- Sunrise: and in principle. This is true, for you know this general for all dimensions, but in practice. Presumably dimensionality should not be too too large.
WH11NE- Sunrise: Okay, so this is Iqm a simple way to build a generator
WH11NE- Sunrise: and also to approximate, if you wish, the density itself. So if what you want to do is to actually approximate some likelihood function or statistical model, then this is a very simple way of trying to do just that. Can I ask a question about the last line? Yes. What about the correlations of the 4 one d. Yes. So the question is, what about the correlations? So here's what's interesting, right is that
WH11NE- Sunrise: this is just a matter of
WH11NE- Sunrise: fact, right? You can simply factorize things this way.
WH11NE- Sunrise: So the way you take correlations in practice when you're when you're sampling, is that? Let's say you have these 4. You have these 4
WH11NE- Sunrise: your networks right. Each of these has its own town.
WH11NE- Sunrise: Okay, each of this has its own town.
WH11NE- Sunrise: And so what you can do is that you can actually, you use. You use this function recursively, you sample from Tau, and you generate, say you generate, say, a Pt. That Pt. Then goes into the next function, and and you and you and you sample a Tau for that second function, and you generate, say, an Eta.
WH11NE- Sunrise: the Pt. And theator, and and so on. So if it's used recursively, you then actually capture the correlations correctly.
WH11NE- Sunrise: And of course, if you take derivative of each of these with respect to its Tau, you can in principle also build the density.
WH11NE- Sunrise: Thank you for the question.
WH11NE- Sunrise: So the next topic I'm going to talk about is physics informed neural networks. Now you now.
WH11NE- Sunrise: on the face of it, you think well, this is very different. Why is this? Why is he calling that a generative model?
WH11NE- Sunrise: Well, I'm using the word generative model. In a very, very broad sense. I'm using any model that can generate ensembles of things. Whatever those things might be. They could be pictures, movies data from particle physics or solutions to equations, for example.
WH11NE- Sunrise: But that's what I'm this is the way I'm using that term 10 small.
WH11NE- Sunrise: So this is what this this actually currently being done by by my colleague, who's the academic director at the African student of matheological sciences in Cape Town, South Africa, and and her students at Ames from Botswana. Oh, no, this random guy who just showed up.
WH11NE- Sunrise: Yeah. So so I'm just gonna talk about the work. That which is still ongoing, and we'll be writing the paper on that soon.
WH11NE- Sunrise: So this, what is what is a physics for neural network? It's simply a way to solve equations. Whatever the equations are
WH11NE- Sunrise: using the computational tools that we developed for machine learning. And we're going to focus on solving a second order. Differential equation as an example.
WH11NE- Sunrise: And there's lots of information about this.
WH11NE- Sunrise: I recommend, actually, there's a there's a very nice Youtube video, by Bob Moseley. That I recommend. So that yeah, there's lots of information about this topic. So what I'm gonna do, I'm going to go through the problem that we're looking at right now as an example
WH11NE- Sunrise: and giving a few details.
WH11NE- Sunrise: 1st of all, what are the advantages of using your using neural networks? After all, the solution. Differential equation is a very well developed mathematical
WH11NE- Sunrise: area, right? I mean, people have been working on that for centuries, literally. And so it's extremely well developed, and one might argue, well, you know what's the point, I mean, we already know how to do that. Well, here are some advantages of using neural network.
WH11NE- Sunrise: 1st of all, solutions are mesh free. They are smooth solutions to some problem, whereas if you use a typical numerical method, you've got to discretize the space
WH11NE- Sunrise: and sometimes discretizing the space can be very difficult.
WH11NE- Sunrise: Right so, and and once you discretize the space, the solutions exist only at the discrete points in that space.
WH11NE- Sunrise: whereas these are mesh free solutions that exists smoothly everywhere. That's 1 of our intention.
WH11NE- Sunrise: It turns out that this particular approach scales quite well as the dimensionality of the space increases and can cope with surprisingly complicated functions.
WH11NE- Sunrise: I said, this has been used, for example, to solve an aspect of the Dyson-schringer, equation, which is quite a ferocious beast.
WH11NE- Sunrise: It's basically an integral differential equation. And so this seems to be what people, finding that it scales well.
WH11NE- Sunrise: but for me the real power of this method, and the reason why such a huge amount of work has been put into it in the past few years is that it provides a way to get a solution with a neural network that is conditioned on the initial conditions on the boundary conditions. Now, why is that useful? Well, if you can do that.
WH11NE- Sunrise: then you can rapidly.
WH11NE- Sunrise: rapidly compute solutions by varying the initial conditions smoothly, and then study the behavior of the solution as a function of the initial conditions, whereas the way you would do that and the way it has been done until now is that you simply solve your numerical system. N. Times with n. Different initial conditions, and then you interpolate in order to build some kind of a smoother function. But this could be done in principle in one. Go.
WH11NE- Sunrise: and you'll see that shortly.
WH11NE- Sunrise: But of course there are disadvantages.
WH11NE- Sunrise: If your goal is to solve a set of equations with just one initial condition. Then it would be utterly foolish to use physics informed neural networks because they're much, much slower to arrive at a solution
WH11NE- Sunrise: right? Numerical techniques are now so good
WH11NE- Sunrise: that that there's simply no point in using that.
WH11NE- Sunrise: That's for a single a single set of conditions. But I said, if you want to solve it for an infinite set of initial conditions, then then by this.
WH11NE- Sunrise: this is an enormous advantage for the for the and the other issue is that the
WH11NE- Sunrise: the theory about convergence of the technique is still being developed. And so lots of this, what lot of what is done is still very heuristic. But that's also true machine learning in general. So
WH11NE- Sunrise: so here's the problem statement, can we model
WH11NE- Sunrise: an infinite set of photon orbits?
WH11NE- Sunrise: This is every every orbit is indexed by by an initial condition
WH11NE- Sunrise: in the Schwarzschild spacetime using a single neural network. So here, here's an example.
WH11NE- Sunrise: Oh, so here's my gray black hole, autistic devices,
WH11NE- Sunrise: and and the units. Here are distances in use of the Schwarzschild radius, right? So the Schwarzschild radius by definition, is a set of one, and and these these orbits are referred to as boomerang orbits. So supposing you're sitting right here at 5 Schwarzschild radii away from the black hole and you fire at exactly the right angle. It will actually circumnavigate
WH11NE- Sunrise: Black Hole and return to its starting point. And there's a particular orbit this one here at 1.5 times the Schwarzschild radius called the photon sphere. So if you happen to, you know, during student during your spring break. You happen to visit a black hole.
WH11NE- Sunrise: do the following experiment.
WH11NE- Sunrise: go to the 1.5 times the Schwarzschild radius, and with bring a flashlight with you
WH11NE- Sunrise: and shine the flashlight in the back of your head.
WH11NE- Sunrise: and don't forget also your binoculars. Look for your binoculars. You'll be able to see the back of your head.
WH11NE- Sunrise: Confirm that, please, and report. So so this is the this is the photon orbit, which is very unstable. And then you have sort of these orbits within the photon sphere.
WH11NE- Sunrise: So these solutions were arrived at by literally just solving
WH11NE- Sunrise: the perfect differential equation for different initial conditions, using standard techniques. So the question is, can we do the same thing with a single neural network that could basically cover all of this and more? That's that's the question
WH11NE- Sunrise: so a little bit more bit of mathematics. So if you go to your standard books on on the Gr and look at the Schwarzschild metric, you'll you'll you'll count to this equation u double dot
WH11NE- Sunrise: plus u dot minus 3 half U squared equals 0. So this last
WH11NE- Sunrise: piece here, that's the the part that comes from. Gr if you get rid of this and you solve that, then, of course, you get straight lines, which, of course, Isaac Newton solution.
WH11NE- Sunrise: And so that negative 3 house over U squared makes all the difference.
WH11NE- Sunrise: Now you
WH11NE- Sunrise: is simply the ratio of the Schwarzschild radius, which is given here, divided by a radial coordinate. And so we're using our Theta Phi coordinate system. But be warned. The radial coordinate is not the distance from U to the center of mass. It's just a radial coordinate because of the walking of the space time around this central mass.
WH11NE- Sunrise: Okay? So that's the question we want to solve. Beautifully simple version. But and question is, can we solve that equation
WH11NE- Sunrise: with?
WH11NE- Sunrise: If so, that the solution is- is contingent on
WH11NE- Sunrise: it's conditioned on the initial conditions.
WH11NE- Sunrise: So I'm gonna I'm gonna show the answers in this coordinate system. So what we have on the vertical axis is U,
WH11NE- Sunrise: right? And so remember, use is Rs. Over R. So when the radial cooled is equal to the Schwarzschild met radius that's equal to one U is equal to one. So the upper. The upper line here denotes the the black hole.
WH11NE- Sunrise: That's what the well, the event horizon, if you will, what I call
WH11NE- Sunrise: along the X-axis we have 5, and and 0 is infinity
WH11NE- Sunrise: right? So at 0 you are infinitely far from the event. Horizon and at U equals one. You are at the event horizon, and this yellow line across is the
WH11NE- Sunrise: it's the foot on sphere.
WH11NE- Sunrise: Okay?
WH11NE- Sunrise: And you can see that they are boomerang, boomerang orbits, because the 1st green line here is at Pi.
WH11NE- Sunrise: But
WH11NE- Sunrise: and and this slide here is at 2 pi. So it means you've gone once around the black hole.
WH11NE- Sunrise: and
WH11NE- Sunrise: and therefore they always ought to be symmetric about 5. So if I just go back, I'll just go back to the previous
WH11NE- Sunrise: pick a truck
WH11NE- Sunrise: right? So what characterizes these orbits is that you start up here, and if you get to this particular point in the orbit that's pi equals Pi.
WH11NE- Sunrise: continue back, you'll have gone 2 pi around the the black hole, and the orbits are symmetric respect to that particular
WH11NE- Sunrise: spectrum pipe. And so that's that's how we're gonna draw things.
WH11NE- Sunrise: Okay, so that's the task. Can we find a single neural network that can actually do all this with suspicion address? So what initial conditions?
WH11NE- Sunrise: We're going to launch the photon at some angle delta with respect to a line from the this position to the center of mass of the of the system.
WH11NE- Sunrise: and and so the initial conditions are are here
WH11NE- Sunrise: right where we have R at 5 or 0.
WH11NE- Sunrise: So you have to make a distinction between the angle delta, the angle of the photon.
WH11NE- Sunrise: the coordinate angle Phi, which starts, you know. So, Phi, this direction is Phi equals 0, and you go around counterclockwise. Is Phi and delta. But these 2 numbers can be mapped into U at 0 and the 1st derivative
WH11NE- Sunrise: at 0. Remember, you need 2 numbers because it's a second order. Differential equation, right? U double.so you need 2 numbers specify uniquely the initial condition. And we're going to call these 2 numbers. We're going to call these 2 numbers, U subscript 0, and the V subscript 0. So u, naught is equal to this guy, and V. Naught is equal to U dot of 0.
WH11NE- Sunrise: And so what we want to build is and see whether it would work is a neural network, that whose inputs are the angle Phi
WH11NE- Sunrise: u, sub 0 and V sub 0.
WH11NE- Sunrise: So that way, we would have essentially a solution for an infinite number of orbits
WH11NE- Sunrise: each index by these 2 initial condition values.
WH11NE- Sunrise: Now
WH11NE- Sunrise: there's a paper which I listed in an earlier slide called The Theory of Connections, math mathematical Papers, actually easy to understand, which is really great.
WH11NE- Sunrise: and but what is really nice about it is that it shows a way to
WH11NE- Sunrise: explicitly include the initial conditions into the solutions, and you'll see why that is important in a second. So if you look at this formula down here.
WH11NE- Sunrise: you'll see that that
WH11NE- Sunrise: U sub naught and v sub naught. These are the initial conditions. So u sub, naught is the function is the solution. U at 5 or 0,
WH11NE- Sunrise: and V is the
WH11NE- Sunrise: 1st derivative of the solution at Phi equals 0, and if you just stare at this for a while, you'll you'll convince yourself that it does indeed satisfy the initial condition. For example, if I set Phi equal to 0. So these F's here are some neural network.
WH11NE- Sunrise: If I set 5 equal to 0, then these 2 terms cancel.
WH11NE- Sunrise: And of course this term disappears because it's multiplied by 5. And sure enough, as advertised user size u sub 0.
WH11NE- Sunrise: If I take the derivative of this solution, U dot
WH11NE- Sunrise: and go for the same argument.
WH11NE- Sunrise: then, indeed, you'll also see that it says U dot of 0 satisfies V. Sub 0. So this is a great way to incorporate explicitly the initial conditions. But notice, however, it's also important to
WH11NE- Sunrise: have the your neural network be a function of the 3 numbers.
WH11NE- Sunrise: Okay, alright. So how do we do it? How do we actually solve this problem?
WH11NE- Sunrise: The reason I love this idea because it's simple, really simple. The idea is simply that you choose the loss function to be
WH11NE- Sunrise: the differential equation.
WH11NE- Sunrise: Here it is, and you square it. So you don't have to worry whether it's positive or negative. That's it. That's that's that's the idea of pins
WH11NE- Sunrise: in a nutshell. You simply choose that to be your loss function. And the reason this is even remotely feasible is because of the existence of things like automatic differentiation.
WH11NE- Sunrise: Right? Without that it would be a nightmare.
WH11NE- Sunrise: This would not be possible.
WH11NE- Sunrise: But but you can you, since we, since we now live at the time when you're able to take the gradient of a hundred 75 billion parameter function.
WH11NE- Sunrise: And it's an exact gradient. It's not approximation, it's not finite difference. It's an exact calculation. It is now possible to to actually do this.
WH11NE- Sunrise: And so here's our Ops function.
WH11NE- Sunrise: And once we have a loss function, we just use the standard methods
WH11NE- Sunrise: that people use for salt. You know, trading machine learning models. And that's it. So this is what we
WH11NE- Sunrise: what we did.
WH11NE- Sunrise: So here is the so the the colored lines are the solutions obtained, using standard numerical techniques.
WH11NE- Sunrise: it orbits and the dash. Gray lines were kind of the the.
WH11NE- Sunrise: the initial guesses of the new network. So this is at the start before training just random random premises of the neural network. And of course, it has no idea what's going on. And then we simply run, we run the minimization algorithm. And this is what happens.
WH11NE- Sunrise: it's like a drunken sailor, but eventually calms down, falls over and sleeps.
WH11NE- Sunrise: So there we are, right. So so after after trading this thing eventually it finds, you know, sufficiently
WH11NE- Sunrise: you know, closest on basin
WH11NE- Sunrise: can't guarantee is the pro minimum. But it's close enough. And you see that this single function because it's conditioned on U naught. And V naught is basically. Now, now I have a solution. So if I wanted to find out, for example.
WH11NE- Sunrise: if I want to study the stability of the solution. Respect to respect to you know. Let's say, how far I am from the horizon. I can study now the stability, because I can take derivatives of my solution, respect to
WH11NE- Sunrise: U naught, or with like V, naught or whatever whatever. I can now study the mathematics of this problem.
WH11NE- Sunrise: right? So that's
WH11NE- Sunrise: so. Yes, it works. And and in fact, people be using this now to do some very interesting things. And as I said, one of the papers that you can read on have Ml, that they are using that to that. They're using the same technique to solve
WH11NE- Sunrise: a piece of the Dyson-schringer equation.
WH11NE- Sunrise: And and in fact, I I am.
WH11NE- Sunrise: I'm quite hopeful that this sort of technique would be actually useful more broadly in theory and other parts of physics.
WH11NE- Sunrise: Okay, so we now come to our last bit of soup.
WH11NE- Sunrise: RTDE, the 1st time diffusion equation.
WH11NE- Sunrise: So we've always, I think we and I all agree right. It's it.
WH11NE- Sunrise: This ability to to sample from a simple distribution
WH11NE- Sunrise: and morph it to the much more complicated distribution that's difficult to sample from is really important in many, many fields.
WH11NE- Sunrise: Clearly I was asleep.
WH11NE- Sunrise: and typically what this, what do we do? We started with this with the with the thing that we all love, the multivariate diagonal 0 mean unit variance.
WH11NE- Sunrise: You know, Gaussian, because that's there are very very efficient techniques for something from that distribution.
WH11NE- Sunrise: And then the idea is is to then
WH11NE- Sunrise: move that to the desired distribution.
WH11NE- Sunrise: And there are lots and lots of methods I'm sure you've heard about flows. We talk just about Iqn, and I want to talk about a particular approach.
WH11NE- Sunrise: I mean, there are many diffusion models.
WH11NE- Sunrise: I will talk about particular one that I find particularly interesting because it's it's a, it's just beautiful.
WH11NE- Sunrise: Okay, so here is my, so this is a 2 dimensional problem.
WH11NE- Sunrise: half a million points, a Gaussian. And we're going to. We're going to solve. We're going to morph this into the desired distribution. A really interesting distribution that I'm sure you will recognize. Perhaps.
WH11NE- Sunrise: if you squint squint.
WH11NE- Sunrise: It looks remarkable like like this guy.
WH11NE- Sunrise: So I took this picture during Sima's desk when Gabriel was sitting in front of this big tray of food. That's right, exactly. Okay. So we're going to figure out how to do that. How can we map
WH11NE- Sunrise: a Gaussian into Gabrielle or the Paso tray full back in time? Gabrielle walks into a diffuser. Okay? So what I find absolutely remarkable is that is that you can actually do the mapping from the Gaussian to the desired complicated distribution
WH11NE- Sunrise: by solving a deterministic all right. That's what's interesting. The deterministic
WH11NE- Sunrise: equation. Now, it happens to be an integral differential equation, which is a bit of a pain.
WH11NE- Sunrise: But basically, if you can solve this equation, so the x, the X
WH11NE- Sunrise: of T. Here is an n dimensional. Vector
WH11NE- Sunrise: and you start at X at T equals
WH11NE- Sunrise: one. And you you sample from the Gaussian. And for every point.
WH11NE- Sunrise: You simply solve the equation to T equals 0. And then that will basically deterministically move that point to another point in that space.
WH11NE- Sunrise: And and this will give you the this, the desired distribution.
WH11NE- Sunrise: And so the problem here is that you you simply solve this equation multiple times for every one of the 500,000 points, and it moves that point to a different position. And you. And so this is really great. Now, the this is not magic.
WH11NE- Sunrise: This is not magic. It's simply because you have encoded in this integral, the information about the distribution, all
WH11NE- Sunrise: the final points.
WH11NE- Sunrise: And that's typically what you're doing with diffusion problems. Right? You have a Monte Carlo simulation of something very, very complicated. You have no analytical expression for the for the problem distribution. But you can actually generate these points. This point clouds.
WH11NE- Sunrise: and then you have, let's say, a Gaussian distribution and task is, I wanted to map my Gaussian point cloud to the point cloud for my simulator.
WH11NE- Sunrise: So somehow you have to embed the point cloud
WH11NE- Sunrise: that you've generated from your simulator into this integral.
WH11NE- Sunrise: and if you do that. Then you can solve this problem. Then you can generate things thereafter. And lambda. And you are just simple functions.
WH11NE- Sunrise: So yeah, so this is 2 papers here now a bit of history. The only reason I know anything about this, and of course I knew about division models
WH11NE- Sunrise: is because last year, about a year ago, my older daughter.
WH11NE- Sunrise: who's a math Prof. At University of Tennessee. She's giving a talk in Tallahassee.
WH11NE- Sunrise: I think it was the American Mathematical society, the southern southeast section.
WH11NE- Sunrise: and of course, you know, like any proud father. I thought, oh, I should show up.
WH11NE- Sunrise: So I went to the talk. It's very, very nice. And I just happened then. After that I just happened to stumble into a section on session, on
WH11NE- Sunrise: machine mathematics and machine learning.
WH11NE- Sunrise: And
WH11NE- Sunrise: in there was this wonderful talk about this subject, and of course, whenever you learn anything new, the only way you can really be sure that you've actually understood it is to do it yourself right? And so I've been studying this for the past year, and it's really remarkable.
WH11NE- Sunrise: So let's suppose you have a Monte Carlo. Approximation of the point cloud that you'd like to generate.
WH11NE- Sunrise: you know instances of.
WH11NE- Sunrise: And let's say, here it is. So here's my, here's my point, cloud. So the X nor in principle can be an n dimensional vector, and I have, you know, big M of those I've generated.
WH11NE- Sunrise: and I'm going to use. By the way, I'm using this, I'm using the notation lambda sub t to be lambda of T, and so on. Just to keep the keep. This notation in
WH11NE- Sunrise: not too heavy.
WH11NE- Sunrise: And so I so I have that
WH11NE- Sunrise: now the thing that's really clever.
WH11NE- Sunrise: And I actually went to talk to. In fact, he's a colleague of mine at Florida. State. Peng bao, mathematician, I'm going to talk to him, and the thing that's really clever is, that
WH11NE- Sunrise: is, let me let me, let's look back. Whoops on the way.
WH11NE- Sunrise: Yeah, the the thing that's really clever is is
WH11NE- Sunrise: as I look here, is that this integral can be approximated with the Monte Carlo preservation.
WH11NE- Sunrise: Right? Since I have a sample that generated from my simulation, I can replace this integral by an average over that point. Cloud.
WH11NE- Sunrise: And, moreover, the probabilities
WH11NE- Sunrise: in inside this integral are unknown. Those are basically Gaussians, individual Gaussians, you know, with particular forms for the for the variances, and so on, and so all that is known. And so this entire thing, this integration, now becomes a 1st derivative with some simple functions here, and a sum
WH11NE- Sunrise: a sum over a point cloud of
WH11NE- Sunrise: quantities that I can write down explicitly Gaussian quantities.
WH11NE- Sunrise: And someone just solves this. And and so here are the details. Of
WH11NE- Sunrise: this problem. So so we generated half a million.
WH11NE- Sunrise: 1st of all, yeah. So so the the size of the point cloud was used is 50,000
WH11NE- Sunrise: that goes into the right hand side. So think about what doing here? Right? It's it's really quite remarkable. On the left hand side you have the derivative
WH11NE- Sunrise: Dx Dt
WH11NE- Sunrise: equals something. And then there's a 50,000. There's a sum of 50,000 terms of Gaussian. So it's a computationally, you know, challenging problem. And then that equation is solved
WH11NE- Sunrise: half a million times.
WH11NE- Sunrise: Right to to map these document points into Gabrielle.
WH11NE- Sunrise: Okay, and I don't have time to talk about how? Why, this particular choice of these of these functions? Mu sub t and lambda sub t. But you can. You can motivate that.
WH11NE- Sunrise: And this equation where on the where, let me go back to here. So this equation can be solved using standard numerical techniques, finite difference techniques. So you write down a finite difference, approximation to the 1st derivative. That's accurate enough.
WH11NE- Sunrise: And you simply just solve it.
WH11NE- Sunrise: Now.
WH11NE- Sunrise: the great thing about the technology that today exists is that is that you can write your finite difference solver using pytorch
WH11NE- Sunrise: right? Using the tensor machinery of pytorch. As a consequence, you can solve
WH11NE- Sunrise: you could solve many points in parallel.
WH11NE- Sunrise: And so this solve is able to solve 10,000 points in parallel on, on some Gpu even though even though there are. There's this, the integral is replaced by a sum over 50,000 terms.
WH11NE- Sunrise: And so when you do that, then, yeah. So you resume. So I thought that
WH11NE- Sunrise: and the last slide, the the term on the right. You're gonna generate within the simulation.
WH11NE- Sunrise: I thought this would be the simulation is going to generate the picture of Gabriela. Yeah. So so yeah. So you take a picture of Gabriela
WH11NE- Sunrise: and you. That's your probability distribution right? And you and you generate. You generate sort of your your point, Cloud. From this. In practice you take like a gray scale, like each pixel or something as a possibility. Or, yeah, that's right. So so literally that I can, I convert to the probability distribution in 2 dimensions.
WH11NE- Sunrise: Kind of
WH11NE- Sunrise: my point. Cloud. Okay? And of course in in a realistic problem. You'd have a you know, some huge, you know, J. On full simulation of whatever you want, and that's that would generate your point cloud that you would then use. Okay, but this is what was actually done in practice. Yes,
WH11NE- Sunrise: yeah, yeah, just getting to great scale.
WH11NE- Sunrise: And and so
WH11NE- Sunrise: so this this was solved on a using pytorch using. Well, I I wrote the solver using pytorch to take advantage of the parallelization and run it on the Gpu. G. 4, Rtx. 1490. I have no idea what that means, but it's something powerful, apparently.
WH11NE- Sunrise: But now you can see the problem. Right?
WH11NE- Sunrise: So this works. This is just literally solving a differential equation. Nothing to do with neural networks. This is just here's a differential equation, integral, differential equation. I replace my integral by Montecolour approximation thereof, and I just solve it.
WH11NE- Sunrise: which is great.
WH11NE- Sunrise: But
WH11NE- Sunrise: it's even work, even running on a powerful machine. And it's only a 2 dimensional problem. It's slow, right? 255 of these points per second.
WH11NE- Sunrise: Now, eventually, you've got, you think if you think about what's going on. You have to solve differential equation. Where on one side of the equation, you have something over 50,000
WH11NE- Sunrise: terms. And because there's a finite difference, approximation, every
WH11NE- Sunrise: step in your finite difference, approximation. You've got to some. You got to sum over those 50,000 terms. So there's a lot of calculation going on.
WH11NE- Sunrise: There's no way this would work at all on
WH11NE- Sunrise: my laptop would probably just melt if I tried it.
WH11NE- Sunrise: So what has been suggested by the people who are doing this work. The mathematicians doing this work is that well.
WH11NE- Sunrise: you take a hit in in mapping, say a million points from the Gaussian to the desired distribution.
WH11NE- Sunrise: And then you turn to your local machine learning expert and say, I have
WH11NE- Sunrise: inputs and outputs. Can you somehow condense this into into a neural network model that will then be lightning fast. That's that's certainly one way. Okay.
WH11NE- Sunrise: But the students is gonna be working with with push for myself and say, Gay.
WH11NE- Sunrise: they're going to something crazier than that
WH11NE- Sunrise: right? What the students can
WH11NE- Sunrise: tell us, I hope by the end of the summer
WH11NE- Sunrise: is whether we can actually solve this differential equation using pins
WH11NE- Sunrise: right and condition it on initial conditions. Because if we could.
WH11NE- Sunrise: if this is absolutely insane. If we could do that, then we would have a system whereby you could. You could. You could map, not just not just a Gaussian to one detail distribution, but perhaps perhaps
WH11NE- Sunrise: the doubts into any a whole class of distributions. Perhaps if this could be done, so we're going to find out by the end of the summer.
WH11NE- Sunrise: Okay, and let's just let's just see our dear leader again.
WH11NE- Sunrise: Okay, so I'll be taking more pictures next year. So okay, so that brings us to the summary. So
WH11NE- Sunrise: you already know that the ability to generate data is just incredibly important in many, many areas of science and finance. And so on. That's just something that what we have to do
WH11NE- Sunrise: and but generating data and kind of measures can be quite problema problematic. But of course, today we have these many different methods, and I've gone over a few of them.
WH11NE- Sunrise: and I'm using the term generative modeling very generally in the sense that anything that that allows you given different initial conditions allows you to generate a whole ensemble of things is a generative model. And of course, today. Alas! People using this amazing technology to do stupid things like, well, let's not go down that road.
WH11NE- Sunrise: And I consider 3 relatively simple.
WH11NE- Sunrise: I mean, compared with noising flows and and the other things that you have learned about here, relatively simple models which are really beautiful, simple idea. And this reverse time, differential equation.
WH11NE- Sunrise: And and I think it's just worth, you know, looking to those and thinking about, you know, could I apply this somewhere? But, for example, you know. Perhaps you know, the theorists among us might think, oh, if I try to use the, you know, since I'm since people have been able to solve the
WH11NE- Sunrise: Dyson-schringer equation, at least a bit of it. Using this technique. Perhaps there are other difficult problems of this nature that can be solved. And it's not only differential equation, basically, any equation that you have, any equation that can be recast as something that's going to be equal to 0.
WH11NE- Sunrise: Right can be solved this way. So it's very, very okay. Thank you.
WH11NE- Sunrise: Okay, hey? Questions, comments.
WH11NE- Sunrise: So the comment on your Gpu, that's up. That's a consumer gaming card. So it's double precision performance is really lousy. But a single precision performance is wonderful. So try it in single precision. Okay? Oh, I should talk to you before. Yeah, okay, very good.
WH11NE- Sunrise: No. I mean, it's how many of you went to Daniel's talk the other day.
WH11NE- Sunrise: Yeah. So just to just do a quick recap. You know, he said, that we should be looking for great stuff. Yeah.
WH11NE- Sunrise: I absolutely agree. We need to start thinking about looking for weird things.
WH11NE- Sunrise: However.
WH11NE- Sunrise: the premise of this talk is that you have this amazing graph neural network that can do track fitting, of course, is just truly astounding. And you can use that to look for weird tracks.
WH11NE- Sunrise: But it's very clear to me from the work that some people doing, including Sergey, that that is not the future.
WH11NE- Sunrise: Sorry, Daniel.
WH11NE- Sunrise: If you're online, the future is.
WH11NE- Sunrise: is is to go raw and raw and really go down to the raw data. But the future is treating the entire event
WH11NE- Sunrise: as as a collection of collection of graphs, where where every piece of detector that's uniform in some sense, is, is a point cloud in a, in a in a certain dimensional space. And an event is just a collection of these
WH11NE- Sunrise: spaces as point clouds.
WH11NE- Sunrise: and the task would be, is this collection of point clouds anomalous in some way compared with this question of point clouds. That's the future.
WH11NE- Sunrise: right? So if you're interested in track fitting and doing clever things, your time
WH11NE- Sunrise: scale for doing that is limited at some point, technology will be such that
WH11NE- Sunrise: that is how we will actually be looking for new things
WH11NE- Sunrise: where Daniel is completely correct, however, is that once we have found the anonymous events.
WH11NE- Sunrise: finally, humans will be useful again, any questions about about this.
WH11NE- Sunrise: So I'm gonna post. I'm gonna post the slides. And I've been trying to somehow put also the the videos somewhere. So they're available also for you. But,
WH11NE- Sunrise: by the way. I hope that this will be something that you might want to look at. So thank you again.
Claire David: Thank you.
WH11NE- Sunrise: Ralph, drawing wasn't hand drawn. Was that sorry? Your ultimate super.