Gaia Grosso: Let's go talk okay, and is going to talk about
Gaia Grosso: speak eyes. I don't know if I'm pronouncing it right correct. Is it correct?
Alex Gagliano (he/him): Speakeasy.
Gaia Grosso: Speakeasy. Oh, okay.
Alex Gagliano (he/him): Very hard to pronounce. I'll explain why.
Gaia Grosso: Okay, so this is Vicky inventing Llm. For supernova science. Just please take it away.
Alex Gagliano (he/him): Great thanks, so much so I 1st want to start off by saying, thanks very much. The organizers for having me, and
Alex Gagliano (he/him): for just having this session. To begin with, I've already learned a lot, and it's funny
Alex Gagliano (he/him): we have talked already about how the landscape is changing very quickly. We are all running into similar issues, and we are addressing them in slightly different ways. So it's already been very thought provoking for me to to have seen the talks today. So yeah, really appreciate it.
Alex Gagliano (he/him): I want to talk about the way that we are trying to solve some of these problems in the domain of supernova science. We'll talk about why we might want an Llm. Agent where we're at so far, some of the things that we've run into, and then where we're going
Alex Gagliano (he/him): and feel free to stop me and ask questions if the organizer will allow for it.
Alex Gagliano (he/him): Okay, so a little bit of context, I study supernovae. These are the explosive deaths of massive stars. They explode in many different ways. But really what we observe in the night sky is a little pixel that changes in brightness as a very distant star explodes.
Alex Gagliano (he/him): we see nothing, and the star explodes, and then the pixel lights up, and all of the physics that we infer about the explosion comes from how that single pixel changes in brightness. So it's a very hard problem. It's a very ill posed problem.
Alex Gagliano (he/him): And right now
Alex Gagliano (he/him): we are at an interesting inflection point in supernova science. So we have gotten better and better at building automated tools to find these explosions. And yet
Alex Gagliano (he/him): we are getting worse and worse at studying them physically. So just to make this point a little more visceral, here is a video of the number of explosions that astronomers have discovered in the night sky since the 18 hundreds. So if we play the movie forward.
Alex Gagliano (he/him): you see that maybe not surprisingly before modern technology. It was very hard to see anything, but really the brightest explosions. As time goes on we find that the sky is completely covered in these explosions. And then, at about the turn of the century, our telescopes come online. Ccd technology picks off. And we're now finding many thousands or tens of thousands of supernovae per year. There's a telescope
Alex Gagliano (he/him): that's going to come online later this year in October, and it will find 10 million supernova candidates every night. Little pixels that flicker in brightness.
Alex Gagliano (he/him): And
Alex Gagliano (he/him): there are not enough humans interested in supernova science in the world to be able to parse all that data and to figure out which ones are the most interesting to study in detail before the explosions fade away.
Alex Gagliano (he/him): And so, right now
Alex Gagliano (he/him): this has become a game of triaging. What we do is we have these massive survey telescopes. They scan the sky and they detect many explosions. And then we have smaller targeted telescopes that
Alex Gagliano (he/him): choose to look at particular locations in the sky to higher cadence. So they take many more images across many different nights to get a better sense of what's going on in the explosions that we care most about.
Alex Gagliano (he/him): and so a collaboration that I'm closely involved in is called the young supernova experiment, or YSE pronounced easy.
Alex Gagliano (he/him): and Yeezy is interested in studying the youngest explosions that we can find. The idea is that the physics of what's happening in the explosion is encoded in information from the 1st couple of days of the explosion. And so basically, Yeezy scans the skies and points to specific supernovae that we find most interesting
Alex Gagliano (he/him): at the discretion of the supernova scientists, and then we schedule it for follow up.
Alex Gagliano (he/him): So right now, what gets done is we have a page where people can look at different explosions that have been seen. There's a bunch of metadata that provides information on the properties of the supernova as we
Alex Gagliano (he/him): currently no, there's information about the brightness over time and its position in the sky, and on the left we have a an excel spreadsheet that humans use to schedule who per night. Usually grad students is going to sift through the new candidates and determine which ones are the most interesting to schedule. Follow up.
Alex Gagliano (he/him): You can imagine that this is incredibly laborious process. It's incredibly inefficient, and it will not scale
Alex Gagliano (he/him): the new instruments. And so we're trying to figure out a way to make this more efficient? Can we automate the organization and inference of physics in the phenomenon that we discovered nightly.
Alex Gagliano (he/him): and that led us to the use of Llms.
Alex Gagliano (he/him): This was the 1st demo of the speakeasy.
Alex Gagliano (he/him): which is what we call the Lm. Running within the unsupernome experiment.
Alex Gagliano (he/him): I built it at the end of 2023, with a grad student at Harvard, and at the beginning it really just looked like calls to Openai. And so there was a back end database where we had our public data.
Alex Gagliano (he/him): and you could ask it a given question about something in the database, like how many transients were discovered. Supernovae were discovered in the last 60 days, and it would translate that to a SQL. Query, execute on the back end database, and retrieve the answer.
Alex Gagliano (he/him): You can then ask. Follow up questions, and of course there's context in the chat. Now, this is very old, Tab, but at the time it seemed very promising.
Alex Gagliano (he/him): And let's wait for it to give that answer.
Alex Gagliano (he/him): Okay.
Alex Gagliano (he/him): And we were at the time trying to push it
Alex Gagliano (he/him): to adopt more flexible queries. So really think about answering questions, constructing SQL. Queries that require multiple joins that were outside of the training set. So here I'm asking it ranges in the sky. I'm asking it about the types of supernovae. I'm asking it to retrieve the names. And this was not on the training set.
Alex Gagliano (he/him): and it takes a while. But it does okay
Alex Gagliano (he/him): until it starts giving the result.
Alex Gagliano (he/him): This also did not fail super well, because of how much it was costing to to use open AI's Api. So we moved to an open source framework. This is the current setup of the speakeasy, so it runs through an interface called Open web. Ui highly recommended. It's free and very useful.
Alex Gagliano (he/him): Anytime a user asks a question. It 1st retrieves via semantic search. The most similar, most relevant publications that have been embedded in advance via a back end database and then the Llm. Decides what back end tools to call to conduct analyses on the data. So
Alex Gagliano (he/him): the tools have been preloaded. They're basic analysis tools that supernova scientists use to understand whether an event is interesting or not. It also has the capability of doing web browsing and retrieving additional papers, and then all of the results of the tool calls and the web browsing results are provided as context to construct its final response.
Alex Gagliano (he/him): So I mentioned, we moved toward open source. Originally we were just using Openai. But now we are going through Olama on the back end, and the really nice thing about Olama is that we are not constrained to a specific model, as the models get better every week we can swap it out. Each one has different strengths and weaknesses. Some are better at reasoning, some are better at tool calling, and it's been really interesting to swap them out on the back end and see how the responses evolve.
Alex Gagliano (he/him): I want to mention this guided generation or structured generation is a piece that
Alex Gagliano (he/him): I think is known to people building these models for users, but is not super
Alex Gagliano (he/him): super well used for the practitioners of some of these models. Structure generation, I think, is incredibly valuable for reducing hallucinations, ensuring output of a particular format. And as we think about guiding Llms toward output that is relevant to our physical domains, I think it's going to become absolutely essential. So I would love to see more work in it.
Alex Gagliano (he/him): And then we are trying to hit a moving target as the models improve, we have improved very rapidly as well. So some of the pieces that we've added in are memories that are user specific citations for both the back end papers and the web sites that are searched.
Alex Gagliano (he/him): It can now interact
Alex Gagliano (he/him): and embed plots and other images that it generates from its analysis in the chat, and it runs a series of validation scripts via prompt Foo. I can talk more about that if people have questions
Alex Gagliano (he/him): I want to end by just mentioning that
Alex Gagliano (he/him): right now in supernova science, we don't really have the data to
Alex Gagliano (he/him): train a model to get good at understanding the connection between language and the data sets that we collect. And I think that this is the primary limitation for adoption of Llm. Models in at least supernova science. Maybe other physical domains as well.
Alex Gagliano (he/him): And so there have been attempts in the past to try and align text and associated modalities of information like images. We have a couple papers
Alex Gagliano (he/him): trying to associate this information.
Alex Gagliano (he/him): But we also just don't have the
Alex Gagliano (he/him): the text built up that is associated with specific objects. And this is a very big challenge, because when we ask an Llm. To give a summary for a supernova, and whether we think it's interesting and what kind of analysis needs to be done. It just doesn't really have the training set to understand what a human would do in that context. So with that in mind, I am now pushing toward building up that corpus to be able to enhance our natural language, queries to allow it to do more flexible hypothesis. Generation.
Alex Gagliano (he/him): And so this website is in very early stages, but it's now live the transientrecommender.org.
Alex Gagliano (he/him): And it's just a website where you can sift through supernovae. Add comments about what you think is going to be interesting. And we are launching this very soon within the unsupernova experiment to really start collecting up the text data for what humans might want to do with a particular supernova, so that we can work toward adding this as relevant context with the speakeasy and future version.
Alex Gagliano (he/him): And then this is future directions. We are expanding toward a suite. Many other speakers have mentioned this, but a suite of different Llms that are specialized for given tasks. We have found that when we overly constrain our primary Llm. For SQL. Queries, for example, it starts to struggle at web browsing or python function, calling. So.
Alex Gagliano (he/him): that is where we are moving. Next, we hope to have a prototype online very soon. And I just want to shout out that initially, we were doing this all
Alex Gagliano (he/him): using free compute resources. Luckily we have secured a grant from Amazon aws, and they will be supporting the gpus that will enable this running in the next generation of speakeas.
Alex Gagliano (he/him): And so I will just say that I think speakeasy is going to be incredibly useful for future astronomical surveys for real time analysis as kind of a level one pass through the data before humans have a look at it.
Alex Gagliano (he/him): We are working on building text annotated events. To better teach a model about what we think is interesting.
Alex Gagliano (he/him): We are building up the benchmarks to validate the open source models that are rapidly changing, and if you are interested in collaborating, please reach out. Thanks very much for your attention.
Gaia Grosso: Thank you, Alex, for the talk. Let's take a question from the audience. Is there any question
Gaia Grosso: there isn't. Okay, thought. Please go ahead.
Loyd Waites: Hi, Alex, I want to ask the tool calling that you have. When do you know when to start using the tool calling? Is it. Just are you finding like a certain relevance level when you're calling your vector database? Or what makes you actually decide to go to the web search instead of just doing the retrieval.
Alex Gagliano (he/him): Yeah, that's a that's a really good question. So when a chat is started.
Alex Gagliano (he/him): the back end database of publications is always queried to give it relevant context to start the conversation.
Alex Gagliano (he/him): So that is something that is hard coded in. We found that that really improves its ability to answer follow up questions. In all of the subsequent questions. The model has basically a chain of thought reasoning where it via structured output decides, do I need to to call a tool. And why do I need to call this tool? And one of those tools is searching the web? If it believes that it doesn't have enough information in its back end publications to to answer the user.
Loyd Waites: Okay, so how do you? How does it make that determination? Is it like, is this like a relevant score from the vector, dB, like what's kind of the cutoff there.
Alex Gagliano (he/him): For the embedded publications.
Loyd Waites: For? Yeah, for leaving the embedded publications. So I know that you do your rag on the embedded publications. You get that to generate a chain of thought. And then, when you say I don't, you're saying it figures out that it doesn't have enough information from the vector. Database. But how does it determine that it doesn't have enough information from the vector database.
Alex Gagliano (he/him): Purely through the chain of thought, prompting.
Loyd Waites: Okay, we we didn't.
Alex Gagliano (he/him): We didn't. We didn't end up encoding a similarity score on the on the back end. You could imagine, like a relevance score. You could imagine coding something like that in with a particular relevance.
Alex Gagliano (he/him): but all yeah, all we did was set up the chain of thought, prompting to say, if, when you believe you have enough information to answer the question, don't call any tools, and then there's there's like a stop gap where it says, like, now I'm done calling my tools. Now I'll move to the next stage, which is constructing the final output
Alex Gagliano (he/him): which I'll send to the user.
Loyd Waites: Okay, that makes sense. Thanks.
Alex Gagliano (he/him): Yep.
Gaia Grosso: Questions.
Gaia Grosso: And so I have a quick one about your slide. 12. So the one before this one
Gaia Grosso: yeah, cause you're talking about this is actually really interesting talking about creating data. Or we, we heard this already in a previous presentation, how preparing data before even training or doing, Rad, it's relevant. So in this case, like, is this web page open to like astronomies or scientists in general like, it's just public. Or do you have, like some experts that are like requested to
Gaia Grosso: to interact with this page? And so how do you regulate this.
Alex Gagliano (he/him): That's a really good question. This is early days for this leg of the project, but the hope is that to build up a sizable corpus, it will be open to the general public. We have seen a lot of successes in citizen science projects in astronomy more broadly, and there, there's a ton of expertise in how you calibrate the responses
Alex Gagliano (he/him): to a particular user. So there are data analysis, methods whereby you can identify this user tends to give
Alex Gagliano (he/him): comments on a supernova that has a low similarity to the comments left by users that we believe to be experts in the field. And so in those ways you can down weight specific sets of comments and tags from different users and kind of recalibrate in the end. The hope is that
Alex Gagliano (he/him): as we scale up there will kind of be a regression toward the mean phenomenon whereby just the sheer scale of the responses that we're getting allows you to kind of average out the variations in people who think nothing is interesting, people who think everything is interesting.
Alex Gagliano (he/him): And and we will really be using the experts to anchor those responses. So within the own supernova experiment, we're chatting about an early rollout.
Alex Gagliano (he/him): so that we at least have several experts that are regularly using the page. But yeah, the hope is that this will become a broader effort than that.
Gaia Grosso: Yeah, that's really interesting. I never really thought about this is like, if you could use like public to kind of fit this kind of pages. So just have, like a little bit more information about this, like, which kind of like feedbacks or like inputs, can you get from like non experts that is still like, sort of reasonable. Is it about like properties of the image itself, like intensity of the pixels? Or
Gaia Grosso: or is it really about? Let's say.
Gaia Grosso: quantitative. Yeah, I'm just trying to figure.
Alex Gagliano (he/him): Yeah.
Gaia Grosso: Yeah, which kind of feedback you can get.
Alex Gagliano (he/him): It's it's a really good question. That is,
Alex Gagliano (he/him): actively evolving. We're trying to figure out what is reasonable to ask of people who are not in the field at the same time. That page, if you go to it now, I set up a very preliminary demo where I walk new users through.
Alex Gagliano (he/him): How do I identify different types of transients? So this, maybe you can't see my cursor, this red and green set of points. Here is the light curve. It changes brightness, it gets brighter, it gets dimmer, and that that shape
Alex Gagliano (he/him): is very predictable for certain science cases, for certain types of explosions. So the tutorial basically walks people through. Here's what it looks like. If we think this thing is going on, here's what it looks like. If we think this thing is going on. If it doesn't look like these types, then mark it as weird. And citizen science projects in the past have actually shown with remarkable accuracy that, like
Alex Gagliano (he/him): a members of the general public, want to engage in science efforts, so they are very active on these platforms, and then B,
Alex Gagliano (he/him): their responses are actually really good if you provide them some basic guidance about what they should be looking for. So I really think that that is where we should be going. Instead of just hoping that the couple of experts that we rely on to do the science normally will build up the corpus sufficient to train an Llm. To do something similar.
Gaia Grosso: Right? This is super cool. Yeah, it's kind of like you're making this as an interactive game with like the whole population. Yeah, I just wonder whether anything like that could be even possible with particle physics as well. But this sounds really super fun.
Alex Gagliano (he/him): Cool, I'd be happy to chat more about it.
Gaia Grosso: Yeah.
Gaia Grosso: So I don't know if Raghav maybe had another question. I don't know when we are running out of time or.
Raghav Kansal: Yeah, I mean, it was a great talk I think we have. We should.
Raghav Kansal: We have run out of time. But I'll reach out yeah.
Alex Gagliano (he/him): Okay.
Gaia Grosso: Alright. So let's thanks, Alex, again.
Gaia Grosso: And yeah, we move to the last presentation of the workshop. That is, from Maria Rosario. I see she's already sharing
Gaia Grosso: kind of nice still loading for me.
Gaia Grosso: Okay.