Gaia Grosso: Context.
Gaia Grosso: it's going to talk about multi-agent research validation and enable using Llms and experience using this for Ligo experiment.
Gaia Grosso: So yeah, please take it away. Thank you. I don't see your slides anymore.
Nikhil Mukund: They don't see my slides.
Gaia Grosso: They were there right before.
Nikhil Mukund: Okay.
Gaia Grosso: But and then just disappeared.
Nikhil Mukund: I see.
Nikhil Mukund: How about now?
Nikhil Mukund: Okay.
Nikhil Mukund: And they're not.
Gaia Grosso: I don't know if it's just me, but I don't see it.
Nikhil Mukund: Right.
Gaia Grosso: All right.
Gaia Grosso: Thanks.
Nikhil Mukund: Yeah, sure. Yeah. Hi, everyone nice to talk to everyone. I'm nikil. I am a postdoc at Mit and the Ligo laboratory.
Nikhil Mukund: I'm also a junior investigator at Nsf. AI Institute for Artificial Intelligence and fundamental interactions. So here I'll try to share my experience which I encountered. But while I was trying to build this sort of multi-agent based systems like Llm. Based systems for Ligo. And so I thought, and it would be nice to share
Nikhil Mukund: and then get some feedback from you.
Nikhil Mukund: So just a brief introduction to like, you know. So we currently have, like
Nikhil Mukund: several like at least 5 gravitational wave observatories, and 2 in us, Ligo, Hanford and Livingstone, and one in Italy, Virco, one in Germany. Geo. We also have an underground Japanese detector called Cagra, and we also have Ligo India, which is currently under construction. So these are essentially kilometer scale interferometric systems that are trying to observe. You know, ripples in space time.
Nikhil Mukund: and we usually look for sources like colliding black holes and neutron star systems. And it's a very like spread out collaboration like we have 1,000 more than 1,000 members in our like over go Cagra collaboration. And the
Nikhil Mukund: and we currently have, like accumulated decades of research from thousands of these researchers. And they are all spread across.
Nikhil Mukund: And the point is like the research that has been generated, and experience that has been, you know, accumulated is is in unstructured and multimodal data formats, you know. So they are spread all around the place.
Nikhil Mukund: and the point is like, whenever we have an issue at the detector in Livingston or in us, the same issue might have happened 5 years back in the other detector. So oftentimes we are sort of, you know, reinventing the wheel or trying to solve a problem that is already solved. And same with the data analysis and the theoretical side. So
Nikhil Mukund: that's why we wanted a way to sort of smartly, efficiently share, disseminate information. And that was the whole point of this kind of work. And I and I am sure this same logic is what other speakers will also. Like, you know, they also had the same intention.
Nikhil Mukund: So on a broad scale, what? What do we
Nikhil Mukund: intent with marvel? So the motivation is like, you know, we we have, like Phd thesis, like we have books, papers like a lot of research articles, a lot of internal lab book lab books that we write every day when we go to the lab and do an experiment. A lot of internal documentations, presentation slides.
Nikhil Mukund: web articles, of course, and also databases that can store information about. For example, the optics that we use or things like that, and
Nikhil Mukund: a lot of codes, you know, in different languages. And then, you know, for different tasks ranging from data analysis to simulating and optics, you know. So we have all this. And typically we want to synthesize, make use of all this, and then a typical researcher would like to generate answers
Nikhil Mukund: and even validate his ideas right? It's not just querying a question, but maybe he or she has an idea, and then wants to, you know. Validate it, you know. Has it been done before? And what are the approaches that I should take and then essentially convert all that into an experimental design or a data analysis, you know, write manuscripts, project proposals, and even outreach.
Nikhil Mukund: So we had this motivation to build some sort of AI co-pilot that can that can help us with this, and also at the time, at the same time be completely private, you know. So we because most of the data is under
Nikhil Mukund: authentic, like authentication. So we don't want to use open. So like, we don't want to use.
Nikhil Mukund: You want to avoid, like, you know.
Nikhil Mukund: open a like, you know, private enterprises as as much as possible. So a private hosted thing. So that was our motivation.
Nikhil Mukund: And so here here is the sort of architecture that we have. This is the sort of planned architecture.
Nikhil Mukund: As I mentioned. We have the task we have the researcher, and this is a multi agent framework in the sense it's not a single Llm that is doing the task we share. So there would be, for example, a primary agent for this project which will get the user question, and it does all the decision making and planning, and then it coordinates with several of the local agents, which are like
Nikhil Mukund: like they are built for different tasks, and then
Nikhil Mukund: and they will have access to sort of several tools which could be Api, or which could be code interpreters. And and they will also have access to the knowledge base. Okay? So, which which are again, you know.
Nikhil Mukund: embedded, which the previous speakers have already mentioned. How they embed different sources of information. We do the similar stuff.
Nikhil Mukund: and then, which have provided as context
Nikhil Mukund: to this local agents. And this whole sort of loop is what we plan. And this is the overall architecture of goal that we are trying to achieve. But the plan or the when we build this kind of architecture, we have ensured that it is. It is sort of agnostic to the sort of
Nikhil Mukund: technical instrument of the data, because we see that many big collaborations will have the same sort of issues to address. So we have made sure that our marvel is sort of easily adaptable to
Nikhil Mukund: other domain and other data sets. So that has been thought about well from the beginning.
Nikhil Mukund: So let's some history. So we we were actually looking into this kind of things in way back in 2,01617, when we didn't, where we did not have any Llms. We were thinking, you know, is it even possible to have such an intelligent assistant? And in by 2,018 we had, like, you know, we were using whatever
Nikhil Mukund: tools were existing that time, using natural language processing. And then we had some sort of an early retriever that we had, and it's still working. It searches across the log books within Ligo, and also the documentation and presentations. But it was still a retriever, you know. It did not have the generative AI aspect, but it was still very useful, and we got a lot of feedback. And people were sort of interested in expanding this sort of technology further.
Nikhil Mukund: And so you know, now, coming t020-23-2023, we had first, st the 7 billion open source Llms available. So we started working on it and then trying to do some proof of concept at that time. And like last year, we, you know, moved to a more multi agent orchestration, and we were setting up the infrastructure and optimizing the server layers of this is actually an engineered
Nikhil Mukund: product, you know. So you have to run it through a lot of optimization testing validation. And then last year, we also started working on this this
Nikhil Mukund: technology called deep search. Because.
Nikhil Mukund: as you know, like technical in technical domain, the questions can be complex. So a single query response, you know, retrieval response might not be sufficient. So we need to have a way to, sort of, you know, decompose the question, and, you know, dig deep, and then find every
Nikhil Mukund: like a better answer so that was what we started in 2024. And like this year, we are like sort of getting internal feedbacks. We're also working on benchmarking and like improving the deep search. And we are like really thinking of scaling it up to the, you know, and serving the entire like a collaboration.
Nikhil Mukund: So I have a initial version of this app online at this address, and then let me go to the thing. So this rag, I guess most speakers have already mentioned how rag works and all that. So I'm not going to repeat. I think it's still like there is still to get it really working on your own data, there still needs to be a lot of
Nikhil Mukund: engineering that we have to do. And then there is a lot of domain specific tricks and tips that we need to do to make it like, you know, very, very accurate. So yeah, so let's skip the rag slide.
Nikhil Mukund: And so, how are we different from other agents. So 1st of all, we are. It's a free and open source technology that we are planning to like. We are planning to open source it in the coming months, and it would be completely based on local Llms, and it would provide deep search capabilities.
Nikhil Mukund: which is something new, and we we try to ensure privacy, and at the same time flexibility such that it can be adopted to other domains. So if I took a look at other such options right now. So
Nikhil Mukund: we have, like the one in blue, are like we have such deep search capabilities, but they are mostly targeted for either paper. Finding the research articles, or like, there's 1 for archive papers, or there's 1 from Alan AI. There's also one from one from Langchain which does open deep research, but it it relies a lot on anthropic, or, you know, open AI Api, although it can still handle
Nikhil Mukund: open source Llms. And then we have the private ones which already already, you all know, and they are expensive. And then.
Nikhil Mukund: you know, you have all the other privacy issues. So we we sort of give. What do we offer? We offer some sort of an iterative rag, but the thing is, we also have a verifier. It's it's rag with verifier, and then we provide sort of research based reasoning. And on the hardware side we use like
Nikhil Mukund: we use whatever resources that we had available when we started this work. So which we had an Arctx 4,090. It's actually a gaming Gpu, but it has like 24 gb. RAM. But then we sort of do this sort of private
Nikhil Mukund: public
Nikhil Mukund: balancing. So if the query relies on open source, openly available data, then we will shift it to the
Nikhil Mukund: public lpus, like language processing units based on grok. And then if it's private data, we use our internal gpus, and that's how we sort of go ahead. So let me show like a quick demo. So just in case you know iterating. So what in one line? If I want to explain this, I would call it marvel is some sort of a search.
Nikhil Mukund: then verification, and then the generative AI.
Nikhil Mukund: But then in an iterative fashion, so it is iterative way to search, verify, and you know, use the Gen. AI, and that's that's how it's envisioned. So a typical question that, you know, Researcher would ask, you know what would be like. Say, someone is interested in knowing the seismic sensors that that are used within advanced Ligo and he or she might be interested in the specialities and the frequencies in which these things
Nikhil Mukund: work. So we actually go through. We verify the question. And then we go through 2 levels of thinking. So it's so because, you know, Llms are already very good at memorizing a lot of facts. So sometimes we might not need to invoke the whole rag architecture. So we do a system, one thinking, and the system will internally
Nikhil Mukund: and, like, you know, come to a point, and it will realize that this is complex and it will automatically switch to the system to thinking which is based on this iterative process that I mentioned. And then it actually goes through several sources of knowledge bases, including, you know, web search, and like semantic search. And
Nikhil Mukund: like SQL, search and things like that as common to other services. And then, yeah, we also make sure that we verify the Urls and things like that, because that's very. We try as much as possible to minimize hallucinations. And then then the last stage is synthesis and synthesis is also very critical, because when you have a lot of data that is pulled in which does not fit in the context of the Llm. There need to be a way to efficiently
Nikhil Mukund: combine them into a coherent document. So we also put some efforts in that kind of stuff, and then here you can see how it brings out different looks into internal documentation. And you know, web articles and come up comes up with all the sources. So yeah, and then we can go back and check all the individual documents that it pulled out, and which is very
Nikhil Mukund: yeah, like. And then here, you know, it shows the right plot that we were looking for. So let me move to the next slide. And
Nikhil Mukund: so the key other thing that we wanted to focus is this deep search. And so the way we implemented, okay, let me see if this message is for me. Okay, yeah. So the way we implement deep search is like we, we use some sort of a Monte Carlo research, which this is a technique that has been used from like, say, deep mind in Alpha 0 and Alphago projects where they were trying to solve like the games, like the chess and like go game.
Nikhil Mukund: And so so the point is like, if you have a very complex query, you would start with the baseline answer from Rag. But then you would start creating sub queries from the node, and create like children like sub queries, then do rag for each sub query.
Nikhil Mukund: and then you sort of expand your node, and then sort of evaluate the answers, the sub queries, and the answers, and the relevance to the original query, and then sort of either go ahead, expand your path, or even through your path and then select a different node and then continue the whole search process.
Nikhil Mukund: And this is what we are trying to see test. And we are. We are actually getting pretty good results. But it's still in, you know, in evaluation process. And we are trying to see if this could be a way to address complex queries.
Nikhil Mukund: The the downside is, it's it's very computationally expensive. So that's something that we want to address.
Nikhil Mukund: So let me, yeah, I just wanted to share some of the issues that we encounter when we do this sort of open source technology. The one main thing is like, you know, open. Such Llms require a significant amount of RAM, and there is a significant difference when you
Nikhil Mukund: you said 8 7 billion plus Llm. Versus, say, 20 billion parameter llm, so you would always want to go to the big better one. But then, at the cost of RAM, and then, especially when you want to do. Multi agent, where you have several Llms, there should be a way to orchestrate all this. So that's 1 issue. So the solution that I find interesting is called quantization aware training, where the Llms are quantized, but in a very.
Nikhil Mukund: very good way, such that they retain their sort of intelligence, but at lower size. And the other issue is like how to summarize when you have such large amount of information that doesn't fit the context length.
Nikhil Mukund: And then, if you look at many of the Llms in this table, you can see that the claimed context length versus the effective context length. There's a huge difference. So we need to take also that into account. And I recommend people going through this technique called Storm from Stanford. That's a pretty nice way to write Wikipedia style articles.
Nikhil Mukund: And finally, since most of her data is in like
Nikhil Mukund: could be in Pdfs. You know, there may be scanned documents. There is a need to do a massive scale. Ocr and Ocr itself is very challenging to do. Because when you have equations and you know technical document stuff, it's not a completely solved problem. And I have compiled the different sources of techniques that are available. And there are open source
Nikhil Mukund: Ocr solutions available, and some of them have really now even started to support vision Llms that are showing significant improvement in how we can extract data.
Nikhil Mukund: So the next step is like we recently got like a seed grant from the office of research computing and data at Mit. So this has helped us to explore future options like going from deep search to deep research, you know, and
Nikhil Mukund: then we also now have, with the increased computational resource. We now have an opportunity to do large scale Ocr, and even large scale benchmarking, which was currently like the bottleneck, was like. We did not have enough resource to do all this, so we would be moving from this current solution of local Gpu.
Nikhil Mukund: like Rtx. 4,090, and a public grok, and to a
Nikhil Mukund: orchid H. 200 gpu. That orchid is kind enough to provide us, and we would be using it to further improve our pipeline test and validate our pipeline. So
Nikhil Mukund: that's what I wanted to say. And I think my last slide is like, you know, we have built this with modularity in mind, and so that we can easily adapt to the changing landscape, because every week you would see a new model coming up. And so we need a way to integrate all the new technologies, and we plan to open source
Nikhil Mukund: the whole workflow so that it can be reused. And then, you know, potentially improved by the community. And yeah, so we are trying to scale our current design and the new Gpu's that we have will help us to, you know, optimize the whole architecture.
Nikhil Mukund: and I am very much excited in collaborating, and then I would love your feedback
Nikhil Mukund: so thank you. I'll take some questions.
Gaia Grosso: Thank you, Nikhil. Other questions from the audience.
Gaia Grosso: Let's see.
Gaia Grosso: Okay, maybe I can start with one. So you said this computationally pretty, let's say expensive to go through like download the graph notes in your iterative search. Right now, like, if you try to use it like, how many people can use it or like, do you? Is it just one user? Do you know, more or less how easy to make it happen like for users to be used.
Nikhil Mukund: Yeah. So the the currently, the version that we have put online use has uses only publicly available. Or like, you know, data like like even the Lego data. So so what we do is like, we sort of, as I mentioned, we do this load sharing. And then for the Grok Lpu part. We we do. We can make a parallel. Api calls.
Nikhil Mukund: okay. While for the Gpu it was resource constrained. So
Nikhil Mukund: if if you have many users trying to query at the same time it would work. But then the latency will be large, and I think these days people don't have people want very quick answers.
Nikhil Mukund: So latency, very crucial factor. So one thing that we are trying to see is like we are trying to see if depending on the query we are trying to. We judge the complexity of the query, and then try to allocate resources efficiently. But we are hoping that with the new hardware that we are getting, we would be able to speed up.
Gaia Grosso: I see? Yeah, thanks. So there is a question from Lloyd as well.
Gaia Grosso: Just go ahead.
Loyd Waites: Hi, I saw your your slide on parsing scientific documents, and we're we're facing something similar. We've been using dockling and and marker, and we're we're just getting started with it. And we've definitely been seeing that marker does a little bit better with equations. But dockling does is able to embed images and figures and things like that pretty easily. Do you have a recommendation on
Loyd Waites: any of the more open source ways to parse scientific documents or kind of things that you've learned.
Nikhil Mukund: Yeah. So I also tried a lot of this. And I find Marker the currently I'm satisfied with Marker. It also has this layout detection. And so, Marker Marker internally uses Surya. That's another model that does the Ocr.
Nikhil Mukund: And then Marker also has this layout detection. The recent versions of Marker also has a vision support, vision, Llm support which, again, can be Gemini, or it can even be the local Quen, like, you know, local Llm models. So I think that's 1 option which I found
Nikhil Mukund: pretty good. At least that's what I use
Loyd Waites: we're I I just started playing with Marker a bit. Is there a way to have it embed? Figures kind of like dockling? Does.
Nikhil Mukund: Right now.
Nikhil Mukund: No, right now. No, so it will just put a placeholder. But I guess you know what you could do is like you could then use.
Nikhil Mukund: probably.
Loyd Waites: Use an embedded model.
Nikhil Mukund: Maybe, you know, if there's an image or a graph, and then Marker gives the placeholder, then you could use another vision Llm. To fill in some details about the image, and then somehow link the image with this description, and then it's a bit of engineered thing. But.
Loyd Waites: Yeah, yeah, that's that's what I was thinking. But I I I haven't.
Loyd Waites: I wanted to make sure that I wasn't going to reinvent the wheel before I tried to do that. So thank you.
Gaia Grosso: All right. I guess we can thank Nikhil again and move on to the next talk. So thank Nikhil, and our next talk is from Alex.
Gaia Grosso: alright! Hi! Alex!