Daniel Murnane: How's that? You can see it?
Gaia Grosso: It's for from my side it's still loading. Okay, now it's there. Alright, thanks.
Daniel Murnane: And yeah, thanks very much. Very nice idea to get everyone in the same room and talk about these.
Daniel Murnane: these exciting kind of tools.
Daniel Murnane: So I am presenting really, on behalf of a lot of people that have done a lot of work. Sometimes mind numbing, scraping work. So they are.
Daniel Murnane: really to be given the credit.
Daniel Murnane: and these are people that have. Some have come. Some have gone over the last 2 years, but at the end of the day
Daniel Murnane: the whole, who the whole assistant, is the result of of all that work.
Daniel Murnane: So the story started 2 years ago. In the Atlas Machine Learning Forum, which meets
Daniel Murnane: every few weeks to talk about
Daniel Murnane: ml developments within Atlas. And and that particular session was about using large language models.
Daniel Murnane: and myself, I had brought a tool that I built.
Daniel Murnane: and Gabriel had also brought a tool.
Daniel Murnane: Mine was very naively called Atlas Gpt. Gabriel had the better name, so we went with his
Daniel Murnane: and credit should go to to Sam for that name. So we we came together. There are a few people that were similarly interested in building this tool. And so
Daniel Murnane: we started working on it.
Daniel Murnane: From time to time, whenever we found the time.
Daniel Murnane: So okay, we had a great name. We had a dream. The rest should be easy. But actually
Daniel Murnane: putting these things into production, as Florian has exhaustively shown, is is actually super hard.
Daniel Murnane: So that brings us to today.
Daniel Murnane: Today, chat list has access to the Atlas Twiki, which is some 20,000 twiki pages.
Daniel Murnane: getlas internal Cds system, which is
Daniel Murnane: tens, hundreds of thousands of internal notes, pub. Notes, Cds slide conf notes.
Daniel Murnane: the indico not all of indigo but the indigo
Daniel Murnane: kind of domains that have reliable content.
Daniel Murnane: All of the the gitlab. Make docs pages
Daniel Murnane: and these are offered in what we call search mode and assistant mode, which is actually, it's interesting. We've converged on similar things as foreign has, and then App Gpt has
Daniel Murnane: which makes sense because we've actually been in contact with Florian and the actually Gpt team and the It Gpt team. And I'll say a little bit more about that
Daniel Murnane: in a moment.
Daniel Murnane: I would. I'll show a demo in a moment. But just to prepare you to say that the ui is fine, but not great. It's custom built from the ground up, which means that we have full control of everything, but also means that it's a bit clunky. We don't benefit from all the nice ui that kind of real engineers have built.
Daniel Murnane: This runs on Cern's platform as a service
Daniel Murnane: at the moment it's using open AI for a mini by default, which is just a trade off between cost and quality.
Daniel Murnane: This is not the final solution. This is a
Daniel Murnane: short term solution. Let's say
Daniel Murnane: we do have a embedding model that is part of the retrieval of many generation, that is, custom trained on
Daniel Murnane: a forum called Atlas Talk.
Daniel Murnane: and a benchmarking system is set up to check the quality of changes to the system. And I'll say more about how that benchmarking works in a moment as well.
Daniel Murnane: This is all running on Cern's database on demand system with postgres and Pg, vector installed.
Daniel Murnane: And we have 2 versions. We have our production app, which we encourage people to use, and then we have our staging app, which.
Daniel Murnane: and try new things.
Daniel Murnane: We have been running surveys, and there is a group of
Daniel Murnane: master's design students, not physicists, thank God, but actual, honest design students, who are
Daniel Murnane: interviewing people running surveys, mocking up designs and uis, and that's that may well lead to a sort of version. 2 of the ui.
Daniel Murnane: we have a qualification task ongoing, which is is dedicated to chat list, so that that was very nice to get kind of
Daniel Murnane: more or less full time tension from from go.
Daniel Murnane: and a lot of the material in this slide deck is produced by Ben from a recent talk at the Ml. Workshop. I appreciate him doing that.
Daniel Murnane: and let's try a demo.
Daniel Murnane: It will, of course, break, because that's how Demos go. But let's try so there are. As I said, there are 2 2 modes. There's the simple search mode. And then there's assistant mode. So simple search mode would say.
Daniel Murnane: electronic. Well, let's have a look at the filters.
Daniel Murnane: We can search across Cds Indico Twiki. Mk, docs, we want to go electro electronic.
Daniel Murnane: the larimeter calibration, for example.
Daniel Murnane: This takes a little longer than oh, actually, it was, that's okay. That was quite fast. I was, gonna say, this takes a little longer than you think, because it's actually searching across multiple databases. So it gives us, broken up by kind of modality. Indigo
Daniel Murnane: split up into both the text search and the vector, search.
Daniel Murnane: The Mk, Docs, twikies Cds.
Daniel Murnane: and of course gives the cell now relevance.
Daniel Murnane: But we can also ask the questions in assistant mode, so it might say. You know what's the
Daniel Murnane: latest flavor tagging transformer model.
Daniel Murnane: This will take a little bit longer, because it's obviously sending off this.
Daniel Murnane: These results about 50 results, I guess to the open AI,
Daniel Murnane: and okay, it gets the answer right? The latest flavor tagging model is is so called. Gn, 2.
Daniel Murnane: This is not a secret anymore. That paper was published
Daniel Murnane: 5 days ago 6 days ago. So I feel okay, pointing out that that is, in fact, the latest public flavor model.
Daniel Murnane: And the nice thing here is that even though there were some 50 results.
Daniel Murnane: we actually return just those sources that were actually used most heavily by the model
Daniel Murnane: as well as sort of hidden behind it all the other sources.
Daniel Murnane: All of this has feedback logging.
Daniel Murnane: and usage logging that is anonymized
Daniel Murnane: because we found out a lot of people ask
Daniel Murnane: maybe silly questions, but no questions are really silly. So we want to protect, even though people are logged in. This is behind.
Daniel Murnane: a login.
Daniel Murnane: We anonymize people's searches.
Daniel Murnane: And there are some okay basic settings. We can switch between different chat models. What is the rag system?
Daniel Murnane: You can add your own Api key, and then you can, in fact, use anthropic models, or gemini, or whatever you want
Daniel Murnane: oops.
Daniel Murnane: So that works. So I don't need to show that video.
Daniel Murnane: What's the big picture?
Daniel Murnane: the big picture is that people want? A an AI assistant Atlas the Atlas Information corpus is overwhelming. For everyone.
Daniel Murnane: People have difficulty searching it, and even when they can search it they often have difficulty
Daniel Murnane: interpreting it, figuring out what is outdated information, and so on. So it seems obvious that we should provide such a tool.
Daniel Murnane: Here's some little snippets, so I'll skip over them. But I will point out that a lot of
Daniel Murnane: the interviews that the design students ran and that the survey returned had to do with coding assistance.
Daniel Murnane: Now, right now, we're not building a coding assistance per se. But this should be on the horizon.
Daniel Murnane: So this talk is really an update on how our model has developed previously. We've
Daniel Murnane: we've even folks and chat lists at those times had pretty crappy answers to pretty standard questions.
Daniel Murnane: We especially Ben and Joe and and Gabriel and the team at Ucl have really spent a lot of time improving the retrieval system.
Daniel Murnane: That gives the model a lot better. Context makes the responses a lot more reliable.
Daniel Murnane: So I'm going to talk to you a little bit about how that was improved.
Daniel Murnane: So the short story is building. This thing is not straightforward.
Daniel Murnane: We need to improve the semantic search system.
Daniel Murnane: We need to improve.
Daniel Murnane: How checklist combines different databases and different types of sources, and whether that can be kept up to date in some automatic way.
Daniel Murnane: This should be familiar to you. What is Rag? We know what it is.
Daniel Murnane: On the left was what the chat lists
Daniel Murnane: pipeline looked like. Say, a year ago we just had one data source.
Daniel Murnane: We had one kind of search system. Now, we have
Daniel Murnane: more than 3. We have 4 or 5 different database types and multiple kinds of search systems. And in fact, we
Daniel Murnane: should add kind of a search agnostic system
Daniel Murnane: with lang graph in the near future.
Daniel Murnane: And one very nice thing that was added is to actually transcribe videos and audio from indigo meetings. So this is
Daniel Murnane: this is ongoing that's embedded. It needs to be included into the system completely. But
Daniel Murnane: this allows us to have basically minutes from recorded meetings.
Daniel Murnane: So this is the state of the data sets that we have everything with a blue line around it is
Daniel Murnane: scraped automatically and update it.
Daniel Murnane: Everything else is a little bit more manual. But okay, for now.
Daniel Murnane: next we have all of the Atlas code bases. Scrape and jira tickets.
Daniel Murnane: Is is on the to-do list.
Daniel Murnane: So I said, there are benchmarks that check where, whether we are improving or regressing. When we make updates to the system?
Daniel Murnane: These are all available on on hippie pi pi, so you can pip, install the chat list benchmarks as well as our scraping system.
Daniel Murnane: and let me just point to those
Daniel Murnane: just really quickly. So in fact, there's 3 we have.
Daniel Murnane: well, we actually 4. We have this scraping, we have embedding, benchmarking and the so-called chains which manages the conversational pipeline.
Daniel Murnane: So I'll just quickly flash those and say, these are available.
Daniel Murnane: The goal was to develop
Daniel Murnane: all of this out in the open and transparently, whilst the data itself is behind a wall, but it should mean that you can
Daniel Murnane: adapt it to your own use case.
Daniel Murnane: So these benchmarks take
Daniel Murnane: kind of well accepted industry processes. For example, we want to take the the twikies, 20,000 twikies, and we want to ask.
Daniel Murnane: given some question that we know should be answerable by some document in the Twiki. Can we actually find that twicky? We find the needle in the haystack
Daniel Murnane: and there are some automatic ways of doing this right, we can take a twicky entry. We can automatically generate with an Llm. The question that should return that twicky
Daniel Murnane: and in doing so we then have a nice corpus of of questions and answers.
Daniel Murnane: And in doing so you can come up with metrics such as this document match metric that that, I'll use to
Daniel Murnane: explain how the model has actually improved.
Daniel Murnane: Which is a breakthrough for us, I mean, a year ago we were sort of vibe vibe
Daniel Murnane: voting or vibe metricing. Checklist. Now, it's completely data driven. We can actually check if some rag voice is is improving the system or not.
Daniel Murnane: one nice thing here is that we created a set of different personas which should represent many of the major use cases
Daniel Murnane: across chatless, and each of those personas might ask different questions of their the data set
Daniel Murnane: and and Ben and Joe and Gabriel went through the painstaking task of actually looking through those questions and answers to make sure that they were sensible
Daniel Murnane: and screened down to just just the the most sensible ones.
Daniel Murnane: And the current data set is mostly using Atlas talk, which is a forum instead of discussion threads.
Daniel Murnane: It would be very nice to do this with Cds as well.
Daniel Murnane: So I talked about Atlas. Talk
Daniel Murnane: at the stock is very nice because it it's completely internal. It's so it shouldn't already exist in industry embedding models.
Daniel Murnane: And it's it's highly specific to the Atlas kind of domain.
Daniel Murnane: And Ben actually went ahead and fine tuned the Mp nets embedding model on Atlas talk.
Daniel Murnane: and you can see that, compared to the sort of arbitrary model on the left, so called old model.
Daniel Murnane: Our retrieval score, our document match score improves
Daniel Murnane: by an absolute value of of 15%.
Daniel Murnane: We generally saw that that fine tuning, these models actually
Daniel Murnane: needs help. We haven't fine tuned the Llm itself.
Daniel Murnane: but the fine tuning of the embedding certainly seems to help
Daniel Murnane: this. All used to be running on on Eos. So this is just a slide to say that we've solved that problem. Now, we're running on a proper database system.
Daniel Murnane: In certain databases of service.
Daniel Murnane: There is an open question about the best way to actually index that database.
Daniel Murnane: And this is still ongoing work. To be honest, I don't think this is this is quite a solved problem yet, but you can clearly see on the left. This is a booty
Daniel Murnane: projection down of the embeddings of the clicky.
Daniel Murnane: and you can see it's clearly not uniform.
Daniel Murnane: There are globular clusters. So it would be nice to say, Well, can't we use this cluster
Daniel Murnane: structure to improve our retrieval speed and our retrieval accuracy. And indeed you can. There are different indices right? There's hierarchical, navigable, small worlds, inverted flower compression. These are different ways of
Daniel Murnane: doing approximate neighbor search.
Daniel Murnane: And maybe some are better suited to different topologies, and certainly it seems like
Daniel Murnane: tuning the hyperparameters of these different indices improves our document match
Daniel Murnane: by a significant amount from 30% to 70%. Again, this is ongoing.
Daniel Murnane: This is kind of the take home message of the empirical piece, which is that by
Daniel Murnane: fine tuning your embedding model, by paying careful attention to the size of your chunking, by
Daniel Murnane: fiddling with the knobs of your index, or choosing a better index.
Daniel Murnane: And what is called here hybrid retrieval, meaning both ex search and emitting search.
Daniel Murnane: You can actually significantly improve your needle in a step.
Daniel Murnane: Search result?
Daniel Murnane: That's encouraging.
Daniel Murnane: And indeed it also passes the sort of vibe check
Daniel Murnane: people are actually reporting that the answers come back. That are more meaningful.
Daniel Murnane: As I said, all of this is on is is Pip installable?
Daniel Murnane: So please do check it out. Then, as a as a final
Daniel Murnane: story. I just want to point to what what could be coming next.
Daniel Murnane: So far, most of this is being developed by physicists, which is
Daniel Murnane: fine because we know our domain. But we might not know how humans work or how design systems work.
Daniel Murnane: So we are working with a Masters group in Denmark who are running surveys, conducting interviews.
Daniel Murnane: trying out different versions of beta testing.
Daniel Murnane: And I would point out simply that they compiled a fairly exhaustive list of
Daniel Murnane: people would wish for, and what main pain points are.
Daniel Murnane: and that's led them to come up with.
Daniel Murnane: Some some very nice proposals for what version 2 of chat list could be.
Daniel Murnane: You'll see that this kind of looks like what Florian showed. That's not a surprise. It's because what Florian showed is a sort of
Daniel Murnane: beautiful industry standard solution to these chatbots rather than our clunky custom thing.
Daniel Murnane: So probably we will be moving
Daniel Murnane: in some direction like that where we use something out of the box.
Daniel Murnane: That said, there are still ways that we can customize it to the Atlas problem.
Daniel Murnane: which is that Atlas has a heterogeneous system of different databases would be nice to offer a hub of databases in one place
Daniel Murnane: and to enable people to search
Daniel Murnane: whether it's through embedding or through graph search the the available office.
Daniel Murnane: And then, of course, I'd be remiss if I didn't say the a word. Agents, people in all of these interviews with the design students regularly said they would love to have a code assistant. They would love to have a way of
Daniel Murnane: interactively navigating through the Atlas ecosystem.
Daniel Murnane: who basically do sort of dedicated
Daniel Murnane: courses of what is available in Atlas software and how to run simulations.
Daniel Murnane: All of these things probably fall into assistance or agent world.
Daniel Murnane: And that's clearly something that we're interested in building.
Daniel Murnane: Yeah. So I'll leave it there.
Daniel Murnane: a lot of this is still ongoing. I'll echo what foreign said, which is totally true
Daniel Murnane: person power is is an issue.
Daniel Murnane: Because there is just so much to build and actually making it production ready is
Daniel Murnane: something that I think physicists
Daniel Murnane: can be good at but in the world of like front end development, that's that's new to us, I think.
Daniel Murnane: So. That's taking some.
Daniel Murnane: That's a it's a large curve to adapt to.
Daniel Murnane: We're interested in agentic assistance. We're interested in in better rag systems, and that's being worked on.
Daniel Murnane: We're working with Yog Behringer at Lbnl, on on building systems to integrate with Pdg, or that fine tune on Pdg.
Daniel Murnane: bot finding is a major thing that people
Daniel Murnane: bang on about they saw a plot 3 years ago. They can't remember it where it is. There's a thousand Cds documents, and you can't find a plot.
Daniel Murnane: So this is something we we really want to work on, and and actually have worked on.
Daniel Murnane: We have a matter most integration already, as of 3 weeks ago, that works pretty well.
Daniel Murnane: But we'd like more integrations into gitlab into Vs code cursor, whatever people use.
Daniel Murnane: And of course we'd like to actually use a locally hosted Llm. Not send of all. Send all of our
Daniel Murnane: private data to Sam Altman, and that solution might be indeed using what act Gpt or Itgpt, providing
Daniel Murnane: that seems like a no brainer.
Daniel Murnane: I'll leave it there. Thanks.
Gaia Grosso: Alright, thanks, Daniel. And yeah, are there questions.
Gaia Grosso: So I don't see any. But I have a re. Okay, I see Giacomo first.st Okay.
Gaia Grosso: want to be the 1st flight. 18. Again.
Giacomo Fedi: Yeah.
Giacomo Fedi: So you manage the benchmarking. Okay, this is a tool that is available. You said so. There are no questions in the side is generating questions based on the document, and then answer the question, and you do the benchmark
Giacomo Fedi: right? Right?
Giacomo Fedi: So
Giacomo Fedi: I'm wondering. So maybe it answer correctly answers correctly. But how do you know if the I mean makes meaningful and answer useful? If questions.
Giacomo Fedi: I mean, do you have any feedback from? I'm from your chat so users can. Can they? Give feedback about the answers.
Daniel Murnane: Currently, the feedback is pretty binary thumbs up or thumbs down. That's a
Daniel Murnane: problem. I would say we would like more qualitative feedback at some point.
Daniel Murnane: Yeah, I mean, with this stuff, there's no free lunch. Right? You.
Daniel Murnane: The more you wanna automate metric system, the less control you have, which is which is exactly why.
Daniel Murnane: you know, we had people in the group actually go through.
Daniel Murnane: the the automated question answers and and screen them for quality. So that still requires a certain level of domain expertise. For sure.
Giacomo Fedi: Okay, so.
Daniel Murnane: Said, you can. You can get some. You can get a little bit of a free lunch. I mean, you can use a very powerful. Lm, for example.
Daniel Murnane: opus for, or whatever you can afford, have that generate the questions and answers, even when your your actual production model might be a lot lighter.
Daniel Murnane: and that way you can.
Giacomo Fedi: Wondering if if the Llm. Would generate questions that a physicist would would do. I mean, but okay.
Daniel Murnane: Yeah, no, that. That's a good question. I think it helped adding, these personas.
Giacomo Fedi: Yeah.
Daniel Murnane: If you can prompt the model very clearly into what a physicist would think like, or what a typical
Daniel Murnane: use case might be. I think that helps. Yeah.
Giacomo Fedi: Okay, thanks. I see. Mauricio wanted to contribute.
Gaia Grosso: Yeah. So I have a another question. That is maybe more about privacy and privacy issues. Because you mentioned you're using, for instance, the Atas Forum, and I mean a lot of resources internal to atas. I go some talks or recordings, and these things
Gaia Grosso: internally within the Atlas collaboration. Did you have to go through some sort of privacy
Gaia Grosso: discussion. Or I mean, what's the policy you're using? And also because you're not just treating this data internally, also sending them out.
Gaia Grosso: Are you just anonymizing everything? Or, yeah, what's the process? You followed.
Daniel Murnane: I mean, the process is a little ad hoc. Every now and again we will present
Daniel Murnane: our latest results to software groups, coordination groups inside Atlas.
Daniel Murnane: there will be the usual questions of
Daniel Murnane: how do we generally feel about sending this information?
Daniel Murnane: Do Openai?
Daniel Murnane: Once people understand that the information is.
Daniel Murnane: I mean, we kind of have to take their word for it. But the small print of open AI says, if you use the Api they do not store the data for training.
Daniel Murnane: okay, that could well, just simply be a lie. I hope that it's not. But I guess we have to take.
Daniel Murnane: They're sort of. They are an enterprise tool.
Daniel Murnane: We are using them
Daniel Murnane: as an enterprise ourselves. We have to assume that they are following their own code of conduct and not training on it.
Daniel Murnane: And so far that has not triggered sort of
Daniel Murnane: a critical audit of our pipeline.
Daniel Murnane: But, as I said, this is absolutely not how we want to which that's into production.
Daniel Murnane: So far it's been just a good enough solution. The goal would be to keep it entirely within cern systems. Yeah.
Gaia Grosso: So you didn't go through any like
Gaia Grosso: process through Atlas, like agreeing and signing off some papers and something anything of that kind. Yet.
Daniel Murnane: No. And this has been
Daniel Murnane: very transparently and and publicly sort of developed
Daniel Murnane: physics, coordination, computer coordination have have been kind of across.
Daniel Murnane: This, I mean, in a sense, it's not public, right? I mean, we're not.
Daniel Murnane: It's still behind a wall.
Daniel Murnane: It's not.
Daniel Murnane: We have to assume it's not going into the public models.
Daniel Murnane: And it's not like Gdpr. Protected information.
Daniel Murnane: So it is a sort of sweet spot of.
Gaia Grosso: Bye.
Daniel Murnane: We have to assume. It's still behind protected walls.
Daniel Murnane: So yeah, as I said, it hasn't kind of triggered a.
Daniel Murnane: The review process.
Gaia Grosso: Okay. So no Atlas users sort of complained or no one raised the issue of like their data being treated by these Llms.
Daniel Murnane: I would say, no one has said.
Daniel Murnane: please stop doing this because I don't want my data shared. People have said
Daniel Murnane: we should think about this and we certainly are thinking about it, and would like to move away from this as soon as possible.
Daniel Murnane: But there hasn't being
Daniel Murnane: and I'm sensing by your line of questioning that this has happened in Cms.
Daniel Murnane: So I'm I'm curious about what that experience was.
Gaia Grosso: And it's just like we are still like under construction, let's say. But we, let's say, we raise these questions recently. And yeah, we're sort of trying to think about. What are the options? That's why I'm asking like, if you already went through this process and you found a way to handle. This could be a good example to follow. But yeah, maybe we should just get together and talk about this like
Gaia Grosso: like certain wise.
Daniel Murnane: I think that would be, yeah, really smart, actually, to have a kind of conventional solution to it. Yeah.
Gaia Grosso: Alright. Okay, thanks. I see. Also, Regava has a question.
Raghav Kansal: Yeah, thanks a lot. It's really, really impressive work in many different directions. Yeah, thanks for sharing a lot to for us to learn from?
Raghav Kansal: I had a the same question. Sorry you already mentioned, like, what resources you're using
Raghav Kansal: like, are these certain Gpus that you're running on? Oh, but actually, now, I've kind of answered my own question. So you you're you're just doing this from
Raghav Kansal: yeah, actually, yeah.
Daniel Murnane: Api, basically. So no, but I mean exactly. You are right.
Daniel Murnane: You've answered your question. So there, there are 2 like machines running right? One is one is that cern the platform as a service that runs just CPU
Daniel Murnane: when we do the embedding search that is actually pretty lightweight transformer that runs on this, on the on the CPU in milliseconds
Daniel Murnane: And then we have right running on locally hosted.
Daniel Murnane: There was a gpu set up at at Ucl.
Daniel Murnane: But still our sort of production version is is doing all of this with Api calls, and it's essentially free in Gpt. 4 h that many is
Daniel Murnane: wildly inexpensive.
Daniel Murnane: If we move to certain resources, then or even potentially
Daniel Murnane: national lab resources. For example, we have be looking at using something like pill model.
Daniel Murnane: If you use way of serving this.
Daniel Murnane: then in that case, yeah, we would use it.
Daniel Murnane: A 100 or something hosted at Cern or in a national lab.
Raghav Kansal: Okay, thank you. And then on on this slide, I was trying to understand what this means that the
Raghav Kansal: so the questions are from the twikies, but generated by the model. So they just trained on all these twikies, and it it then learns to generate more questions.
Daniel Murnane: No. So maybe that was not so
Daniel Murnane: clearly phrased. So we have a list of the most viewed twikies which
Daniel Murnane: which we can pull with some Api calls and
Daniel Murnane: we generate the questions from those meaning, even a twicky entry.
Daniel Murnane: even a persona of a particular kind of student or physicist.
Daniel Murnane: What other set of questions you could ask from that twiggy and expect to get full answer from it
Daniel Murnane: doesn't make sense.
Raghav Kansal: Yeah, yeah, thank you. And then the last one, I thought the design project was really interesting. And
Raghav Kansal: that's something we usually see in a in a physics. So who are? Who are the interviews for these are these interviewees, I mean like, are the other at this, like
Raghav Kansal: business or or general. And what's like the
Raghav Kansal: the the plan for updating the Ui, are you working with this? Is there an it department going to help with that? Or is it
Raghav Kansal: Atlas members.
Daniel Murnane: Yeah. So the process as part of the survey, we put out a call for people that were interested in
Daniel Murnane: going through interviews and Beta, testing
Daniel Murnane: surprisingly. Quite a few people were happy to give up
Daniel Murnane: the time to be interviewed and do Beta testing. I was sort of surprised. I assumed easy Atlas. People
Daniel Murnane: don't wanna do that stuff, but
Daniel Murnane: people happy to give their opinions. That's actually not surprising.
Daniel Murnane: And yeah, they they've over the last few weeks they've been running lots of of interviews, and this is all being put together into a report that
Daniel Murnane: should be finished sometime next week.
Daniel Murnane: And from that report.
Daniel Murnane: and all of the the sort of mock ups I mean, these are a little bit. These are from a month or so ago, so my expectation is, we should have a lot more of these
Daniel Murnane: iterations. You know different ab versions of these uis that are hopefully.
Daniel Murnane: directly addressing the wishes and pain points of students and and scientists.
Daniel Murnane: how exactly that gets implemented into a v 2 is
Daniel Murnane: not yet clear to me whether
Daniel Murnane: whether we use an out of the box thing like
Daniel Murnane: Lauren showed this open web ui.
Daniel Murnane: and customize it. Whether we continue to build from scratch
Daniel Murnane: I'm not sure yet. My preference is probably just to do whatever is easiest.
Raghav Kansal: Okay. Awesome. Thank you.
Gaia Grosso: Alright, so I guess we can thank Daniel again and move to our 3rd speaker for the 1st session.
Gaia Grosso: Who is Kay? I think.
Ke Li: Yeah. Can you hear?