2026-04-17-pm-foundation-models-multimodal-ai-cancer-research

Date: 2026-04-17 Primary topic: virtual-cells Cross-referenced from: bioinfo-tools, virtual-cells Word count: ~11,668

Auto-generated captions

This transcript is Vimeo auto-generated English. Expect misheard names and technical terms. Use the ripgrep/search approach for exact quotes.

Good afternoon. So just before I start, the QR code here is to submit online questions. So if you want to scan it briefly now and then you don't want to stand up to ask your question in public, you can do so via the QR code. So good afternoon and welcome to session ED03. I'm Charlotte Bonner. I'm an assistant professor at EPFL. I'm affiliated with Computer Science and Life Science, the AI Center and the Cancer Research Institute, and I'm super excited to share this session on foundation models and multimodal AI for cancer research. So cancer is inherently multimodal, so no single data type, let it be pathology or any other type of imaging, genomics or patient records really tells the full story on its own. And the promise of multimodal foundation models are really that they may finally let us aggregate everything that we can collect of a patient, and at the scale and in particular, the biological resolution that we need in order to understand the disease. So the question this session takes up is how do we build such multimodal foundation models? How do we connect them to agentic systems? And then how do we basically design them in a reliably enough way so that they can be used for cancer research and eventually patient care? So in the next 90 minutes, we will have three complementary talks that walk through the question of virtual patient representations with those multimodal foundation models. We talk about models that reason over disease dynamics and scientific knowledge. And last but not least, agentic systems that can plan and in particular, verify their own reasoning. Each talk is followed by a discussion window, so if you have questions or questions here will be dedicated to those minutes. So let's get started. I will actually give the very first talk. I will speak on virtual patients with multimodal foundation models and agents. So I will go to the next slide. As I just mentioned, cancer biology is multimodal. So we have not only multimodality, but often patient data that is really on a resolution of sometimes month, otherwise decades. And we can look at health records that are usually in natural language. We can look at genomic information. We have sometimes, especially for beyond standard of care patient, drug screening information. And then we have multiple ways of looking at tissue from the routine histopathology to, at this point, spatial omics, spatial proteomics, that give us really a molecular insight into a patient's tissue. And so the data collection I just screened here on the lab on the screen is not just an invented multimodal profile that we might have. In fact, it's the data corpus that is now sitting on the Swiss supercomputer that is from the Swiss Tumor Profiler study that looked at multimodal patient profiles to understand what do we need in order to best understand a patient biology and find the most accurate treatment. So how do we incorporate everything into some representation that we can grasp, something that we decode or call a virtual patient? We need to be able to look at all those modalities, combine the information that we have, and in particular, we need to deal with the fact that most of the time we will have only a subset of those modalities, and we need to complement some modalities with others. How do we do this across the scales of biology, where we have information on the molecular, on the cellular, on the tissue, and on the organ level? How do we integrate on the one hand, but how do we use those representations to now forecast in silico or simulate the what if? What if I would give this patient treatment A or treatment option B? And then ultimately, we need to incorporate everything to have some profile that lets us basically retrieve similar patient profiles. A patient that is very similar to a past case that I've seen where treatment A worked very well, or to find the best options or inclusion criteria for clinical trials. And this is a tremendous effort and really something that requires innovation on AI itself. So that's exactly what my lab is focusing on, building AI systems that allow us to build those virtual patient profiles. And what we think it needs is, on the one hand, multimodal foundation models about which we will talk today. It requires generative AI. That's something we will particularly cover in the second talk of this session. Can we now forecast the what if and simulate what would happen? Or even generate one modality from the other? And then last but not least, nothing is worthwhile if we have no systems to read all this information. We, at this point, actually know that those multimodal foundation models capture more than we think, but we have no way of extracting it. And we might use agentic system, or I will show you some initial generation of systems that decode us or give us AI-derived biomarkers that can be used for clinical decision-making. So a virtual patient is not built overnight. So we will start with a virtual tissue platform. How can we build those multimodal foundation models for tissue biology where we have so many different modalities that measure complementary but very different aspects of tissue? The goal again is to integrate all those different modalities that we have, use it to enable in silico forecasting. Is this the best foundation we have to now simulate the effect of a treatment? And then also extract novel biomarkers, and I will show youThose systems will enable us to actually guide what needs to be collected of a patient. Do we always need a spatial proteomics screen? No. Right? Many decisions can be simply made from routine pathology. But sometimes it's interesting to look at the molecular readout of protein A or protein B. Which is the most optimal panel? And so on and so forth. Many questions that AI can actually help us design. Let's look at classical methods that have found quite a success recently that are built on histopathology. So a generation of histopathology foundation model, what is special about histopathology, it's an RGB image, so an image that consists of three channels, red, green, blue. So something in AI we are very familiar with. We have vision transformers, in particular, those developed by Meta that are incredibly powerful and turns out we can use the same architectures for histopathology. So we are looking at histopathology in a way that we have a massive data corpus, many different cancer types, many different hospitals, thousands and hundred of thousands of slides. And now we are tiling those images in small patches and learn internal representations that are informative enough that we can use very small linear regression or linear probing to ask questions such as cancer subtyping, cancer grading, and so on and so forth. But the moment we use a different modality, if we want to get a molecular insight into a tissue and we move to a data type that is not just an H&E, we struggle. Why? Spatial proteomics, for example, is not an RGB image. It's an image where every assay looks at a set of proteins, and it might in clinical study A look at protein ABC, and then the next clinical study actually looks at protein DEF. And we actually have a lot of those proteins, so we have not three RGB, but actually sometimes 40, 80, 150 plex protein panels. So here it already starts. How do we build foundation models for those type of data modalities? And then right here I showed you just an initial cohort. We are at this point at 30 clinical studies where we train our foundation models. Here you see an initial set of 15. There's not a single study that has the same set of proteins. So we really need to step back and think, what is an AI architecture that is able to digest this complexity of data? And then in particular, histopathology records very different information from spatial proteomics. How can I use methods or how can I build multimodal foundation models that meaningfully integrate information from both that I can probe with either histopathology or spatial proteomics? So that's the very first part, and we already see that for integration, we need to think about new AI architectures. Thinking about those principles led us to write a perspective. You see many authors, many ideas. How do we build multiscale multimodal foundation models for cell biology, ultimately for cancer biology? And many of the ideas that we laid out in this perspective published in Cell are now in the architectures that I will present to you. So here's a multimodal foundation model, the very first foundation model that is truly multimodal. So it's not just learning a representation of one modality, a representation of another modality, but actually a joint representation that can be queried from either modality. And you see here that this is not any more a vision transformer developed by Meta for RGB data, but it's actually a transformer developed for biological samples. So from the level of tokenization, how do we define a unit that these transformers process until the loss function that defines the learning signal, this is all designed to be tailored for biological tissues. We need to decode the protein identity because the model has to process more than just this is channel red, this is channel green, this is channel blue. We are able to process here in the same way histopathology, but also the complexity of any random input of a set of protein markers measured in a spatial proteomics experiment. So this is an example of a multimodal foundation model, and I will show you in the next second what it enables us to do. How is it trained, right? For this, we need to collect datasets that are of a certain scale, and you can see here why those histopathology models are trained on hundreds of thousands of data. We have very, very few patients for which we have multimodal data. We've collected here the largest open source dataset that has, I think up to now, it's actually 5,003. The one that will be published is 4,605 patients for which we have spatial proteomics and histopathology. It requires also actually compute of quite a scale. So those models are actually trained and much larger than standard vision transformers because the data that it has to process is just simply much more complex. And we see that the more data we have, the more parameters those models need. How do we do this? Right? My lab is very small. We are working on Switzerland supercomputer, so national infrastructure that is built for cancer biology or that benefits cancer biology because those models get more and more demanding. It's one model with many capabilities, right? So this is the beauty of foundation models. We have a representation that is general enough to now ask many questions. We can look at simple classifier questions like what is the outcome? What is the cancer type? What is a certain mutation? But we can also look at the-- like do a tissue analysis. In particular, we can segment as well as cell type all in one shot.We can look at virtual staining. We might have only an H&E and want to decode what has been measured. And our model I will show you is actually better than models that have been only solely trained on this task. But we can also decode all the other proteome of the proteome that has not been measured. So we measure five of the minimum protein panel, and we actually just virtually stain the rest. But what I'm most excited about is that this type of model can actually guide experimental design. What is the most optimal protein panel? When is a histopathology enough? When is a spatial proteomics experiment needed? Last but not least, but we now understand that those models have actually extremely good patterns to guide clinical decision-making, and in my opinion, a new generation of biomarkers that we should think on how to bring into the clinic. So here you can see one example of the data annotation of our cell typing and then competitor foundation model. You see that those foundation models, when you give them the information of which protein is encoded, it learns robust enough representation that you can, as in this case, do complete transfer of cell typing. So it's a new lung cancer tissue that the model has never seen. It's cell typing without a single label. So this is really complete transfer of capabilities on the downstream task. And you see, of course, there's some differences to the ground truth, but also the ground truth is no ground truth. It can virtually stain, so go from H&E to virtually stained model. Of course, this doesn't work very well for many proteins, but it gives us a proxy that might be valuable enough. And you can see it is as good or better than models that have been solely trained on this one task. Because we now know that foundation models have to learn such a general representation that they actually learn a robust or sometimes more robust representation than specialized models. These are kind of tasks that are interesting, but we see that many of the other models can do them when they are specialized on the task. So let's focus on capabilities that only emerge in the setting of foundation models. One of them is that we use them as a foundation for in silico screening. So now we have tissue data that we can map into what we call a virtual tissue representation. It's a robust representation that captures what? It captures spatial environment and a spatial arrangement of cellular identities. We can now use a treatment encoder and now use a generative model to forecast how those state or cell tissue states change over time. We can read out how this tissue has changed, and we are using here a set of biomarkers that is currently used in the molecular tumor board from those drug screening assays that are informative markers for clinicians to understand if a treatment has worked or not. This work is, in fact, funded by an AI 2050 fellowship by Eric Schmidt. It's very experimental AI work on this end, and I will not dive deeper because the second talk will very much focus on those type of models. But how do we have now patterns of those models that are actually very informative for clinical decision-makings? We have here the luxury of a clinical study by Wang et al. 2023, where we have triple-negative breast cancer, so patients that are subject to immunotherapy and chemotherapy. And in particular, we have here tissue samples before, on, and after immunotherapy. And we have them with signal of what is a respond and what not. You can see here a learned representation of this foundation model, and now we ask ourself a question: Do those foundation models learn patterns in the data that are extremely informative if this is going to responder or non-responder? So these are the representations of the model before treatment. And in fact, we can extract minimum signatures that are corresponding to responder and non-responder signals. And here you can already see it in the color coding. For responder pattern two, it's looking at a particular subset of immune cell types, and same for RS one, that is very close to those apoptotic tumor regions. When we look at those, we see that they are much more informative than any of the classical clinical stratification schemes that the clinicians that we are collaborating with ask us to run that they are currently using in the clinic. And in particular, it is much more significantly better than the biomarker of the original nature study that had really been focused on finding an informative biomarker, but is a biomarker that will not work on the next clinical cohort because it has a different set of proteins, and because it's just so tailored to one dataset, it simply does not translate. We, on the other hand, can go to the next clinical cohort. Again, it's a clinical cohort that the model has never seen. We take the same set of biomarkers, and again, we get a stratification into low, high, and medium-risk patient. So those who have a lot or not a lot of those signatures, that is very informative and in fact, leads to a better kind of stratification than, again, the original cancer cell study that basically proposed a new biomarker, as well as the clinical stratification schemes I mentioned before. So these biomarkers have interesting patterns. I can also show you more detailed what they actually capture. They capture very well-known biology things we know, but just things that are hard to extract by eye or by manual analysis.So last but not least, how do those models guide clinical decision in a set of how do we design an assay? Now we have models that learn those representations, and those representations help us to ask many or answer many different downstream tasks we are interested. What we want is that the foundation model knows what it knows and asks us for more whenever needed. So we need to have a system at test time, at deployment, that basically is able to understand which information it needs. So it's a sequential acquisition problem, asking for more whenever needed or stopping when done, that uses reinforcement learning as many of the test time algorithms also used for language models at test time, and now sequentially basically acquires to be good at any type of downstream task. This is a method that we built, it's called EchoK, that is now able to tell us exactly, for example, which markers to acquire in a spatial proteomics experiment. Of course, those methods are not necessarily sequential, but we can just run it for five and then stop. And it gives us an informative pattern of markers to acquire, and here we evaluate on how good it is at cell typing. And you can see that this is already good with a plex of eight, right? Instead of the full 40 plex. So it's a minimum assay that is good enough for any downstream task, including here evaluations on cell typing. How do we deploy those models, right? We need to be able to give agents access to those models so that they can use them. And all the analysis that we currently do, automate them, and actually put them into a place where they can easily be used also in a tumor board by clinicians. So we've started thinking about those agents in a molecular tumor board that have to be able to reason across multimodal system and longitudinal data. Very first step here is designing a benchmark, right? That actually ask hard enough questions so that GPT fails, because those questions are not just found in the Internet. So a very first Neurt paper from last year focused on developing a benchmark. And then here we have an agentic system that is able to interact with those foundation models and basically solve the benchmark. Very first results, but the most of the processing is still unpublished, is that whenever we give those models or those agentic system, here you see for many type of vision language models, access to those tools, they improve. But this is still very simplistic systems, and we are still working on the bigger one. So with this, I just wanted to give a brief introduction on how to build such multimodal systems. How does it look like for the case of tissue biology? What innovation does it need on the level of architecture? What efforts on data collection? And then ultimately, we will hear much more in talk two about generative models, and in talk three about agents. And with this, very happy to take some questions. We have approximately five minutes. Hello, thank you very much. Very interesting talk. I had a question about how you integrate your protein data, because having worked with antibodies and being new at that, I know that there's a lot of complexity when it comes to working with them and how well did you stain, did you-- all these other parameters and all this noise that you can get and cross signal reactivity. So if you're having all these huge panels to train your data, I would imagine that it's pretty hard to maintain the quality control. So how do you guys handle that? Yeah. So we do train those models at this point across five spatial proteomics technologies, right? So I think we need to get used to the fact that data is messy and that data is noisy, and we simply need to build systems that can deal with it. So if we think about everybody's thinking the Internet is messy, right? We still have very powerful language models from them. If you look, Instagram data is very monotonic because it's a lot of food, yes. So you do initial processing, right? And kind of sub-sampling the amount of food images you want to have your models trained on, but then you still train them on the corpus of Instagram. And so same here, we are encoding proteins by giving them access to a protein language model that's not ideal, right? We can do much better. For now, we are basically tokenizing the identity of a protein using a protein language model that's been trained. And now we basically are trying to just train our models on as much diverse data as possible, which is quite hard, right? We are only at eight cancer types at this point. We are at five different technologies that have all differences in their antibodies and in their measurement technology and so on and so forth. So the diversity here helps us. In fact, I didn't show you all the details, right? But the model is able to get rid of technical artifacts. So we see that there is a way for those models by the amount of data it sees to distinguish between technical artifact and biological signal. I think a lot comes by just scaling those models, and we see a lot of performance the more data we add. And so the initial submission is on 15 clinical studies. We are currently at 33. Most of them are public, and we will release them in a way that they can be used so more and more people develop better architectures. Thank you. So let me check if there's some online. Okay, so there's a lot. Yeah. So there is a question how it compares to recent methods that basically do purely virtual stainingSo basically, this method can do, among many capabilities, virtual staining. Other methods that have been previously published that focus only on virtual staining basically solve some of those tasks. In fact, the performance of those models we compare to, if you only have a model trained only on virtual staining, often the performance is actually less good than having models that train on many different tasks, and this is also something we see here. In fact, virtual staining, I'm still very skeptical because if there's not the signal in H&E, then even a foundation model that has been trained on multiple modalities cannot infer what is simply not present. So I think what is much more realistic is to be in a setting where we have a histopathology, an initial set of markers from which the model has enough cross-attention signal to actually incorporate or reconstruct the rest. So I'm very skeptical of methods that do purely virtual staining. Yeah. Hi. Yeah. Great talk. Thank you so much for this information. I have a question about the scale of the data. Yeah. When you train foundation models, for example, analogously on the internet data or the coding data, there's a whole lot of it. What do you think this number of patients, is that sufficient for- No. It's a beginning that allows us to train those foundation models. But we did some initial analysis on scaling. So I think we don't yet know how scaling laws look like in biology, especially in terms of multimodality. Multimodality is not like in vision language where we have modalities that perfectly overlap or more or less overlap, but in particular in biology, we have all modalities that give us different angles on the underlying ground truth that we will never measure as a ground truth. So scaling laws are still to be questioned and to be analyzed, but if there are scaling laws, we are really at the tiny tip. And there's a lot of data. It's just not in the public domain. Thank you. Hello. I was wondering how with the especially multi-model foundational data you handle conflicting biology. Especially across your models, if some of them have conflicting biology, how do you handle it then? What's the logic you use, especially with virtual digital twins? Yeah. So, what we see is that I can just argue for what we've analyzed so far. So histopathology and spatial proteomics, for example, give you different angles on a tissue. One focus is much more morphology. The other one has actually less morphological signal, but much more molecular information. And we can actually see different signals if you give the model one, the other, or both at once. So I think there is a certain level of conflicting biology, those models can find basically patterns that you have a constraint optimization problem that basically fit both criteria to some degree. So I think I wouldn't call it conflicting. It's just like setting different constraints on what might be actually the case. Yeah. Thank you. Hi. Really interesting talk. When you're developing your foundation model, you can have your different modalities go through a lot of different layers and be massaged into a lot of different representations. In your work, when do you find it is best to combine different modalities, like much earlier in the process of the algorithm- Yeah ... et cetera? So we compare basically this, what we here have is an early fusion scheme, something that we also have in GPT-5. GPT-5 is a truly multimodal system. We compare this to late fusion schemes, so what we actually had in just GPT-2, 3, but not everything is known, where you basically have two independent encoders and then just a late fusion scheme. Those models, they actually don't learn how to connect the information between modalities we compare against them. And basically, the capabilities that you have in having multimodal systems to much improved segmentation, for example, because you have both morphology of H&E and the cell type identity of the molecular stains. This you don't achieve when you have those late fusion schemes because they actually don't really capture. And even in those systems that are early fusion like ours, we still see what we call shortcut learning. So that the model actually learns shortcuts too. And it's really hard to get rid of those shortcuts. We are still working on this. So I think, just a broader perspective on where AI has to go, we need to have really good ideas on how to do proper multimodal learning to actually satisfy requirements that we have in biology. Yeah. So I have another question, if that's okay. Yeah. I imagine these different modalities have different loss functions tied to them and what might optimize each best. So how did you go about finding a multi-objective function- Yes. Same ... to combine all these things? That's a huge design scheme- Yeah ... to explore. We are out of time. Sure. But also there, I think if we look at the loss function that have worked so far in language is the next token prediction. Biology, difficult if we don't have sequences. Masking also for a tissue not ideal because if you just mask a T cell, why would the model know? So I think also I wrote a recent essay, it's not out yet, with Aviv. I think the space to explore and what type of learning signals we need to give the model is as crucial as the architecture itself. Awesome. Thank you. Thanks so much. So we are unfortunately out of time. I will be here after the session, and there's also a lot of questions online that we have not covered. But I would like to move to the next speaker. So Serena Yeung Levy, she's a assistant professor of biomedical data science at Stanford, and she really works at the frontier of multimodal foundation models and generative models. So her talk will focus on modeling disease dynamics and reasoning over scientific knowledge. So two of the capabilities that we will increasingly rely on as models move from pattern recognition towards really scientific inference. So Serena, the stage is yours. Okay. Hi everybody, and thank you very much, Charlotte, for the introduction. So I'm an assistant professor in the Department of Biomedical Data Science at Stanford. And as Charlotte mentioned, a lot of my research focuses on multimodal foundation models, especially coming from the perspective of visual and image data, which is a lot of my background. And so that's the perspective I'll be focusing on today. And just to set the tone for what we'll discuss, I just wanted to show this figure, which captures a lot of this diversity of imaging data. Of course, imaging is an essential modality across many aspects of biomedical discovery, but one of the major challenges remains how can we extract and interpret the fine-grained information across all of these different data modalities, which is furthermore, not only in 2D, but also 3D and time-lapse as well. And in addition to this, these imaging data is also heterogeneous and scattered across many different sources and formats. And so we have many imaging repositories and databases where it is challenging to combine all of this data together and reason jointly over them. And we also have a lot of data in scientific paper and literature that also has images together with loosely matched text and accompanying information about these images. And so today what I wanted to talk about is giving an overview of several different types of generative models, two in particular focused multimodal generative models. One, the first on the left is focusing on models that produce imaging as output. And so these are types of virtual cell models, but now focusing on image generation. And then on the right, these are visual question answering models. And so these are multimodal large language models, where we are going to produce text as output, but the input is going to reason over imaging data. And so these are different ways in which imaging data can be incorporated into different classes of generative models. So I'll start first here, and in particular, I'll talk about these generative image models in the context of building AI virtual cells. This is actually a nice figure from the perspective Charlotte mentioned that she led. But I thought it's a nice figure that captures this idea of where we want to be able to simulate the effect of different types of either chemical or genetic perturbations, and to be able to eventually use these in these virtual experimental loops. And so when we think about trying to simulate these perturbation effects, one of the widely recognized challenges is how can we disentangle the true perturbation effects from real-world experimental batch effects, in particular, given the populations of control and perturbation cells that we have across different experimental conditions and plates and wells here. And so what we can think about is this problem here, which is essentially trying to be able to map these populations of control cells to perturbed cells under different perturbation conditions. Right? And so technically, we can think about this problem as trying to perform distribution to distribution modeling to map across these different distributions and to try to simulate and learn their transformations. And so I want to share one state-of-the-art type of approach that is well-suited to this problem, that is a widely used approach in machine learning, and in particular, flow matching, which in flow matching, what we are trying to do is to learn a continuous velocity field that transforms from one source distribution to another target distribution. And in particular, flow matching is behind a lot of the state-of-the-art visual generation models today. And so when you see, for example, state-of-the-art image generation or video generation, these all have these similar principles in these cases, trying to typically map from a noise Gaussian distribution to image distributions or to video distributions. These are actually just some examples of state-of-the-art models. But one thing I want to mention about flow matching, why it has been very successful in modern AI is because of its scalability. Because we'll see in just a little bit, but it is easy to be able to train at very large scale, and to achieve these foundation scale models. However, at its base, flow matching is designed to transform between arbitrary source and target distributions. So it actually very naturally fits this problem that we're talking about, trying to go from the source to target cell populations. However, the actual setup of mapping from arbitrary source to target distributions has not been very well studied or demonstrated in practice. And so I think that's an area where in this particular problem setting, I think there's a lot of opportunity to study further. And so how does flow matching, how can it work in this particular setting? Well, so CellFlux was a work that demonstrates how this can be done. And so here you can see examples of a control cell image population and then perturbed cell images. And the idea here is that we want to transform from one to the other through an ordinary differential equation. And then what we will do actually is to learn the velocity field that corresponds to this, and we'll learn a neural network that approximates this velocity field. And so ultimately, it becomes this very simple to optimize objective function that can just be trained at scale over large amounts of data samples.And then once we have this learned, then at inference time, we can simulate this perturbation transformation by just rolling this forward and applying this velocity field on source control images. And so here just shows some examples of qualitative results of this initial CellFlux work that if you see the control images on top, ground truth perturbed images on bottom, and CellFlux in this row of images was able to outperform previous baselines on this image generation task. And one nice thing about using this type of flow matching is that it also, because we're learning this velocity field, you can also visualize these interpolations, the transformation from our source to our target distributions for individual cells here. And so this is something that, although hasn't yet been studied very deeply, I think is an opportunity to also potentially investigate whether there may be nice insights that may be able to be gained by these types of visualizations. And then, so I just want to also contrast here that this is this kind of source-to-target distribution mapping that allows us to do this, where standard noise to target doesn't give us the same capabilities. So as I mentioned, this is still a setting where there's still room for further study, and so for example, there are challenges with data sparsity here because now we have not only real data in the target distribution, but also the source distribution, as opposed to just Gaussian noise here. And so for example, there has been also work where we have looked at trying to inject stochasticity into different aspects of the flow matching process, which was able to improve the training of these models. And then, as I earlier mentioned, the ultimate goal is for us to be able to scale. And so flow matching gives us this scalable paradigm. And so this is something that we have looked at in CellFlux V2. This is a work where we looked at what can we do as we try to increase more and more training data, and are we able to observe improved performances. And so here you can just see examples of how these generations look as we scale from 300,000 images to 11.6 million images here. And it's really this, if we're able to scale successfully, that we can achieve these virtual cell foundation models. And so scaling laws, these are really key to the powerful performance that we've been able to achieve with modern large language models. This is just an example where if you're able to see the kind of behaviors that you want as you scale larger and larger, and to see these behaviors, then you're able to have also this confidence that as you collect more data or to scale different aspects of your model, the types of behaviors you might get. And so, the good news here is that as we are scaling up CellFlux, that you can see some of these good scaling behaviors that give us promise of the ability to continue to improve our performance. And then here is also showing, now looking more specifically at more biological accuracy metrics, where we are looking also at mode of action in addition to just the image generation quality. And then finally, in addition to just imaging here, so a lot of existing efforts in virtual cell modeling have also focused on transcriptomics. And so there's a large body of work there. And however, what I think is also very important is being able to jointly model these together, imaging and transcriptomics together. And so this is just an example showing that we can now also naturally extend these frameworks to be able to jointly model these modalities together, in these cases, obtaining latent representations of both, being able to then jointly perform flow matching over these combined representations. I think how do you best represent these jointly? That is, I think, a open question where there could be a lot of improvement here. But even naive ways of just combining these and concatenating these sorts of embeddings allows you to perform this type of joint modeling. Okay, and so this is essential for complete virtual cell modeling. Okay, so here I've discussed some of these generative image modeling. And now in this second part of this talk, I will introduce some work around trying to produce visual question answering models, and in particular, to have scientific reasoning capabilities. So the premise here is that there's a lot of interest in being able to use LLM, and in particular, multimodal LLM models when we deal with microscopy data, to be able to aid us in our scientific investigation process, potentially referencing literature, helping us with different types of reasoning tasks. And so we'll talk about where we are with these types of models, how well are we doing and what is continued to be needed towards getting us to these virtual scientific research assistants. And so one thing that was needed in this domain was to first really understand how well can current VQA models perform multimodal scientific reasoning, and in particular, for researcher-level scientific research and discovery.Right. So there's been a lot of effort in benchmarking. So benchmarking is critical to driving progress across these large foundation models. We have all the key ingredients, including data, benchmarks and evaluation, algorithms, of course, and compute. And so in benchmarking, there's been quite a lot of effort in trying to generate question-answering data sets, going from, in the science domains, from grade school-level questions to undergraduate and graduate level. There has been some that are also focused, some on text, some with images, and trying to get at this research level. But I think the challenge is that there really doesn't exist easy, good sources of research-level, scientific microscopy-based reasoning. Not the same way as you might have exams corresponding to particular other domains. And so if you look at this and on the bottom here, you can also see these Bloom's levels, estimated Bloom's levels across these benchmarks, from recall to the higher levels of analysis and evaluation. And so a lot of the existing benchmarks still was covering primarily these earlier Bloom's levels. And so MicroVQA was a work that tried to make progress in covering that space of higher-level reasoning, and to truly be able to assess researcher-level capabilities. And so here, our goal was to basically try to understand what can be a brain dump of the kinds of questions and interactions that a biologist researcher might want to have with a multimodal LLM about their microscopy data. Right. And so let's see. And so what this benchmark was able to do was to ultimately extend benchmarking to these higher Bloom's levels. You can see here, this filled in, really covering these higher levels of analysis and evaluation and just much less of the lower one and two levels. And so this-- Sorry. I think there might have been one slide missing here that I intended. But what we did here was to basically try to get a brain dump of asking a diverse group of biologist researchers, how might you want to treat LLMs as a smart colleague? And through a lot of discussion, a lot of the answers came down and boiled across several major categories. So you can see here, expert visual understanding. Can we compare image sets? Can we identify abnormalities? Hypothesis generation, looking at causal mechanisms and functional implications, and then also experimental proposal. Can we suggest new experiments and also address technical issues? So how can you also improve the image capture itself? And so, ultimately, there were over 1,000 questions that were curated into this benchmark across diverse fields and biological image types. And with these experts, it was this process of collecting the questions, but then also being able to format it into a format that's appropriate for benchmarking. And so this is an actually important technical aspect of creating these question-answering benchmarks. Because you can imagine that there's many ways in which you can have naive multiple-choice questions with very easy distractors or things that suggest at the right answer without truly testing the capabilities. And so there's a lot of work in the space of benchmarking on how can we have good principles for how to design these types of questions. So you can see here the types of original expert questions and answers that you might get. And then here, we'd have this multi-step process of turning those into exam cell multiple-choice questions. First, through just converting it to the multiple-choice question format, and then refining these in a way to address many different types of issues, like especially language shortcuts. You have appearance cues in questions. You have silly distractors, language biases, other things like this. And so what you can see here is that this benchmark is still hard for current state-of-the-art models. This was at the time of the work, comparing against existing state-of-the-art models, both commercial and open source. And then as models have advanced, and so you can see here from o1, o3s, other models, GPT-5, that there is improved reasoning continuing to make progress on this leaderboard. And so that shows that as they are focused on increasing reasoning capabilities, how this performance is also able to improve on these leaderboards. MicroVQA has been used by these various frontier labs to measure their performance in scientific reasoning. And so you can see here, for example, Gemini 3 Pro, getting the current state of the art on this benchmark. But there's still ways to go. And so I think this suggests both the fact that these frontier models will continue to improve, but that there's also a lot of opportunity to understand where are they failing, why, and what's really needed to close these gaps.And in particular, both in what was found in analysis of Micro-VQA errors and I think in general in VLM, vision language model benchmarking and analysis research, is that the vision part of these multimodal large language models, the perception is still quite fuzzy and the ability to do visual reasoning here. And so while LLMs are, I think, quite advanced in many ways, vision language models, although they are making big strides and they are able to do a lot, there's still a challenge to understand a lot of the fine-grained reasoning, for example, from just even perceiving fine-grained details from things like counting, that these are all still things that are relative weaknesses. And a lot of this also goes to the types of how do we represent visual data in these multimodal large language models? And how do you encode the vision data to be used as part of this reasoning? Okay. And so, as we work on trying to improve multimodal scientific reasoning, and especially as we work on both methods and open source models in the research and scientific communities, it's also important to have data to be able to work on these problems. And so, scientific articles are a very rich source of multimodal data. They have a lot of images as well as accompanying textual information in many different forms, both in captions, in main text, and inline text references. And so one good source of open-access scientific literature is PubMed Central open access, about six million open-access scientific articles. And so BioMedICA is a work that endeavors to make all of this PubMed Central open-access scientific literature accessible for vision language model development, training, evaluation, and it contains in total 24 million images and caption pairs, 30 million figure references, all the full article texts, as well as some other associated metadata. And in particular, this data was extracted, also annotated and transformed into a kind of AI-ready data set that's easy to use. But in particular, also hosted in web streaming format on Hugging Face, and so you can work with subsets of this data or all the data without having to download everything locally. And BioMedICA is also labeled in a semi-automated fashion with a taxonomy of different types of global and local categories. And so this allows data filtering and balancing to be able to develop different tailored foundation models or for different training data recipes. And so this can be useful in various ways. This is showing some examples of using BioMedICA to train state-of-the-art clip embedding models, to be able to train auto-regressive visual question answering models, as well as in retrieval RAG systems, in this case for some biomedical guidelines question answering. But this is a resource that we hope can be useful to the community in working further on multimodal vision language models. Right. And then finally, I think there's a lot of opportunity for then work in how do we, given these big resources, actually develop the best environments and training to be able to perform capabilities like accurate question answering in scientific papers, which also involves things like being able to perform accurate hopping across this data, traversal, factual question answering. And so I think there's a lot of opportunity to develop different training environments for methods like RLVR with reinforcement learning here with verifiable rewards. And so I think this is also an opportunity where there can be a lot of utility in domain knowledge and understanding how these data are best structured and maybe best leveraged, given we have only limited amounts of this data that contains such rich scientific knowledge, and how can we develop training environments that allow us to learn with the most signal possible? And so this is just one example of trying to create an environment now for factual question answering across many different paper abstracts. I think there's opportunity for a lot of work along these dimensions. Okay. And so I think we are just out of time. So I hope it's given you some sense today of some of these ongoing opportunities for continued work, both along generative image models as well as multimodal large language models that can also reason over images as input, and this type of multimodality in generative models. So with that, I'd like to just thank some members of my lab that have contributed a lot to some of this work, as well as many collaborators that were part of the works that I discussed today, and happy to take any questions. The question here. Flow matching models are a bit notorious for performing poorly out of distribution. Do you feel like your flow matching model performed well because your prior distribution wasn't actually an image? And moreover, what is the size of the corpus of your images? Did you have some simulated oracle to get a better distribution? How did you go about that? Yeah. So we looked at different degrees of out-of-distribution. So to have, for example, whether it's perturbations or images, some in distribution, for example, training on a perturbation that was seen, versus a completely unseen perturbation. So we looked at it at different levels, and of course, the performance gets worse the more out of distribution you are, but I think it's still overall promising trajectories here. And then that's also something where the idea with scaling is that the goal will be to tackle some of that, and so we do see improved performance as we go from on the hundreds of thousands to the millions. But there's, I think, a lot of continued opportunity to scale. Yeah. Cool. Thank you. Thanks. Maybe I'll go here first and then, yeah. All right. Great talk. Thanks. On the CellFlux paper, I was just curious, do you think, and IMPA or any other work think that working in image space is necessary because most of the evaluation is going to be done, FID scores, whatever it is, in feature space, and whenever we do image analysis, we extract features. That's how we get to the actual root of the problem. So is there a route of just working on a flow matching model that is directly in an embedding space rather than wasting time on pixel generation and stuff like that? Yeah. So actually in the CellFlux v2, the actual flow matching is done in the latent space. Okay. And so I think that especially when you want to scale, it is helpful to model it in this latent space. But of course, you may want to decode, to see what- Sure. Yeah ... your image looks like. So I think it's, yes, a good point. Cool. Thank you. Mm-hmm. Go ahead, sir. Hi. It's a great talk. So, I'm really interested in the CellFlux MM. I'm wondering how does it fill the somatic gap between the visual morphological changes and the transcriptomic signature or molecular signature? Yeah. So, I think the idea here is that there's both complementary information from these different modalities. And so, I think one direction is that when you try to work on modeling them jointly together, you can try to leverage a data you have from both. I think a very interesting direction is taking small amounts of data where you have in those data sets both modalities, but also very large unimodal data sets. So there's much more data that's unimodal and trying to pull all of these together to try to achieve the improved models. And then the other direction is also trying to go from one modality to the other. So, for example, you may have cheaper imaging data and you want to also try to then predict what the transcriptomics may look like. And so I think ultimately, at the core of good performance of these models in general is having strong representations, and those strong representations that allow both joint modeling, but potentially also decoding between them. Okay. And another quick question about the flow matching. I know that the flow matching will provide a continuous and a very smooth trajectories, but how do you model some discrete transitions, like the apoptosis, and other biological process? Yeah. So I think this is a good question, and I think it is a challenge. I think that both we have to understand better what kinds of trajectories are being learned. So first of all, we are modeling these in the latent representation and to try to then understand what's being learned here. But I think also there is then also opportunity to look at these at multiple layers, like hierarchical modeling. So right now we may be looking at particular single cell transformations, but maybe you can look at what happens over also populations of cells and how those may evolve. And then I think there's also additional methods that will be needed to come in to properly model all these things. Thank you. Sure. Thanks. Hi. Great talk. You partially answered my question. I was also interested in the CellFlux MM. So it sounds like for the imaging and transcriptomics data, you have a small set of matched data and then a much larger set of non-matched data. So where you have the matched data, how are you ensuring that you have enough power to actually get a foundation model out of it? And then where it's not matched, so much of your talk talked about doing these distributions across different batch effects and things like that, so how are you handling that where you have unmatched transcriptomics and imaging data? Excuse me. So, I think you're right. I don't think the matched data pair data is enough to train... I mean, we don't have the data to train as large scale foundation models as we want. So I think the goal will be to take-To learn a single model, but that learns from matched data where that exists and learns from unimodal data where you have that. But you can bridge across them because you have each modality across both of these types of setups. Awesome. Thank you. Okay. Take a- First of all, thank you very much for this interesting talk. I just want to check the time because I can't see the time. Last one. Okay. I have this question. So I'm interested in training a vision language model, but my medical data consists of several images, but with one caption. The caption mentions some of the important findings from a few images. And do you think, is there a standard way to train vision language model in this case? So you're saying you're having multiple images- Mm-hmm. ... with a single caption? Yes. So yeah. So I think that the naive thing is, well, each image is associated with this caption. Or you have one big image, if this was a multi-panel image with a caption. But obviously, it will create noise. You may have signal overall. You may still get some model if this is a part of your data. But I think actually one promising route is caption rewriting. So are you able to, through some knowledge, you know something about how these images are different, but to be able to refine those captions to better fit each image individually. And so one possibility is using, in fact, you can use LLM sometimes to help with this or some agentic system where you're injecting some knowledge. But ultimately, if you want to say that these images correspond to different semantic concepts, then you will probably want to refine your captions to embody that. That makes sense. Thank you very much. And also, I want to mention that when I started studying deep learning, I saw separate videos of you from YouTube, and it's good to see you here. Thank you. This is a Stanford class on deep learning, so thank you. Thank you very much. Thanks so much, Serena. So we will have a final speaker, Michael Moore. He's an assistant professor of medical AI at ETH Zurich. So his work focus really on the development of agentic systems for medicine, and in particular on agents that can plan, evaluate, and verify their own reasoning. And that's exactly the capabilities that we will need before any of those systems will be deployed alongside physicians. So Michael's flight got canceled. The airline is on strike, so he's joining us remotely as a recorded talk. Let's see if this works. Welcome to my talk about reliable medical AI: agents that plan, evaluate, and verify. First of all, I'm very sorry I cannot make it in person. I assure you I have not been uploaded, but my flight had been canceled short notice. AI is permeating our everyday lives, if we want it or not. But there is a positive side. It will and should change healthcare as we know it to the better. It will help us better integrate multimodal data that might be spread across different systems and different formats, and will allow us to diagnose patients more comprehensively and earlier and help us also target therapies. But there are also some big obstacles in AI in medicine field. For instance, we have been working with models that require a very fixed type of data modality input, solve a very narrow problem, and are of limited explainability. I believe we need to work towards AI systems that are more flexible, that can ingest and produce different modalities interleaved, that can adapt and learn from context, and that are able to reason explicitly with biomedical knowledge and clinical intuition. This vision we first presented three years ago in a perspective, where we presented a new paradigm for how foundation models can unlock generalist medical AI. Meanwhile, there has been new challenges arising, especially around confabulated and unverified outputs, and here we really need to develop new methods, new strategies to ground and verify outputs of AI systems. The fundamental problem is that, despite recent advances of AI systems in the medical domain, many systems typically rely on language models, and they are just estimating a distribution over the most likely next word. And they are not necessarily grounded in the world. But this is a problem, because if a patient presents with a specific symptom, there might be an entire myriad of different differential diagnoses that might be rare hiding behind this patient. And to unravel this case, we need to actively seek information and collect various types of tests that might be different from case to case. So it is not enough to just learn about all the disciplines and all the diseases, et cetera, in a book, but we actually need to interact with environments.And this behavior is actually much harder for language models than previously assumed. We showed that already two years ago, that if you provide a language model, you confront it in some type of agent clinic where a doctor and patient are all played by language models, and you provide different tools to the doctor. It will be much harder for such a doctor agent to come up with the right diagnosis if the case has to be interactively worked up as compared to those case profiles where you have these exam-style benchmarks, where all the information is being presented on a silver platter. So here you see one by one how many models drop drastically in performance if they have to actually do the workup on their own. This brings us to where my research lab comes in. So we want to develop and evaluate flexible medical foundation models as well as AI agents that are grounded in multimodal biomedical knowledge. This covers different branches of topics, which we will go into more details next. To briefly tell you who I am or what I have done so far, I actually started out with medicine, pivoted then into a machine learning PhD at ETH Zurich, and then got completely lost to the machine learning and AI field at Stanford, where I did my postdoc for working on medical foundation models. And now since fall 2024, I have the pleasure to lead the medical AI lab at ETH Zurich. Okay. Now back to business. So how do we really unlock medical AI agents? What do we even mean with agents? We have heard this term a lot recently. And another question we want to also tackle here is how can we make them reliable? Because we don't want our AI agents to go delete our entire email inbox. So the first step is we need to make sure that our AI agents have access to some high-quality external knowledge sources, so they have to be grounded. But how do we do that? Conventionally, a language model would just take in a question, for instance, some instruction, and produce an answer. And in a more agentic environment, language agents would actually have some kind of environment at disposal where it can use different types of tools, query against databases, use calculator, and so on, to then come back with a more grounded or well-informed answer. And it also can take actions in this environment. So let's zoom in a little bit on how an agent should retrieve information. How did language models do that previously before we were talking so much about language agents? So a very classic paradigm is that we take a query provided to a language model, and then that language model is able to search some database of documents. This could, for instance, be your guideline documents that you care about. And then we retrieve some documents that are most relevant and then prepend that to the input to the same language model, and that will then produce an answer that has access to these very specific relevant guideline documents. This sounds all nice in theory, but the problem is that currently we use just a massive amount of text chunks that are completely unstructured. So retrieving from this is actually quite hard and quite leaky and noisy. What we would like to have is a more organized representation of the data, where there's a neat hierarchy, and we know exactly how to navigate the space to retrieve the needle from the haystack. I think this is also relevant in oncology, where it's really hard for an AI system to retrieve the exact single case study or paper that was relevant in a molecular tumor board. So to address this in a general medical domain, we propose this Myriad dataset and also approach. What we have been doing here is we basically try to get away from this unstructured approach of surfacing information to language models and structuring knowledge at scale into query-response pairs. This then enabled us to build a large-scale atlas of roughly six million medical question-answer pairs, each pair being grounded directly from a passage in the peer-reviewed medical literature. We actually find that Myriad on its own, just as a knowledge corpus, can be swapped in, and it can improve standard basic RAG applications. We see that it also has improvements in agentic RAG, hallucination detection. It can change how we do literature analysis and train retrievers. In this example here on a more challenging benchmark, MedExpertQA, we see that compared to standard RAG, we can improve quite a bit by just changing the representation of the same knowledge. This is not changing the content of the knowledge. It's just changing the representation in this Myriad format. AndI think it's also important to augment not only language models, but also to augment humans. So we also have some demo where we show this atlas and where humans can navigate this, explore, look up the QA pairs, click on the original sources to then provide feedback. So we now learn that we can improve language models and language agents by grounding them and by improving the methods of how we ground them. But how do we really make them more reliable? For this question, I would like to pivot over to a related project that we published last year at a leading natural language processing venue as an oral presentation. Which is now instead of connecting a language model to a neatly organized corpus of information, we're actually trying to take it a step further and say, well, it's not enough to just give a language model access to a neatly organized corpus of knowledge. We actually want to check individual reasoning steps that a language model produces. So previously, we just had this setup where a language model would make any sort of statement and on its own would have to go and check in the literature. Now, we try to disentangle those two tasks and have some kind of policing language model, which we in the field would call a process reward model, that would analyze each individual statement or reasoning step that the base language model does on the left. It has the ability to check the literature and guideline, et cetera, to then come back to the language model and say, "This step is correct. This step is wrong." And just by training such a verifier or such a process reward model, we can actually improve many different language model by just hook plugging it in at inference time. And this was quite interesting because in the general, like math or code domain, we don't see this generalizability. We're also following this work up with endowing this critique or this reward model with more capabilities. And this we then call a process reward agent, where you can see on the left, language model might perform an entire tree search over different hypotheses and reasoning steps, whereas now online in real-time, this process reward agent can interfere and steer this reasoning process by checking individual steps against guidelines. So to move to the last part, I would like to quickly tease with some other exciting applications of AI agents that we have been working on very briefly. I'm sure that Charlotte will also talk about it, so I will not only highlight it here briefly. We are quite interested in actually advancing the benchmarking of agents for molecular tumor boards. We already had some first work at NeurIPS last year, and we're also very interested in advancing also agents in the multimodal radiology space, which I think also has some great applications for oncology. In this work that's very old, it just appeared on archive today if everything went well. So in this work, we built an agent that would analyze step-by-step 3D imagery. So here, chest CT. And we find some very encouraging results because this agent has the ability to faithfully represent what happened in the reasoning trace or in the entire agent loop, so to speak. Great. With that, I would like to come to my final summary slide. So I'm convinced that the medical domain is still very hard for AI, even for foundation models. So we cannot just assume we can purely scale away all the problems, just throw more data at bigger models. So we really need research. We need new methods to really increase reliability and trustworthiness of those systems. External tools can really augment language agents and make them more reliable. We can use that just as a test time. We can use it during training. There's many different ways how those external tools can be introduced to improve multimodal reasoning. I hope I could also convince you that it really matters how we represent knowledge that our agents ingest and how we really surface it to them. And I hope I also could get you excited about new avenues, how we can really carefully check individual reasoning steps such that the future of language models and language agents is not just one where we care about the final accuracy, but where we really have this defensible trace of reasoning steps that on their own really make sense and build up to a conclusion that is defensible. Great. With that, I would like to thank you so much for your attention. I'm sorry once again that I could not present in person, and I'm also very grateful for all our great collaborators, and on the left you see our very small and growing team. So unfortunately, thanks so much, Michael. Unfortunately, it's 3:00 a.m. for him. He didn't stay up to take your questions. I don't dare to answer yours. So we will close the session just very briefly. So thanks so much to both Serena and Michael, and in particular, your questions that kind of completed and well-rounded the session. We covered foundation models, how they tackle multimodality. We tackled generative models and agentic systems, and I think you very much see that in the end, it needs a combination of those, and we are very much in active research of building up the benchmarks, building up the models, building up the problems. So I hope you enjoy the rest of ACR. There will be an AI panel where you see my face again on Tuesday. And otherwise, thanks so much for attending.