Skip to content

2026-04-20-am-decoding-cancer-ecosystem-pathology-tme-heterogeneity

Date: 2026-04-20 Primary topic: single-cell-spatial-omics Cross-referenced from: bioinfo-tools, single-cell-spatial-omics Word count: ~13,534

Auto-generated captions

This transcript is Vimeo auto-generated English. Expect misheard names and technical terms. Use the ripgrep/search approach for exact quotes.

Welcome to the session of Decoding Cancer's Complex Ecosystem. My name is Hoifung Poon. I'm from Microsoft Research. Very excited to welcome you to this session. And so, as we all know, cancer, one of the really primary challenge is that it's very robust, right? And so the first generation chemotherapy tried to target mitosis, tried to kill dividing cells. Toxicity is high, and efficacy is low. Then came the genomic revolution and the targeted therapy, leading to the notion of precision medicine and precision health. But still, cancer has the blessing on natural selection, and often find out ways to compensate, to escape, to actually get roaring back, right? So leading to all the relapse and so forth. Immune system is one health system that also have natural selection on its side. So, immunotherapy is a really promising direction for taming cancer once and for all, right? So indeed, we have seen, obviously, many of you know way better than I do, right, that when immunotherapy works, oftentimes the response is much more durable. The challenge still is that right now, a lot of the immunotherapy is still a blunt instrument, right? So checkpoint just released the brake, and the overall response rate is still far too low. And the current companion like PD-L1s and TMB, they are not that good at actually stratifying patients, right? And so the question is how can we actually go much, much better and to do much, much more precise stratification of the patient? Now, tumor microenvironment is really one kind of 80/20 ways to actually start to unraveling a lot of those kind of secrets about why can the patient respond or not responding, right? And when we look at the tumor microenvironment, it's a very complex ecosystem of immune cell, tumor cells, stromal elements, and so forth. So there is lots and lots of clones and heterogeneity, and also there is a lot of temporal complexity, right? So the tumor microenvironment constantly changing in respond to immuno pressure, in respond to treatment pressure, right? So understanding this kind of tumor microenvironment grammar thereby could really start to shed light on how we actually can understand the cancer response, treatment response, and eventually figure out ways to actually make immuno-oncology much more precise. So, the goal for this session is try to overview all this complexity, right, from mechanical to temporal complexity, and then try to discuss a little bit about how artificial intelligence can potentially provide us some of the very powerful new tools, equip a human expert with some of the new capability to try to uncover this kind of tumor microenvironment grammar. So, I'm very excited to have the first speaker of this session, Dr. Ken Pienta, who lead the Cancer Ecology Center at Johns Hopkins. So he will basically introduce this eco-evolutionary lens for understanding the tumor microenvironment. Now, unfortunately, Dr. Pienta has a family emergency, so he couldn't actually come here. So he actually recorded a video, so I will play that video now. So, again, apology that we cannot do a live Q&A with Dr. Pienta here, but he is more than happy to communicate with you guys offline, right? So after Dr. Pienta give his talk introducing the tumor microenvironment ecosystem, I will talk a little bit about how AI might be able to help decode that tumor microenvironment grammar. So that will be the plan for the session. So now, let's play the video from Dr. Pienta. Hello, everyone, and thank you so much for allowing me to present this talk to you by video conference. I'm very sorry that I can't be there today with you, but I really appreciate your indulgence in letting me present to you virtually. I'm going to give you a talk today about tracing the eco-evolutionary path of tumors to understand cancer, but really what I'm trying to do is set you up for the next talk by your chairman. So here we go. I have really no relevant disclosures to this discussion except to say that I'm a professor at Johns Hopkins University, and none of the entities I consult for are relevant to this talk. I wanted to say welcome from the Cancer Ecology Center at Johns Hopkins, where we're harnessing ecology and evolution to redefine cancer risk detection and treatment. The issue we have is that 20 million people a year around the world are diagnosed with cancer. 10 million of those patients are cured with local therapy, but 10 million patients progress with metastatic and therapy-resistant disease and ultimately die of this disease.What is this idea of convergent evolution and how do these different cancers, all of which start independently in different people, converge to this lethal phenotype? Well, convergent evolution is the independent evolution of similar features in separate species and, or at different periods of time. It leads to analogous structures that have similar form or function but were not present in the last common ancestor. Good examples of this are eyes, teeth, wings and especially stress response, which is important for us here today. The convergent lethal phenotype of cancer leads to 10 million deaths per year worldwide. What is that convergent lethal phenotype? Well, number one is metastasis. Cancer spreads from the primary tumor to other parts of the body and the second part of the lethal phenotype is therapy resistance. Cancer is resistant to all known forms of systemic treatment even before it's been exposed to those treatments. So we try to look at this problem a little differently. Generally, we think of cancer as a disease of mutations, but cancer is no more a disease of cells than a traffic jam is a disease of cars. A lifetime study of the internal combustion engine would not help anyone understand our traffic problems and tumors are the same way. This was a very famous quote by David Smithers from 1962. So what is cancer? Well, cancer is a disease of uncontrolled proliferation by transformed cells subject to evolution by natural selection. We think about this because nothing in biology makes sense except in light of evolution. But nothing in evolution makes sense except in light of ecology. Evolution is change in heritable variation over generations, but ecology are the external forces that influence adaptation. Evolution acts upon phenotype, not genotype. The evolution that we observe in each patient is a result of cellular adaptation to those changing ecological forces, i.e. the tumor microenvironment and the patient macro environment, that cells that adopt a successful phenotype outcompete those with a less optimal phenotype. So our challenge is to integrate evolutionary principles to improve patient care. Remember the ecology. What selective forces are at play in a patient? How do they change over time and space? What adaptive cellular phenotypes are selected for? With intervention, how do we change the ecology of the adaptive landscape of the tumor? And don't forget, treatment kills more than just cancer cells. So what is an ecosystem? An ecosystem as defined by Eugene Odum in the '30s is that any unit that includes all the organisms, i.e. the community in a given area interacting with the physical environment so that a flow of energy leads to defined trophic structure, biodiversity and material cycles. Ecosystems do not exist independently, but interact in a complex web of ecological relationships connecting all ecosystems to make up the biosphere. Way back in 2008 now, we first defined tumors as ecosystems, that cancer cells as a species coexist in a complex habitat with other host cells. These cells interacting with their habitat create an ecosystem, and this ecosystem in turn exists within a larger environment. The host patient is the tumor's biosphere. And we created these tables that everybody's now familiar with looking at how we can think about man in a ecosystem like living on the land with cancer cell, living on the extracellular matrix, interacting with all organisms as a man, all cells with cancer. Our community is, for us, an ecosystem, whether we live in a city or a farm. The cancer cell community is the microenvironment, creating meta communities, metastases and living within the host biosphere or the patient. Cancer has several strategies to evolve traits in response to stress that it's constantly feeling in response to how the tumor microenvironment is changing. And these strategies we're going to talk more about, but they include tumor cell heterogeneity, proliferative quiescence, evolutionary triage, and self genetic modification. But the bottom line is that cancer creates a new ecosystem that now we refer to as the tumor microenvironment. We need to understand how, we need to understand how to disrupt this process because remember, it's easier to drain the swamp than swat a billion mosquitoes one at a time. And cancer cells are actually ecologic engineers, and there's two kinds of ecological engineers that result in niche construction. Allogeneic engineers modify environments by mechanically changing materials from one form to another. Think about beavers that construct dams. Autogenic engineers modify environments by changing themselves. Oak trees, as they grow, provide more and more living space for various birds and small mammals.Cancer is both an allogenic and autogenic engineer. Think about prostate cancer modifying the bone matrix, and the growing cancer alters the environment. Cancer creates its own beaver pond, attracting new species like tumor-associated macrophages and cancer-associated fibroblasts. Does this engineering affect genetic evolution, causing an increased rate of clonal selection? Well, if you look at ecological inheritance in increasing the rate of clonal selection, a feedback relationship between natural selection and niche construction, when organisms affect their environment, that change can cause a shift in what traits are being naturally selected for. The effect of niche construction is especially pronounced in situations where environmental alterations persist for several generations, introducing the evolutionary role of ecological inheritance. One example of this on Earth are yam cultivators in West Africa. They cut clearings in forests to grow crops, but that resulted in much standing water, which attracted mosquitoes and increased the rate of malaria. This caused a modification to the selection pressure for the sickle cell allele that protects against malaria. So does the cancer swamp increase the rate of clonal selection? Do we see that the rate of intrinsic genetic instability changes due to ecological inheritance? So let me run you through a few examples of how the cancer swamp increases or accelerates clonal selection. Gatenby and Gillies pointed out what's called the breeder's equation applied to somatic evolution, where they showed that harsh TME conditions like hypoxia, acidosis, and a heterogeneous extracellular matrix creates steep fitness gradients. Alvarez and colleagues pointed out that if you looked at in vitro versus in vivo, clonal selection through barcoding, that there were about 17,000 stable barcodes in vitro, but if you find those in tumors, there was a 73% reduction in clonal complexity versus the starting population and the 300 times expansion of the top clone. There have been, again, four converging mechanisms that have been documented. The steep fitness gradients, for example, through the hypoxia, immune-mediated pruning, acidosis-driven mutagenesis, and positive feedback loops because selected clones remodel the tumor microenvironment to entrench their advantage. So environmental stress can drive monoclonal dominance. You see this in even precancerous lesions, as shown by Zhang et al., where they demonstrated a polyclonal lesion with multiple completing clones, selection pressure through the tumor microenvironment leading to monoclonal dominance. So in summary, the swamp does drive selection, and there have been multiple examples of this across time. And this is why it's so important to think about how we can model this, as you'll see in the next talk. The tumor microenvironment is not merely a bystander. It is a primary engine of cancer evolution. So I want to switch gears a little bit and bring this home with thinking about why is lethal cancer lethal? Why do we die from cancer? This is a patient that died from prostate cancer, had all these lesions in their liver, but they had normal liver function. People don't die in general of organ dysfunction because the cancer metastases grow and overtake that organ. The tumor microenvironment kills us. At death, we estimate that a kilogram of tumor is lethal for a patient. But that kilogram of tumor is spread out across the patient in metastases. And that kilogram represents one to two trillion cancer cells of the total of 50 trillion cells in the human body. It's not about cancer cell mass. We need to understand the context within cancer occurring. That context is the human patient host. And so I want to remind you that cancer has no negative feedback loop. As long as it has enough energy to grow, it doesn't turn off like a normal cell does. So without a negative feedback loop, the healthy pond devolves into a swamp. Why do we die from cancer? Cancer poisons the biosphere of the patient, through cachexia syndrome, through thrombotic syndromes, through skeletal pain, through respiratory failure, and the common theme here is all of these are a result of cytokine overproduction. And we call this cytokine overproduction the miasma of cancer. MiasmaWhat a great word. It's a highly unpleasant or unhealthy smell of vapor. And so when you think about swamp gas, we all have heard the stories of fires in the swamp lighting because of flammable ecologic swamp gas. But it's really methane from dying algae and other fermentation. But lethal cancer swamp gas is a result of many cytokines leading to these different syndromes. So again, we actually published this almost 20 years ago now, looking at the different cytokines that contribute to what we call cancer swamp gas. IL-6, produced by macrophages, T cells, muscle cells, osteoblasts, adipocytes, and cancer cells. TNF alpha, otherwise known as cachexin. Look at all these cells that make TNF alpha. Cancer cells barely make this list. It's the cells of the tumor microenvironment. Matrix metalloproteinases made by macrophages and cancer cells. And all these cytokines lead to things like the cachexia syndrome, where the cells are released from the tumor microenvironment, causing lipolysis of fat and muscle wasting, et cetera. So what I want to leave you with here is this idea that the cancer swamp, i.e. the tumor microenvironment, poisons the patient. That's why we need to understand and be able to intervene in this better. So our challenge is integrating evolutionary ecology principles to improve patient care. In the clinic, if we consider the ecology with treatment type and timing, maybe we can exploit it to attain tumor control and perhaps eradication. At the bench, let's move beyond mutation and epigenetic silencing and think about what adaptive phenotypes are targetable and manipulatable. We published a recent paper on this last year, which I urge you to take a look at, where we summarize this, "Defeating lethal cancer: Interrupting the ecological and evolutionary base of death from malignancy" in CA, written with my colleague Sarah Amend, as well as a patient, Pat Gooden. So there are many ecological evolutionary paths to evolutionary rescue following therapy, meaning resistance recurrence. Selection acts on heritable phenotype, not on genotype. What are the adaptive strategies at play, and how do you shift the selective pressure? So there's phenotypic heterogeneity. Among the heterogeneous population, there already exists a clone that has a trait to confer resistance, but it has to be picked. It has to have a pressure that allows it to flourish, like chemotherapy. And if a cell happens to have a drug efflux pump, it's going to be the one selected for. Evolutionary triage. Cells with better, i.e. a more adaptive phenotypic traits, are selected over those that are less fit, though not necessarily absent adaptive trait. Darwin's finches, for example, are a great example of this. In a tumor treated with anti-androgens, over time, cancer cells upregulate the number of ARs are more adaptive. Epigenetic plasticity. The adaptive trait that is selected for is the cell's capacity to alter its phenotype in response to its environment. Good example, cancer cells that downregulate presenting MHCs. And of course, dormancy. The adaptive trait that is selected for is dormancy. Cancer cells are only susceptible to anti-cancer therapy if they are undergoing cell division. So here's an example of a non-proliferative dormant cancer cell sitting in the bone of a patient. So these eco-evolutionary strategies can be potentially adapted to treat cancer. And we, and along with many others, are thinking about how to do this in the clinic right now and will continue to develop these, especially if we can adopt better models of the tumor microenvironment. Cancer ecology is necessary to understand lethal cancer all the way from the selection from a precancerous lesion to the metastatic therapy-resistant phenotype. So thank you for listening from the Cancer Ecology Center group, and I wish I was there to answer questions in person. Thank you very much. So once again, apologies that Dr. Pienta cannot come here due to a family emergency, but he will be really keen to interact with you offline. So next, I would build on top of what Dr. Pienta talked about, the complexity in the tumor microenvironment and the cancer progression and resistance, and then talk a little bit about how the confluence of digital transformation and AI revolution could potentially provide us with new tools, new capability to do this. So before I dive into the detailI would just give you a little bit of my background to set up the context for myself and this talk. So I actually come from a straight computer science background. Started my PhD in machine learning NLP two decades ago, back when AI was actually in the last ice age. So never imagined AI could be so hot one day. And then after PhD, I joined Microsoft Research, which coincided with the 10-year anniversary of Human Genome Project. At that time, I didn't know how hard biomed and health is, so got really excited and really decided to start looking into whether AI can help precision health. I was hired as an AI researcher, but told my boss I want to work on cancer. Miraculously, he didn't fire me on the spot. So here are basically a little bit of a decade and a half of the battle scar in learning. Actually, often thanks to a lot of our collaborators, some of them sitting in the audience, and actually teaching me and educating me. So really looking forward to picking all of you guys' brain. So I have a few speaking and advisory gig. And so as we all know, right, an immunotherapy is really a very promising direction by mobilizing the immune system, which are naturally equipped with natural selection to really counter the sub-cancer robustness. But right now, even the best-in-class of immunotherapy are still far too low response rate, right? And so the question is that what can AI do? So here I would basically summarize metaphorically, on the war on cancer, AI potentially can equip us with three new capabilities. So the first one is a microscope. A new microscope. So when you think about one of the really exciting prospect of studying those tumor microenvironments and so forth, is that there is lots and lots of new modality of data coming in, right? So starting from multi-omics and also nowadays with a lot of high-content imaging like digital pathology, spatial omics and so forth. But the pattern hidden inside those data is very, very complex. And so here, AI can potentially empower human expert to see much better, right? Another new capability is really a crystal ball, right? So Dr. Pienta talk about how we need to think about how cancer is essentially in a temporal longitudinal journey, right? So just containing, putting cancer in remission is far from being enough, right? And so how can we actually understand the whole evolution trajectory? So here, AI can potentially allow us to start to be able to model extremely high dimension, extremely long timescale dynamics. And finally, AI can also give us a wind tunnel. Right? So by start to allow us to start simulating a lot of those kind of experiment and hypothesis generation and testing, right? Because one of the fundamental limitation in biomedicine is that making every single step forward is very expensive, very time-consuming. So the more we can actually build up the model for these processes and be able to actually do a lot of groundwork in the computation, the more potential we can speed up this progress. So, in a nutshell, if you think about the entire technical program that we have been looking into using AI to advance the precision health, right, is you can sum it up as learning the language of a patient. So, for example, thinking about a patient journey. Starting from, you typically have a non-invasive radiology modality like CT, MIs and so forth. Then with biopsy resection, you start to get digital pathology, multi-omics, and so on. Each modality is trying to tell you something about the patient, but if we don't understand the language, then they just sound like gobbledygook, right? On the other hand, if we can start making some progress to understand mastering those languages, then we can start to construct this virtual patient, a world model that as a digital twin, to start being able to forecast disease progression and treatment response. So obviously, the trillion-dollar question is, can AI deliver? So there has been obviously lots of hype, lots of skepticism, right? And I personally find it quite useful to think about gen AI application as spanning a spectrum from what I will loosely call productivity gain, all the way to creativity gain, right? And basically from automating chores, mundane chores on the left-hand side, to all the way automating discovery. And many of you have heard about the Amara's law, right? It basically observed that oftentimes when technology disruption happen, at the beginning, we often overestimate the short-term progress, and that's actually often lead to the hypes, right? So, for example, we are still very far away from actually attaining the moonshot on the right-hand side. Even though we are already getting some pretty promising direction. And on the other hand, we can also underestimate the long-term progress, even though major transformation already happened. And right now in biomedicine, the major transformation already happening on the left-hand side, on the productivity gain. So in this talk, I will essentially start by focusing on this overarching frontier area, which is AI can enable medical scientists to begin to simulate clinical trials and towards building a virtual patient, right? And we will start with the productivity gain to show you some of the extremely low-hanging fruit that arguably already ready today or even yesterday. And then go all the way to the right-hand side, the moonshot, where we are far from actually conquering the moonshot, but there are several promising directions.And so many of you know better than I do, right? So medical discovery basically advances one clinical trial at a time. This process is extremely manual, very inefficient. A single phase III trial can take years to execute, often costs more than $100 million. And what we actually do is to actually collaborate with Providence, some of the folks sitting in the front row, to build this AI called TrialScope that can enable medical scientists, like the Providence researchers, to utilize the real-world patient journey, and then simulate clinical trial virtually. So we demonstrate that this kind of virtual trial can be just as accurate as a real trial by comparing to the gold standard trial for marquee cancer treatments. For example, the KEYNOTE-10. So many of you know that was one of the original trials for Keytruda, one of the best in cancer immunotherapy. Now, obviously, by doing this kind of simulation, we can do it much cheaper and faster. Now, of course, I'm not advocating actually that RCT will be going away anytime soon. But what this actually demonstrates to you is that the real-world patient journey has lots of valuable information, and also AI now finally gets to a stage to begin to glean that information now. And one of the prerequisite conditions, obviously, is the rapid digitization in biomedicine. So starting from genomic sequencing, then arrived the electronic medical record. So every day, there are literally billions and billions of data points documenting a patient journey from diagnosis to treatment to outcome, exactly the kind of journey that you study in clinical trial. Right now, these data are basically primarily used for billing and operation, and then get locked up in the attics. But what if we can actually take those kind of data readily available, and then using AI to unlock all the stories that all those patient journeys are trying to tell us? So this could become effectively a population scale free lunch. So one of the challenges, obviously, is that a lot of these data are highly unstructured. So for example, cancer patients may have hundreds of clinical notes. But AI have already advanced to such a state that they can enable scientists to structure this information with high accuracy and at scale. So every year there are 20 million new cancer patients. Traditionally, you will hire abstractor, registrar, curators, and so forth. And each cancer patient could take hours and cost you hundreds of dollars to just pull out that basic information from those giant haystacks of clinical notes. Now, with frontier AI-powered universal abstraction, you can do that in seconds and cost you a few cents. So what that means is that at Providence, for example, with our collaborators, we basically have this every night, when the data get back up from the live Epic system to the data lake. The TrialScope AI agent wakes up at 1:00 a.m., so basically processes all the incoming notes and extracts all these fine-grain medical events. And then you can basically tally precisely the window of opportunity when basically a patient satisfies all the eligibility criteria and right before the first disqualifying event happens. So that window of opportunity for oncology, for example, obviously often is very short. But now actually the technology is already here to be able to do this kind of twenty-four-seven monitoring. And what this means is that not just empowering clinical trial matching one step, but actually you can start to rethink the entire clinical trial process, starting from feasibility, site selections, and screenings, and so forth. You can also use this to, A, assist your clinical trial design. For example, given a particular criteria, how many patients might be eligible? What if you change a particular criteria? How many patients? How would that change the number of eligibility? Right now, a lot of this are just wild guess. But now you can pretty much tell precisely at every single moment in the history how many patients might fit your criteria. And you can change your criteria, and then a second later, get all the statistics. And then you can also start to look into external control, paramedic trials, and electronic data capture, and so forth. I also want to give a shout-out to Jimeng and Kenji AI. They do a lot of great work in this space. Personally, I'm also very excited about the prospect for frontier AI to supercharge health equity. So for example, even in the US, 85% of cancer patients are actually treated in community center, rural hospital. They often don't have the resource to do these kind of matchings and so forth, and they largely preschool from the clinical trial. But now at least the technology is already ready to level the playing field to actually democratizing this high-quality healthcare to everyone. Now, obviously, the technology is just one small bit. So it makes what impossible yesterday to become possible tomorrow. But we still have the hard work layer ahead of us, to rethink the workflow, ecosystem, business model, change of management, and so forth. So far, I've been telling you kind of super low-hanging example of how can we actually harness frontier AI in the text modality to basically derive massive productivity gain. Now, this is in the grand scheme of things, on the other hand, is just an appetizer.So all those clinical texts. Actually, many of you know better than I do. They are just highly imperfect human translation of patient ground truth. So over the years, I started make a lot of friends in medical folks, and they start sharing inside joke with me. So my radiology friend will tell me that sometime they will actually see different thing on the same image two weeks later. And my pathology friends tell me you can turn pathology slide 90 degrees, sometime you see different things. So these are some of the smartest people in the world. But the challenge is that our visual cortex didn't evolve in the jungle reading MRIs. But now come the prospect of AI to potentially arm them to be able to decipher all those really subtle kind of patterns. And here is also a good place to insert some of the sort of clarification about Frontier AI's capability. So which often lead to some confusion. So first of all, AI is no magic. So it really sort of like data algorithm and compute. So the Frontier AI already get to very good at the text, including biomedical and clinical text, as we show earlier. Partially the reason is because there is a gigantic amount of biomedical text in the public web. Starting from 40 million PubMed paper, you also have all the WebMD, all the podcasts or whatever. So that's why Frontier AI are getting very good in the text modality. But once you go beyond the text modality. Once you go to all those omics and imagings and so forth, there isn't that much public data for multi-modal, longitudinal patient journey for good reasons. And that's actually when Frontier AI can be blind and deaf. And that's actually exactly the exciting growth area for us to bridging those competency gap. Another point I want to also make is that no matter how high value a single modality is, it's often insufficient. So only by piece together all those modality can we recapitulate sort of a holistic patient journey. So this lead us to basically start thinking about precision health in probably a little bit nerdy way from a machine learning point of view. So we can think about what we try to do here, is trying to learn a function that's starting from a multi-modal input for the patient, and we try to predict two really important medical events, disease progression and treatment response. Now, obviously, the challenge is that a patient journey is not a snapshot, but actually a longitudinal time series. Most frustratingly is that most of the information we hope to see actually are missing. And even the observable one, they are highly biased and highly noisy. But this is arguably why GenAI is particularly promising for precision health. So the Gen in the GenAI signify generative models. So they basically describe the probabilistic approach to try to model the joint probability distribution of all the relevant clinical variables. Now, it sounds like we sign up for a much harder job. But if we can actually make even some modest degree of progress, that means that we now have a principal way to handle all those noise, uncertainty, biases, and so forth. And it give us a principle too, to compress all those observable information into this compact patient representation and patient embedding. And also, we can now infer those missing information using standard probabilistic inference. That's exactly what probabilistic model do. And then predicting the future event, it's just a special case of inferring this missing information. So this motivate us to start thinking about can we actually really, using all those real world population scale, real world data, try to pre-train frontier biomedical models. Now the GenAI disruption means that for the first time ever, we can start to imagine building this really high fidelity world model as a digital twin to model the patient journey. Now, on the test time, if we have such a high fidelity virtual patient, what it can do is that it can enable us to do patient-like reasoning at the population scale. So when somebody have a cancer diagnosis these days, if they have resources, the first thing they are going to do is to get a second opinion. But if you have this virtual patient, you can snap your finger and immediately get a million second opinion. You can interrogate all those most similar patient. You can look at what have been tried, what kind of intervention have been tried on them, what are the longitudinal outcome. And you can immediately improve the patient care. And then on the flip side, you can compare the non-responder versus the exceptional responder and start to uncover, for example, why those 80% of patients don't respond to immunotherapy. So even the best doctor cannot internalize a billion patients' journey, but we can now potentially have AI to empower them to do exactly that. So this is a very exciting prospect, but obviously the forward path is anything but easy. So one of the fundamental challenges, as many of you know, in AI, one of the really powerful tool in machine learning is called supervised learning. And one of the major bottleneck there is that you have to carefully curate all this kind of example input-output pairs. So again, AI is no magic. You need to teach the AI to do stuff, and you need to show them how to do those thing. And now this kind of so-called supervised training data become your major bottleneck. So you can think about our entire 15 years of research program is trying to figure out whether we can leverage some data that already readily available and then try to make it as little as possible to curate those super precious training data. So I basically liken that to having a free lunch. So in the rest of my talk, I will basically give you four recipe for free lunch.So the first free lunch actually come from all those unannotated data point themselves. So any moment you get unannotated images, protein sequence, and what have you, that's precious, actually, self-supervised training instance. So you can actually basically turn it around to basically teach the AI to do stuff. So one very powerful general scheme is basically this sort of encoder-decoder framework. So you can think about really a majority of machine learning and AI is just a glorified dimension reduction. Because once you can actually, for example, reduce let's say a billion dimension digital pathology slide to a 500 dimension vector, then actually learning how to make sense of those pathology slide will become much, much easier. But the challenge is how do we do this dimension reduction, right? And familiar PCA, they only can get rid of the linear dependency. And now the promise with deep learning is that actually you can get rid of arbitrary-- You can model and get rid of arbitrary non-linear dependencies. And so again, back to the encoder-decoder framework. So you basically introduce a pair, modality-specific pair of encoder and decoder. The encoder do the dimension reduction. But you also want to make sure that you don't encode arbitrary crap. So that's what the decoder come in, forcing it to be able to try to reconstruct the input as accurately as possible. So now you can ensure that the representation you learn actually capture as much of the original structure and insight as much as possible. So on the text modality is very, very special. So somebody already poured billions and billions of dollar. So I highly recommend reframing from tampering with it until you know what you're doing. But the good news is that you can start from all those very powerful frontier model on the text modality. So we can separate the two regime. So this is a very powerful paradigm, but in a specific modality in biomedicine, there are also a lot of idiosyncratic challenges. So go back to solve the tumor microenvironment. So, obviously one of the holy grail would be that can we figure out, uncover this tumor microenvironment grammar between tumor cell, immune cells, stromal elements, and so forth, and then really unraveling all the secrets there. So in theory, we actually have exactly the right tool from AI to do this. So many of you probably familiar with Transformer, which is a workhorse of modern gen AI. And for images, so Transformer actually, the way that it works is actually really simple. So you're given a image. Basically what Transformer do is to try to pass messages between every pair of the pixel. And then you heard about all those deep learning, many layers and so forth. They basically facilitate message passing for many rounds. So after you do that, every pixel actually know everything about every other pixel. And this is exactly why Transformer can be so flexible, so powerful in modeling arbitrary kind of interdependencies. But this superpower actually come with one big Achilles heel. In order to do the pairwise message passing, that means that your computation will scale quadratically in the input size. And unfortunately, digital pathology is some of the largest image in the world. So a single standard pathology slide, if you do a standard Transformer, it will cost you a billion times of compute compared to your favorite cats and dog web images. So here, basically what we try to do is to steal one idea that originate from speech recognition, which also have to deal with this kind of long context, and then adapt that to medical imaging space. So the idea is this is called dilated attention. The idea is actually very simple. So think about the pairwise message passing. If you have a local neighborhood, very small input, it doesn't actually cause you too much trouble. You can definitely do that. So after you do that, then every pixel actually know the neighbor very well already. Now the quadratic start to kill you when you try to pass message between your local neighborhood and some neighborhood very far away. But if every pixel already know the local neighbor very well, that means that you can just elect representative for every neighborhood, and then focusing message passing among the representative. So this basically give you a principle way that you can actually turn the knob to essentially dramatically cut down the computation, but still try to maximally preserve the fidelity of the computation. So we actually adapt this to digital pathology. We have some really, really wonderful collaborator from Providence and University of Washington. I also want to highlight Bernie, Ram, and Brian sitting in the front row. So we are very excited to basically develop this Giga-Path model that is the first pathology foundation model that can truly scale Transformer to the gigapixel whole slide images. So one of the things that we actually set out to do is that we try to throw all kinds of hard problem to it. And then in particular, we try to look at can we train sort of a pan-cancer signature across 41 cancer types. And to actually correlate H&E images with some of the biomarkers. So, initially it sound a little bit crazy because those 41 cancer type have very, very different morphology. But the good news about AI is it never complain. So you can throw anything to them. And indeed, we find some pretty interesting signals. And then, my friend, Gabrielle, actually sitting there, he has been leading this very amazing project at Mount Sinai and Memorial Sloan Kettering that actually demonstrate in a global prospective trial that you can actually take this idea and turn it into kind of real world deployment. So in particular, what they observe is that even in the US, despite strong guideline-A quarter of the lung cancer patients never get EGFR mutation status sequence before the first line, largely due to resource constraint. So that means that their treatment likely is suboptimal. And what they actually show is that they can actually just fine-tune the gigabyte model and then predict those EGFR mutation status just the moment you get that tissue sample and take the microscopic image, you can make that prediction, and amazingly with very high accuracy. So very excited to see this kind of real-world application. And this is one of my favorite graphs to share. Actually, I can look at it all day because a lot of folks that tell us their favorite applications, so really encouraging. So far we talk about the first free lunch. Now imagining tomorrow you wake up, you already train, everybody already train amazing foundation model for each modality. But our job is far from being done. Because there is no guarantee that those semantic space uncovered by each individual modality will actually align with each other. So in this sense, each of those modalities is like a distinct language. So the CT could be a French, the digital pathology could be a Hindu. And very soon we will have a United Nation problem here. So in actual united nation, in actual translation, what people do is that they actually introduce this resource-rich language, typically English, and using it to serve as this so-called interlingual. Basically as a bridge across all those hundreds of languages. So for example, you may not have any parallel data between two language A and B. So thereby in conventional supervised machine learning paradigm, you cannot do any translation. But with the interlingual as a mediator, you can potentially translate language from A to English and then from English to B. So this is actually exactly how commercial translation engines scale to hundreds of language in the world. So what we are proposing is to let's steal that playbook and apply that to multimodal reasoning in biomedicine. And obviously the very obvious interlingual modality is text. We already have very powerful frontier model. And for the text modality. Moreover, for any modality under the sun, the study of that modality, often you leave some paper trail. So for example, when you have radiology images, you often have a radiology report. For pathology slide, you sometimes have pathology report. So those are become your free lunch. So we can reusing the encoder and decoder portraying the first stage and then focus all our computation trying to train this very lightweight adapter or translator, to ground the modality from individual modality to the text semantic space. You can do this modularly for individual modality. And after you do that, number one, all those semantic space will now suddenly get much closer to each other. And also the second implication is that those frontier model, keep in mind, they read every single published paper and every single comments in the web. So they have incredible capture of knowledge, the current consensus knowledge, and also incredible reasoning capability. So now by actually aligning individual modality to the text semantic space, you basically get for free using the frontier model as a unified knowledge representation and reasoner to reason across all those modality to actually interpret individual modality. In the interest of time, I wouldn't go into too much detail. If you are interested, you can check out some of our early work in LLaVA-Med that basically provide the first general recipe for how to do this in biomedicine. Since then, we have extended, we have demonstrated in multiple domains like radiology. Other folks have applied it to digital pathologies and so forth. And, more recently, we applied this paradigm to actually train this model called BioMedParse that basically is the dream for a lazy person like me. So to basically just talk to the image, and then have it to do the holistic image analysis. So BioMedParse can basically capture nine major image modality, 64 major object types. Originally, we actually use a title, "Segmenting Everything Everywhere All at Once." So after the Oscar movie, unfortunately the Nature Method editor think this is too sensational, so we have to go back to a much more boring title. But the editor still feel quite excited about the paper, so she also published an independent review paired with the publication of the paper and say some very encouraging words. And as usual, the most exciting prospect is to see actually people try to use it and find some of it useful. So far we talk about the first two free lunch. These are based on modality that relatively abundant and frequent. Unfortunately in biomedicine, or maybe fortunately, is that we actually have a long tail of emerging super high value modality. And that basically expand our potential scope for uncovering the secret in biomedicine. But for those long tail modality, oftentimes we don't get that many data point, let alone any text accompanying them. So for example, spatial omics. In particular, something like multiplex immunofluorescence. They basically can give you simultaneously map out multiple protein activation status in individual cells. So they basically give you a very detailed digital map about the tumor microenvironment grammar.Now, there is only one catch, right? So they are very costly and expensive, and time-consuming to generate. So generating the data, for even just a single tissue for even the top labs like our collaborator at Providence can take days and cost thousands of dollars. And now you can do the math and imagine what if we want to generate the data for every single person in the globe, right? That will bankrupt the humanity. So in collaboration, again, with our Providence collaborators and University of Washington, we developed this multi-model AI called GigaTime. So basically what they try to do is to start from a Rosetta Stone of 40 million cells with pair H&E and also the spatial proteomics, the MIF. And then we try to learn a translator to go from the dirt cheap pathology slide to actually the very hard-to-get spatial proteomics. So our Providence collaborator generate this super precious, the first batch of this kind of Rosetta Stone. And then we basically try to teach the AI to learn from the correlation there. So here in this example, you can see in the middle column, that's the ground true MIF. And then on the right-hand side, those are the kind of AI-simulated MIF. Now obviously, again, AI is no magic. We will not get to perfect anytime soon, and is highly varied by the protein channel. But as you can see, amazingly, just from such a relatively small training set, and essentially just the first attempt, we were able to already reconstitute a ballpark of the information in the spatial site. So some of you might ask, how can this be possible, right? Again, AI is no magic. So there is no magic behind. What is going on? So how can you start from essentially H&E basically give you nucleus and cytoplasm, basically the morphology, how can you actually get to the cell state? So it turns out that actually this is related to an idea that almost date back to the dawn of AI and linguistic. So, I was originally training as a AI NLP person. So this is something that basically, at the time, every NLP people talk about, is so-called distributional hypothesis. So imagining tomorrow you receive in your email a big chunk of Martian text, right? So you don't understand every single word in it. But what you observe is that you have these two word, A and B. And they often appear in the context nearby words that are actually the same. So the distribution hypothesis then basically hypothesize that those two word probably have related meaning. So sometime they could be synonyms, sometime they could be acronym. So in fact, a big part of our early days, NLP, was actually using this idea as inductive biases to basically cluster words and so forth. What's actually really exciting, and that's by the way, also lead to the so-called word embedding. So otherwise, why would it make sense to predict next word from the previous word? This also lead to ultimately the language model, the LM, and all the gen AI first. But fundamentally, it originate from this idea. And turns out that what's really amazing here is that this idea seem to also be applicable to cell biology. In particular, when you think about a T cell, whether it's active or exhausted, just by morphology alone, it's very hard to tell. But active T cell often appear obviously in supportive environment, and exhausted T cell often appear in highly regulated environment. So thereby, even if you can just barely make out the core cell type in the surrounding environment, it can already give you some hints about whether the cell, what kind of state it's in. Now, obviously, just like distribution hypothesis is not far from perfect in the text front. It's far from perfect in the cell biology front, but still, that can already give you some signal. And, so that's kind of like what's the secret behind the scene. So with our Providence collaborator, we apply GigaTime to 14,000 cancer patient, essentially as the first populations experiment. And then we basically run GigaTime on all the available H&E pathology slide and generate over 300,000 virtual MIF size. Now, that basically give us essentially sort of like a way to conduct really the largest spatial proteomics study that were previously completely unthinkable. So we can basically looking at all those kind of cell states in the tumor microenvironment and correlate with the longitudinal outcome you care about, the progression, tumor response, and so forth. So that really solve the-- I would say, GigaTime obviously is exciting first step, but it's just a baby, tiny, tiny baby step, and there was a gazillion way we can improve. So finally, in case you are not completely full yet, so let me present you the final, but arguably the most important free lunch, which is the patient journey themself. So many of you are probably familiar with GPT-4. It basically learn by predict next word, next word, next word. Now it turns out that we can actually try to do the same thing for virtual patient. What if we can serialize the patient journey and try to predict next event, next event, next event? Now just like in the language model form, if you do this in a small scale, nothing much happen. But what if you can do this at eight billion patients' journey? Then there was this whole notion about emerging capability in the larger language model world. Basically saying that if you have large capacity in that model, you pre-train on large enough data, they can actually uncover a lot of sort of insight and intelligence that you-- It's sort of like undergone a phase transition that suddenly those capabilities start to seem to be emerging and become very, very high competency.So as a tiny baby step, we team up with Epic and Yale pre-train this foundation model for virtual patient called Curiosity. And this is built on top of the amazing work that Epic folks and the Cosmos community have done with 300 million patient journey, right? So essentially, we train the largest virtual patient model with one billion parameter for the model, pre-training on over 100 billion medical events, very fine-grained medical events, right? Fine-grained diagnostic, medication, procedural, lab, and values. It took over 40,000 GPU hours. Now, of course this kind of large-scale pre-training require lots of resources, right? So we can't go to the senior leadership on both sides to say, "Hey, give us lots of money and do a lot of trial and error." So one of the important exercises that we have to do is to conduct this so-called scaling law study. Basically, what it gives you is two things. You basically establish a best practice, right? So to actually search among all those... A lot of black art in gen AI is to figure out the hyperparameters, right? So there was a laboring of a gazillion hyperparameters, and actually doing those, tuning them actually accurately would make a huge difference. And this kind of scaling law basically is using small pre-training run, right, on small data, smaller model, to basically figure out the best practice to do that. Another thing is to using that to establish empirical laws to actually extrapolate, like what if we can scale this up to next order of magnitude, 10x larger, 100x larger? Can we actually get exponential ROI, or do I get diminishing return, right? So, again, in the language model world, right? That's exactly how the early GPT model were using this kind of study to justify why we should spend a billion dollars to train the next generation. And here we obviously do a much smaller scale for structure EHRs. One of the amazing thing for this kind of solve generative model is that it can basically conduct a very generic pre-training. And it try to capture as much as the correlation structure as possible in the patient journey, right? And then out of the box, it can actually perform lots and lots of different things that are not specifically designed for, right? So, for example, Epic folks have actually painstakingly curated 78 tasks for evaluation. They also trained supervised method for each individual task. Now, what we did is that we take the Curiosity model out of the box, just do this brain death simulation for the future, right? You can simulate as long as you want. You can simulate as many parallel universe as you want to start quantifying the uncertainty. And amazingly, out-of-the-box, this model, a single model, actually match or outperform those supervised method in 78 real-world clinical tasks. So again, there are gazillion of way to improve, right? But this is really a very promising first step. And Epic actually is getting very excited, and the whole Cosmos community start rolling out this model to the Cosmos community. So I want to conclude by adding one major caveat, right? So, this is one of the key reason why RCT is not going away anytime soon, right? So, the real-world patient journey, they are free lunch, right? But there is a reason why they are not Michelin three-star restaurant yet, right? And one of the big problem is that these are observational data, right? So it's subject to confoundings. Now, obviously, this is also an area where actually AI can potentially also provide you with a potentially much more powerful way to model those underlying distribution, to model those potential unknown confounders, right? To basically go way beyond, for example, some of the standard tool, like propensity scoring and so forth. So, basically, hopefully, I give you sort of three things, right? Microscope, crystal ball, and wind tunnel. That's some of the area that AI could potentially contribute. So I'm really looking forward to picking of your brains. And I want to highlight my team at Microsoft Research and they do all the real work. And also all our amazing collaborator without whom we can't obviously do any of this. So thanks very much, and we should have some time for questions. For anyone with question, feel free to come to the mic. Thank you for a fantastic talk. This is Mohammed Gadamosi from the University of Florida, College of Pharmacy. I'm quite interested in using generative AI to predict the next event, but maybe on a smaller scale than the patient. I think there's a huge interest in the evolution of a tumor, maybe in response to a particular therapeutic. And there's a lot of methods that are currently out there. Some people have used agent-based modeling to achieve this goal. I haven't seen a lot in the realm of generative AI. Could you maybe comment on the ability of generative AI to achieve that outcome and maybe compare it to standard methods that are utilized now, like agent-based modeling? Mm-hmm. What's the trade-off there? I think there may be some challenges with interpretability- Yep ... with generative AI that agent-based modeling may not face. Yeah, absolutely. That's a wonderful question. You will notice I barely mentioned agent, even though that's kind of like today's buzzword, just like big data once upon a time is, right?Agent is a very, very powerful paradigm, but I would say agent is kind of orthogonal to a lot of this kind of modeling. So agent, agentic AI, actually, you can think about the trial scope that I mentioned in the first example, right? That's basically one embodiment of what agentic AI can do. So essentially, you have lots and lots of tasks. Right now, you have to manually do them, right? What agentic AI actually is to decompose those tasks and be able to carry them out. Now, this is nothing new. So if you look up agentic AI or distributed computing and so forth, it dates back decades. What's new today is that this frontier model become very powerful in terms of being able to understand natural language instruction. So that's the first very important part. In the old days, you have to actually specify in domain-specific language to specify, "Here's what I want to do. Here's how you should plan, here's how you should execute," and so forth. But now you can actually just say it in natural language. Another thing that some of the frontier AI unlock is that in capturing a lot of those knowledge. So for example, actually, AI can go through all those 30 million paper, identify all the possible experimental setting, what are all the procedures and so forth, which traditionally are very hard to do manually, right? So all those are basically the way. And then agentic, obviously the ultimate goal is can we do this kind of science in the loop, right? So sort of really facilitate, okay, given the... Let's come up with hypotheses, let's actually then execute it. Let's look at the result. Let's go back to revise our hypothesis and so forth. So again, agentic AI, I would say, is orthogonal. It's basically taking any sort of digital knowledge work, and now you can have a very intelligent, very powerful, completely untiring assistant to do... You can control a million assistant to help you do it, right? So, that's the agentic part. But the modeling part is still sort of challenging. How do you actually model the temporal kind of dependency, the temporal dynamics? That's not like you just out your agentic AI, it will solve the problem. Because a lot of those modalities and so forth are not actually available, again, in the public web, right? And your data point may be very few. And so the standards of the image encoding or any of those encoding may not be very good for this kind of data, right? So I would say, as you can see in sort of a lot of this kind of paradigm, what we try to do is to find places where you have abundant data, and then using that-- Again, 99% of the underlying work is just doing dimension reduction. Then once you do that, then you're actually using mobilize, your very precious small data to building on top of this already very well-learned representation. Then you can actually sort of do the modeling much better. And then agentic basically take away all the manual step, but it's completely orthogonal, I would say. I see. One follow-up to that. In terms of predicting tumor evolution, we may not have a lot of images. I agree with you, the data is quite sparse in that regard. But there is a lot of natural language that is out there. A lot of people have written reviews on how therapy affects tumor growth and- Yeah. -affects the pushes towards a hot tumor or pushes towards a cold tumor. Yep. Is it possible to leverage that information in an AI model that would be able to predict tumor evolution in response to therapy? Absolutely. If you see the free lunch, right? So they basically, again, come from the places where you have abundant data. One abundant data source is basically all the knowledge, right? Basically, all the healthcare common sense is captured by all those frontier models. You can ask it everything about all the published guidelines and knowledge base and so forth. All the experimental settings and so forth. Of course, you need some guardrail to shape them, but fundamentally, the information are in those one trillion models. And then another kind of source of information is all those images, they actually have related images. So you can also get a lot of those related image. For example, that's how, for example, Gabriele and team build on using the gigapath to actually model sort of pathology slide. They are using in just specific kind of scenario that are very different from even the original provenance data, but they can show that actually because of this pre-training, because of the dimension reduction, that already is a very good representation. So those would be probably two obvious kind of starting point. Absolutely. Thank you so much. Yeah. Thanks. Maybe over there. Yeah. So Mike Saracci, Boston G. On one of your last slides, you went through some unexpected predictions of events you didn't necessarily train for. And one of the things you said really caught my ear, which was that you allow for the simulation of an unlimited number of universes in order to help the model understand how precise it was. So what that has me thinking about, the question that is pervasive for me is, as we move into an era where we're going to be running these virtual trials, what's the right way in each of those instances to estimate and then communicate the precision with which the result is going to be representative of what the reality, biological reality is going to be? I'm sure this is something you've thought through, could you talk through where you think that portion of the field should go? Because I know the way that we communicate that precision is going to have to be fairly standard across- Yeah. Absolutely. So great question. Just to repeat the question for the online audience. So I think that what you're asking is, given this virtual patient, obviously the Holy Grail would be to simulate the future trajectory. You can also simulate many, many parallel universe, right? But how do we do this virtual trial if we want to really kind of turn that into sort of the standard to advancing next generation medicine? Is that kind of the gist of your question? Yes. And importantly, how do we communicate how accurate the answer that we generate in silico is? Yeah, absolutely. So, I would say that, again, AI is no magic, so I wouldn't say AI actually fundamentally change how evidence-based medicine works, right? We still try to figure out the causality, like if we do this kind of intervention, what might happen to the patient, right? So given this intervention, what will be the status change? So I would say a lot of the frameworks and so forth, I would say is still kind of applicable in my mind. But I think what AI can do is probably three or four dimensions. One is that it can take away a lot of the mundane chores work, right, and then start to make them much, much efficient and faster. The other one is the other aspect to your point about the accuracy is that we need to avoid being kidding ourself, right? Because one of the challenge is the confounders, right? If the virtual case arm and the virtual control arm fundamentally are very different in some of the aspect that might influence the treatment response, then we are host, right? So, but here again, you can think about propensity scoring is one kind of poor man's approach where you model just a few features. But with this kind of more all-encompassing virtual patient, you can start to using that to replace some of those kind of limited representation. And we now no longer need to worry that much about we missing a lot of the unknown confounders, right? Basically, you try to incorporate every single aspect of it. And then another kind of potential prospect is we can start to model potentially much longer longitudinal scale, like you can sort of virtually simulate. Now, get back to your accuracy. I would actually argue that a lot of the gold standards still remain, right? So we probably, at the end of the day, still need to run some of the maybe much smaller, much more faster kind of RCT, right? So looking at the actual outcome with the intervention. But what AI can basically is hugely leveraging all those real launch, all those real-world journey to hugely magnify the value you can draw from those small set of examples. Another aspect I want to add is right now, if you look at how clinical trial conducted, right? We say, "Oh, we need to add this condition. We need to add that condition." Very soon you are left with no patient, right? And we also know how all those condition comorbidity actually intertwine. So, the hope is also that can we actually be able to model all those together. Now, one of the caveat obviously, is that this is a high dimension model, right? So many people call it black box and so forth. So you no longer have the comfort zone to say this is just reduced to some simple rules, right? Like A go to B and so forth. So that is indeed a potential challenge, but I would say it partially is because biomedicine is very complex, right? So does that answer your question? Sorry. Not completely, but it was fantastic. Thank you. Maybe I can follow up. Yeah. Love to. Yeah. Yeah. Hello, this is George Aquara. I'm from Scripps Research. I wanted to ask a question regarding, well, we'll see all these foundation models that you have built, which is fantastic on how we can learn from unsupervised, less labeled data and so on. I want to have a little bit more details on two aspects. One is, so are all these model after they are trained, are they public or will they become public at certain point? And also, seems to me that they are all based on public data. If I understand, is there any concern also on the privacy potentially or can they reveal something or potentially re-identify in an unexpected way participants and create some problems in that respect? So... Yeah, absolutely. So I would say privacy and compliance is obviously our most important aspect for the patient, right? So, I will also highlight another potential risk is that when a virtual patient get really, really good, right? Imagining you have a better and better crystal ball, right? On one hand, you can use it for precision health. On the other hand, it could also be used to deny your claim, right? So, I would say basically what we are trying to do is to de-risk this technological approach to say, what can AI possibly do with this kind of data, with this kind of algorithm, with this kind of compute, right? But I would say there is a larger societal problem. Question is like, how should we think about this new era when AI is much, much more powerful compared to prior statistics and supervised machine learning methods, right? So, and exactly to a point, there were some of the traditional risks that will be very small. Just like for example, in the past, even if you have some public information, without a search engine, somebody would never find it, right? Now, with a search engine, every single piece of information you cannot escape, right? So, there is a risk and challenges here, and the question is how with the society to do that. Now, back to some of our immediate work, we have a tradition try to-- because we already take all those compute, take a lot of expertise, take a lot of to do train those model which-As long as possible, we try to make them openly available, right? Now, some of them, I would say they are not trained on public data. For example, our joint work with Providence on the GigaPath, GigaTime. But we carefully want to make sure that the model they release are only on those unannotated images without any PHIs and so forth. So we want to make sure we minimize risk. And also the whole release process, we try to put it in a controlled way. But again, I would say that beyond that, we are just a small potato in the big cap community. I would say it ultimately will take regulators, payers, and the whole society to figure out how should we rethink about how we regulate this, how we actually go about this, right? Because there are tremendous good we can do, but there is also tremendous risk that we have to be aware of. Thank you. Yeah. Thanks. Yes. My question is really sort of in line with that question. I've always been told there's no such thing as a free lunch. Just because you're feeding one to me, I shouldn't necessarily eat it. So with curiosity-- Sorry, this mic is really small. For curiosity, I'm concerned. I'm Canadian. We're in a public healthcare system. I lived in the US for many years in an insurance-based provider. Is there concerns that insurance could just take this, make some sort of very conservative or not conservative version of this model, and then start denying people care? And is that something that you guys have talked about, and whether or not this is really a path you should be going down with- Mm-hmm ... given the constraints of hallucinating and predicting things in the context of this type of a large model? Yeah. You actually packed several good questions there. So let me gradually unpack. The first thing, I'm a machine learning guy by training, right? So there is this machine learning 101, it's called no free lunch theorem, right? So basically, it's like any learning is only possible with some inductive biases, right? So when I say free lunch, I'm more talking about from a sort of like a techno business front, right? It's basically like in order to directly generate those kind of data for the purpose of training the model that you want, it's pretty much economically infeasible, or maybe nobody would do that, right? But the free lunch there is you already have some other related data. Again, not directly the high-quality data you would like to have, but some other related data. And if you can put some of the smartness to connect those two modalities together and leveraging those information source, right? By leveraging AI to pull out those information, then you can using that to generate a lot of the new signal that power your original application. So that's what I mean by free lunch, right? Now, I think your question specifically to curiosity, I think there are several different questions, right? So the first question is actually more related to Giorgio's question, which is how should we think about using this responsibly? So this is absolutely sort of the top priority that for Epic Cosmos and us, right? So, many of you are probably familiar with Epic Cosmos, right? So they are actually governing by this 300 health system that form this consortium. So every single thing, both the data and the model, and also all the compute, are happening within that tenant. So we actually don't get any data outside. Just like when we work with Providence, right? We don't get any data to outside. We work with our partner inside their private tenant, and then to conduct the computation and everything. So at that level, we make sure that all those computation are actually within their control and governance, right? And then, the Epic Cosmos work by essentially governed by this Epic Cosmos governing council, and it's for research only. And so the model is basically now releasing to all the Cosmos participants and so forth, right? So it's really that consortium are governing how they do it, and certainly something that you talk about is absolutely some of the really important top of mind. I would say the last aspect you raised is also a very good question, which is the real-world patient journey again, mix up lots and lots of confounders, right? So a lot of the coding, ICD coding, right, obviously, it's a well-known thing. It's like they may not absolutely reflecting the diagnostic, right? And then there is a lot of idiosyncratic healthcare practice challenges and so forth. So once again, I'm not claiming magic here. Far from it, right? So what we try to do is even starting from sort of relatively straightforward sort of today's technology, what kind of insight can we unveil from those kind of data? What can we already be able to do? The next question is, how do we, number one, scientifically, how do we counter a lot of those confounders and so forth? And then societally, how do we actually-- now that this capability is we demonstrate that this is possible, how should we collectively think about how to regulate it? So at least that's my naive sort of two cents for now. Hey, thank you. Great session. I'm Amitai Jacob from AstraZeneca. I have a little bit maybe sort of a more technical question, but I think that can serve the community. So if you look at the left, we have all the modalities, and if we just from the last few days, two, I think, amazing paper, one on a new RNA foundation model called Authoris from Bo Wang, and another from Goodfellow, that is-On the DNA mutation? Yep. Right. And I guess that next week we're going to have more. Right? I'm wondering, and I think you guys in Microsoft have a lot of technical capabilities to help the community understand how these kind of newer presentations can be plugged in and show the deltas. Mm-hmm. That could serve many, including AstraZeneca. So I was wondering if you've been thinking about that on the strategy side, on the technical side, and how, as a big tech, you could maybe lead or- Mm-hmm ... share with us your thoughts around that. Yeah. That's a fantastic question and certainly something we should chat more about offline. Right? But I would say that if I think about what will be some of the wave for disruption for GenAI to start going to cancer research, but also more broadly, biomedical discovery, right? I would say the first wave is already going on. So I call it productivity gain. Now, obviously, everybody talk about agentic AI, right? And that's indeed where the first wave will be agentic AI, right? So this, again, anything you can think of today that you can just describe to some intelligent intern, right? That you have, and to be able to do. Now, obviously, anything you could manipulate on the file systems or anything, your VM, your cloud computings and everything, this agent can do it for you. Right? So in the past, you have to painstakingly do a lot of those kind of groundwork. But actually now, having this kind of agentic AI system can already do it very, very well. Now, Claude already show how this can be do for empowering the programmer, or replacing some of the jobs, right? So the question is, can we also tailor-make that for the biomedical discovery process to really empower the scientists, right? To actually, right now you might have to waste hours or days or weeks to do some of the really, really boring stuff. But now you can free them up, right? So maybe replacing the boring part of the job, but now free them up to do the way more pleasant and the creative part of the work. So I would say that is absolutely a platform question, exactly like you said. And really, I would say, again, certainly some of the technology company can deal with some of the scale problem. But I would also say the stakeholder, like you guys, who actually are in the trench, who know what are the important things. I will flag one important aspect is the benchmark. Again, AI is no magic, right? So if we don't know how to hold them accountable, then it's very hard to evaluate progress and so forth. And that's actually how I imagine some of the key stakeholder, end-to-end stakeholder like you guys can come up with those kind of benchmarking. And then the potential platform provider can actually then-- Once you have a goalpost, it's not hard for the AI folks to actually engineer it to actually get to those goalposts. Then I think we can start to think about the more exciting part beyond the agentic wave, right? So to say, how can we not just do a shallow, like tool use, a plug-in kind of use, right, of the model? How can we truly actually integrate all the insight captured by this model? Right now, the amazing thing is that there are, like you said, there are so many of this model. Each of them reflect one small piece of the universe, one piece of the silo, right? Can we have a sort of unified medical brain that capture all those? That will also be- This was actually my next question. Your thoughts about moving from a late fusion- Mm-hmm ... or feeding multimodal embeddings in a late fusion- Yeah ... versus one big beautiful model. Yeah. I think my opinion is that we are on a journey. Yeah. And in the end, we're going to have a one world model for patients, right? But then what's your thoughts about this journey and the- Yeah. This is a question actually we have debated even with our own CEO three years ago, right? So I would say that it's probably one-- What I mean by one, it's more like one system that actually can capture all this kind of knowledge and insight, right? But just like think about me, right? Standing here. I'm actually a number of trillion of cells, right? So with different kind of organs and... So actually look under the hood. There are so many kind of underlying kind of components, right? But still, hopefully, I function like a coherent human being, right? So I would think about that actually is where we want to get to. It's a unifying system that actually are coherent and consistent in terms of the goals and everything. But underneath it, there could be lots of kind of like components. The reason we would try to do late fusion, there are several reasons for that. Number one is that it's very, very hard to fine-tune those humongous foundation model, right? Like now MiFauns probably 10 trillion parameter, right? You really need to know what you're doing to do fine-tuning. And also oftentimes they are training on tens of trillion of tokens and very hard to move any needles. On the other hand, once you go beyond the text modality, there was so much big growth area, right? So for example, that's why with just 1 billion parameter in the Curiosity, even when we try to pre-train some from scratch- Mm-hmm ... for those structure EHR, actually the language model are not very good, right? So anything beyond the text, there is much easier to make progress and that's why we want to try to decompose that and do the late fusion approach. But obviously nothing is completely set in stone, right? I would say right now we are still very early. So again, go back to the original goalpost. What are the hard problem? And that will be my default approach. Thank you. Thanks. Awesome. Thank you very much. Sorry this dragged on a little bit late, but thanks for everyone who attended the session. This concludes the session. Thank you very much.