2026-04-18-pm-3d-tissue-imaging-and-cancer

Date: 2026-04-18 Primary topic: single-cell-spatial-omics Cross-referenced from: bioinfo-tools, single-cell-spatial-omics, virtual-cells Word count: ~16,035

Auto-generated captions

This transcript is Vimeo auto-generated English. Expect misheard names and technical terms. Use the ripgrep/search approach for exact quotes.

Okay, everyone. Thank you very much for coming to this educational session on 3D cell imaging and cancer. We've located our second speaker, but I'm sorry to tell you that our third speaker, Carolyn Larabell, who was going to talk about quantitative 3D views of whole cells using soft X-ray tomography, is not going to be here today, and we've not managed to secure a video, I'm afraid. So what we're going to do is Ashley and I are just going to extend a little bit our talks, and then we're going to do what we're supposed to do in these sessions anyway, which is to have a more extended Q&A and panel discussion. So as we go to that last piece, I'll welcome anybody who's interested in that to come forward, and it'll make it easier for us to use the preset microphones. So before we start, let me just say, and some of this was written up in the introduction, which I had made. Just a moment. Ashley. Ashley. Okay, thanks so much. I got lost. Yeah. No. Do you want to sit up here or you want to sit down there? We'll get everything working. You have 25 minutes. We were just checking that she wasn't going to talk 2D. It is going to be 3D, so I think we're safe. So I wanted to just say an introductory remark before I start on my own session, and it was in the comments that I'd written as an introduction. I think it's important as we talk about 3D imaging today, to acknowledge a few things. First, that 3D imaging is intrinsic to everything that we think about in radiology, so it's all tomographic. So that has to be there. Secondly, as I mentioned, pathologists have traditionally had a kind of 3D mindset, and it manifests itself in the fact that diagnostic pathology on a tissue level will look at many sections before coming up with a summary view, and also will typically focus up and down on a single section, particularly using what's called high powered field, and the high powered field then allows you to get a 3D view. And I think what's different in what you're going to hear about today is taking the same fundamental concepts but trying to make them more systematic and a little bit more quantitative. And you're going to see in the two talks, and then hopefully in our follow-up discussion, a sense both of where we are and where we're going, and how this inevitably relates to what's going to be a much broader range of two-dimensional methods. So, with that, let me start in on my presentation, which is going to be about highplex 3D microscopy. I want to start off a little bit with a problem statement here, talk a little bit about spatial profiling, and 3D imaging. Then I'm going to tell you about three biological applications, necessarily fairly quickly, identifying cytotoxic tumor cell interactions, looking at pathways of tertiary lymphoid assembly, and then neuroimmune interactions. And the logic connecting these is really going from the tightest spatial scales to the largest scale. Let me also mention here that if anybody's interested, all the papers related to this, some of them are just being posted to bioRxiv this weekend, can be accessed via this QR code, and I'll pop it back up at the end of my talk. So just to set the stage for this, the motivation that we've had in the group to try to think about this kind of spatial profiling technology is not simply to get deeply into individual cell types, which we obviously recognize is important to understand in the immune system, what type of cell, its activity state, but also to get access to location. And here's a tracing that I made of a primary melanoma, as it turns out, with just some of the kind of features that you encounter here. Does it matter whether the T cell is infiltrated or is in the surrounding stroma? What is T cell exclusion look like when you have tumor domains of different sizes? What about the role of the vasculature, for example? And then clinically about these clustered assemblies, tertiary lymphoid structures, or SILTs, for example, in the colon, in which a lot of T cell programming happens, T and B cell programming happens outside of the primary lymphoid organs. The other thing, of course, on a much smaller spatial scale, sort of imagined here on the right, and we'll see some images of this, is trying to get access to true anti-tumor reactivity, and we're going to look at this in today's talk, surrounding T cells. And of course, in a human setting, there's no way that we can use tetramers. We don't typically know what the antigen is, and it may in fact be quite diverse. And so actually working out which subset of T cells might be reactive in the tumor environment is a challenging problem. So the methods that I'm going to talk about today are of a family of methods. This is an open source version that we built, but it's the typical method that's used here. And they're these cyclic accumulation technologies, and so you end up building a high-dimensional image. They can be as high as 100 or 150 dimensional. Some vendors claim yet more. And they're typically built up by sequentially collecting lower dimensional images. And the individual cycles can be anywhere from, say, three colors up to 20 or more, but the idea is to use a microscope optimally to get at molecular information that allows you to subtype individual cells. And then what I'll show you today in a paper that we just published at the end of last year is that you can take all of that and now move it into various forms of three dimensions. And I'll tell you a little bit about why you might want to do that. This is on a 35-micron section of a primary melanoma. And in this case, we have a 54-plex image, which allows you to go fairly deeply into individual cell types, and activities.And so if we look across sort of the range of applications, and I think both Ashley and I will discuss this, is you have different ways of trying to get access to the third dimension. Serial section, in some ways, is the simplest, but it's also fairly time-consuming and difficult. I'll show you some of that data. Light sheet microscopy is used almost universally in model organisms and has only just come into the fore now in humans, and I'll show you that. And then various forms of high-resolution confocal microscopy. And you might ask, why would I need a range of techniques? All of which, by the way, are happily in the facilities we already have in our institutions, is because you're trying to scale all the way from intact tissues on a millimeter scale down to the functional properties of cells at the levels of their organelles. So a couple of quick basics here. If you just imagine in any 3D imaging, typically these are mounted to microscopes, inverted microscope here, so the x, y axis, the z axis is typically the most difficult one, and it's what we mean by 3D as opposed to 2D. But it's important that you also think about the size of the specimen as well. Diagnostic histopathology requires whole-slide imaging, which is going to be a couple of centimeters of tissue if the specimen is that large. Anything on sort of TMA size is not going to work at all to look at things of the size you're going to see about today. There are various ways to get access to that third dimension, and those of you who are microscopists know there are many more than I'm showing today. The two I'm going to mention are optical sectioning microscopy, so using the capacity of a microscope to focus on a fairly thin slice of tissue. I'm going to show you images where we take 250 sequential optical sections, and you can then build a three-dimensional image. And those microscopes are in routine use in pre-clinical studies around the country, but again, not much applied in the areas we're interested in of human histopathology. And the other one, which is our set of volumetric methods, very diverse in terms of instrumentation, use light-sheet imaging, and those are directly volumetric. So they can collect large volumes of tissue down to cellular resolution, and by far their widest application is in studying the mouse brain and similar neuroanatomy. The other thing that most of you, I think, will be aware of is that the numerical aperture, which essentially is how fat is your lens, and the fatter your lens, the more expensive it costs, so anywhere from 4,000 to 20,000 for the big fat lens here. The numerical aperture ultimately determines resolution. And so the point of showing you this is that 3D imaging is identical to high-resolution imaging. Because as you get a higher resolution in x/y plane, you get a higher resolution in z, and you therefore need to collect the third dimension information. And you can see that simply here. So this is lenses going from $1,000 to $20,000. Here's sort of the imagined object here, so sort of delta function. And you can see that you get a narrower and narrower sort of reimage of this point spread function. And this is what you would see in x and in z. So the third dimension, the one along the optical axis, is always the hardest to sample. And you could say, well, what does looking at a dot have to mean? Well, here's ECAD actually on a melanocyte, an invasive melanocyte, and you can see, you could imagine that the ECAD is simply made up of an appropriately distributed set of point spread functions, the so-called convolutional aspect of optical microscopes. And so that's sort of what it looks like. So if you're going to cut a thick section of tissue, this is about 30 microns, and we'll see why you're going to do that, stained just for two colors. You can see on the left, with just standard microscope, wide field microscope, it's a pretty fuzzy thing. On the right, you're going to see the confocal optical sectioning, one focal plane shown here. So the 3D equals resolution. Resolution equals the ability to see deeply into individual cells. So what we're going to look at now is sort of the first set of results we got out of this a couple of years ago now. And the goal here is to try to find that morphology in a T cell which is associated with active killing. And what we realized, if you go to a standard five-micron histological section, they can sometimes be 10 if you go to the vendor show, but five is the norm in diagnostic histopathology. These are cut from FFPE tissue, so everything I'm showing you today is designed to work with standard histopathology workflows. And what you'll find is essentially no nuclei and no cells, almost without exception, are intact in a five-micron section. If you go to around 35 to 40 microns, you could say, why not go much thicker? Well, one thing is we're trying to preserve clinical specimens in many cases. But that thickness is suitable that about 70% of cells in a typical epithelial cancer will now be intact. And so we'll ask ourselves whether that's important, and you'll see, I think, pretty clearly how it is. And this is the work of Clarence, Alex, and Daniel, and so I'll try to credit the people who did this as I go along. So let's now go into one of these three-dimensional images, 130 optical planes here. You're going to see a video of this in a moment. We pulled out just one dendritic cell and two T cells. We got 50 channels on this, so we have a sense of phenotyping, and we're going to ask ourselves the question, did it matter whether I chop the cell in half or not? And so what I'm going to do is take this complete image, and this is side on, by the way. And here it is again. I'm going to chop it up as though this were sectioned like a standard five-micron tissue section. And then I'm going to create a virtual 2D image. This is actually much sharper than you would normally have seen, but nonetheless, it serves the purpose. And if you do that, you're going to find is that you get a very high rate of misidentification of critical immune cell lineages. And the reason for that, of course, is that-These are highly polarized markers. So if you chop a cell into many slices, you're going to miss the vast majority of the LAG-3, for example. In many other cases, you'll find that signals are actually on top of each other because cells lie on top of each other in tissue, and so you perceive what was in the upper cell to lie in the lower cell, and you're going to end up with, for those of you who use these methods, our favorite cell, which is the epithelioid T cell, the T cell that carries E-Cad, for example. And in Xenium images, for example, that can be up to 40% of T cells, and it's probably because of this lying on top of each other. And then finally, of people who are developing reagents, you're often going to find that you have the schmutz that you can't get rid of. And we now realize that the schmutz you can't get rid of is often just part of a process of a cell whose nucleus you're missing and you don't recognize it as such. So what really has come out of this thick section imaging is two things then, the capacity to exploit resolution to get at function, and critical, because remember, in the human tissue, we sometimes go pre-treatment, post-treatment, but we don't typically have a time dimension. We don't have a CRISPR knockout in there, so morphology is what's reporting on function for us. And here, it just gives you an example, and again, pulled out of one of these images, of a neutrophil on the left and a fairly round-looking T cell on the right, and you can see individual organelles. And those actually can be informative, as we'll see, for activity, particularly the cytolytic granules. So the idea behind now looking for active T cell killing is we have a somewhat bewildering number of markers. And depending on when and where you wish to publish, you will choose the appropriate subset. Currently, obviously a lot of interest in the early lineage determinations. But there are a whole bunch of other classic features, like the fact that you have granzyme B-containing granulysin or perforin-containing granules. Those are going to be oriented relative to the microtubule organizing center. You would expect in this type of imaging to see a separation of around 20 nanometers between the T cell and its target, and eventually, for at least a subset of cells, you'd hope to catch the actual cytolytic interaction underway. So the question is, could you do this with this type of imaging? And so here now let me show you what that dataset that we collected looks like, and there are a whole bunch of them, but I'm just showing you one. We're going to peel back some of the layers so we can see what's going on. Using the channels I'm not showing you, we can actually go through and determine the likely function of each of these cells. We have a memory T cell, a helper T cell here, an effector T cell, a dendritic cell, and a tumor cell. And one of the things we've learned in doing this imaging, by the way, is the average T cell in a tissue is making between five and 10 functional contacts with its neighbors. And so here what you're looking at actually is that T cell with LAG-3, so checkpoint marker up. It nonetheless actually is highly cytotoxic, so it actually contains those oriented green cytotoxic granules. They're trying to kill, as we interpreted here, that Mart1-positive tumor cell. We're rotating so you can see different sides of this. But nonetheless, even though this is a cytotoxic interaction, there is this strong PD-1, PD-L1 inhibitory reaction going on in the exact same cell, in this case mediated by a dendritic cell. So to the first approximation then, we can actually pull these types of interactions out, and I've just given you a few of them here. And morphologically, we can determine whether they have cytolytic granules in the right place. Interestingly, in these tissues, we can't really image the cytoskeleton satisfactorily. But using super-resolution imaging, again, a standard feature on these types of instruments, we can look at membrane contact distance down to 30 to 50 nanometers. And so we believe that there we're capturing a functional immune synapse. And by going back and forth, work underway, back and forth between organoid experiments and what I'm showing you here, we hope to functionalize these kind of critical morphologies. Interestingly, the average T cell, and we looked at this in colorectal, and ovarian, and melanoma cancers, has a distance to its immediate neighbors of around one micron. So typical membrane-membrane distance is in these tissues around a micron. When a T cell's invading into an epithelial surface, its interaction distance will be about 400 nanometers. And then we think when it makes a functional synapse, that goes down and stabilized over about five square microns, and it gets down to about 30 nanometers, which is consistent given the resolution of these instruments with what we know from electron microscopy. There are a lot of shape differences you can also capture, and these are difficult in what is standard tissue profiling out on the vendor show there because the resolution is insufficient, but also because you aren't capturing whole cells. So this is just to show you what a T cell actually looks like in tissue. It's relatively rare to see them round. Here we're actually imaging them based on CD8. They often have this extended structure here, which pathologists tell me they often call squiggles. But in highly activated regions where they're interferon positive, you can see they form these very distinctive spike-like protrusions. And that's the kind of information that, to be frank with you, we don't really know how to interpret. It's universal in what we do in tissue culture. Morphology is how we look at all these things, and so we now need to, beyond marker space, functionalize contacts and functionalize morphology. And that's the state of the art of this type of imaging. So let's look at a slightly larger spatial scale. And this is a project that we've been doing in lung tissues, two wounds that do not heal, a lung adenocarcinoma and, of all things, tuberculosis. And I'm actually going to show you some TB pictures here, but bear with me, because there's a fundamental similarity in both. And what we're trying to look at are tertiary lymphoid structures. And I think-- So this is going to be now both T cells and B cells in big lumps. We know TLSs are very important in the action of immunotherapy, but also in natural tumor surveillance. And they're typically organized in a developmental series, excuse me, that goes from a kind of early TLS, where you'll see an aggregate of B cells, but you don't have the formation of a germinal center, to the beginnings of a germinal center, and then finally a complete germinal center, where you have follicular dendritic cells and other characteristic features.And if you just go through and you start to section happily through, and you go ahead and just image this, you'll find that a single follicle or a single, in this case, TLS, can actually be called any one of the three different developmental stages, depending on where you slice it. And of course, there's no mystery in this. The thing looks like an olive, and so whether you hit the pit or not depends on where you cut it, and also just how big the pit is. And so, the degree of error varies to some extent with the tissue section. So, Emmanuel, Jerry, Zoltan, Sohail, and Brigitte in my lab went through, and we've actually been doing large-scale serial reconstruction here. So you could ask why serial reconstruction, and it's because we want to span up to a couple of millimeters of tissue. And I'll show you in the end how we can do that without a serial reconstruction. But this allows us to do a multi-omic approach. And the tissue here is a couple of square centimeters, so we're looking at around 60 million cells. And we have sufficient antibodies that you can begin to subtype these cells. And so this is sort of what that kind of reconstruction looks like. And one of the immediate things-- By the way, if you're interested in this, if you flash the QR code later, you can get to the bio archive paper. So this is actually what a typical TLS looks like, both in LUAD and in this case, in the infected lung. And that it's an extended structure that's not individual balls, but actually they've all sort of grown together. They look like a Jerusalem artichoke actually to some extent. And they can contain multiple germinal centers at different developmental stages. This is not something that we had expected to see. But we've now built up 470 of these individual structures, and here you can get sort of a picture of what that looks like, and we know what cells are in them, et cetera. And then that we can ask ourselves, could we put these cells in any kind of a logical developmental series? And so the method that we've used is called optimal transport. So I think many of you are familiar with trajectory-based methods that are used in transcriptional profiling, and the idea is you have a high-dimensional transcript profile, and you try to find the next profile in a series, different cell, for example, that would be closest to it. Here, we're actually doing what optimal transport was initially developed for, which was building forts in the 19th century. And the thing we're minimizing is called the Earth mover's distance, and it's in fact still called the Earth mover's distance. And in this particular case, we're saying if we have two structures, we want to find those that are closest to each other if we allow only cell birth, cell death, or cell movement. And birth in this case can also be delivery from the vasculature. And so that kind of mechanical and sort of physical idea allows you now to build up a trajectory series. And this is what that trajectory looks like ultimately. So you start down here with a very, very small, and I'll show you a picture of this, a very small, kind of partially formed aggregate. They then go-- And you'll see they're actually sort of two kinds of attractors. And we know that these ones are actually around the vasculature, and in this case, these are around necrotic domains in the tissue. So we actually see a distinction. But the critical take home, I think, for this is that all of what we have in two dimensions been concentrating on early, primary, secondary, vast literature on that is in fact squished in one little corner. And because we can't correctly perceive the larger architect, we've missed all the variance in green, if that makes sense. That third category is in fact far more variable than the other categories are from themselves. We can get some confidence in these cases of this type of trajectory just by looking at physical properties. So along one of these trajectories, the things get bigger. We can see they get more compact. So we can look in the core of the TLS and say whether the cells are getting more compact, and that kind of compaction is a known feature of maturation. And then we can look at particular cell types, like here, the interactions of T cells or B cells with follicular dendritic cells and their maturity. So in terms of those features, we sort of built the kind of plausible developmental trajectory. And then you can ask yourself, "Well, what's different between these things?" And it turns out that actually what's fundamentally different, and we're looking here at how many B cells do you have, how many immature follicular cells, and how many mature follicular cells over time. And you can see that one set of these structures matures very early. Those are the ones deeply embedded in vasculature. And the other set, the one that's proximal to necrosis, develops substantially later. And so this would be the beginning of saying that this kind of TLS reconstruction can give us a view of the maturation state of these, and obviously you'd now like to look at this, for example, in the presence of an absence of an ICI therapeutic intervention. Finally, we can go into the imaging I showed you before. And just to give you a piece of eye candy, this is a compact secondary tertiary lymphoid structure. This is what the follicular dendritic cells look like, by the way. They have these propeller-like structures. This is the high endothelial venule, which goes right through the middle of them and delivers the cells into them, and these contain both proliferating and active B cells, and then on the periphery, proliferating, and we think program T cells. So sort of the ability then to drill down. Now, the challenge with what I showed you just a moment ago is that this kind of serial cell reconstruction does not really scale. So I showed you a couple of specimens there. You're going to want dozens. And so the method that's used almost universally in preclinical biology, particularly, as I said, in neurobiology, is light sheet fluorescence microscopy. And it can look at specimens several millimeters thick. And so this is our first development of that. Again, now going back to cancer, looking at colon, similar team of people. And what's neat about this is that as you get these three-dimensional images, they're fantastic tools for tracing individual elements within them. And what we're tracing here are the individual nerves, and we can look at ganglia, et cetera. So let me just show you what an image like that looks like. So here you're looking at something that's two millimeters thick, and we've imaged all in one, 30,000 cells. This is a human-- Sorry, excuse me, 30 million cells. This is a human FFPE tissue, and this is done by cyclic immunofluorescence light sheet fluorescence microscopy. And so you can see the nerves, both NGFR here in tub three are the best markers. Pan cytokeratin will get the top ofOf the epithelial layer. And then here we're drilling through, and you can see the CD4 marker coming up so that you can actually see the neuro immune interaction. So this is sort of the final type of imaging that I want to show you today, and it basically can be coupled with what I showed before. So you can do large-scale light sheet microscopy, and it's actually pretty fast. Actually, once this is ready, in a couple of days, you'll be able to get a couple of samples at about 20 plex, and then you can couple it in with what I showed you before. So we take the specimens, do the light sheet microscopy with cellular resolution, re-embed, remount, slice again, and then do the high-resolution one that I showed you before to get at the structures of particular things. So how would you do that in a TLS? You'd sort of image the whole thing, and then you'd go right for the middle because you know where to cut, and that's where you'd look with this final type of imaging. So let me conclude by saying a word about H&E. So there are a lot of reasons why you don't just want to use molecular-type imaging the way I've emphasized today. You actually want to go back to H&E imaging. And one of them, of course, is the deep insight that you can get into morphology from 150 years of diagnostic histopathology. The other one is that even when you go to high plex, you actually only catch about 70% to 75% of the cells with a definitive marker. So there are a lot of things that you actually don't see. So there's a big value in embedding even high plex molecular in traditional histopathology images. And we'll hear more, I think, later also about the ways you can use AI to analyze these sorts of images in greater detail. So the kind of thing we often do in two dimensions is actually an Orion microscopy, is we actually collect same section H&E. And we have visualization tools, public domain tools like this, and you can just drag your lens around, and the top is a bunch of colors, and the bottom is a pathologist's favorite image. And as you can imagine, this allows a kind of cross-training. But for today, what I wanted to show you is you can take exactly the same thing and you can now move this into three dimensions. But the trick in this case is not real H&E, because real H&E on a two-millimeter piece of tissue isn't going to work. Instead, this is what we call virtual H&E. So it isn't as good as H&E. You wouldn't go and do a diagnosis on it. But you can see that it actually gives you a great view of the tissue architecture. And it's computed using a more or less fancy method, directly out of autofluorescence and a nuclear stain. And again, this can be done on many microscopes. And so, just to give you a sense of what the application would be, slightly different color scheme here. Here's synthetic H&E. You'll recognize these as colonic crypts here, and you can see the individual T cells. So you can take a cell type here, which we have only one color, and you can embed it into the full sort of complexity of that tissue. And as you hear more, you can then compute on those tissues and get all sorts of features. And in this case, we have sort of a boring one, which is the volume of individual crypts and whether they're undergoing fission events. So this type of technology, we think, actually is broadly scalable. I don't think any of these are diagnostic. But this actually can be done relatively inexpensively. And when you're done, you can re-embed and put it back on the shelf and come back to it some other day. So in conclusion, I hope I've shown you that this kind of profiling can have lots of variants, from what you might be familiar with, five-micron FAP frozen. You can integrate this in various ways with H&E and transcriptomics. I've shown you a bunch of thick section imaging methods, and these can be implemented on a really wide range of different instruments. So one should think about this not as a specific technology I need to go and buy from vendor X, but just sort of a branch of histopathology or a branch of microscopy that's broadly applicable. And then finally, you could ask, should I do this for everything? So I think there's a certain hype around 3D tissue imaging, and the answer is clearly no. If you're going to do a large clinical cohort or a large mouse study, you're just not going to 3D image everything. And so the concept, still vague in our minds, you can imagine doing some of this with machine learning, goes back to a sort of stereology idea, is a limited amount of sampling in 3D gives you the capacity to understand 2D images much, much better. And we actually are learning that there are sort of shadows in a 2D image of a 3D structure that we couldn't appreciate without the 3D ground truth. So let me then close by sort of thanking-- I've tried to thank the funders first, but I also want to thank the instrument makers who've come along and helped us with this. So a lot of them have loaned us instruments we couldn't have afforded otherwise. And the methods that are here, if you're interested, are all public domain. And I think you could just apply them by going to your friendly, local, NIH-supported microscopy facility. I've tried to recognize the people who did this work, but I want to say it's been incredibly dependent on pathology collaborators. So our group's been really lucky to have been befriended by pathologists on several continents. And I want to call out particularly Sandro Santagata. And then finally, a software and data analysis team whose contributions I haven't fully recognized. Thank you very much. So what I'm going to do is I'm going to entertain five minutes of questions, and then we'll hear from Ashley, and then recall that we have an opportunity to come back at the end, and actually do a more extended Q&A series. So, if there are any questions, please come up to the microphones. Great presentation. It's really exciting to see all these new techniques and what we can do now. And my question is kind of a curiosity. How it works for tissues with a lot of inflammation, like tumor, when you have a lot of destruction of the morphology, is these all techniques able to identify that? And also, what is your preferred method to... conserve the tissue, like PPFE, frozen tissue, and how that influence the background of the tissue, like the signal? Yeah. A couple of fantastic questions. So, one of them is one of the most interesting things in a tumor setting actually to look at is the structure of the necrotic and perinecrotic environment in an advanced tumor. That thing is just chock-a-block with cytokines, and so T cells. And it generally is filled with dead neutrophils. So that can actually be imaged very nicely using that synthetic H&E I showed you. Antibodies won't stain it. Synthetic H&E pulls it out beautifully. With respect to tissue preservation, again, it depends very much what's necessary there. So I think this 30-micron thick slice for any kind of research study is pretty good. You can usually, for a solid tumor, sacrifice that. It's less than what you sacrifice to do exome sequencing, by the way. But I think if you want to do that very thick light sheet, you sort of have to dedicate a specimen to that. So for the moment, I think in a research setting, it's best with things like colorectal cancer where sadly, by stage III, you have a lot of blocks. So, and then finally, you asked about frozen. We work with FFPE. Frozen works fine. Just remember, anytime you use frozen, the antibodies are different, the buffers are different, the concepts are the same. Okay, I think what we'll do is go on to Ashley's talk. So I'm very pleased to be able to introduce Dr. Keman, who's going to talk about AI approaches to 3D imaging. We'll entertain questions after hers, and then we'll go into an extended sort of discussion. And we have until 2:00 for that. Thank you. Thank you all. My name is Ashley Keman. I'm a quantitative biologist at Johns Hopkins, and today what I'd like to focus on, now that we've had this nice talk about imaging, is how we can apply AI and analytic technologies to these images to quantify them. So what I'm going to do today is give a quick high-level sense of the types of imaging we do. When is 2D imaging ideal? How can we go beyond 2D, and what common methods do we apply? Which distinct advantages do things like intact imaging and sectioning-based imaging have? How AI has changed the game, the fact that a lot of these imaging methods date back as long ago as the 1850s, and why, in my opinion, it's AI and quantitative imaging that has made this the booming field that it is today. And then I'll talk about specific applications of AI, which models we use, and with what applications. So just to begin with, to talk a bit about what is 2D imaging, how is it used, what benefits does it have? 2D imaging, it's the foundation of today's anatomic pathology in the clinic. It's how patients typically get diagnosed. The most common stain that you'll see is hematoxylin and eosin, H&E, which uses simple pH to stain the nuclei cytoplasm collagen in tissues. Today, we use a lot of advanced imaging on top of this, things like cyclic IF to do highplex imaging, multi-omics. All of this can be done in 2D, and one of the biggest benefits of 2D is that we can analyze big data sets. So if the question you have needs many, many patients, tens to thousands of patients, typically 2D is the way to go. On the second hand, if the question you have has to do with connectivity, has to do with how cells communicate and something like 2D, say, is going to cut the body of a cell, is going to not sample what you need, that might be a time when 2D is insufficient. We have two main technologies used in the field. One is sectioning based. That's what my team uses a lot is histological sectioning going all the way down a clinical block. You also have intact imaging, things like CT, quantitative phase imaging, light sheet imaging, each of those with its own distinct advantages that I'll get into. While these have benefits, they also have downsides, the biggest of which is it can be tedious and expensive to collect this data, so it's infeasible to think of studies on the scale of thousands of tissues. This is something that you might look at tens of patients, to look distinctly and deeply into individual samples. So just a quick visual sense of what's done today in the clinic. And I want to give a quick defense of 2D pathology because as decades of diagnostic anatomic pathology have shown, it can be quite effective. It allows pathologists to diagnose patients, to stage patients, and to see actually a lot about the TME in tissues. So, in the clinic, what's done if tissue is excised, is that tissue will be cut into many small pieces, put into individual FFPE wax blocks, and individual 2D sections will be cut off of each of those blocks. A pathologist will look at each of those blocks independently, so it's a misconception that 2D pathology means looking at one slide. Pathologists typically look at many slides, just not continuous. They'll typically use H&E and will use isolated immunolabeling, IHC, if in a specific diagnosis it's needed. So in some diseases, they use Ki67. In some diseases, they use things like ECAD, CK19 to subtype diseases. So 2D is good at looking at multiple locations in a block, but it's not so good at connectivity. So maybe, on one section, you see what looks like two bodies, but on the second section, you would see that they combine, and it's the same cell. 2D is not going to get you that. On the second hand, we have technologies like CODA used by my team, which uses consecutive H&E sections. This allows us to image a volume of tissue instead of individual 2D slices. The upside is that now we have connectivity. We can see spatial association between tissues quite well, but you can imagine the downside that if we pickblock K and block A, B, and C had the diagnostic entity, we'll miss it because we only focus on one specific location within a block, even though we can deeply analyze that location. So just to sum it up, when is 2D imaging useful? Again, in the clinic, it's been shown that it's quite useful, although people have been publishing individual examples of when, even in the clinic, 2D imaging can fail. 2D is also quite useful, again, when you want to look at many patients. So looking maybe deep within a patient to ask a specific question might not be as helpful as looking at many patients. And finally, when someone has applications that they want to apply multi-omics assays, things like Visium, things like highplex IF staining, it can be quite expensive to apply this to multiple sections, so oftentimes, depending on the question, it might be best to apply it to many patients as opposed to many sections of the same. And now I'll give a few examples of when 2D may not be sufficient. This example is looking at cyclic IF staining in TMAs. So a TMA is a plug of a block of human tissue, and on a single slide you might have two dozens patients' tissue, all laid on the same tissue block. So with each section you can sample tens of patients. The downside of a TMA is to be able to fit so many patients' tissue on a slide, you have to cut a small piece of each block. So in this study, Lin et al showed quite nicely that depending on what you quantify in these images, something like a TMA is insufficient to ask the question of how inflamed is this specific sample? Maybe you could ask the question if you look at tens and tens of samples, how inflamed is colon tissue typically? But you cannot ask how inflamed is this patient's colon, because as you look down multiple sections of the same block of tissue, that inflammation can change wildly. Between slide to slide, you might have five times as many immune cells as you go down. To give a second example of when 2D pathology may be insufficient, we can think about collagen alignment. So if you look at the video on the left side of the slide, the blue label is showing you ductal epithelium, and the pink is showing you collagen. And this video is made of thousands of consecutive H&E slides with an AI model applied to them to label cell types. And each one of these sections, we can pull out individually and look at what the pathologist would've seen if they saw this slide. And you can imagine, if we go in and we pull out the slide that the duct was cut axially, we can see one angle of that duct. And if we go down about 10 sections, we can catch a slide that the duct was cut longitudinally along its long axis. And then we can quantify on each of these images, how well-aligned does the collagen look. And we did this, and we showed that, in fact, the collagen encasing this duct, this blue tube, is highly aligned. But if you catch the axially sectioned duct, which is the most likely angle to cut this duct at, it does not look aligned. And if you quantified that, you would come to the conclusion that this collagen is not aligned. So this is an instance in which 2D pathology is going to give you inconsistent data, depending on the angle of sectioning that you catch. The next example I want to give is one that specifically has impact in the clinic. When pathologists diagnose needle biopsies, tiny plugs of tissue taken out of patients, a tiny amount of tissue is taken, and even less of that tissue is used diagnostically. Pathologists will look at maybe one 2D slice of that needle biopsy. And what this team did is they applied open top light sheet imaging to an intact needle biopsy. And then by looking section by section, they showed each of those synthetic H&E slides to a pathologist and asked them to assign the Gleason staging slide by slide. And they showed that even within a tiny needle biopsy, depending on which section the pathologist looks at in the clinic, the outcome would change. Which is significant because, say, if a patient is assigned a Gleason 9, the action, the actional output is going to change as opposed to if it was, say, a Gleason 5. So this shows that in some cases, a 2D H&E slide in the clinic is not enough to make an actionable decision. And then finally, I want to show an additional example using H&E imaging. In this case, the video that you see on the left is a whole human fallopian tube, kind of coiled up and laid into a mega cassette. And in the fallopian tube, a lesion develops called STICS. And if you look in studies that estimate the incidence of STICS in the healthy population, some studies will tell you one in 1,000 healthy women has STIC lesions. And the next study tells you one in 50 women has these lesions. And it changes wildly based on the study that you look at. So in this application, we took a whole fallopian tube and exhaustively sectioned the whole thing. It took weeks, and it was quite expensive to do H&E and IHC down thousands of adjacent slides. But the amazing thing is this is the only way that you can exhaustively map all of these STIC lesions in a whole fallopian tube. And with this, we could then go back and synthetically sample that fallopian tube and cut 2D H&E images and estimate how likely is it using the existing clinical pipelines, they call it SEFIM, it's the way that the fallopian tube is sampled in the clinic. How likely is the SEFIM technique to catch these STIC lesions that we could find in this automated mapping? And we found that the 2D sampling of the fallopian tube exactly explains the fact that we miss these lesions in the clinic. Not to say that in the clinic, they should be exhaustively sectioning tubes because the goal of the clinic is not to find these small indolent lesions, it's to identify the active STIC lesions that can become invasive. But the point that we make with this is that you cannot use the same methods to diagnose patients as to come up with these incidences, because what we show is 2D is insufficient to find all the STICs. In fact, it's likely that all healthy women have these indolent STIC lesions.And that methods such as this exhaustive mapping may be needed to come up with those estimations. Okay, so I want to now talk about the types of imaging we use and then get into the foundations of those technologies and how the use of them has changed with application of AI. So I think like you saw in the last talk, the main methods applied to image tissue can be split into sectioning-based methods and intact methods. While today we think of these as quite visual fields, you can see all the videos on these slides, they actually have foundations dating back all the way to the 1850s. If we split them in two, we can think of them as intact imaging and sectioning-based imaging. If we take one as an example, an intact imaging method may take a block of fixed tissue, say a clinical FFPE block, and apply index matching solvents to make the optical densities alike within that sample, apply specific antibodies to label specific components of the block, and then image them. Advantages of this include that you can tag specific cell types that you want to see. You can image all the way down to single-cell level and even within the cells, and you maintain continuity. The fact that you don't cut these blocks means you can image, say, collagen filaments and quite subtle anatomy within the tissues. One of the biggest downsides, I would say, is that multiplexing is quite challenging. Say if you see something amazing within this block of tissue and you want to go back and apply something new to it, like xenium, you would have to take that block, go back and cut it in the clinic, and hope that you can identify the slide that had the entity you wanted to see. On the second hand, we have sectioning-based technologies that you could take the same block of tissue, section it the same way as done in conventional 2D pathology, and then apply a sequence of techniques to each of those sections. Advantages include that you can quite easily apply multi-omics to this. You can do cyclic IF, H&E, conventional staining techniques, as well as today's advanced and expensive multi-omics. While some of the disadvantages can be that the act of cutting the tissue can exhaust clinical blocks. Also, that the sectioning is quite challenging and time-consuming, but at the end, it's one of the only ways to effectively multiplex that we have today. Okay, so if we think of the tissue atlas community, a lot of us think of it as something that just developed. But what I want to show you is that the foundation of what we do even to today and the questions that we ask dates back millennia. The field of anatomy, people think of it today as having begun in ancient Egypt. The Egyptians scientifically thought about how to dissect the human body to develop methods of mummification. Even histology, hematoxylin, which is a plant-based dye, was used dating back to the 1750s. I was able to find studies on tissue sectioning, multiplexing, and index matching, especially applied to developing tissues, dating back to the 1860s and 1870s, as I'll show. And then even what's changed since then has been imaging. You might think imaging is one of the latest things we've done. You can find light sheet imaging studies going back to 1911, CT imaging back to the 1950s. So as I'll make the case today, the biggest thing to me that's changed is AI and quantification. So on the left, you see a study talking about tissue sectioning. This was published in the 1880s. It was the development community that actually came up with a lot of the technologies we apply today. And in this study, what they wanted to convey is that in a developing mouse, the angle at which you section the tissue changes the things that you can study. So if you cut sagittally, you can see specific angles of specific tissues that might help you in studying, say, the GI system. And next to that, you see a study that was some of the initial foundations of light sheet imaging and was giving the best list of index matching chemicals to apply to match the optical densities in eggs in developing mice. And what I find so fascinating about these studies is that even though they ask a lot of the same questions that we ask today, today we think of this as quite a visual field. But a lot of these studies, this was the only image in the whole study, and it's because it had to be sketched by hand by the scientists. So even though they likely looked at the same types of thing that we look at today, it wasn't as communicable. So a heavy focus of these studies was on explicitly detailing the methods that they applied so they could be duplicated by somebody else, because it was the only way to convey what they saw, because the imaging and the quantification was missing. This example I specifically love. This was published by someone named William Johnson in 1899. He was in the anatomy division at the Johns Hopkins Hospital. And what he did is he took a block of human kidney tissue and sectioned all the way down the block, and then sketched to himself the functional unit of the kidney called the glom. It's a bundle of blood vessels. And he took those sketches, and then he built an upsampled wax model of what this looks like. And then he sketched that wax model and published it in "The Johns Hopkins Magazine." Next to that is a study that's almost finished by a student in my lab named Lucy DeQuette. And what we did is we took a block of human kidney tissue, and we sectioned all the way down the block and stained it with H&E. And instead of manually outlining these functional units, we labeled them with AI. But you can see we asked a lot of the same questions. The thing that I think distinguishes what we can do today and what was done by William Johnson in 1899 is that we have imaging and we have quantification. But I think it also highlights one of theThe easiest fallacies that we can make, which is that even though we have a lot of advanced imaging techniques today, we need to watch that we don't just ask the same questions that someone could have asked in 1899. So to go beyond just anatomically mapping things that maybe someone mapped by hand a long time ago. Okay. So as I've shown, a lot of the imaging and the technologies and the tissue sectioning and staining is quite an old field. What I believe has changed is computing, imaging, and quantification. So especially with advances in GPUs with NVIDIA in the 2010s, the AI boom of the 2015s, and today, the fact that we have these LLMs and even agents that can begin to just do it all so we don't even need to code. To me, while the imaging and the chemical methods have existed a long time, it's this quantification that has allowed tissue atlasing to become as big of a field as it is today. And in fact, AI helps us with many specific methods as I'll get into, what types of models do we use and what specific questions can we ask with these models. So just to give a few specific examples, open-top light sheet imaging, as I've shown, is an effective means to image whole pieces of tissue. So this is again a needle biopsy. And this can be a way of looking at H&E down a continuous stack of slides, but with computing methods, you can take this-- This image actually didn't look like H&E. It was a model that was applied to make a synthetic H&E image on this. On top of that, segmentation models can be applied to segment the glands inside this tissue. As a second example of a sectioning-based method, thick tissue sections can be imaged when, say, an individual nucleus as you saw in the last talk, when someone wants to study the whole body of a cell, that can be missed in an individual H&E slide. So one of the advantages of these thick tissue sectioning methods is that you can see appendages of nuclei and begin to quantify those. This is something that maybe someone would have been able to see this back in, who knows, the 1910s, 1920s. They wouldn't have been able to image it until the 1950s and '60s, not at this scale, say, until the last decade. But they wouldn't have been able to quantify it until maybe the 2020s. And finally, the CODA technology that we use at Johns Hopkins uses the foundations of H&E staining and imaging with AI applied to those H&E images to automate and quantify what pathologists have looked at visually the past decades. And in this, we apply segmentation methods as I'll get into. We can also apply multiplexing to look at inflammation down blocks of tissue. So in this slide in the video spinning, you see a lesion called panin that develops in the pancreas, and we can apply IHC staining to map immunological hotspots and cold spots on these cysts. Additionally, because we have so much adjacent slides, we can go back and apply DNA sequencing to combine the anatomical mapping that we can do with genomics to show that all these independent cysts that we find in the same block of tissue develop independently because they have independent somatic mutations. Okay, so what AI methods do we use to quantify these images? I've shown a lot of videos, a lot of applications that we do, but specifically, what AI methods do we use? I'll quickly go into how we use segmentation and at many scales, but using a lot of the same model backbones, how we use pathology foundation models, and how we do synthetic staining and the main benefits of these methodologies. So to begin with segmentation, nucleus segmentation is one of the smallest scale segmentations people apply to tissue images. This is a type of model in which the nuclei of cells in H&E and IF and tissue-based imaging methods can be outlined. It's something called instance segmentation, that not only do we label each nucleus, but we know which nucleus it is. And this is helpful because we can use this to detect cell type. So up along that wavy line in the image, you see the epithelium. This is the whole gynecological system of a mouse, and the epithelial cells can be distinguished both by shape and by location in the slide. Immune cells, say, just behind the epithelium, can be distinguished in H&E using the size and the intensity of the nucleus. Multiple methods have been published to do nucleus segmentation as well as GUIs, applications that make them easy to use even by people who don't code. A second type of segmentation, not at the nucleus level but moving up a scale in anatomy, is tissue-level segmentation. So this is an H&E section of developing human kidney, and in H&E alone, we can label 18 cell types using segmentation. This is a method that typically involves a human manual annotation that can be used to fine-tune a model such as a CNN, a VIT, and today a lot of people use the Segment Anything Model developed by Meta. And finally, going up to whole body scale, segmentation has also been applied to non-invasive medical images such as CT. So in the example on the left, you see things like the kidney and the GI system segmented in CT imaging. And on this side, you can see a glioblastoma automatically segmented in images. So what all of these methods have in common, the cell-level, tissue-level, and whole body-level segmentation, is even though they look quite distinct, all of them use the same model backbones. They all use the same CNNs, the same fine-tuned models, and the same initial weights fine-tuned on ImageNet typically. A second model I want to quickly talk about is pathology foundation models. The idea of these models is to get as close to doing what a pathologist does as possible. So these models typically don't need fine-tuning. You can download them and use the existing weights and apply them to many tissue images of many tissue locations and many applications. So what these do is they take a H&E slide and cut it up into small tiles, and then out of each tile they make you embeddings, which is just a fancy way of saying a long list of values that you can use to do a lot of things with. You can fine-tune models to estimate patient outcome, you can segment tissues, and you can do a lot of additional applications.As an example of how pathology foundation models have been used, we've been developing this unique set of tissue at Johns Hopkins of about 1,000 patients with panNET, a malignancy in the pancreas that develops out of the islet cells. And we've applied pathology foundation models both to imaging and to text to pull data out of the clinical notes that pathologists look at, as well as the H&E images. And we can use this both to identify automatically how long patients will live, but also to locate the location on the slide that tells the model the patient's outcome. And it's able to tell us that the venous invasions, as well as inflamed locations indicate patient outcome. Just to skip ahead a bit, synthetic staining, as you also saw in the last talk, is an effective means of taking advantage of a lot of the latest imaging methods. So what you see all the way on the left on this slide is called quantitative phase imaging. It's a way to quickly image thick tissue blocks, but it gives you this image that doesn't look like anatomy to a lot of people. So using something called a GAN, a model that competes against itself to develop a synthetic image, this team was able to make these quantitative phase images into H&E, which allows them to now apply the many existing models that can be applied to H&E images to this quantitative phase imaging. A second example is a diffusion model developed by my team and just posted online this week, in which we took H&E images and applied a conditional diffusion model to make H&E into synthetic multiplex IHC. In this case, applying this model to the islets in the pancreas to make H&E images into insulin and glucagon-stained tissue. Why this is special is because now with H&E image alone, which is inexpensive and exists in all the hospitals in the US and in pathology, we can take H&E and make it into this expensive immunolabeled image, in this case, to study development of type 1 diabetes. I'll skip this. And the final type of AI I want to talk about, in this case, is called optical flow. If we have individual tissue sections, how can we fill in the gaps between missing slides? So in this application, if you think of Z sections, Joshi et al. took this data set and could synthetically make the missing slides in sectioning-based methods. So say in a technology like CODA, in which you use H&E and skip some in-between slides, you have missing anatomy. So how can we use AI to put back synthetic images to fill in the gaps in those tissues? And they showed that this can be applied to H&E, but also to imaging methods such as IHC, CT, and light sheet imaging. And finally, if we think about the multi-omics community, technologies like Visium allow us to see deep into tissues to look at the gene expression of cells. But a lot of these methods come with spot-based, so they have gaps in between the spots. And this method, called iSCALE, used a VIT with an MLP head to fill in the gaps between the Visium slides. So allowing us to take an image which was incomplete and make a continuous slide with AI. So in conclusion, as I've said at the beginning, the tissue atlasing community, it's booming today, but it's actually founded on old ideas that date back to the 1850s. What AI is allowing us to do is to help us quantify, to visualize, and to combine multi-omics in new ways, and to link that to clinically meaningful data. And as we continue to use AI, I think we'll be able to go beyond individual planes and think about how to combine these modalities. Thank you. Yeah. If anyone has any questions, please come up to the mics. Hello. Two excellent talks that just blows you away. Very practical problem when you're working with clinical material. In this case, we focus a lot on prostate needle biopsies. The pre-analytic variables that you can't control the way you can if it's a mouse model, how do you handle things like the time between the recut and whatever molecular or immunohistochemical or immunofluorescence analysis that you're doing? Do you have sort of boundaries, or do you have storage conditions? What do you find is the most practical way to deal with that issue? Yeah. It's a good question. I do think it's one of the biggest challenges with analyzing clinical tissue is you often don't know. You don't know how quickly was this tissue fixed. You don't know, was it left in the sun? A lot of the guides you see online of applying, say, Visium, they estimate how old of tissue you can analyze. But I've seen applications that we take 1990s tissue, and the quality is beautiful. And we have some tissue collected a month ago that we don't know why, but it's- Awful ... unusable. It's awful. I think so much of it has to do with the pieces of the pipeline that we can't see. And it's difficult to solve, to be honest, unless you had the whole pipeline kind of monopolized. So what do you do on a day-to-day basis? You've asked for blocks. You've asked for recuts. Do you want just to move into whatever your analytical method is within days, within weeks? What do you find to be something that you can do routinely? Yeah, I would say with my own applications, we often focus on uncommon diseases. So it's impossible to say, "Okay, this tissue's no good. Let's wait till the next one comes." Because we maybe need to analyze the 20 patients that have been seen since 2000 with this disease. So it's not so much about how can we change the way we collect tissue now to make it all as good as possible, though that is also something that needs to be done, but also how can we salvage these unique and invaluable samples that have been sitting since the '90s, the '80s, and what can we still get out of those today? Okay. Well, thank you. Hey, Andy. Yeah. Imagining us going into this general discussion session. You got a question? Thank you very much. I was going to-- Totally agreeing with you. I don't think we have the solution to the pre-analytical variables. What I would say, and it would be interesting to see if Ashley says this, one of the beauties of the H&E reconstruction or even the synthetic is it's actually pretty robust. So we too go back- H&E is tough. Sorry? H&E is very robust. It is. And we also find that protein imaging, we've done-- Some of the sections that I showed there are 15 years old, right? That we have to face the block to get off the oxidation. But proteins are robust. The biggest difficulty is in transcriptomic profiling. And typically, less than two weeks from sectioning is a requirement. So I think it's why this sort of emphasis that everything will become transcriptomics is sort of unrealistic. What we found, and it's interesting, here's the same, we found about a third of specimens in a cohort that we collect within a surgical nurse setting, so it's a little fancier. A third of them have pre-analytical variables that we still haven't managed to fully understand. And so in those cases, we do have methods to try to get at least minimal information. We've published some of those, too. Well, thank you. Well, thank you so much for a wonderful talk, Dr. Creamer. So I have a question that comes from a problem that I'm facing and the project that I'm currently working on. So even I work on pancreatic cancer, and so we know that there is a lot of heterogeneity in the morphology of PDAC. There are well-differentiated glands, and then there are diffusely infiltrating tumor cells. And it is only plausible for us to imagine that these two morphologic features have very distinct biologies. So I was wondering if you do have a solution to segment these two units in the time of image processing even though they have a similar biomarker profile. For instance, I would expect to see pan cytokeratin in these diffusely infiltrating tumor cells as well as the well-differentiated glands. So even when they express the same profile of biomarkers, how do you morphologically segment them based on AI? Yeah, that's a good question. I would love to talk to you at the end about this, too. We actually have an ongoing study looking at some of the same questions, and how we've been distinguishing them is by size and continuity. Okay. That we think of these big tongues of invading glands as kind of collective epithelial-like invasion. But then we can also distinguish this mesenchymal-like invasion when you have just a few clumps of cells invading by themselves. And in H&E, they honestly look almost nothing alike. Even cytologically, the nuclei don't look that alike. But the biggest, cleanest way that we can distinguish them is just by connectivity and size. How many cells is in this collected unit? Got it. Thank you so much. One-- Ashley, if I could. Why don't you sit at the- We can do the joint questions. We can go to the joint question session, if that's good. We have quite a while. So anyone, if you also have ideas, please feel free to speak up. We have-- Because we-- No, it didn't come. Yes, we don't have the video. So we have 20 minutes to engage in a broader discussion. I was going to say, one thing that-- I showed a picture of a bunch of T cells of different shapes, and they sort of seemed to be sitting in some landscape. That actually was similar to what Ashley described. It was actually an UNI model embedding. And what's interesting is these things are just visually totally different. These machine learning AI models can neatly discriminate them. A human can discriminate them, but none of the morphometric features that we've used in the pre-machine learning era are any good. Like, is it prolate? Is it ellipsoid? How long is it? How fat is it? And so to some extent, there's been in your history there, 30 years of tissue morphometrics that have been very disappointing. And I think what's sort of exciting now in these quantitative analyses coming out of machine learning is they're able to-- They're incredibly intuitive, if you've ever tried them. They're able to recapitulate classifications that you would visually agree with and then extend beyond that. And so I think that's a very big step forward. And one setting that I've seen is in melanoma diagnosis. I'm a PhD, not a diagnostician, but if you look at the AGC7 to 8 manual, we lost three molecular features. So the number of lymphocytes is no longer a diagnostic feature, and the number of proliferating cells is no longer a diagnostic feature. And the speculation is not that those are not important for cancer, but it's just extremely difficult for histopathologists moving quickly to quantify those types of features. And so that would be a case where the model acting like a pathologist would really be supportive. And of course, that's how we do Pap smears. So we know in a sense that it will work. So I think you have the right problem to bring up. Yeah. Thank you so much. Nice talk, Peter and Ashley. I was thinking that, have you ever tested a real-world case where the pathologist may have a disagreement or it's very subjective interpretation, and see how your AI tools can help make it more accurate? Is this mic on? Yes. Okay. Yeah. I think I have a good application that we've done just that, and it's not so much a question of beating the pathologist. It's not testing, saying a pathologist is not good. I think pathologists have shown a shocking ability to link what they can see visually in a slide to patient outcome. But we have one ongoing study that we wanted to ask, how difficult is it to identify some things pathologically? So we took thick blocks of human tissue and cut about 50 sections each, and specifically found blocks with something called vessel invasion. So it's the beginning of metastasis when cells invade into a blood vessel. And diagnostically, it can be used in staging. If pathologists see cells inside the vessel, it means the patient may have metastatic disease, even if it wasn't found on imaging. And what we found is if we isolate a vessel invasion in a block of tissue and we collect all the images below that moment following the blood vessel, only about half the time can a pathologist definitively call this a vessel invasion. Because in so many instances, it just doesn't have all the diagnostic qualities to call in the clinic, even though we know it exists because we can follow it continuously. And I think it shows not that pathologists should be able to identify this because, say, if the endothelium of the vessel is missing, it's impossible in that slide. You couldn't do anything about it. But can we now use AI to identify something else holistic in that slide that a pathologist could use as a cue? Yeah. No, I agree with you that sometimes the diagnostic features could be missing, but sometimes they also could be extraordinarily difficult to make a call. We've had an example recently where we were looking for a structure called multinucleated giant cells in the lung that show up as they're a fusion event for macrophages and epithelioid histocytes. And it's tricky to call because it's a three-dimensional structure. By the way, all the tissue profiling technologies throw it out because it's a cell with many nuclei. And what was interesting, we had three pathologists annotating for us so we could get an inter-rater reproducibility, and then we had a foundation model scoring, actually learning from them iteratively. Very much as Ashley said, I think it was difficult to score in many cases, but what was really interesting was the model and the humans, the three humans on actually three different countries, came to a much better understanding in the end by working with the machine. So that was actually really interesting. We ended up in that group with a pretty good consensus view that neither the machine initially had achieved because it has hallucinations, et cetera, or the individual humans had done by annotation. So I also think there's going to be a way in which this will help as a kind of teaching tool, because as it exists now, as you probably know, is you get 10 negative slides and 10 positive slides, and you train on that. This, I think, will be a much more interactive way to do it and actually be very exciting. Yeah. Thanks, Peter. Thank you. That was a very good presentation. I'm Bob Edwards, Pittsburgh. I'm a surgical oncologist who has a lab that takes out a lot of tumors and fallopian tubes and infiltrating lesions looking for CD8 infiltration. And what I've been struck by is there's three major dilemmas that we have. The lack of correlation of the STIC lesion with the development and bulk of ovarian cancer, the inability to characterize CD8s in tumors that have involuted, and then the presence of TLSs and whether or not they're prognostic for a given cancer. So I know these are big, broad questions, but when I was looking at the AI reconstructions, I was thinking maybe the way we're choosing to do our sectioning now reflects our inability to understand the three-dimensional relationship. And are there-- This is a little off what you were talking about, but it was inferred. Maybe the approach to processing the tissue needs to be changed to look more carefully at what it is we're actually sectioning. Rather than doing serial sections, maybe we need to do targeted sectioning. I don't know. I wonder what you thought about that. Yeah. No, I think it's a good question. Especially as I think in the tissue atlasing community, both within fallopian tube and any place in the body, in the skin, in the GI system, people have been thinking about these lesions that we all get as we age. And as we've now been pushing so much money into the field, I think we find that all of us develop so much as we age that just stays benign. If we exhaustively looked at a fallopian tube, we find STICs even in-- One of those women was 25 and we found two STIC lesions. I think it shows that what pathologists see in the clinic with techniques like CFIM is the lesions big enough to be caught by that kind of even sampling. And it's not to say that the small lesions need to be found. Maybe they don't if it's likely to stay indolent. But does some technology exist that might not be pathology, like tubal washing, something else that could be done that could know if you have a STIC lesion, but maybe you don't need to know the location? Because we now know we all have these, so how do we identify the active ones? Yeah, no, that's great. I think probably both of us would be happy to work with you on some of those surgical specimens. We too are looking at STICs.The other thing to say on that is, so we have noticed actually in the example I gave of TLSs, is they're actually, if you got a single section that didn't go, for example, through the germinal center, and you were going to ask yourself, could you determine that that was a true TLS as opposed to a loose aggregate? And the answer with some precision is yes. So there are features that we don't currently evaluate in 2D sections that are sort of like Plato's shadow there, sort of informative. And I think that's the sense that this kind of 3D data sets will help us become a ground truth. Another example is we noticed in the paper that actually Ashley showed, Lin et al., we actually looked at the siculation that's seen often at the leading edge of a colorectal cancer, the budding it's called. And these have been interpreted as little spheres of cells that are part of the metastatic process, but that's not correct actually, they're the ends of fingers. And so, again, as with the colon, that view and that view have been interpreted as totally different structures, but they're simply the two orthogonal views. So I think all of those things are going to help us interpret the 2D data. One other thing we haven't talked about very much, by the way, is cost. So, one of the things that we've been doing in 2D 20-plex imaging is trying to push the cost down to $500 per section or lower, and I think that's going to happen. The H&E methods are really quite inexpensive. And so I think the moment you want to scale or apply in your setting, they're going to have to be inexpensive costs. I think doing $15,000, $18,000 per slide, exciting though it is, is really very limiting. So, the cost aspect of this is a big deal, because it really limits the science and medicine that can be done. And I think that also shows the value of these synthetic staining methods, that if you can do the highplex labeling and the H&E on the same slide, if you can build, say, an H&E to a multiplex model, now you can go back to all those clinical H&Es that have been banked, scan them all, and then do one of these multi-omic studies at scale. I think it's the only way, is no one's going to have half a million to stain a thousand patients' tissue, but can we use AI to scale up these methods? I wonder if in the future, the biggest barrier we're going to face is, beyond the cost of doing it, is going to be a kind of ethical barrier, right? Our hospitals are all holding onto these images now, both because they see profit, but I think they're concerned that big for-profit companies are going to exploit them in ways. So I think we're going to have to think carefully about how we can ethically, as we have done with the Human Genome Project, share this type of data. Yes. Hello, Andreas Boser from Milton here. So we have recently outlined and published a method how you can do the clearing of a material, and then afterwards sectioning it, and then do a multi-omics approach, let's say, with hundreds of antibodies on the same section from this part. But due to the procedure of the clearing and afterwards the rehydration, we still obviously have distortion of the tissue. And now we are still struggling with the part, how can we best superimpose the optical sections with the physical sections after the maximal analysis, let's say. Is there a good method, software, which you would suggest how you can best get rid of this morphological distortions to better superimpose both? Yeah. It's a good question. I can give you my best attempt at a solution that we've been employing. I've been helping link quantitative phase imaging to H&E, that it's the same issue, that you have the H&E doesn't exactly line up at the whole slide level to the phase imaging. Excuse me. And so what we've done, if you think of a lot of the synthetic stain models, they don't need the whole slide to be flawlessly aligned, they need high-quality tiles. So the solution we have, as inelegant as it is, is a tile-based alignment, is let me take this piece of the H&E and align it to the image, and the next piece of the H&E and align it. And as long as you do that kind of systematically one by one, I have found that it's one of those times that a manual step gives a lot of value at the end, is having a student look back at 10,000 matched H&E and IHC images and tossing out the bad ones, it adds so much quality to the model. Even if it's a diffusion model, a GAN, anything, the quality in is quality out. There's obviously also Ashley's CODA method. For lung reconstruction, we've actually found we've had to do more non-rigid alignment as you do, and we've even been using the ANTS setup. So that's actually used for tomography in CT imaging, and it's got a great toolbox for doing nonlinear sort of morphing. The trick there is you got to segment everything first, because once you morph everything, at the single-cell level gets a little screwy. But the tools we have, we'd be happy to share with you. But they were inspired by the tools you built, so. The methods we were using is called ANTS, A-N-T-S. You wouldn't have heard of it, but it's universal in CT imaging, in experimental C. So there's a whole series of tomographic alignment tools for 3D. If you write to us, I can show you what we've done. Thank you very much. Christoph Cooper from Aachen, Germany. Also, really, I'd enjoyed the two talks. I have two very different question. The first one is more related to the sectioning. So sectioning of 35 micrometer FFPE tissues, I think is quite tricky. I would really love to learn how you do it, and particularly if there are any artifacts in the 3D tissue that you induced by this type of sectioning, if you've controlled for that. Another question is related to the 3D data that you presented and deanalysis. So most of the time what I saw is that you mainly use2D data analysis types, and then, the Uni2 model is trained on 2D sections. So obviously, of course, a foundation model in 3D doesn't exist yet maybe, but particularly because you mentioned connectivity, what would be other approaches maybe to analyze the 3D structures that you described? Why don't you take 3D and I'll do 35 microns? Yeah, it's a good question. And to be honest, we typically have been applying foundation models 2D image by 2D image. I think the CT space is a good place to look to see what might change moving ahead. That as opposed to taking individual 2D tile inputs, they go cube by cube, so you have that Z context in mind. I think it would be amazing if somebody could develop a foundation model that could do the same. But you can imagine the imaging they would need as an input isn't something that a lot of teams have. No, I think one thing there is it would-- unfortunately, with all the disruption that's happened, is there is no good repository for these data. So like you, we put it on S3 and we let people download, but that's expensive. Neither in Europe, which frankly is far ahead of the US in a lot of these imaging things, but nothing at the NIH has been sustained actually enough to make it possible. Because I think if you had all these data sets, it would inspire groups to come in and build that. Let me briefly describe-- Happy did the protocol for 35, we can show you. It's interesting, you cut it, it curls a little bit as you would imagine, depends on the wax. We lightly relax it with a little bit of heat, and we float it in superglue. So in fact, we superglue all these things down, which seem completely insane, but was invented-- or not invented, was developed by a student in my lab. And so actually there are glues to hold it down. And you can actually do all sorts of tricks, it turns out, with glue and histology. So, UHU might be the glue you would use in Aachen. Thank you very much. Yeah. Again, the protocols are online, but happy to give you more. Peter, I was interested in the light sheet microscopy that you were looking at, and you just mentioned that it can go as deep as a millimeter in thickness. How deep do you think in the future you could go? Because examples like melanoma invasion is commonly more than a millimeter in Breslow's depth. Do you think it could go deeper than a millimeter? So it already does, right? So there were two issues there. One is there are many types of light sheet microscopes, right? And the deeper you go, even with very good clearing, the less likely you are to be a true single-cell resolution. But the real issue for us is we were trying to preserve immune cells, as you would be interested as well, I think. And so we don't do delipidation. So clearing typically uses strong delipidation and then refractive index matching. And that's how in a mouse brain you can go a full centimeter or more depending on the instrument. So we found with non-delipidation-- So delipidation, there is the occasional CD marker, immune marker that will survive, but most don't. So we've gotten about 20 of them to work with a non-delipitating clearing method, and that practically is limited to about two millimeters. So it's all in the chemistry. So there's every reason to think that it can be extended. There is the additional issue there that we're doing cyclic imaging, so we are trying to push the five-color limit that Ashley mentioned. We've gotten up to 20 colors, and that brings in these distortions, et cetera. But it's all engineering. Those things are just engineering solutions. I think fundamentally, the tissue is capable. Fundamentally, the technology is there. So, I think it's completely solvable to get the entirety of an invasive stage 2B melanoma. Is that what you're looking for? And another question, unrelated. Where does the pathologist fit into this equation? Because they're used to looking in two-dimensional space. Is it to do with education of what the tissue now looks like in three dimensions to then educate the pathologist to maybe use this for future diagnosis? Yeah, I think it depends on the specific study. We do have some studies in pathology education. The vessel invasion one that I mentioned that we want to build an atlas of vessel invasions and make them all public so that pathologists can see easy cases that I could teach you in two minutes what a vessel invasion looks like. It looks like this. All the way down to the challenging cases that maybe none of the pathologists that we tested them on could identify, but we know based on continuity that it is. I think that's an instance when ideally this type of data could be fed back into the medical education system to educate pathologists. A second case is, like you mentioned, linking pathologists to the AI and seeing can the AI model teach pathologists something that they could use in the clinic. So in the 2D foundational model application that I showed today, we've been taking text inputs and image embedding inputs and seeing how well does a text-only model do when it's looking at extensive pathologist notes. So how well do pathologists do today? And then next to that, we have an image embedding model that does this beat the text model? And if it does, can we make back what they call attention heat maps so the model can tell us what location in the slide did it look. And if we can see something consistent about what the model is looking at, can we give that back to pathologists and say, "Okay, tell me now, what was the model looking at? And is that something that you could begin to make note of in the clinic?" And in fact, in this application, we find what pathologists call the leading edge, and depending on the shape of that leading edge, it's indicative of the outcome. Which is something pathologists kind of subjectively know, but they don't make note of it today in the clinic.So this is a place that we can have that feedback loop. Fantastic. Thank you. Yeah, let me just add to that. I think, first, you're not a pathologist, and I'm not. So in my lab, this is known as path-splaining. It's a bit like mansplaining. It's non-pathologists explaining pathology. But our pathologists, everything we do engages them as it does at Hopkins. But that's not the norm, right? I think out on the floor, if I could be a little controversial, it is seen as all a branch of genomics, right? Because of the spatial transcriptomics. So I think really figuring out how to engage them. One difficulty is that pathology departments-- So we work with the Brigham and Women's pathology department, which was among the most storied in the United States. And in modern American medicine, it's becoming a service center rather than a research center. It's all about revenue value units. And so I think re-engaging that community in research should be an incredibly high priority from a kind of national funding priority, because you're just not going to understand what we've shown today without people, without surgeons, without oncologists, and they're not in the room. It's bewildering to me. Absolutely. You need pathologists on your side to get this. Agree. Yeah. And the radiologists and the surgeons, right? But it's not how our funding in this country is currently constructed. So maybe we'll take one final comment or question, because we should wrap up. Just fantastic tools and fantastic pictures and looking at biology and everything, but I keep getting back, maybe it's from too many years of tumor boards, of false negatives, false positives, sampling errors, and gold standards , particularly for imputing data and that sort of thing. And I'm kind of fuzzy where this is all really going from that standpoint. Let me just say that it's a fantastic thing to bring up, right? I think there is quite a bit of trying to develop analytical accuracy in this. It's coming along more slowly. This isn't as well-developed as laboratory medicine from that perspective. One thing we have actually, in our own work, been much criticized for is sort of looking at the kind of questions you showed, which are really diagnostically difficult. Like what is the transition from abnormal morphology in the skin to the first melanocytic lesion? Where would you make that call? Where in a breast cancer have you got the first DCIS? I don't think either of us imagine that we're answering those questions yet. What we're trying to do is put not just a bunch of tools together, but put together the conceptual foundation of addressing those tools reliably. So if imagine right now exome sequencing is a clinically practical tool that is used universally, there was a lot that had to come along, discovering all the driving oncogenes, developing the methods, and I think we're sort of in that early phase, developing the methods. I do believe, as Ashley has alluded to, that bringing a more quantitative analytical framework that's still human or physician in the loop, is going to be incredibly powerful in this field. Because AI is nowhere is it more powerful than in dealing with images. So see this as a clinical application coming along. Having said that, very simple things, we can now count mitotic indices with AI very quickly. We can fully quantify tumor infiltration using these tools. So there are some simple things that are ready to go, but all the issues you brought up, as with the question of pre-analytical variables, they haven't gone away. Yeah, I listened to an excellent talk a while back by Elliot Fishman on AI and CT imaging, and how he put it was, and we can make the analogy to pathology, but the goal of these AI tools, it's not, can we eliminate the pathologist and can we do what they can do so we don't need them? Especially at academic hospitals, you can find a specialist in anything who's going to today absolutely beat any AI that we can develop. But as an example, my family lives in Wisconsin in a township. It has a population less than 1,000. So when they go to a hospital, they see a GP who's not a specialist in anything they have. And if my dad has, say, a skin lesion, can we get a Johns Hopkins pathologist's mind and embed that in the AI and give those tools to these small-town hospitals that so many people go to? And it's not going to make the diagnosis, but can it flag his case and say, "You should look at this a second time. You should maybe suggest this patient go to Mayo Clinic," and help take the academic hospitals down to these small towns? It's interesting. I was buttonholed by a bunch of pathologists who are now all retiring, and have studied leprosy and these rare tropical diseases, and those diseases, when they were more prevalent in the Western world, everybody went to the London School of Tropical Medicine to be trained. Nobody's being trained to diagnose them anymore. And so in fact, we're behind where we were in the 1950s. I think those are things that could also probably be embedded. Yeah. Well, thank you everyone. Thank you, Ashley, for a fantastic talk.