CS50 Video Player
    • 🧁
      • 🍬
      • 🍉
      • 🍿
    • 0:00:03DAVID MALAN: Well hello, world.
    • 0:00:04My name is David Malan, and this is the last of CS50 seminars
    • 0:00:08here at Harvard in fall of 2022.
    • 0:00:11As you might know, we've been filming live the class again.
    • 0:00:14And all of this past semester is lectures
    • 0:00:16and now seminars will end up on edX and YouTube
    • 0:00:18and other platforms on January 1 as part of what will be called CS50x 2023.
    • 0:00:24In this particular seminar, I'm so excited that we're
    • 0:00:27joined by Andrew Sellergren, an alum of Harvard who actually took CS50
    • 0:00:31the very first year I taught it, for better or for worse.
    • 0:00:35I think it's fair to say that surely everything he knows
    • 0:00:39can be traced back to me, it would seem.
    • 0:00:41But that's actually not the case because if you read the description for today's
    • 0:00:44talk on applications of artificial intelligence or AI in medicine,
    • 0:00:49Andrew has been amazingly self-taught.
    • 0:00:52In fact, when he was in college here only
    • 0:00:54took CS50 but then went on nonetheless to work for Google in various roles
    • 0:00:58since.
    • 0:00:59Only just a few years ago did he dive into and teach himself
    • 0:01:02the world of machine learning, which he now
    • 0:01:04applies to real-world problems in medicine that are so incredibly
    • 0:01:08impactful that we're so glad that he's here
    • 0:01:11today to share not only that same trajectory, but a little bit
    • 0:01:14about what he now does at Google.
    • 0:01:16So without further ado, here is CS50's own and now Google's Andrew Sellergren.
    • 0:01:23ANDREW SELLERGREN: Thanks.
    • 0:01:24Thanks a lot, David.
    • 0:01:25That was a really nice introduction.
    • 0:01:28You said you would be funny, but I guess you didn't come through on that.
    • 0:01:33No I really I know he exaggerates, but I do honestly trace a lot of this back
    • 0:01:41to CS50 in particular.
    • 0:01:45Let me go ahead and start sharing, and I'll kind of tell my story as well.
    • 0:01:50OK, awesome.
    • 0:01:54Great, yeah, I'll start by welcoming everyone and, thank you for joining.
    • 0:02:01It's really exciting to be here, and it's really, really cool
    • 0:02:05to see all the faces from around the world.
    • 0:02:07That's been a real joy of mine to watch as the course grew.
    • 0:02:12So as David mentioned, I took the course for the first time.
    • 0:02:16I guess I should, I am going to date us by saying this, but it was 2007.
    • 0:02:22It was my senior year in college, and I was a pre-med.
    • 0:02:26I was getting ready to go to medical school.
    • 0:02:29And my last year at Harvard, I was taking courses
    • 0:02:34that I wouldn't normally take just for fun and CS50 was one of them.
    • 0:02:38I actually already knew David from a previous venture
    • 0:02:43and was really excited that he was going to be teaching it.
    • 0:02:46So it was everything I wanted in terms of getting
    • 0:02:51me excited about a new whole set of knowledge that I didn't have yet.
    • 0:02:58And I wouldn't say there's not like a lightning strike
    • 0:03:03there, where all of a sudden I decided I wanted to be a software engineer.
    • 0:03:06I won't tell the story that way.
    • 0:03:08But it was the beginning of a lot of lifelong learning.
    • 0:03:13And that's the advice that I can give that is probably
    • 0:03:17best for anyone is that as long as you are continuing to learn,
    • 0:03:23challenge yourself, you're excited about what you're doing,
    • 0:03:26you don't necessarily need to concern yourself
    • 0:03:28too much with what that makes you whether that
    • 0:03:31makes you a software engineer or a scientist or an artist
    • 0:03:36or what have you.
    • 0:03:37It's really just about a skill set that you're gathering.
    • 0:03:41And that's really what the start of it was with CS50.
    • 0:03:44And I don't exaggerate when I say that that was what gave me the confidence
    • 0:03:51to learn all of these things.
    • 0:03:53So CS50 is in large part about learning the fundamentals, the programming
    • 0:03:59languages that you might begin with and some
    • 0:04:01of the very foundational concepts of it.
    • 0:04:04But it empowers you to learn anything you want after that.
    • 0:04:08You can always trace back and say, well, I remember this concept
    • 0:04:12from CS50 or yeah, I got stuck at this problem set,
    • 0:04:15but then I worked my way through.
    • 0:04:17And that's been invaluable.
    • 0:04:19And to be honest, I think it continues to serve me even today.
    • 0:04:25There's just, there isn't anything that I try to limit myself
    • 0:04:28and say, oh, I can't learn that or oh, that's not
    • 0:04:31what I'm supposed to be learning now.
    • 0:04:33It's really just something that I'm always challenging myself with.
    • 0:04:37And CS50 was a huge part of that to begin with.
    • 0:04:40So the course itself has grown tremendously
    • 0:04:44since then, which is also really exciting.
    • 0:04:48It was still really cool then, but it's gotten a lot bigger.
    • 0:04:52Oops, OK.
    • 0:04:54So today, I'm here to talk to you about what I currently do,
    • 0:04:57which is AI for medical imaging artificial intelligence
    • 0:05:00for medical imaging and a tool that we just released along with some research
    • 0:05:05that we did behind it called CXR Foundation, CXR for chest x-ray.
    • 0:05:11And I'm a member of the Google Health AI team.
    • 0:05:15So I do promise I used to look like that.
    • 0:05:19David can attest that I did look like that at some point.
    • 0:05:22But I guess I, the pandemic hit pretty hard.
    • 0:05:26So I'm a software engineer.
    • 0:05:30A little bit more about my journey, too was
    • 0:05:32that after college when I decided I didn't want to go to medical school,
    • 0:05:36I wasn't really sure what I wanted to do.
    • 0:05:39I did end up teaching CS50 a bit.
    • 0:05:41I taught a few of the other courses that David
    • 0:05:43offers, really enjoyed it really, used it as an opportunity
    • 0:05:47to keep learning myself.
    • 0:05:49And then, I applied for a job at Google as a technical analyst,
    • 0:05:55which is some kind of data analyst for the ad traffic quality team.
    • 0:06:00And we were looking for invalid add activity on the AdSense and AdWords
    • 0:06:06networks.
    • 0:06:07And that was also a great opportunity because I learned a lot of skills,
    • 0:06:12and there was a big push to scale yourself.
    • 0:06:15So if you wanted to have a bigger impact,
    • 0:06:18but then you could write some scripts and some tools of your own
    • 0:06:21that would make you scale better than that was a huge benefit.
    • 0:06:27So that's where I also started learning more of my coding skills.
    • 0:06:30And that was the time when I decided, hey, I really like this.
    • 0:06:35And I want to do this more frequently, or I want this to be my core job.
    • 0:06:40So I transferred into software engineering in 2014.
    • 0:06:44I worked on the Google Fit app for a couple of years, if you've used that.
    • 0:06:50Then I, in 2017 I transferred to the Google Surveys team.
    • 0:06:56In both of those teams, I worked a lot on infrastructure,
    • 0:06:58so large scale distributed systems, getting data
    • 0:07:02to flow from here to there, kind of like plumbing.
    • 0:07:06And it was a fantastic experience, obviously
    • 0:07:10at Google because of the scale of the data that we were working with.
    • 0:07:14And it also just hardened my skills in terms of writing good code,
    • 0:07:19writing good tests, so on.
    • 0:07:21And then in 2019, I decided that I wanted
    • 0:07:25to pivot back a little bit toward health and something
    • 0:07:29that I knew was intrinsically meaningful to me.
    • 0:07:32So I reached out to the team working on some of the health
    • 0:07:38AI ventures at the time.
    • 0:07:40I joined in an infrastructure role, and I was working
    • 0:07:43on some of the infrastructure there.
    • 0:07:45I had no modeling experience, but I wanted to learn.
    • 0:07:48So again, I just kind of picked it up, picked up the mantle
    • 0:07:51and just said, OK, I'm going to learn this.
    • 0:07:54I took Andrew Ng's course on machine learning for Coursera
    • 0:07:59and then just dove in and just really started modeling extensively.
    • 0:08:04And it's been three years.
    • 0:08:06I would definitely not say I'm an expert,
    • 0:08:08but I'm really excited with what I've learned.
    • 0:08:10And the cool thing about the AI field in general
    • 0:08:15is that it's advancing so rapidly that pretty much no one can keep pace
    • 0:08:20with it.
    • 0:08:21So if you are trying to stay on the cutting edge of it,
    • 0:08:24you may be ahead of some other people who have studied a lot longer
    • 0:08:27but haven't been keeping up with the state of the art.
    • 0:08:31So that's where we find ourselves now.
    • 0:08:33And I would also be remiss if I didn't show my great team here
    • 0:08:40at least in one slide.
    • 0:08:42This-- most of these are my co-authors on the paper.
    • 0:08:46They run the gamut from pure software engineers who
    • 0:08:52work on machine learning theory to product managers to we
    • 0:08:58have radiologists in there.
    • 0:08:59We have clinicians clinicians, scientists, research scientists.
    • 0:09:04And this isn't even all of our co-authors.
    • 0:09:06These are just the ones at Google.
    • 0:09:08We also collaborated with a few different hospital systems.
    • 0:09:12So it's really a huge team effort.
    • 0:09:14And that also hopefully gives you inspiration
    • 0:09:16that you don't have to be a pure software engineer
    • 0:09:19to contribute to things like this.
    • 0:09:22And I'll get into how valuable they were in each of their different roles
    • 0:09:26and why it's important to come together in this interdisciplinary way.
    • 0:09:31Because you don't get great results without that kind of teamwork.
    • 0:09:38So you may know some of our work on Google in terms of AI.
    • 0:09:46And it was not too many years ago that Sundar put out
    • 0:09:51a missive basically saying that we would focus on AI first and a lot of things.
    • 0:09:58So Google has tremendous research experience in this area.
    • 0:10:04And we can use this AI to improve experience in core Google products
    • 0:10:08like Google Photos, Google Translate.
    • 0:10:10So you may have used photos where you can search for personal photos
    • 0:10:15with tags, or you can even search for objects.
    • 0:10:18You can search for bike or banana, and it will find it.
    • 0:10:23And then the Google Translate functionality
    • 0:10:26is something I've used myself where you can read a menu
    • 0:10:30in a different language in real time.
    • 0:10:32It's fantastic.
    • 0:10:35And we're also using AI to tackle hard problems in the physical world.
    • 0:10:40So Waymo is taking on computer vision in the realm of self-driving.
    • 0:10:46And we'll be talking a little bit about computer vision and AI,
    • 0:10:50and it's not too dissimilar from what they're doing there as well.
    • 0:10:55And with Google Health and AI, our mission
    • 0:10:59is really to show what's possible and bring these benefits of AI to everyone.
    • 0:11:05So we do conduct research that advances the state of the art in the field.
    • 0:11:08And we're applying AI to new products, new domains.
    • 0:11:13But one of our larger goals is to make sure that everyone can access that AI.
    • 0:11:19And that's what I want to talk to you about today
    • 0:11:22and why I tried to reiterate that no experience is necessary.
    • 0:11:26I think if some of the topics we talk about today may catch you off guard,
    • 0:11:32then you can always go back, study a little bit, come back to this.
    • 0:11:35But the goal is to bring that to everyone both in terms of resources
    • 0:11:40and in terms of knowledge.
    • 0:11:43So a few of the examples here, you see there's
    • 0:11:46some work we're doing on device with Coral.
    • 0:11:50There's the brain mapping.
    • 0:11:54There's colonoscopies and prediction of sleep patterns.
    • 0:11:59So all of these things are in our wheelhouse.
    • 0:12:04OK, so before we get started, I know that a lot of you
    • 0:12:10also don't have experience with artificial intelligence
    • 0:12:14or with ML, machine learning in general.
    • 0:12:16So I wanted to give a very quick intro to neural networks, which
    • 0:12:20is the type of machine learning and artificial intelligence
    • 0:12:23we'll be talking about today.
    • 0:12:26However, I want to give a big shout out to Brian Yu
    • 0:12:30and to point, you all to that course.
    • 0:12:32Because I went through a little bit of the content so far, and it's fantastic.
    • 0:12:38I'm going to be borrowing a few of his slides
    • 0:12:40from his lecture on neural networks.
    • 0:12:43But it's a great way to bootstrap yourself in terms of learning what ML
    • 0:12:48and what AI can do.
    • 0:12:50So but let's just talk about neural networks for a minute.
    • 0:12:54Here, we have an illustration of a neuron.
    • 0:12:57So these are the fundamental building block of your brain.
    • 0:13:02And these neurons are connected to each other,
    • 0:13:06and they receive electrical signals from other neurons.
    • 0:13:09So there's two important parts there, basically.
    • 0:13:12One is that they communicate with each other.
    • 0:13:15So there's billions and billions in your brain.
    • 0:13:18They process input signals and then they have this notion of being activated.
    • 0:13:23So that in the brain, that corresponds to an electrical signal.
    • 0:13:27But we will talk about it in the context of an artificial neural network
    • 0:13:31in a second.
    • 0:13:32So here you have a two neurons that are communicating via the synapse.
    • 0:13:37And that's some kind of electrical signal that's going between them.
    • 0:13:40And together, the two of them plus billions of others
    • 0:13:44may combine to do much more complex calculations.
    • 0:13:48So that's an oversimplification, but it's a useful one.
    • 0:13:52So when we talk about artificial neural networks,
    • 0:13:55we're talking about a mathematical model for learning
    • 0:13:57that's inspired by these biological neural networks.
    • 0:14:01So you can think of it as just one large math equation.
    • 0:14:06It takes inputs on one end and gives outputs on the other.
    • 0:14:10And this is based on the structure and the parameters of the network.
    • 0:14:14So these neurons, let me go forward for a second.
    • 0:14:19This neuron, this biological neuron you can report it placed with just this.
    • 0:14:24We'll have a circle here, a single building
    • 0:14:27block for this large neural network.
    • 0:14:29And if we want to do more complex things,
    • 0:14:31we can connect them together and connect many, many, many, many, many together,
    • 0:14:36right?
    • 0:14:37So the-- let's go back for a second to here.
    • 0:14:41So the other aspect of this neural network that is important to note
    • 0:14:46and will be good for our sake is that they can learn.
    • 0:14:50So we start them with basically completely random,
    • 0:14:54they have no knowledge.
    • 0:14:56But through a process, which I think I can point,
    • 0:15:00you again to Brian Yu's course through a process called back propagation.
    • 0:15:04We give it the outputs.
    • 0:15:06And then we let it look back on all of its parameters and say, OK, how can
    • 0:15:12I tweak these so that I predict those outputs a little bit better?
    • 0:15:17And that is the magic that makes these neural networks work.
    • 0:15:23These parameters are basically, you hear them called weights.
    • 0:15:29But they essentially are coefficients in a very large math equation.
    • 0:15:34So if you were to write that out end to end,
    • 0:15:37you would be able to put in your inputs.
    • 0:15:40And then you would get your outputs just like you do in a equation like y
    • 0:15:45equals mx plus b, except it's many, many millions or billions of these neurons.
    • 0:15:53So what do we use this for?
    • 0:15:55So Brian also put together these nice slides about a classification problem.
    • 0:16:02So a lot of the problems that we face in machine learning
    • 0:16:06and in health for machine learning are classification problems.
    • 0:16:10That just means that we're trying to say something is of one class or another,
    • 0:16:16in this case, red or blue.
    • 0:16:18In health, it might be diseases present or disease is not present.
    • 0:16:25So with this distribution of data that you see here,
    • 0:16:28it's fairly obvious that we could design a classifier just by drawing
    • 0:16:33a line between the red and the blue.
    • 0:16:36So in our neural network, then it would be very simple.
    • 0:16:41It would just be a line.
    • 0:16:42And it's a nice visual representation of how we can separate that data.
    • 0:16:46And so then you say anything below the line is blue.
    • 0:16:49Anything above the line is red, and we go from there.
    • 0:16:54What about a more complex set of data like this?
    • 0:16:57So obviously, we as humans can very quickly
    • 0:17:00draw a circle around the red in there.
    • 0:17:02That's actually a little bit more complicated
    • 0:17:04for writing an equation, a math equation for this.
    • 0:17:09One thing that I'll mention here that you
    • 0:17:11may hear in other lectures on this topic is that these neural networks
    • 0:17:16are non-linear.
    • 0:17:17So that's a huge advantage to them to be able to learn not just
    • 0:17:22linear relationships but have these so-called activation
    • 0:17:25functions, which are linear.
    • 0:17:27And they allow it to learn things like this,
    • 0:17:29which are a little bit more complicated in their nature.
    • 0:17:33So we'll put a pin in this for now because I will actually show you
    • 0:17:38much more complex data that can be even reduced
    • 0:17:40to something that looks like this.
    • 0:17:43But very, very briefly and very broadly speaking,
    • 0:17:48our problem is no different than this.
    • 0:17:50We're trying to separate the red dots out from the blue dots.
    • 0:17:55You may have a more complex, maybe there's more than two colors.
    • 0:17:58There's green there's yellow, et cetera.
    • 0:18:00But it's fundamentally the same problem.
    • 0:18:07I guess actually this would be a good time to pause for questions
    • 0:18:10if there are any, or.
    • 0:18:17I should just mention that we have plenty of time,
    • 0:18:20and I will be going fairly quickly.
    • 0:18:24This will be a big survey through not just ML and AI
    • 0:18:29but medical imaging as well.
    • 0:18:32So please don't hesitate to stop me.
    • 0:18:34And mostly, I want to show you what's capable, what we can do with this
    • 0:18:41and to get you excited about the field
    • 0:18:49So let's now talk about AI development for medical imaging.
    • 0:18:53And as I mentioned earlier, the big focus of Google and Google Health AI
    • 0:18:59is to reduce barriers to AI development.
    • 0:19:02So for us to have impact, we want more people to be involved.
    • 0:19:06We see ourselves as an enabler.
    • 0:19:08That means both as I said, in terms of resources as well as knowledge
    • 0:19:14and skills, so we're very well aware that Google has resources for modeling
    • 0:19:20that most other places don't.
    • 0:19:22That comes in the form of accelerators for the actual modeling as well
    • 0:19:28as data.
    • 0:19:30And we want to share that with the world.
    • 0:19:32We want to share the benefits of that with the world,
    • 0:19:35and similarly the knowledge and the skill set.
    • 0:19:38So we're very lucky to have the best in the world that we can work with.
    • 0:19:44But we want to make this a little bit more
    • 0:19:47accessible to those who are just getting started.
    • 0:19:50So today, we're going to talk the use case or the case study
    • 0:19:54that I'm going to look at first is chest x-rays.
    • 0:19:57And that's my main focus.
    • 0:20:00But it's really just an example of what could be possible.
    • 0:20:06And this could apply to many different modalities like CT imaging, MRI,
    • 0:20:13ultrasound, pathology slides, even just natural images
    • 0:20:19from a camera for things like your skin conditions.
    • 0:20:23So some of the principles or most of the principles
    • 0:20:28are pretty applicable through everything.
    • 0:20:32So let's see.
    • 0:20:39Oh, I do see some questions.
    • 0:20:43DAVID MALAN: Andrew, I've been jotting down some questions as we go.
    • 0:20:46I can leave them in when it finally.
    • 0:20:48ANDREW SELLERGREN: Yeah, I think it's a good time to pause anyway.
    • 0:20:51Does this-- Fred says does this mean that in the future,
    • 0:20:54radiologists will not be human?
    • 0:20:56I don't know if I would go that far.
    • 0:20:58I think, and this will be clear from some of how
    • 0:21:02I describe our collaborations later.
    • 0:21:04But I think the future is really a combination of the two.
    • 0:21:09We get our best results when we work collaboratively with radiologists.
    • 0:21:15They lend their knowledge to us, and then we lend extra tools to them.
    • 0:21:19And we really want to understand the problem from their perspective.
    • 0:21:22And from there, we may design a tool that works well with their workflow.
    • 0:21:29But we are not intending to replace it.
    • 0:21:32We don't think that that's the best path forward at the moment.
    • 0:21:37But we do want to help save them time and help bridge
    • 0:21:40that gap for a short supply.
    • 0:21:45So let's see.
    • 0:21:50Yeah, I think we'll have some as far as resources,
    • 0:21:53we'll definitely have some links that I can provide you
    • 0:21:57and we can send out that would get you started in this field of machine
    • 0:22:03learning and in health AI.
    • 0:22:06With how much precision can medical AI identify disease, e.g.
    • 0:22:10Cancer cells?
    • 0:22:11It's a good question.
    • 0:22:12I think it's getting more and more precise every day.
    • 0:22:17So and it's maybe scaling even exponentially.
    • 0:22:23So there-- we have a team for example that
    • 0:22:28works our genomics or genetics team that works
    • 0:22:33on deep variant and deep consensus.
    • 0:22:36And those are things that enable you to do virtually real time sequencing
    • 0:22:41of either disease or the person.
    • 0:22:46And that helps with a precision treatment
    • 0:22:50or the availability of a treatment.
    • 0:22:53The precision medicine on the scale of say, pathology slides or radiology
    • 0:23:01images, those are getting more and more real time.
    • 0:23:05Obviously, they're getting larger.
    • 0:23:07But then they're still able to run particularly fast.
    • 0:23:10So it's an exciting time to see how far we can push the field.
    • 0:23:19Instead of predictions, what type of AI would
    • 0:23:21run for decision making in health?
    • 0:23:23That's a really good question.
    • 0:23:24I don't know.
    • 0:23:25My field is not necessarily in the NLP or the natural language
    • 0:23:32processing or conversational AI, but obviously that field
    • 0:23:36is advancing rapidly as well.
    • 0:23:39And I think the ability to add reasoning to some of these AI tools
    • 0:23:46is super compelling.
    • 0:23:48And I think it's going to get better.
    • 0:23:51I think there are some data sets like medical question answering data sets
    • 0:23:56where you test its ability to reason.
    • 0:24:00And again, I think that's still where humans shine in large part,
    • 0:24:05but it's getting better every day.
    • 0:24:10Let's see.
    • 0:24:15Are there models not based in biological brains?
    • 0:24:20Yes.
    • 0:24:21I mean, I think I would actually argue that if you want to dig deep into it
    • 0:24:26and go down a bit of a rabbit hole, the models don't really
    • 0:24:31perfectly mimic biological brains.
    • 0:24:35I think we learn more about how they complement each other
    • 0:24:38or how they mimic each other every day.
    • 0:24:41But it's a bit of an oversimplification.
    • 0:24:44I would also argue that your brain is far, far, far, far more
    • 0:24:49powerful than the neural networks that we're training.
    • 0:24:52So that's a pretty cool thing to learn when you maybe
    • 0:24:55come into this field thinking the opposite that oh, machines
    • 0:24:59are much better.
    • 0:25:01They may be better at certain tasks, but we are much better generalists.
    • 0:25:07And there's no better example of that than my daughter
    • 0:25:09who watching her learn is amazing.
    • 0:25:13Because she may not have even seen something before.
    • 0:25:17But she already has a knowledge of how to interact with it.
    • 0:25:22And that's a pretty cool thing.
    • 0:25:27Did Google perform research on EEG, EKG, EMG?
    • 0:25:31I think we do have some research there.
    • 0:25:34I'm not sure of the current published status of it.
    • 0:25:36But yeah.
    • 0:25:40So let's see.
    • 0:25:44DAVID MALAN: Andrew, I can take notes on some of the questions
    • 0:25:47and weave them back in if some of these you'll touch on anyway.
    • 0:25:49ANDREW SELLERGREN: OK, yeah that's--
    • 0:25:51wow there's, a lot of--
    • 0:25:54yeah, we will do our best.
    • 0:25:55Let's I guess we'll move forward and maybe you
    • 0:25:58can weave some more in as we go.
    • 0:26:00DAVID MALAN: Perfect.
    • 0:26:01ANDREW SELLERGREN: OK, so going back to AI for medical imaging
    • 0:26:07and as I mentioned, we're going to use chest x-ray here as a case study.
    • 0:26:12But a lot of it is going to apply to other modalities,
    • 0:26:15all kinds of medical imaging.
    • 0:26:17So why chest x-ray?
    • 0:26:19Well, chest x-rays are ubiquitous.
    • 0:26:23They're highly accessible, highly available there's about a billion
    • 0:26:26taken around the world every year.
    • 0:26:30And they're very inexpensive.
    • 0:26:32They're also non-invasive meaning they're not requiring
    • 0:26:36surgery or something like that.
    • 0:26:38They have a fairly low radiation dose I believe as well.
    • 0:26:44So it's a great modality for that.
    • 0:26:48It's also used for a very long tail of rare conditions.
    • 0:26:54So we'll look today at five or 10 that are fairly common
    • 0:26:59and that are kind of standard ones that you may see on a lot of chest x-rays.
    • 0:27:03But there's a huge long tail.
    • 0:27:05There's a lot of rare conditions which makes chest x-ray very useful,
    • 0:27:10but it's also very difficult then to interpret them.
    • 0:27:14And that's where the radiologists shine on the human side of things.
    • 0:27:19But that high quality interpretation is difficult to do and to train
    • 0:27:28for, doubly so for training a model to do that.
    • 0:27:33So that's where this work that we'll talk about will shine.
    • 0:27:39So there's also a short supply of radiologists.
    • 0:27:43There are definitely areas of the world where there's far fewer
    • 0:27:46available than there are images taken.
    • 0:27:50And there's also variability between the experts and the sites.
    • 0:27:55So you may have different imaging equipment.
    • 0:27:57You may have different interpretations of the same image.
    • 0:28:04So these are all challenges.
    • 0:28:05And then as I mentioned with Google and where
    • 0:28:08Google's forte is in building these very, very large models.
    • 0:28:15So they require very large curated data sets.
    • 0:28:20So for our purposes for this model we're talking about,
    • 0:28:24it was about a million chest x-rays, which
    • 0:28:26even on the scale of machine learning is pretty small.
    • 0:28:30A lot of the natural language processing ML
    • 0:28:35that you see today may have been trained on multi-billion or web scale data
    • 0:28:41sets.
    • 0:28:42Whereas a million chest x-rays is a very large medical data
    • 0:28:45set but it's a small data set for natural images.
    • 0:28:49But for anyone to have a million chest x-rays is pretty difficult.
    • 0:28:56And it also requires extensive fine tuning.
    • 0:28:58So some of the very specific models that we developed like our tuberculosis
    • 0:29:05model, they require six 12, 24 months of software engineering time.
    • 0:29:11And that's a whole team of people working on it, trying different things,
    • 0:29:15going back to the drawing board.
    • 0:29:18We call this fine tuning.
    • 0:29:20And it's not something that everyone can replicate.
    • 0:29:23So this is where again, once we've done this in one use case,
    • 0:29:28how can we help generalize it and let it benefit
    • 0:29:33everyone and a bunch of other tasks?
    • 0:29:37So let's-- we can play a little game here if people want to be interactive.
    • 0:29:46So how challenging is this task, and not just for machines, but for humans?
    • 0:29:54I wanted to show a few examples of chest x-rays.
    • 0:29:57You can Google around for things like these.
    • 0:30:00But here here's one.
    • 0:30:03Can anyone spot the abnormality here?
    • 0:30:07I see one person saying pneumonia.
    • 0:30:11Anyone else have any guesses?
    • 0:30:20Right chest smaller, left upper lobe, actually,
    • 0:30:24yeah, I would say left upper lobe is pretty close, so I'll give that credit.
    • 0:30:30So there's a nodule here, which you can see is very, very difficult to see.
    • 0:30:36A nodule is a small mass of tissue in the lung.
    • 0:30:41In the case in this case, it's less than three centimeters.
    • 0:30:44And it can be benign or malignant.
    • 0:30:46So it may or may not be even noted on a radiology report.
    • 0:30:52It could be something that they decide to biopsy and figure out
    • 0:30:55if it's benign or malignant.
    • 0:30:58But as you can see, it's not super easy to catch.
    • 0:31:03And so training an AI classifier to do it is not trivial.
    • 0:31:07OK, how about this one?
    • 0:31:09Anyone can spot the abnormality?
    • 0:31:17Oh, wow.
    • 0:31:18Salt has got it, pneumothorax.
    • 0:31:21Yes, so on the right here you have what looks in some ways like more normal
    • 0:31:29it's a Black space on the chest x-ray.
    • 0:31:33But this is actually a collapsed lung.
    • 0:31:36So air has leaked into the space between the lung and the chest wall.
    • 0:31:40And this would require intubation.
    • 0:31:45And it's a very serious issue.
    • 0:31:50I'll pause here just to note one thing about these chest x-rays.
    • 0:31:54So when we're talking here, we're talking about frontal chest x-rays.
    • 0:31:57There's two different ways to take those.
    • 0:31:59There's AP, anteroposterior and PA, posteroanterior.
    • 0:32:06So this is whether the x-ray machine.
    • 0:32:10Comes from above or below and there's some interesting implications
    • 0:32:15for that when patients are sicker, they tend
    • 0:32:21to have one type because they're laying down,
    • 0:32:24and they may not be able to have an x-ray standing up.
    • 0:32:27So they-- this is actually something that confounds our ability to classify.
    • 0:32:35It may just learn the type, the orientation of the image as opposed
    • 0:32:39to the actual diagnosis.
    • 0:32:42Similarly here with the pneumothorax, this one is untreated so far.
    • 0:32:48So there's no intubation whereas pneumothorax would normally require
    • 0:32:54intubation and immediate treatment.
    • 0:32:58So then when you take the chest x-ray, anyone who has a pneumothorax
    • 0:33:01is going to already have some lines and tubes, which
    • 0:33:05make it very easy for the AI to learn well as long as there's lines and tubes
    • 0:33:10then I know it's a pneumothorax.
    • 0:33:11So that's a big problem.
    • 0:33:16I think I saw a hand raised, but.
    • 0:33:20DAVID MALAN: There is a question in the chat.
    • 0:33:22Amelia is asking Andrew is this the same person, and what about their age group?
    • 0:33:26ANDREW SELLERGREN: No, this is not the same person as the nodule one,
    • 0:33:30no it's not.
    • 0:33:31You can see there's a pretty big difference in their anatomy.
    • 0:33:35I have the citation here below.
    • 0:33:39So you should be able to look that up.
    • 0:33:41I'm not sure about the age group of them.
    • 0:33:45OK, so this one I'm not even going to ask
    • 0:33:49you to try because it's very, very difficult,
    • 0:33:51but it was my example of a very, very difficult case that
    • 0:33:55may not be easy to catch.
    • 0:33:57So down there in the bottom right of our screen but on their left
    • 0:34:03is a rib fracture.
    • 0:34:05So this is the type of thing that can very easily be missed,
    • 0:34:08and it's also very hard for AI to pick up on.
    • 0:34:12So it exemplifies some of the challenges we face here.
    • 0:34:20Here's a question I can address really quickly.
    • 0:34:22Would you recommend any specific deep learning structure to the task?
    • 0:34:26Is Vision Transformer a good choice?
    • 0:34:28I'll try not to go too much down the rabbit hole
    • 0:34:31here because it's a very interesting time.
    • 0:34:34The type of model that we've trained here
    • 0:34:36is something called a convolutional neural network.
    • 0:34:39So again I'll point, you to Brian's amazing course,
    • 0:34:42which talks about convolution.
    • 0:34:45Convolutions are you can think of as a filter for the image.
    • 0:34:49So it will progress through the image and in some ways sort of compress
    • 0:34:55it to that end.
    • 0:34:57And this has been the standard since about 2012 for vision
    • 0:35:01anyway for a number of reasons.
    • 0:35:03Convolutional neural networks have built in what
    • 0:35:08they call translation invariance.
    • 0:35:10So it doesn't matter where in the image that it's happening, it can pick it up.
    • 0:35:21And that's a huge advantage for something like vision tasks.
    • 0:35:28That being said, so they've been this the go to for 10 or more years
    • 0:35:33at this point for.
    • 0:35:35So only recently in the last two years, vision transformers have emerged.
    • 0:35:39Transformers is another type of architecture
    • 0:35:42that has been used heavily in natural language processing.
    • 0:35:47And this just works in a little bit of a different way.
    • 0:35:51It doesn't have some of the same what we call inductive biases
    • 0:35:55as the convolutional neural networks.
    • 0:35:58But it also has advantages like its ability
    • 0:36:01to using a method called attention, it can basically look at the whole image
    • 0:36:08And it correlates every block of the image with every other block
    • 0:36:12of the image, which is a big advantage.
    • 0:36:16However, they require a lot more data in the case of vision
    • 0:36:21in order to kind of bootstrap themselves and get used to the task.
    • 0:36:26So they've definitely proven very performant
    • 0:36:31on some of the natural language, I'm sorry, natural image tasks.
    • 0:36:37They are starting to prove even better for the medical image tasks.
    • 0:36:42I've tried them on some of the same tasks that I'll show here,
    • 0:36:46and I got slightly worse results than I did
    • 0:36:48for my convolutional neural networks.
    • 0:36:50That doesn't mean they are worse or that they always will be that way.
    • 0:36:54It could just be a matter of I didn't tune them properly.
    • 0:36:58But that remains to be seen.
    • 0:37:06So is there a nodule in the right lobe here?
    • 0:37:08To be honest, I don't know I don't think so.
    • 0:37:11If you're talking about this, I think that's maybe an artifact of the image,
    • 0:37:16but I could be wrong.
    • 0:37:17I'm also not a radiologist I'm doing my best impression here.
    • 0:37:22But so this last one, we'll see if we can spot the abnormality here, anyone?
    • 0:37:39Yeah, I'll go ahead and give that I hope I'm saying Myla or Mila, pneumonia.
    • 0:37:47Yeah, so this one is COVID-19.
    • 0:37:50So they refer to it technically as peripheral and lower lobe predominant
    • 0:37:55rounded airspace opacities.
    • 0:37:57But for the most part, that will amount to pneumonia
    • 0:38:03in the bottom parts of these lungs.
    • 0:38:07So this is an important motivator for some of the tasks
    • 0:38:11that we're going to look at for our model here.
    • 0:38:15And it's just a reminder of what chest X-rays can do.
    • 0:38:20So as I mentioned earlier, our mission is
    • 0:38:23really to enable others to train better custom chest x-ray models.
    • 0:38:27We know that other outfits don't have the same data, the same setup,
    • 0:38:32the same computational power.
    • 0:38:36So our mission is to advance science and deliver impact.
    • 0:38:41And we define impact as being the end result, actual clinical effects.
    • 0:38:49In some-- in most cases, those are going to be taken care of
    • 0:38:53or affected by others, not by Google directly.
    • 0:38:58And that's what we're looking to do here with our ability to train these models.
    • 0:39:03So this is a three-fold problem.
    • 0:39:07We want to decrease the training time for these models.
    • 0:39:10We want to improve their label efficiency.
    • 0:39:12That means we just want to be able to use fewer images
    • 0:39:15to do the same tasks so as opposed to having the hundreds of thousands
    • 0:39:20or millions of images, can we do it with 1,000, 100, 10 images?
    • 0:39:27Reducing model complexity, so the convolutional neural networks
    • 0:39:31that we trained here are on the order of some of them as large as 500 million
    • 0:39:36or even a billion parameters, a billion weights.
    • 0:39:41And those are massive.
    • 0:39:43We're also talking about huge high resolution images.
    • 0:39:48Original chest x-rays are somewhere between 3,000 by 4,000 pixels.
    • 0:39:55A lot of the literature will downsample them all the way to 224 by 224.
    • 0:40:01With our resources, we're able to train on images that are size 1,024 by 1,024.
    • 0:40:09So that's another big advantage.
    • 0:40:15So what is our method here in order to try to affect this third party
    • 0:40:19impact that I mentioned?
    • 0:40:20What we want to do is just basically do as much as possible on Google's side
    • 0:40:25as we can before we hand off something resembling a model
    • 0:40:28or something resembling the training to the third party
    • 0:40:33to the person who might actually be doing the end training.
    • 0:40:37So there's a fair amount of text on here,
    • 0:40:41but let's just let's just break it down fairly simply.
    • 0:40:46So going back here when I mentioned--
    • 0:40:53So this is a rough diagram of a neural network.
    • 0:40:57And you can see we're probably looking at moving left to right,
    • 0:41:03which is through these neurons.
    • 0:41:05But you can see let's say you were to turn it on its side,
    • 0:41:09it resembles something like a pyramid.
    • 0:41:11The way that we're envisioning this here and a useful analogy
    • 0:41:16here is to think of it as a pyramid.
    • 0:41:18So on the bottom, we have our chest x-ray image,
    • 0:41:21which we feed into the model as pixels, just as numerical values.
    • 0:41:28And we slowly hone in on our classification.
    • 0:41:33So the actual output of the model is going to be just one number generally
    • 0:41:37that will be a number probably between zero and one
    • 0:41:39that represents our prediction.
    • 0:41:43So usually, it would be something like if it's closer to one,
    • 0:41:47then we are predicting that the disease is there or the condition is there.
    • 0:41:52And if it's zero, it's not there.
    • 0:41:56So here within our pyramid, these layers in here are artificial neurons.
    • 0:42:02So we have many, many, many, many layers of them
    • 0:42:05and again millions billions of them in there.
    • 0:42:08Somewhere in there toward the top of it, we
    • 0:42:11have a layer that is not quite the end result,
    • 0:42:16but it has a lot of what we call features.
    • 0:42:21So these are still just floating point numbers.
    • 0:42:24And we can't exactly point to one thing that they represent.
    • 0:42:29But they are useful for the diagnosis.
    • 0:42:33So they're not quite as interpretable as say oh, this is a left lung,
    • 0:42:40and this is a right lung.
    • 0:42:42But you can think of it as something resembling that.
    • 0:42:46And we're going to call this the embedding.
    • 0:42:48This is in our case, an embedding is really just a string of numbers.
    • 0:42:54It's maybe 1,000 1,300 2,000 numbers in a row.
    • 0:43:01And these are our features for the chest x-ray.
    • 0:43:05So another useful way of thinking about it is this
    • 0:43:09is just a very fancy image compression algorithm.
    • 0:43:13So we put in a chest x-ray image, which is 3,000 by 4,000 pixels.
    • 0:43:18And what we got out of it was 1,000 numbers.
    • 0:43:21So just like you might compress an audio file or a video file on your computer,
    • 0:43:29this is sort of a compression of that in a way
    • 0:43:32that we're trying to retain as much of the useful information as possible.
    • 0:43:38So let's take that definition of embedding and go forward from here.
    • 0:43:43OK, so what was our technique here?
    • 0:43:46It's pretty much as simple as it sounds.
    • 0:43:49So I don't want to complicate this and make
    • 0:43:52it sound like it was too, too fancy.
    • 0:43:55What we have is what we call a CXR pre-trained network,
    • 0:43:58a chest x-ray pre-trained network.
    • 0:44:00So in a typical setup for machine learning,
    • 0:44:04you often do what's called transfer learning, which
    • 0:44:07means that you take a model that was trained on one task,
    • 0:44:10and you try to train it on another.
    • 0:44:13You fine tune it on another task by exposing it to new data.
    • 0:44:18And generally, that is an advantage that tends to get better results.
    • 0:44:24The story, the truth of it for medical imaging
    • 0:44:28is that may not be all that useful.
    • 0:44:31So the state of the art in all kinds of machine learning and computer
    • 0:44:36vision out there is usually done on a data set
    • 0:44:38called ImageNet, which is I think 14 million natural images of 1,000
    • 0:44:46different classes.
    • 0:44:47And everyone competes to get the best score on this.
    • 0:44:52Then what we do on in medical imaging is we usually take
    • 0:44:56a model trained on those, and then we expose it to medical images,
    • 0:45:01and we get good results.
    • 0:45:03But the question is, do we get better results
    • 0:45:05than if we just started from nothing?
    • 0:45:07And I think the jury is still out on that, actually.
    • 0:45:10Because it may speed up the time it takes for us to train it.
    • 0:45:15But I'm not sure that the end result is a better model.
    • 0:45:19So what we're proposing here is basically
    • 0:45:21kind of adding in another step where we train things
    • 0:45:25on more specific data for your task.
    • 0:45:28So we have the ability to train it on a million chest x-rays.
    • 0:45:32So then when you take your model, it will already
    • 0:45:36know what a chest x-ray is.
    • 0:45:38It will have some way of reasoning about that.
    • 0:45:41So then on the right side, we're going to take
    • 0:45:44this pre-trained network, which we're using to generate these embeddings.
    • 0:45:49And then we can do a bunch of different things.
    • 0:45:51So the typical setup is on the bottom right.
    • 0:45:56There's a task specific network, and we can fine tune the whole network.
    • 0:46:03So this is usually going to get the best performance,
    • 0:46:06but there's a few concerns here in our case.
    • 0:46:09One is that we trained on private data from a few different data sets.
    • 0:46:14And we don't want to expose that data in any way to the outside world.
    • 0:46:20So releasing the model may be a risk in that sense.
    • 0:46:26So what can we do instead?
    • 0:46:28Well, we are able actually to just expose the model as an API.
    • 0:46:34So you can call this model with your images, and you get back embeddings.
    • 0:46:42And then from there in these strategy 1 and 2,
    • 0:46:45you can see you can actually train a very small model on that
    • 0:46:50and get decent results in any case.
    • 0:46:54So the way we were visualizing this is, again hopefully very instructive.
    • 0:46:59If this is the pyramid, we're sort of just chopping off
    • 0:47:03the head of the pyramid.
    • 0:47:05And then we're asking you to fill to fill
    • 0:47:09in the space so as opposed to having to build the whole pyramid yourself.
    • 0:47:14So this is where this really cool animation comes from.
    • 0:47:18On the left, you can see the traditional approach
    • 0:47:21is to build a pyramid for each different task.
    • 0:47:25So you have to start all the way from scratch or from ImageNet,
    • 0:47:29and it's going to take time.
    • 0:47:31On the right, we gave you a pyramid with the head chopped off.
    • 0:47:35And you can just add your own different heads that represent different tasks.
    • 0:47:42And what we found is that this approach, you
    • 0:47:45get really good results even without having started from the very beginning.
    • 0:47:52So I'll very quickly talk about this is more highly technical side of things,
    • 0:48:00but this is the type of learning that we did for this.
    • 0:48:04So when you start your machine learning education,
    • 0:48:08you will probably learn a lot about cross entropy
    • 0:48:10loss, which is kind of the typical task or objective for classification
    • 0:48:18problems.
    • 0:48:19But there's a lot of other options out there.
    • 0:48:22Some of them are contrastive learning.
    • 0:48:24And this is just what it sounds like.
    • 0:48:26So you're taking to two images, and you're comparing them to each other.
    • 0:48:31And then you're giving the model some information
    • 0:48:34about what those two images represent.
    • 0:48:37So in this case, or in the images we have
    • 0:48:41here, what we're doing is we're telling the model
    • 0:48:43that these two images are both of a dog.
    • 0:48:46And so we want to learn some representation,
    • 0:48:52some embedding of this image that brings those two things together.
    • 0:48:57And then we want to push all the cats away.
    • 0:49:00So this is the same for our medical imaging task.
    • 0:49:05What we're going to do is we're going to say that these two images are abnormal.
    • 0:49:10So that means that they as you saw in the previous slides,
    • 0:49:13maybe they have a pneumothorax, maybe they have a nodule,
    • 0:49:16maybe they have a fracture, they're abnormal.
    • 0:49:19So put them together in our embedding space,
    • 0:49:23and then push away any ones that are normal.
    • 0:49:25So anything where a chest x-ray doesn't have anything that requires
    • 0:49:29follow up or is of note.
    • 0:49:32So the nice thing about this is that most of our radiology classification
    • 0:49:38tasks can be reformulated in this way.
    • 0:49:40So you can imagine that if you could identify a chest x-ray as normal,
    • 0:49:46then we're kind of done with it.
    • 0:49:47The decision tree is basically you move on.
    • 0:49:51But the abnormals is where we want to start separating things out.
    • 0:49:56What's interesting about the loss, and this
    • 0:49:59is a perfect example of why I love this field is
    • 0:50:02because even doing something like this, we
    • 0:50:06get separation of the abnormal images.
    • 0:50:09They get separated from each other to some extent.
    • 0:50:12So the fractures get separated from the pneumothorax.
    • 0:50:15And I'm not really sure why.
    • 0:50:18It's something that we want to actively consider.
    • 0:50:21But here's a visualization of that.
    • 0:50:23So if you remember, I pointed earlier to this red and blue
    • 0:50:26and how prescient that would be for our classification problem.
    • 0:50:31So here, we're using a visualization technique
    • 0:50:35called T distributed stochastic neighbor embedding, or TC.
    • 0:50:40And this is just a very fancy name for a way of taking our long vector,
    • 0:50:46are array of 1,000 floats.
    • 0:50:51And then we're compressing it, or we're projecting it down onto two numbers,
    • 0:50:58two axes, x and a y so that we can visualize.
    • 0:51:00It so it's definitely an oversimplification,
    • 0:51:02but it helps us as humans reason about it a little bit, right?
    • 0:51:06So what we're seeing here is that on the left,
    • 0:51:09if we take a model that was trained on ImageNet, and we do this technique,
    • 0:51:14you can see that there's not really very good separation.
    • 0:51:17It would be very hard for you as a human to draw a circle around just
    • 0:51:22the blue dots that excludes.
    • 0:51:24The red dots and just the same for a machine,
    • 0:51:27it would be very hard to do that.
    • 0:51:28On the right, it's a little bit easier.
    • 0:51:31It's not perfect, but you can see you could
    • 0:51:33imagine drawing a little circle around the blue,
    • 0:51:35and that's a separation from the red.
    • 0:51:38And this on the right is our chest x-ray network.
    • 0:51:42So you can see that even without having done any work,
    • 0:51:46we have a pretty good separation.
    • 0:51:47We already have a pretty good way of classifying this airspace opacity
    • 0:51:52class.
    • 0:51:53So that's a pretty cool result.
    • 0:51:57And I'll say here very quickly just to talk
    • 0:52:04a little bit about the metrics, what we're visualizing here
    • 0:52:09or what we're graphing here is something called AUC, area under the curve so
    • 0:52:15actually, more on that in a minute.
    • 0:52:17I will say that what we're showing here is
    • 0:52:21that we're able to get basically the same performance as some
    • 0:52:25of these pre-trained networks that are fine tuned on the task at hand.
    • 0:52:32Our model with a small layer trained on top is the red.
    • 0:52:38So you can see it's best in show up to an order of magnitude
    • 0:52:44around 10,000 images.
    • 0:52:46So only there, if you have more than 10,000 images
    • 0:52:50do you start to get an advantage from training on all of them
    • 0:52:54and training a large network on them.
    • 0:52:55Otherwise, you may be better off doing our simple approach, which
    • 0:53:00is much faster.
    • 0:53:01And in the case of two really vital diagnoses
    • 0:53:06here, tuberculosis and COVID-19, we get better results
    • 0:53:11when we use our small models, and that's because we have so few images.
    • 0:53:17So I'll bring up this slide so we can talk about some of the metrics
    • 0:53:22as I mentioned.
    • 0:53:23So here, we have AUC is on the left.
    • 0:53:27And that's area under the curve.
    • 0:53:29So what is that curve?
    • 0:53:30Here is that curve.
    • 0:53:33It's called the receiver operating characteristic curve.
    • 0:53:37So it represents the trade between finding
    • 0:53:41true positives and false positives.
    • 0:53:44So once you have your number in a classification model
    • 0:53:48that's between zero and one, you have to decide
    • 0:53:51what is your so-called operating point.
    • 0:53:54So that's the threshold at which you say, OK above this is positive,
    • 0:53:59below this is negative.
    • 0:54:02And that's a whole can of worms to try to figure out your operating point.
    • 0:54:08You might think OK, let's just start with 0.5 well most of the time,
    • 0:54:11it's not perfectly well distributed.
    • 0:54:14So maybe 0.8 or 0.7.
    • 0:54:17So you have a choice to make as to where to pick that operating point
    • 0:54:21to kind of get the best trade-offs.
    • 0:54:24So in some cases, you may want to have--
    • 0:54:28you may want to optimize for more true positives or a higher
    • 0:54:33true positive rate and then a false positive rate or a lower
    • 0:54:38false positive rate.
    • 0:54:39So this graph, this receiver operating characteristic curve just
    • 0:54:43represents the trade-off between these two.
    • 0:54:45So everywhere along this line is a possible operating point
    • 0:54:49that you can pick.
    • 0:54:51And each one is going to have, again different trade-offs.
    • 0:54:53So when we talk about what scores we're giving our model,
    • 0:54:57where we actually just go ahead and add up all the area under this curve.
    • 0:55:02So that the better this curve is, the more up into the left it is, right?
    • 0:55:07So this AUC here is on the order of 0.9, 0.92, 0.95.
    • 0:55:13You can see most of the graph is underneath the curve.
    • 0:55:18And that's a good thing.
    • 0:55:21And also here, we have graphed a few different radiologists.
    • 0:55:27So this is from having them review the same images,
    • 0:55:31we can show that our performance is at least as good as them.
    • 0:55:36Because they fall somewhere on the curve or below the curve generally.
    • 0:55:41But what's fascinating about this is that here, we've
    • 0:55:44trained on only 100 images.
    • 0:55:46We trained a very simple model that took five minutes.
    • 0:55:49And the train and the test sets are from different countries.
    • 0:55:55So another common practice in machine learning is to train on certain images,
    • 0:56:01and then you hold out some images so that you
    • 0:56:05can be sure that you are testing on something sort of fair.
    • 0:56:09So in this case, our training images came from the US,
    • 0:56:14and our test images came from China.
    • 0:56:16So we can be sure that it does pretty well
    • 0:56:18even if we're using different demographics, different imaging
    • 0:56:23equipment.
    • 0:56:25So that's a really good result. And then here, we have even more ROC curves.
    • 0:56:31And we trained a model on COVID-19 severity.
    • 0:56:34So our ability to predict whether a COVID-19 patient would need to be
    • 0:56:41ventilated or go into the ICU.
    • 0:56:45Again, the interesting thing here is we only have 500 images.
    • 0:56:48So I'll pause here again to talk quickly about the sum of how
    • 0:56:54this research came to be.
    • 0:56:55In March of 2020, we were looking at developing an abnormal normal model,
    • 0:57:01so as I mentioned, this ability to separate abnormal chest
    • 0:57:04x-rays from normal chest x-rays, and then when the pandemic hit,
    • 0:57:09we wondered could we do something with this model
    • 0:57:11to help with the scarcity of testing that was available at that time.
    • 0:57:19We talked to our partners at Northwestern,
    • 0:57:21and we found that they had about 500 images.
    • 0:57:24And it was March of 2020 so it was very new.
    • 0:57:27They didn't have a lot of images.
    • 0:57:30So what can we do with that few images?
    • 0:57:32Well that's where we started to look at these very low label,
    • 0:57:36low number of images case studies.
    • 0:57:41So the result that we showed was that yes, you can actually predict
    • 0:57:44some of this stuff with chest x-ray.
    • 0:57:46But I also want to highlight that this conversation, we did not put anything
    • 0:57:53out there because we, having spoken to our radiologists, our clinicians
    • 0:57:58on staff the determination was basically that this is not
    • 0:58:04a part of a normal clinical workflow.
    • 0:58:06So even though you could diagnose COVID-19 from a chest x-ray,
    • 0:58:11it would be a strange thing to have people
    • 0:58:13come in who had symptoms of COVID and then take a chest x-ray of them
    • 0:58:17and then diagnose them.
    • 0:58:19That's not typically what would happen.
    • 0:58:20And so it wasn't super useful off the bat.
    • 0:58:24However, it was a great way of demonstrating
    • 0:58:28that we are able to train these models that can very quickly adapt.
    • 0:58:32And that continues to be true even today because the landscape of COVID-19
    • 0:58:38is always changing more so than anything we've ever seen before.
    • 0:58:41So now we have patient populations that are
    • 0:58:45changing in terms of the vaccine rates, their demographics for who's
    • 0:58:52being affected, the different variants.
    • 0:58:54So all of these things are things that are going to challenge our model,
    • 0:58:57but if we're able to quickly adapt with a small number of images like this,
    • 0:59:02then we're well suited.
    • 0:59:04So I'm almost to the good stuff.
    • 0:59:07I will say one other thing just to set up the--
    • 0:59:11so another type of-- so I mentioned earlier this idea
    • 0:59:14of contrastive losses, and we're taking two images and bringing them together.
    • 0:59:19Another cool thing that you can do is take
    • 0:59:22not two images, but one image and one piece of text and bring them together.
    • 0:59:28And you tell the model, these are similar, even though they're different.
    • 0:59:32One of them's an image, one of them's text.
    • 0:59:34You can bring them together in this embedding space.
    • 0:59:37So this was developed originally for the clip paper,
    • 0:59:41but it's now been applied to chest x-rays with the convert and check zero.
    • 0:59:47And we'll just hold that thought for now because it will become important
    • 0:59:51in a minute.
    • 0:59:53So what can we do with these embeddings?
    • 0:59:56We can do a few things, and I'm going to do-- help you do all of these right
    • 0:59:59now.
    • 1:00:00So let's get to it.
    • 1:00:02A zero-shot image classifier, what does that mean?
    • 1:00:04Well zero-shot means that the model has not seen anything
    • 1:00:11or has not been given a label of something before.
    • 1:00:13So you could imagine training something on Fracture,
    • 1:00:19and then you ask it where is the pneumothorax?
    • 1:00:24That would be an example of a zero-shot?
    • 1:00:26And it's a very difficult problem, obviously because machine learning
    • 1:00:30models benefit from having seen labeled images beforehand.
    • 1:00:36So what if you don't have that?
    • 1:00:38Well, this is what we'll look at here.
    • 1:00:40Another case is text image retrieval.
    • 1:00:42So what if you wanted to look up images based on the text that you input?
    • 1:00:48So this is what I'm demonstrating here on the right and what we'll build.
    • 1:00:51What if you wanted to build a radiology report generator?
    • 1:00:56We can do that, too.
    • 1:00:58And I want to pause here also to really thank
    • 1:01:01the folks that put together these two large open source data sets MIMIC-CXR,
    • 1:01:08it comes out of an MIT.
    • 1:01:09CheXpert comes out of Stanford, and they're really, really useful
    • 1:01:15for developing these kinds of models.
    • 1:01:18So let me go ahead and switch to Colab so we
    • 1:01:23can get some cool demos going here.
    • 1:01:27I have taken more time than I expected.
    • 1:01:30David, do you have any questions you want to feed in right now?
    • 1:01:36DAVID MALAN: No, I think folks are probably
    • 1:01:38about to really enjoy the hands-on part.
    • 1:01:39So we should forge ahead.
    • 1:01:41ANDREW SELLERGREN: Ok, sounds good.
    • 1:01:42So if you're not familiar, this is Colab.
    • 1:01:45It's based on Jupiter, which is also available.
    • 1:01:49And it's basically just a playground for you to run Python code locally
    • 1:01:55or you can have it connected to a runtime that's hosted in the cloud.
    • 1:02:01So I will make some version of this code available.
    • 1:02:05I need to clean it up because it's embarrassing right now.
    • 1:02:07But it will serve its purposes now.
    • 1:02:11So here, there's just some instructions for setting this up to run locally.
    • 1:02:17And I will I've also prefab some of this.
    • 1:02:21So here we're just going to go ahead and import a bunch of libraries
    • 1:02:24that we need, a few of them are TensorFlow, SK Learn, Pandas.
    • 1:02:29A lot of them are very common data science libraries.
    • 1:02:33CSR foundation is the name of our tool that we released
    • 1:02:36and is a library on GitHub and on PIP.
    • 1:02:40PIP is our library dependency management system.
    • 1:02:44So I've already installed this.
    • 1:02:45It also includes a bunch-- has a bunch of dependencies.
    • 1:02:48So it takes a little longer to install.
    • 1:02:51So now, I'm connected to my local machine here.
    • 1:02:56And I'm just going to go into this folder for MIMIC-CXR.
    • 1:03:01This step unfortunately takes a little bit too long.
    • 1:03:03But what I'm doing here is just reading in the radiology reports.
    • 1:03:06Because we're going to use those as inputs to our model.
    • 1:03:12And here, we're also going to take the radiology reports,
    • 1:03:16and we're going to just clip into a section of them called
    • 1:03:18the impressions section.
    • 1:03:20So that's usually some kind of a summary that the radiologist
    • 1:03:24has given that is less detailed but more focused
    • 1:03:28and will be good for our training.
    • 1:03:31So here, I've also just again, done some more pre-processing of this.
    • 1:03:37But I'll show you what this looks like in a minute.
    • 1:03:40OK, so let's examine our embeddings.
    • 1:03:43So as I mentioned, these are just arrays of numbers.
    • 1:03:48So to prove that point, here's one of them.
    • 1:03:51So these range in value.
    • 1:03:54I don't know that they have a min and a max.
    • 1:03:57You can normalize them and so on, but they could be negative.
    • 1:04:00They can be positive.
    • 1:04:01There's about 1,000 of them in our particular model.
    • 1:04:04And believe it or not, they contain all of the information that we need.
    • 1:04:10So here, I have my labels file.
    • 1:04:14So for every one of my images, I have an identifier.
    • 1:04:17I want to load up the radiology report that corresponds to this.
    • 1:04:23So I've again, prefabed this.
    • 1:04:26But it's really a lot of this is working in Pandas, which is a data science
    • 1:04:32tool that is fantastic.
    • 1:04:35And it gives you back these things called
    • 1:04:36data frames, which is just a table.
    • 1:04:39It has rows and columns like an Excel spreadsheet.
    • 1:04:43And it allows you to have things associated with each particular image.
    • 1:04:48So here, I have our DICOM ID.
    • 1:04:51DICOM is a type of is a medical image format.
    • 1:04:55And then I have a bunch of these labels representing
    • 1:04:59different conditions that are diagnosed on chest x-ray, atelectasis,
    • 1:05:05cardiomegaly.
    • 1:05:06Cardiomegaly is an enlarged heart.
    • 1:05:08A lot of these other ones, edema consolidation,
    • 1:05:10atelectasis, they pertain to lung conditions.
    • 1:05:15There's fracture.
    • 1:05:17So this is the PA versus AP as I mentioned,
    • 1:05:22is the orientation of the person and the imaging equipment.
    • 1:05:28And finally over here, so I'll make this a little bit bigger.
    • 1:05:32Well, actually it's probably easier to just look at the impression section.
    • 1:05:35So the impression section is again, our boiled down version
    • 1:05:38of the radiology report.
    • 1:05:41And you can see, actually one big, let me go ahead and make more.
    • 1:05:47I'm just going to ask for more examples here.
    • 1:05:52So you can see this is a good example of one of the problems
    • 1:05:59that we'll have with our data, which is that it's very templated.
    • 1:06:03So when a radiology report is normal, they'll
    • 1:06:08often say something like normal study or no acute intrathoracic process, which
    • 1:06:13means nothing's going on in the chest and no acute cardiopulmonary process,
    • 1:06:18nothing's going on with the heart and the lungs.
    • 1:06:21The other ones that are for abnormals are going to be a lot more involved.
    • 1:06:28And we're going to, let's see if I can, I have to do a little bit of whoops.
    • 1:06:52OK this is just to get a little bit wider view of each of them.
    • 1:06:56So you can see there's a lot of text here.
    • 1:07:01And we're going to try to get our model to understand most of this text.
    • 1:07:06So this is not too interesting again, just setting up our frame a little bit.
    • 1:07:16I'm going to skip this part, skip this part.
    • 1:07:21So the first thing that we'll do is what I
    • 1:07:23mentioned earlier, which is we can take our embeddings,
    • 1:07:26and we can train a very small model.
    • 1:07:28So you can imagine actually, let me go ahead here's
    • 1:07:35our GitHub, which I will have links to, and you'll
    • 1:07:41need to if you want to use this tool, you'll need
    • 1:07:43to just fill out a very quick form.
    • 1:07:44But it's just kind of getting to know who you are, nothing too heavyweight.
    • 1:07:51So if we look at our model here, OK, so the important part is actually here.
    • 1:08:03So we're putting together just two of these layers that
    • 1:08:05are called dense layers.
    • 1:08:07So that's this line of those circles that I mentioned, those neurons.
    • 1:08:12And there's just two of them.
    • 1:08:14So that's it.
    • 1:08:15That's how complicated our model is.
    • 1:08:17And with Caris, which is a library that works on top of TensorFlow,
    • 1:08:23it's as simple as just saying give me one layer, give me another layer.
    • 1:08:27And then I put my model together.
    • 1:08:29So the rest of this stuff is a little bit.
    • 1:08:32You don't need to know too much about it right now.
    • 1:08:35It has to do with things like your learning rate.
    • 1:08:37But I just wanted to point that out to show you how simple the model is.
    • 1:08:41OK, so now, I'm going to start this training so that it will oh oh, OK, I'm
    • 1:08:57going to start at training so that it can run while I'm talking.
    • 1:09:03So what I've done is I've given it my labels, which are
    • 1:09:09for the condition called cardiomegaly.
    • 1:09:12And they're just one for positive, zero for negative.
    • 1:09:16These are just all of our hyper parameters.
    • 1:09:20So batch size, how many images are we going to look at once?
    • 1:09:23512.
    • 1:09:25How many epoch are we going to train?
    • 1:09:2620, we're going to go through the whole data set 20 times.
    • 1:09:31So what's nice about this, you'll see the first epoch,
    • 1:09:35the first pass through all the data is going to be pretty slow.
    • 1:09:39But then I'm going to cache the data set, and it will fly.
    • 1:09:44So as it's training here, you can notice the training loss is going down,
    • 1:09:49which is good.
    • 1:09:51The AUC, which is going up it's getting up to 0.81.
    • 1:09:54This is all for the training set.
    • 1:09:57So what we mostly care about is our validation set.
    • 1:10:00We want to know how does this do on data that it hasn't seen yet.
    • 1:10:06So we'll leave that off for a minute.
    • 1:10:09I wanted to ask David, are we able to go a little over time if I.
    • 1:10:14DAVID MALAN: Think we're OK.
    • 1:10:15Folks don't mind sticking around with us.
    • 1:10:18Happy to finish up.
    • 1:10:19ANDREW SELLERGREN: Yeah, sorry.
    • 1:10:20I underestimated how long this would go.
    • 1:10:23But I definitely, I'm able to stick around.
    • 1:10:25So if people have extra questions and so on.
    • 1:10:29DAVID MALAN: Absolutely, we can leave some
    • 1:10:31of the questions that were asked in the chat for the very end.
    • 1:10:34So folks can join or leave as needed.
    • 1:10:36ANDREW SELLERGREN: OK, sounds good.
    • 1:10:39OK, let's see.
    • 1:10:42Other the things I wanted to call out here, let me,
    • 1:10:49oh I don't have any so two of the things that I didn't--
    • 1:10:57that I glossed over here were your learning rate and your optimizer.
    • 1:11:02So these are the things broadly speaking,
    • 1:11:05when you're doing machine learning, and it's doing back
    • 1:11:08propagation you're taking steps toward your end result.
    • 1:11:13And these just determine how big those steps are.
    • 1:11:16And the learning rate particular defines how big that step is.
    • 1:11:20You can do things like a learning rate function
    • 1:11:22where it will get smaller over time.
    • 1:11:25So that you're taking smaller steps sort of getting closer to your result.
    • 1:11:29OK, here we go.
    • 1:11:30So a couple of interesting things here before we move on,
    • 1:11:34you can notice that over each of the epochs
    • 1:11:37are this is our training that you see.
    • 1:11:39And it's getting much, much better, getting almost close to one,
    • 1:11:42which is the best possible result.
    • 1:11:44Our training loss here is constantly decreasing.
    • 1:11:49But our validation loss, so it starts off decreasing,
    • 1:11:54and then it basically plateaus.
    • 1:11:56It even starts to increase a little bit at the end.
    • 1:11:59And our AUC for the validation set stays about the same.
    • 1:12:04But it's still good, it's 0.86.
    • 1:12:06So this is an example of overfitting.
    • 1:12:09When your model, even one that's this small,
    • 1:12:13but the embedding is being as feature rich as they are,
    • 1:12:16it learns too well in some ways.
    • 1:12:19It's picking up on things that don't translate well to other data sets.
    • 1:12:25So we call this overfitting.
    • 1:12:26And there's a bunch of different ways to handle it.
    • 1:12:29A lot of techniques are called regularization
    • 1:12:32that try to address that.
    • 1:12:34And that's a lot of what we do as machine learning specialists.
    • 1:12:40So here, we can just very quickly look at model summary.
    • 1:12:43We'll give you a quick breakdown of the whole network.
    • 1:12:48So again, it's very, very small.
    • 1:12:51It has two dense layers of 512 and 256 neurons.
    • 1:12:57And the total number of parameters is less than a million.
    • 1:13:00So that's a pretty small network.
    • 1:13:04So let's do some more interesting stuff.
    • 1:13:07So now, as I mentioned before, we often take one data set, we train on it.
    • 1:13:11We validate on another.
    • 1:13:13We test on another one to see how it's doing.
    • 1:13:15So the MIMIC data set comes from MIT.
    • 1:13:20I can't remember which hospital systems, but it's mostly ICU patients,
    • 1:13:26so intensive care patients.
    • 1:13:27CheXpert is a data set that comes from Stanford,
    • 1:13:30so all the way across the country, different imaging equipment,
    • 1:13:35different distribution of patients.
    • 1:13:39So we're going to actually test our model on CheXpert
    • 1:13:42without having trained on it at all.
    • 1:13:45We want to see how well it does when it hasn't seen the data yet.
    • 1:13:48So I'm going to go ahead and load up our train
    • 1:13:52and our validation sets for CheXpert.
    • 1:13:56And this is another labels file that I've created.
    • 1:14:01You can see these numbers correspond to the various conditions,
    • 1:14:07cardiomegaly among them.
    • 1:14:10So now, I'm going to gloss over this a little bit
    • 1:14:13because I need to start the training and then we can come back.
    • 1:14:29OK, this is even messier than I remember.
    • 1:14:35So how are we going to feed our radiology report into the model?
    • 1:14:42Well, we need to convert it to numbers just like the images numbers.
    • 1:14:46So that's all that this stuff is doing here.
    • 1:14:50Here, I'm just setting up a little bit of a way for us
    • 1:14:52to use CheXpert as the evaluation set.
    • 1:14:58So now, our model is a little bit slightly more complicated,
    • 1:15:03but it's not too much so.
    • 1:15:06So we have a couple of layers here that are going to get added onto our image.
    • 1:15:11And then those a couple of different layers
    • 1:15:13are going to get added onto our text.
    • 1:15:15And then we're going to take the outputs of those two and compare them.
    • 1:15:19And that's going to be our task.
    • 1:15:22So here, in order to get numbers from our radiology reports,
    • 1:15:31we're going to pass them through this preprocessor.
    • 1:15:34So we're going to tokenize them in the language of natural language
    • 1:15:38processing.
    • 1:15:40So basically, that's just a dictionary of OK, this word is the number one.
    • 1:15:46This word is the number two and so on.
    • 1:15:48In fact they're actually word pieces.
    • 1:15:50So they're syllables or a few one part of the word.
    • 1:15:55And this is all a bunch of fancy stuff that just says, OK do that assignment.
    • 1:16:01Now, I mentioned the clip model.
    • 1:16:04This is where the magic is happening.
    • 1:16:06I'm just going to, let's gloss over that part.
    • 1:16:10All this is doing is passing our image through one side of it,
    • 1:16:13our text through another.
    • 1:16:15And again, this loss function here is a cosine similarity.
    • 1:16:22So it's going to say how close are these together.
    • 1:16:26The farther apart they are, the more, we want to bring them together.
    • 1:16:31And that's what our loss function is going to be.
    • 1:16:34This is just another learning rate function.
    • 1:16:38So here's our data set in TensorFlow.
    • 1:16:42You can basically define your data set sort of iteratively
    • 1:16:48where it reads in these files.
    • 1:16:51And then it processes them as you go.
    • 1:16:54And that will be a little bit faster than
    • 1:16:56if you tried to load it all into memory.
    • 1:16:59So let's look here.
    • 1:17:02I'm now initializing our model, our clip loss model.
    • 1:17:09I'm giving it a cosine decay, which is a type of learning rate.
    • 1:17:15I'm giving it the AdomW optimizer.
    • 1:17:19And then I'm just calling model.fit on it.
    • 1:17:24So each iteration is actually going to go for about 500 steps.
    • 1:17:29Then it's going to pause.
    • 1:17:30You can see it's paused right now.
    • 1:17:33And it's going to evaluate on a validation set of CheXpert.
    • 1:17:39So but what's interesting here is that, and I maybe
    • 1:17:44buried the lead a little bit here, but CheXpert, we
    • 1:17:47don't have the radiology reports for it.
    • 1:17:51So we have the radiology reports for MIMIC
    • 1:17:54and we're using that to train the model.
    • 1:17:56So how do we get a result from CheXpert at the time of evaluating our model?
    • 1:18:02Well, I glossed over this before.
    • 1:18:04But let's go back here.
    • 1:18:06The way that we're defining our labels now for CheXpert
    • 1:18:11is this so-called zero shot or kind of a soft prompt, right?
    • 1:18:18So we get to pick whatever we want to define our labels.
    • 1:18:23So if you want to search for a right pleural effusion
    • 1:18:27or a left pleural effusion, you can do that instead
    • 1:18:29of just a label of effusion or not.
    • 1:18:32You can specify moderate cardiomegaly, severe cardiomegaly.
    • 1:18:37And we're going to do a little bit of math to turn this into a prediction.
    • 1:18:41So over here on the right, this set of prompts
    • 1:18:48is going to say, OK, this is normal.
    • 1:18:50So as I mentioned earlier, this sort of template
    • 1:18:52of saying no acute cardiopulmonary process, that means nothing
    • 1:18:57is going on there.
    • 1:18:59And there's no pleural effusion.
    • 1:19:02So we can get our prediction from this.
    • 1:19:07Let's see how our training is going here.
    • 1:19:12So we got our first result. I'm actually going to just halt it
    • 1:19:16there since in the interest of time.
    • 1:19:20So you can see we're getting really good AUC
    • 1:19:24scores even though the model has not seen
    • 1:19:26this data set or these exact phrases.
    • 1:19:31We're sort of asking a question of the model that hasn't been asked before.
    • 1:19:37And one cool thing that we can now do with this,
    • 1:19:42so what if we wanted to write a plain text chest x-ray search engine?
    • 1:19:46So as I mentioned earlier, we have the ability
    • 1:19:50to query now our CheXpert data set where we don't have the radiology reports.
    • 1:19:58What we do is we feed in our query, which is in natural language.
    • 1:20:03We turn it into numbers, and then we ask the model
    • 1:20:07how similar is this to the image?
    • 1:20:12And when we do that, we can then rank them and say here's the top five.
    • 1:20:18So I have a query here of large bilateral pleural effusions.
    • 1:20:24And this is going to give me results that have some of those things.
    • 1:20:30And if we wanted to look for a, let's see, a normal chest x-ray,
    • 1:20:44we can do that as well.
    • 1:20:52So as you can see, these are a lot more normal.
    • 1:20:56So we just built a search engine for our chest x-rays.
    • 1:21:00You could imagine this would be useful if you were a radiologist
    • 1:21:03and you want them to look up an example of a case,
    • 1:21:05but we didn't have radiology reports for it.
    • 1:21:09So I am going to have to hurry through this last one.
    • 1:21:12So let's just get to it.
    • 1:21:15For my next trick, we're going to train a radiology report generator.
    • 1:21:22So this is not going to be perfect, but it's
    • 1:21:25just a pretty cool demonstration of what we have,
    • 1:21:28what the possibilities are here.
    • 1:21:31So the setup is actually quite similar to our clip model
    • 1:21:34where we have a text model over here on the side in addition
    • 1:21:42to our vision side of things.
    • 1:21:45In this case, it's going to be what we call an encoder decoder.
    • 1:21:50The encoder is actually not used.
    • 1:21:52The decoder is a BERT, which is a type of transformer,
    • 1:21:57which is used a lot in natural language processing as I mentioned.
    • 1:22:03So we're basically going to give it our image embedding.
    • 1:22:08We feed that into the text model, and then we
    • 1:22:12say give us some text for this, which is a crazy thing
    • 1:22:19to do to be honest to ask it to do something like that.
    • 1:22:25So you can see it's going to take a couple of minutes to train.
    • 1:22:28But thankfully, I did pre-fab the results I hope.
    • 1:22:35One interesting thing to note here, I had
    • 1:22:38to reduce the batch size considerably.
    • 1:22:40It no longer fits in my GPU even, which is
    • 1:22:42why it's also slower, some of the things that you just sort of wrestle
    • 1:22:46with as an ML practitioner.
    • 1:22:50And let's just say this is my final trained model I hope.
    • 1:23:07So now, I'm going to run through our validation data set.
    • 1:23:11And I'm going to ask it to give me the radiology reports.
    • 1:23:14And I'm going to compare that to what the actual ones are.
    • 1:23:19So to be honest, it's surprising even to me how well this works.
    • 1:23:23So here's a normal one, no acute intrathoracic process,
    • 1:23:26no evidence of pneumonia.
    • 1:23:30Here's an example where it's not working,
    • 1:23:32which is it's always good to start looking at.
    • 1:23:34So Dobhoff tube is one of a type of tube for I
    • 1:23:38think it's for feeding, a feeding tube.
    • 1:23:41And you can see here, instead of Dobhoff, off it says do something.
    • 1:23:44So it doesn't quite know what to do with that.
    • 1:23:47Same here, possible cavities, well, it says possible cat.
    • 1:23:51But it's getting there.
    • 1:23:55And it's pretty crazy that it can do this with just our embeddings.
    • 1:24:02And it gives you a sense of what you could do with yours.
    • 1:24:06So I do I want to quickly, before I wrap up what's so to read more about this
    • 1:24:15and get more involved, we did publish our paper in Radiology
    • 1:24:20it's called Simplified Transfer Learning for Chest Radiography
    • 1:24:23Models Using Less Data.
    • 1:24:25There's the free API, which I'll have links to where you can,
    • 1:24:29and we'll have a collab where you can use some of these free data sets.
    • 1:24:32MIMIC and CheXpert, you can also work with, which for Mimic,
    • 1:24:36you'll need to go through a training.
    • 1:24:38For CheXpert, you, I think you need to sign some agreements.
    • 1:24:42But they're also available to you.
    • 1:24:44Our goal really is to accelerate research.
    • 1:24:47We're always looking for collaborations if you
    • 1:24:49know people, if you are people that want to collaborate and build something
    • 1:24:52cool with this, please reach out.
    • 1:24:55What's next?
    • 1:24:56We want to do this for more modalities.
    • 1:24:59We want to do it for potentially ultrasound and some others
    • 1:25:04that I think would benefit a lot from this I also have some links here.
    • 1:25:10There's a software engineering internship.
    • 1:25:13Applications are open now through December 1
    • 1:25:16for undergrad, graduate, PhD students.
    • 1:25:19There's some connect with Google resources.
    • 1:25:21So this one references students.
    • 1:25:24And I think it's focused on the US, but there also is a link.
    • 1:25:28And maybe I can break them out.
    • 1:25:29There's links in here for around the world
    • 1:25:33as well as industry so a few others that are maybe
    • 1:25:35more specific to whatever group you identify with
    • 1:25:40and some other resources, Google Careers, OnAir.
    • 1:25:44And yeah, again, all of these will be linked for you.
    • 1:25:47So I want to pause there.
    • 1:25:51Sorry I went even longer than I expected,
    • 1:25:53but I do want to stick around for questions.
    • 1:25:58I'm able to say around four 20, 25 minutes
    • 1:26:02depending on how long people want to stay.
    • 1:26:06But again, thank you so much.
    • 1:26:08I really hope this has inspired you, gotten you excited about it,
    • 1:26:11also giving you the confidence that this is something you can do.
    • 1:26:15And you shouldn't be overwhelmed by it or feel
    • 1:26:18like it's beyond your reach you absolutely can do it.
    • 1:26:22And CS50 will give you the tools to do it, so.
    • 1:26:26DAVID MALAN: Thank you, Andrew so much for joining us,
    • 1:26:29and maybe a virtual round of applause with your favorite emoji reaction
    • 1:26:32or physical hands, so good to reunite.
    • 1:26:35If I could ask just one question on behalf of the group
    • 1:26:38and then turn things over to you and Carter
    • 1:26:40so that I can get to another meeting myself if you don't mind.
    • 1:26:43But we'd love for folks to stick around for more one-on-one Q&A
    • 1:26:46if you would like.
    • 1:26:47I think one question for the group that was recurring in the chat was just
    • 1:26:50how folks can learn more, not just about these resources
    • 1:26:52that you have on the screen.
    • 1:26:53But if they want to learn more about AI, like you did on your own,
    • 1:26:57if they want to learn more about AI in medicine, are there courses?
    • 1:27:00Are there universities?
    • 1:27:01Are there other learning resources you would recommend specifically?
    • 1:27:05ANDREW SELLERGREN: Yeah, I would definitely
    • 1:27:07as I said before, I would highly recommend
    • 1:27:09CS50's AI for artificial intelligence in Python.
    • 1:27:15The Coursera course is good, too.
    • 1:27:17I would be remiss to not mention that.
    • 1:27:19Machine learning was the original Coursera course.
    • 1:27:23O'Reilly has great books.
    • 1:27:24I always, when I'm trying to learned something new,
    • 1:27:27I always go for O'Reilly first.
    • 1:27:29Those are the ones with the funny images of animals on the front.
    • 1:27:33Those are usually a good, and honestly YouTube.
    • 1:27:38There's just so much content these days of people walking you
    • 1:27:41through things, getting you bootstrapped and those things.
    • 1:27:45But maybe I can come up with a list of resources, too,
    • 1:27:49and we can post that with the rest of these slides and so on, David.
    • 1:27:53That would probably be good.
    • 1:27:54DAVID MALAN: Absolutely.
    • 1:27:55I've been sharing the slides in the chat,
    • 1:27:57and we will post everything at cs50.ly/zoom as well for folks
    • 1:28:01afterward.
  • CS50.ai
Shortcuts
Before using a shortcut, click at least once on the video itself (to give it "focus") after closing this window.
Play/Pause spacebar or k
Rewind 10 seconds left arrow or j
Fast forward 10 seconds right arrow or l
Previous frame (while paused) ,
Next frame (while paused) .
Decrease playback rate <
Increase playback rate >
Toggle captions on/off c
Toggle mute m
Toggle full screen f or double-click video