This is your FBI. This is your FBI. An official broadcast from the files of the Federal Bureau of Investigation. Presented as a public service by the Equitable Life Assurance Society of the United States. To your FBI, you look for national security. And to the Equitable Society for Financial Security. These two great institutions are dedicated to the protection of you, your home, and your country. Tonight, the story of a crime against society. The confidence game. There are many ways for a thief to steal. There are many forms of larceny. They range from the simplest stick up to the involved manipulations of a big time swindle. This present war period has created a fertile field for the swindler. Increased salaries, increased employment set up a ready market for his get-rich-quick activities. On May 22, 1934, Congress, by passing the National Stolen Property Act, put into the hands of the FBI an effective weapon against confidence men. In spite of this, however, swindlers still pursue their losing game. That drama is an old one, and it's worked with little variation. Its first act might begin as this one does, and the office of a respectable businessman. He is greeting a visitor. Sit down, Mr. Perry. Thank you, sir. Thank you. Well, warm enough for you? I should say so. Phew! You know, I thought they called this the Windy City. I haven't even felt a breeze since I've been here.  Well, sir, now I don't mind if I do. There. I feel better already. Oh, well, how long are you planning to be here in Chicago, Mr. Perry? Just long enough to complete my business with you. You mean you came here just for that? Exactly. Well, as I told you in my letter, Mr. Mitchell, this is a really big proposition. Yes, it must be.  I suppose I give you some of the details. Yes, I wish you would. Well, as you know, I'm the American representative of British Farm Equipment Limited. Yes. Our headquarters, of course, are in London. Oh, I know of the first. My principals ask me to contact you and find out just how much interest you might have in working along with us on a very ambitious post-war program. Here in the States? Yes. Confidentially, we plan to open a factory here as soon as possible. I see. We'd like you to take charge of the plant. Me? Me take charge? That's right, sir. But why should you be interested in me? Well, we made a complete survey of all men in the farm equipment field. It should flatter you to know, Mr. Mitchell, that your name came out at the very top of the list. You don't say. So that's why I'm here. Well, I certainly am flattered. Now, I'm not authorized to close any deal with you. I merely came to sound you out and turn in a report. Of course. What can I tell the Home Office, Mr. Mitchell? Are you interested? Yes, very interested. Good. Sir Richard Barton, our financial director, is arriving in this country in the next few weeks. He'll undoubtedly want to get together with you. Can I cable and say that you're willing to talk to him? Of course. Thank you, sir. Well, this has been a very satisfying visit. The first act in any swindle must sound completely authentic. Must convince the victim, must stand up under any investigation he might make. The facts the confidence man gave regarding the farm equipment company were true, with only one exception. He did not represent the company. His victim, naturally, did not know this. So, several weeks later, an answer to a summons from the swindler, Mr. Mitchell traveled east to meet one Sir Richard Barton at a hotel in Newport, Rhode Island. Some more bread, Sir Richard? No, thank you. And how about you, Mr. Mitchell? I have some thanks. What was I just talking about? The patent rights on that tractor, Sir Richard. Oh, yes, of course. We have exclusive rights, you know. The tractors can be produced at an extremely low cost. We can undercut the whole market. You mean here in this country? All over the world, Mr. Mitchell. Where? I can tell you, confidentially, that the British government is interested in our company to quite the large degree. That we don't talk about though? Of course. Frankly, that's the main reason for my visit to America. Oh? I have to contact your State Department on some cooperation matter. I'm doing it through our own embassy. I see. I may have to ask you to come to Washington with me, Mr. Mitchell. Why? Because of your extensive knowledge of this type of manufacture. You might be called on for some off-the-record testimony. Who had I? I'd like to help any way I can. Fully for you. I had an idea that you liked a... Say, yes? What is it? What's this? What? A wallet. I felt it with my foot. It was under the table. Is it yours, Sir Richard? No, it's not mine. Uh, Mr. Mitchell? No, it doesn't belong to me. Well, what do you think I should do? Look inside, Perry. See if there's any identification.  Well, then, here. Let me have it. Yes, sir. Probably dropped by whoever was dining here. Oh, I see. Look here. What? One thousand dollar bills. Five of them. Hey, Colin. Here's an identification card. Lee Jackson Surf Hotel. He's a guest here. Definitely. A pretty careless fellow. I should say it's fortunate to him that we're the ones who found the day. Yes. Perry, contact Mr. Jackson. After dinner, have him come to my suite, will you? Oh, answer that, will you, Perry? Yes, sir. Another drink, Mr. Mitchell? No, not right now. I'll spend a night to dial it. Yes? The name is Jackson, though. Lee Jackson. Hello. You told me to desk. Somebody up here found my wallet. That's right. That's right. Come in. Thanks. This is the man who lost the wallet. Oh, how are you, sir? Okay. You're Mr. Jackson. That's right. Well, I have your wallet here. I think for obvious reasons, you should identify its contents. Sure. Will you check the items, Mr. Mitchell? Oh, yes. Yes, of course. Five one thousand dollar bills. Mm-hmm. Any order of receipts? Three, I think. That's right. An ox card and a dress book. Two letters and a racetrack badge in my name. I think that's all. Does that tally, Mr. Mitchell? Yes, yes, it does. Very well. Let him have the wallet. Thanks.  I know it. I'd like to do something for you. Oh, that isn't necessary. Look, take this grand and cut it up between you. Thank you, Mr. Jackson, but we're really not interested. Definitely not. Right, Mr. Mitchell? Of course. Well, I've got another idea. Yes? This you can't thief about. Look now. I'm around racetracks, which you call a betting commissioner. Yes. I'll take this grand and put it on a horse for you. The right kind of a horse. Please, Mr. Jackson. The wins, we cut up the profits. Now, what's wrong with that? Well... How about it? What do you think, sir Richard? Oh, I suppose we have to make some compromise with Mr. Jackson. And you, Mr. Mitchell? Well, I have no objection. Okay, gentlemen. It's a deal. Act two of the swindle is always the buildup. The victim is let in on the ground floor, has given something for nothing usually in great abundance. Mr. Jackson, the latest addition to the rapidly growing confidence ring, returned to the hotel suite the following evening. Returned with a $3,000 profit. His confederate suggested that the money be bet again. He agreed to this, and the second horse won, and the profit was $14,000. Another horse was played, and another. Until at the end of three days, the trio had a profit of $120,000. As a figures, gentlemen, it's amazing. Amazing. Well, where do we go from here? Well, I... I don't know. What do you think, Perry? You've got me. Mr. Mitchell? $120,000 is a lot of money. Yes, indeed. That's $30,000 apiece. I know, Mr. Mitchell. Can I put in my two cents worth? By all means, Jackson. Please do. These horses I've been playing weren't exactly stiffs. Naturally, I had to have information. Of course. I can just about guarantee you that the information is going to continue. Really? I'm expecting a bookmaker up here. You'll have to pay off money with him. Meantime, do a little thinking for a minute, and see whether you want to take one more step. Very well. Look, I don't have to do any thinking.  In which direction, Perry? I'm for playing one more horse. Well, it's very sporting. And by Joe, I think I'll stick along. I still say $120,000. There's a lot of money. Oh, come now, Mr. Mitchell. After all, we're gambling with their money. That's true. Well, him. Why hesitate? Well, all right. I agree. Capital. Capital. That's probably the book. Come in. Mr. Jackson. Come in. Come in. Come in, Frankie. OK. This is Mr. Perry, Mr. Mitchell, Mr. Barton, Frankie Stevens. How do you do, Mr. Stevens? I've been holding a stockholder's meeting, Frankie. Just about made up of money. I've played $120,000 back with you. Oh. Just like that, huh? What's the matter? Well, I come here to have a little talk with you about that $120,000. Does this mean he's reneging? Now, look, Mr., I don't operate that way. I've got your dough right here in this little bag. But first, I got to find out something. Well? You guys have been getting a free ride. How? Suppose you had lost $120,000. Would I have collected? Well, would I? Well, that's hardly the point. It's very much the point, Mr. In fact, that's the only way you're going to get your money. What do you mean? Give me $120,000. And I mean cash. Then I'll make my pay off. Oh, look, Frankie. That's my deal, boys. Now get it up. After the fourth member of the conference ring, the bookmaker had departed from the hotel suite. A conference was held. A conference which resulted in the victim, Mr. Mitchell returning to Chicago, accompanied by Perry, to negotiate a loan to pay his share of the bookmaker's demands. Now the second act of the swindle has been completed. The victim is primed for the kill. Obtaining $30,000, $30,000 in cash, Mitchell and Perry returned to Newport, returned to the surf hotel, returned for the final act. Now let me see. You've given me your $30,000, Perry. That's right. I have a draft here for $60,000, which was sent to me from New York. I'm putting up Mr. Jackson's share, you know? Yes. And you have your money, Mr. Mitchell? Yes, right here. Splendid. Let me have it, please. Oh, what for? Perry has agreed to go to the bookmaker's establishment. He'll show him our capital and collect our profits. Oh. Oh, very well. What about the bet? How's that? I thought we agreed that we were going to take a stab with one more horse. Oh, of course, Mr. Jackson. Of course I... It completely slipped my mind. Perry will still have to bring the dough to the book. Well, how much are we betting? Our winnings. We don't touch any of this money that we're putting up, this credit money? Of course not, Mr. Mitchell. What's the horse, Jackson? Oh, yeah, here, I... I wrote it out in a slip. You play him the place. Is that for a second? Yes. What's the name of the horse? Why? Well, shouldn't we know who we're betting on? Do you know who we were betting on before? Do you know what horses put us 120 Gs ahead? No. Jackson's right, Mr. Mitchell. He has more than established his honesty to date. Let me have the money. Very well. Here you are, Perry. I'd better be getting over there. We'll wait for you here. Good luck, old boy, and good hunting. Did they announce any results yet, Mr. Jackson? No. Are you sure you're tuned to the right station? Yeah, yeah, we'll get it in any minute now. You know, this is most exciting. Do you agree, Mr. Mitchell? Yes, a bit nerve-wracking, too. Well, I always say, as voting chance is great. It's your race at Jamaica, off at 321 and a half. They win a post-3, Sunboy. Is that our horse? A second, post-11, Victory Ride. That's us. That's our bet. Well, we won? Yes, sir. We picked him in the right slot, too. We played him the place. Congratulations, Mr. Jackson. Congratulations. This is wonderful. You should pay better than three to one. Heavens, we've made a fortune. Come in. Billy, you're just in time. We won, Mr. Perry. We won our bet. On the level? Yes. The horse won? No, ran second, but that's good enough for us. Second? Yep. You mean it didn't win? What's the difference? We still collect. Oh, that's awful. What's wrong? Oh, I... I did a terrible thing, gentlemen. What do you mean? I bet on him to win. You did what? But I told you... I know, I know, but he was ten to one. I figured you've been lucky right along. Perry, that's disgraceful. Oh, yes, you haven't heard the whole story. What do you mean by that? I... I bet our credit money, too. What? Why are you... I'll keep away. Jackson, don't. Stop it. Perry has a very bad heart. Jackson. Good heavens. Perry. Perry, are you all right? He's unconscious. Yes. Mr. Mitchell, get a doctor, please. There's one on the second floor. Second floor? Yes. Hurry. Very well. All right, boys. Let's blow. We momentarily close the Federal Bureau of Investigation file on the Confidence Ring. We'll return to this case in just a moment. We Americans must think very highly of life insurance because we buy more of it than all the rest of the world put together. 70% of the life insurance now enforced on this globe is owned by citizens of the United States, all of which goes to prove that we Americans believe in self-help and self-reliance, believe in watching out for our futures, believe in educating our own children and in taking care of our families. Of the large number of Americans who feel that way, 3,200,000 have joined forces as members of the Equitable Life Assurance Society of the United States. Please take particular note of that word society in the Equitable's name. It means that the Equitable is not a profit-making enterprise, has no stockholders, is entirely owned by its members, that is to say, by its policy holders. Dollars entrusted to the Equitable Society to build security for its members are put to work in ways that benefit the whole country, are invested in homes, farming, and in scores of the key industries on which our national security and prosperity depend. By serving its members, the Equitable Society serves America. And now, back to the file on the Confidence Ring. The Confidence Game has an ancient and dishonorable history. Its existence can be traced back to the very beginning of time. Through the years, through the centuries, the pattern has varied, but one element has always remained, the gullible victim. Mr. Mitchell returned to the hotel suite in Newport, Rhode Island to find that his erstwhile companions had left, had disappeared along with his $30,000. He was stunned and bewildered he returned to his home in Chicago.  When I finally located the doctor, I brought him right to the suite. To my amazement, I discovered that everyone was gone. Including the men who had been assaulted? Yes, sir. What did you do then, Mr. Mitchell? Well, to tell you the truth, I was too mixed up to do much of anything. You inquired for them at the hotel desk, see if they'd checked out. Eventually, yes. They hadn't paid their bills. They'd just left the suite and never returned. I see. I wound up paying for it myself. Now, this money you gave them, the $30,000, what were the denominations of the bills? 20s, 50s and 100s. Did you by any chance keep a record of the serial numbers? No, I regret to say I didn't. You see, at the time I... Well, I just didn't suspect. I understand. How did this man who called himself Perry first contact you? He wrote to me from New York. Did you keep the letter by any chance? Yes, I have it on file in my office. We'd like to see it, surely. How accurately do you think you could describe these men? Oh, quite well. I'm sure of that. Then, suppose you give us the descriptions. We'll send them to Washington, have them checked, and the general appearance file. The general appearance file is an excellent example of FBI thoroughness. This file consists of photographs and descriptions of approximately 15,000 outstanding criminals. Special agents for the identification division checked this file for information on the confidence ring. They removed punch cards of all persons falling into the general classification of the descriptions furnished. The general appearance file is an excellent example of FBI thoroughness. They removed punch cards of all persons falling into the general classification of the descriptions furnished. They placed these cards in a sorting machine. A machine adjusted to sort and finally select by mechanical means cards bearing descriptions similar to those of the suspects. Within a short time, the machine had done its job. Photographs which bore the same numbers as the cards that have been mechanically selected were sent by air mail special delivery to the Chicago field office. Can I see you for a minute, Russ? Yes, come in, Lou. I've just been with Mr. Mitchell. I had him check over the photographs. Any results? Yes, he positively identified all four men. Good. He really was up against a pretty strong team. Here, take a look at them. Are you well? The old boy there, the one who calls himself Sir Richard is one George Willis, an English confidence man. I see. The rest of the gang all have quite impressive records, too. I received a report from the Providence office this morning. The agents did a check at the Newport Hotel. Are they uncovering it? No, there was no indication as to where the gang had gone. Well, how about Scotland Yard? We heard from them late yesterday afternoon. British Farm Equipment Limited is a legitimate company. How about Sir Richard Barton? Oh, he was financial director of the company, but he died in 1940. This gang stayed reasonably close to the facts. Yes. Oh, by the way, Mr. Mitchell gave me that letter he received from the man who called himself Perry. May I see it? Oh, surely. Here you are. Thanks. Take a look at the letterhead. British Farm Equipment Limited. I doubt that it's genuine. New York address? Probably a vacant lot. Yes, but the letter was mailed from New York. I know. If this stationery is faked, it might have been printed there, too. You're right. Let's send it on to the New York office and find out. A slender lead, a letterhead. A letterhead that might have been set up in any print shop in the country. But the letter had been mailed from New York. So agents in that city were given the assignment of checking with every print and stationery shop in the five boroughs. It was a slow job, but tedious job. A seemingly endless job. But a job that finally brought results. Lou. Yes, Ross? A teletype just came in from New York. On the Mitchell case? Yes. What's the story? That letterhead really paid off. How do you mean? Well, first of all, they checked the address. You were right on that. It was a vacant lot. Well, that figured. They were making on the print shops really uncovered something. They picked the men up? No, it wasn't that strong. But they did find the printer who'd done the job. He said it was for a man named Phil Renville who lived in a midtown hotel. I see. The agents went to the hotel, contacted the manager and found that Renville had checked out just this morning. What a break. Judging by his description, Renville is really Mr. Perry. The hotel manager also revealed that three other men had shared a suite with him. Then they're all still together? It would appear that way, yes. Any idea where they went? I believe we have. Good. The hotel manager recalled that Renville had done some business with the transportation desk. So the agent checked there and found that Renville had bought four railroad tickets. Well, we'd better have that train covered. We don't have to. Why not? We're going to cover it ourselves. Why? Sir Richard and Company are aboard the 20th century. They're due in Chicago here tomorrow morning. Ah, this is good to stretch the legs again, eh, boys? Yeah.  I am. This early morning deal is murder. How long await do we have between trains? A couple hours. Shall we take a stroll about town? We'd better get over to the other station first. Other station? That's where the chief leaves from. Oh, we ought to get our bags over there. Very well. Mr. Perry. Huh? Mr. Perry. It's the umche. Good heavens. Hello there, Mr. Perry. Sir Richard. Greetings, old boy. Mr. Mitchell. Well, this is a 10 strike. You've no idea how worried we'd been about you. About me? Yes. But in the world, it never happened to you. Whatever happened to my $30,000? Oh, you already know that. I mean, what really happened to it? Look, Sir Richard. Yes? We're wasting time. We have to make that train. I think you should answer Mr. Mitchell's questions, gentlemen. Who are you, sir? We're special agents of the FBI. Just stay together, Oliver. May I ask a question? Yes. What is it? Can we dispose of four tickets on the cheap? The Swindler's drama is an old one, and it's worked with little variation. It's three acts of the meeting, the build-up, and the payoff. But since 1934, since the passage of the National Stolzmann Property Act, the FBI has been able to add a convincing finish, a finish that was not in the Swindler's script. This finish is apprehension and arrest. Confidence, men, as well as all criminals have blot on the American way of life. Special agents of the FBI will continue to ferret out these individuals. And with cooperation of the public and law enforcement agencies, bring them to justice. You'll hear the disposition of tonight's case in just a moment. Will you join the Equitable Society in a salute? A salute to a man so familiar to you that you tend to forget how important his wartime services are. A salute to the hard-working merchant who runs your corner store. Yes, your butcher, grocer, druggist, and the other small but important businessmen on America's main streets deserve an overwhelming vote of thanks. In spite of shortages, point rationing, and manpower difficulties, they're still doing business at the same old stand. Still proving that in a land of opportunity the right kind of men can always get ahead. The Equitable Life Assurance Society of the United States is proud of its many associations with these typically American institutions. And we firmly believe that the millions of dollars of Equitable Society funds that help finance business may truly be called investments in victory, just as are the other Equitable Millions that have gone into war bonds and war industries. In wartime, Equitable Society dollars are fighting dollars.  In tonight's case, a federal grand jury convening in Boston indicted Barton, Perry Jackson, and Stevens. They were tried, found guilty, and sentenced to eight years in a federal penitentiary. The incidents used in tonight's broadcast are taken from the files of the Federal Bureau of Investigation. However, all names used are fictitious, and any similarity thereof to the names of persons living or dead is accidental. Tonight, the music was under the direction of Van Cleave. Your narrator was Frank Lovejoy. This is your FBI is a gerrydivine production. And now this is Carl Frank speaking for the Equitable Life Assurance Society of the United States and inviting you to tune in again next week at this same time for this is your FBI. This is Infantry Day, a day dedicated to the Doughboy, the Foot Slogger, the indispensable soldier who storms enemy beaches, knocks out pillboxes, meets the foe face to face and hand to hand. No war was ever won without infantry, and no infantry ever surpassed the heroic American troops we honor now. But remember, they can't reload their weapons with words of praise. So be practical about Infantry Day. Cheer G.I. Joe and his leaders by giving them ample material aid for the tremendous task ahead. On Infantry Day and every day, buy war bonds in the Seventh War Lone Drive. This is the American Broadcasting Company.


So our next speaker this afternoon is Piers Keen from the University College London and Moorfields Eye Hospital by the NHS Foundation Trust. If you are a reader of nature medicine, then over the last few years may have noticed a number of very interesting papers on machine learning in oftalmology. And Piers is a driving force behind this work. So there's a big collaboration between Moorfields and Google DeepMind with exciting results. And I'm sure we'll hear more about these in a minute. So Piers is a medical doctor by training. He's now an associate professor at the Institute of oftalmology at the University College London and holds an appointment with the Moorfields Eye Hospital. He has won a number of awards in 2015, a clinician scientist award from the National Institute of Health Research and this year, a UKRI Future Leaders Fellowship. So we are very excited to welcome him here and to learn more about his research. Thank you for joining us, Piers. Thank you. Thank you, Karsten. It's a real pleasure to speak to you. And I'm gonna talk to you, I'm gonna talk to you about what I believe is the potential for machine learning to transform oftalmology. I think it's fair to say that there's a lot of hype around artificial intelligence and machine learning for healthcare. But even despite all the hype, I think there is potential to do something quite exciting if we approach things in the right way. And one of the things that I'd like to try to make a case for over the next hour is that oftalmology is one of the medical specialties that's at the forefront of these transformation. And I think that there's a lot of our experiences we can share with other medical specialties about not just the development of machine learning tools, but also the validation, the implementation and the adoption of these systems. So Karsten, we've already summarized my background, but I have a dual appointment. So I'm a consultant oftalmologist. So like a senior oftalmologist at Moorfields Eye Hospital in London. And Moorfields Eye Hospital is the oldest eye hospital in the world. And one of the largest eye hospitals in the world. At Moorfields, I specialize in the treatment of retinal diseases. So conditions such as age-related macular degeneration, which is the commonest cause of blindness in Europe. And diseases such as diabetic retinopathy, which is the commonest cause of blindness in working age populations in many countries in Europe and around the world. As you said, I'm also an associate professor at University College London at the Institute of Ophthalmology. And I lead a clinical research group at UCL, which is focused on AI-enabled healthcare. So we have a lot of work that we do with industry. So in particular with DeepMind and with Google, but we also have a lot of work that's separate from industry that's purely in an academic setting. And I'll tell you a little bit about both of those things during this meeting. And then the other thing is that I'm very privileged to be funded by UK research and innovation as a future leader. And this is a relatively new funding scheme and funding organization. And so it gives me the rare opportunity in that I'm a medical doctor, I'm a clinician, but essentially I have 80% of my time protected for research. And I think that that's the thing that has really facilitated my ability to work with machine learning collaborators because it would be very hard. And I'll bet some of the people listening would have experiences working with clinicians where it can be challenging in particular when the clinicians are working 100% of their time in clinics and surgeries and other things. The other thing that I would say is that I have acted as a consultant for DeepMind. I'm an act as a consultant for pharmaceutical companies such as Roche and Genentech and I'm involved in some machine learning work with them as well. But I think given that I'm gonna be talking a lot about the collaboration between Morfields and DeepMind and Google, it's important that I volunteer that financial disclosure at the start. Now, in terms of an overview of what I'm gonna discuss over the next 45 minutes to an hour, probably more than 50%, maybe two thirds of the talk will be about the Morfields DeepMind collaboration and some of the research outputs of that collaboration. Karsten, you mentioned some of the articles in Nature Medicine and I'm gonna talk about some of the results there. But then in the last third of the talk, I'm gonna focus more on work that has come out of that collaboration but is independent from DeepMind and from Google. And so in effect over the last five years, we have gone from at Morfields, we have gone from knowing nothing about clinical AI to having a lot of expertise and infrastructure to do clinical AI projects. And one of these is something that we call the Insight Health Data Research Hub. And I'll explain what that means. Another thing that I'm particularly excited about is called the ELTS Eye Study. And this is the idea of using the eye as the window to the brain, sorry, to the rest of the body. And then lastly, I'm gonna speculate a little bit about an emerging ecosystem for clinical AI. And in particular, I'm gonna give some of my thoughts about the potential for automated deep learning and automated machine learning. And I think that I'll be very interested to see, to hear what people's comments and criticisms are on that, particularly given that this is an audience with machine learning experts. And so I may try and say some kind of controversial things to just provoke a little bit of discussion, but I'm very happy to be pushed back against in some of those comments. So first of all, let me just set the scene. What is the problem that we face in ophthalmology? Why do we need new technologies and innovation and why do we need AI? Or maybe this is just hype and AI is just another way for us to publish some papers and get some grants and things like that. Do we really need it? And so this figure is something that I love showing to clinical audiences, particularly when I'm speaking to doctors from other medical specialties. So in 2017, ophthalmology overtook orthopedics as the number one busiest of all the medical specialties in terms of clinic appointments in the National Health Service in the UK. In fact, nearly 10% of all clinic appointments in the NHS are for ophthalmology. That constitutes nearly 10 million appointments per year, and that's a number that's going like this. If you plot orthopedics over the last five years, it's like this. If you plot ophthalmology, it's like this. So in some senses, and maybe this is not an appropriate metaphor, but in some senses, we're drowning in the number of patients that we need to see. And if you go to an eye clinic, whether it's in London or Vienna or Boston or Beverly Hills, it's going to be very, very busy. I can guarantee you that. And it oftentimes standing room only in the waiting rooms. Now, the upshot of that is that there are some patients who are potentially losing sight and even potentially going blind because they cannot be seen and treated quickly enough by human experts. For example, a retina specialist like me. Now, this was particularly illustrated in January of 2020. There was a lot of media attention in the UK because there was a very tragic case towards the end of 2019 that I appreciate it's sort of anecdotal evidence, but nonetheless, it highlighted the challenges in the system that we face. And so this was a case of a woman in her 30s. I think she was 36 years old in the south of England. And this woman was pregnant and she also had advanced glaucoma. Glaucoma is an eye disease where you have high pressure in the fluid of your eyes that damages your optic nerve. And it's a common cause of blindness. Now, what happened with this woman is that she needed to have urgent glaucoma surgery performed. And for whatever reason and for whatever chain of events, there was delays in her getting the surgery and she lost her sight completely. She went blind. Now, I cannot think of a more tragic scenario than a young mother who has lost her sight and perhaps it could have been prevented if the intervention had been done in a timely fashion. And so I believe, as I say, and it's not perhaps not a surprise, I believe that the latest advances in machine learning can at least begin to address some of these problems that we face. So essentially, this is the reason why I initiated this collaboration between Moorfields Eye Hospital and the artificial intelligence company DeepMind. Now, I should highlight to people in this audience, particularly if you're asking any very detailed technical questions later, is that I'm coming at this from the background of a clinician. I don't have any formal machine learning or data science or computer science training or anything like that. And so I contacted DeepMind just because I knew that there were people going blind because they couldn't be seen and treated quickly enough. I knew that there was massive advances taking place in the machine learning community over the last five, seven, 10 years or so, in particular with regard to deep learning. And I knew that we had huge data sets of atomic images and I'll tell you a little bit more about those. And so I was looking around for collaborators who had deep learning expertise. And so initially I was looking for some academic collaborators back in sort of 2014, 2015. And I didn't really find anybody that was particularly interested at that time, although I have subsequently. And so actually what happened was I was reading, I was reading a profile about DeepMind in Wired Magazine in July of 2015. And I had known about DeepMind because I've seen the media attention when they had been acquired by Google. But what really resonated with me was that two of the three founders of DeepMind are alumni of UCL, and which is of course the university affiliated with Moorfields and two of the three of the founders are from London. And of course the other thing was that their headquarters are based in the King's Cross area of London. And for those of you who know London, that's only two stops away from Moorfields on the Northern line on the London Underground. So it's like a very, it's a very short distance between the hospitals. And so I approached them and Moorfields then sort signed a formal research collaboration agreement in July of 2016 with them. And essentially what we wanted to do is develop a deep learning system that could look at these eye scans, these scans of the neural tissue at the back of the eye, which is called the retina. And these eye scans are called OCT scans or optical coherence tomography scans. And OCT is kind of like an ultrasound but it measures light waves instead of sound waves. And in doing so it gives very high resolution three dimensional images of tissue. So the axial resolution of an OCT scan is approximately five micrometers. So if you think about it, that's an order of magnitude higher resolution than a CT scan or an MRI scan. And OCT has become the dominant image in modality and ophthalmology. In fact, we do more OCT scans than all other ophthalmic imaging combined. And then we do more than a thousand OCT scans per day at Moorfields. So it seemed to me that this was a nice overlap between a pressing patient need and a situation where we had large amounts of data that could be amenable to machine learning. So the research collaboration agreement actually, I mean it was signed in the middle of 2016 but it had actually begun when I had read the article in Wired in July, 2015. And in the article, the part of it described, one of the co-founders of DeepMinded, Mustafa Silliman, who's leading their applied division. And he said, I'm interested in applying AI to healthcare, to climate change, to energy consumption, making the world a better place. And so he was the person that I contacted to initiate this collaboration. And the exciting thing for me was that, just a few days after I sent him a message, I found myself sitting in his office and telling him about the potential of this collaboration. But I remember going in and saying, we've got so many patients, we're doing thousands of scans per day or per week. People are going blind. And he said to me, well, how many OCT scans do you have in total? And I didn't really have an answer to that. And he said, what are the different systems? What's the quality of the scans? What proportion of the scans do you have labels for? You say that diabetes uses a common cause of blindness, but how many of the scans are from diabetic patients? What are the file formats? Are they proprietary file formats? Are they open source file formats? All of these questions. And to be honest, if I look back now, I feel embarrassed because I was so clueless. I didn't really have any answers to the questions that he was asking me. And he asked me equally important, he asked, well, what are the ethical and the governance requirements? Because this would involve sharing these scans with the company that's owned by Google from the NHS. And again, this was a collaboration. I just knew that there was this urgent clinical need and this technology could address it, but I didn't know all that would go into such a project. And so actually it took a lot of teamwork behind the scenes at Morefields to begin to figure out at least some of the answers to those questions. And an interesting thing that I would say actually that I've observed over the last five years is that deep mind and Google are very commonly approached by clinicians who are in a similar situation to me and they know that they've got some big data set, they know that they've got a pressing clinical problem, but in the meeting with them, you know, deep mind or Google or sometimes me would say to them, okay, well, can you answer some specific questions about the data set? And they usually can't. And they say, at the end of the meeting, they say, okay, we'll find out the answers to those questions and we'll get back to you in about a week's time or two weeks time with all the answers and then we'll proceed. Nine times out of 10, you never hear from them again because they find it very challenging to get the answers to those questions. So when we, I think we were good because I think we were kind of tenacious and when we did answer all those questions we could put in place a data sharing agreement and we ended up sharing back in 2016 about 1.2 million anonymized historical OCT scans with deep mind and that began this collaboration. Now, the thing is that, you know, this is a sensitive and appropriately sensitive area because this is, like I said already, this is sharing NHS data in the UK with the company that's owned by Google and a multinational tech company. And so from the start, we wanted to be as transparent about what we were doing as possible and to make sure that we did everything right or do our very best to be cautious and careful and respectful of the use of people's data. So from the start, if we put a section of the Moorfields website, which is dedicated to the collaboration and it's still there now and you can go and have a look at that and there's videos and there's FAQs and there's things like, you know, if you want to opt out of this research, here's the email address of our information governance officer, giving as much information and being as transparent as we can. If you go to the waiting areas in Moorfields, you can see that we've got sort of flat screen TVs to try and see people happy while they're waiting, while they have long waits for their appointments sometimes. And one of the things we do is we have screen savers that come up intermittently and one of the screen savers is about the collaboration and it tells people what we're doing and it points to people to where they can learn more about the work that we're doing. The other thing that we did was that before we had done any sort of experiments whatsoever, we published our protocol in open source machine learning or open source peer reviewed publication and then we could finally sort of get to the science and it was very exciting for us then in August, 2018, Karsten, as you've mentioned already to publish our work in the Journal of Nature Medicine. You can see that there's more than 30 different co-authors here and this reflects the fact that we had experts from UCL, experts from more fields and experts from DeepMind and the key point is that this wasn't just giving them the data and then coming back to later with a publication, this was very much a research collaboration from the very start and I don't think it would have been possible if any of the three groups were not contributing. Now, it was also very exciting for us because it was published on the cover of Nature Medicine and of course this is AI so there's a lot of hype and it was very exciting that it got global media attention. Now, you know, it's thrilling, I think for me it's a thrill to see your work getting widespread attention. It's also a little bit awkward because the fact of the matter is that we have not saved the sight of millions of people yet and in fact, what we've published here in Nature Medicine is a proof of concept that a machine learning algorithm can do this and in fact, if you were to email me after this talk and say, can I come to more fields and see it in action, you wouldn't be able to because it's not in action yet. We're not using it in the real world yet and what I've come to learn in the last few years is that going from the research publication or the conference presentation to implementation in real life is really, really hard and in some ways, doing the first step is kind of the easy step and before this, I always presumed that the hard part was publishing the paper and that's not the case. So, in any event, what we did was we created you know, a deep learning framework that consists of two neural networks. The first neural network was a segmentation network. We'd feed in the raw OCT volume and we trained it with about 900 manually segmented OCT volumes and that allowed it to create this intermediate representation that could be fed into a classification network that we trained with about 15,000 labels and ultimately would give a classification output such as urgent referral, semi-urgent referral or a diagnose up to 10 different retinal diseases. So anything that you would expect an ophthalmologist to be able to look at and see on an OCT scan. Now, we created the tissue segmentation map as an intermediate representation for a variety of reasons, but one of the nice byproducts of that is that it gives some clinical interpretability to the model and also it gives a range of kind of interesting quantitative metrics for clinicians when they're treating these patients. So this is just an example of the video of the model running. And so at the bottom row, you can see an OCT volume and the bottom right, you can see the figure legend of about probably about 10 or 15 different anatomic parameters that are segmented. This is a patient with diabetes. This patient in his 40s got poorly controlled diabetes and you don't have to be an ophthalmologist to see that the retina is very swollen to waterlogged. This is called retinal edema. And in the top row, you can see referral suggestion is semi-urgent referral and diagnosis probability macular retinal edema 98.5%. Now, we wanted to evaluate the performance of the model. So we got 1,000 new patients that had presented to Morfields that had had a macular OCT scan done at presentation. And let me tell you, this was a retrospective evaluation. So this wasn't perspective clinical trial and that's an important distinction in terms of validation of these systems. And essentially, we ran the algorithm on those 1,000 cases and we looked as a primary outcome at the errors on the referral decision. And what we found was here that it had about a 5.5% error rate on the referral decision. This is the triage decision because remember our use case is not replacing the retina specialist. Our use case is targeting those people with cytraine disease to get them in front of a retinal specialist who can actually make the diagnosis and get the treatment. Now, we want to benchmark the performance against human experts. So we got eight human experts and experts one to four are senior retinal specialists at Morfields and experts five to eight are optometrists at Morfields who have a little bit less experience in looking at these scans. We locked them in a room and said, look at all the 1,000 scans and give your triage decision, your referral decision and give your diagnosis and all of those things. And we looked at their errors and the gold standard that we took was when the patients had seen a retinal specialist and they had not just had an OCT scan but they had, you know, fluorescein angiography and a range of other tests and then they were treated and followed up for a six month time period. And that was our reference standard. Now, how did the human experts do? Well, what we saw was that the algorithm did better than all eight human experts except for experts one and two. When it was better but it wasn't statistically significantly better. Now, one thing that I've learned over the last five years is that whenever you see the hyper and AI and you see superhuman performance or a headline like beats the best doctors, when you dig into the details, you often find that the doctors are not doing a task that they do in their day to day life or they're doing a task with one arm tied behind their back and it's not a fair comparison with the real world. And indeed, this is not a fair comparison because in the real world, the human experts would never make a triage decision based on an OCT scan alone. They would know the visual acuity of the patient. They would probably have a retinal photograph and they might have, they would likely have some history, some details of the patient. Now, to make it fairer and to simulate that, we repeated the experiment with the human experts and we gave them all of that additional information and locked them in a room again and asked them to reanalyze the stands but in a different order and after a time period. And it wasn't a surprise to us then that the human experts, that improved the performance of all the human experts and it got to the point where expert number one was able to equal the algorithm using the OCT scan alone. And let me tell you, experts number one and two are world famous ophthalmologists who have more than 20 years of experience each. Now, interestingly, the algorithm did as well as it did with just the OCT alone but adding that additional information didn't really improve the performance of the algorithm significantly. So what did it get wrong? Well, I'm not gonna go into the confusion matrices here but what the BBC picked up on was that AI did not miss a single urgent case. Now, from my perspective, what was interesting is that when I looked at the expert number one, the human expert number one and saw that they had a 5.5% error rate, well, what did they get wrong considering that they are one of the world's leading experts? Well, actually the cases that they got wrong were ambiguous cases. And an actual fact, when we looked post hoc, maybe our human experts label was correct and our reference standard was incorrect with the benefit of hindsight. But of course, we can't change it after the fact. Now, interestingly, when we looked at the errors that the algorithm made, that was also the case. So a lot of its errors were not really errors when we looked at it after the fact. Now, where are we going with this? Well, at the moment, we're trying to bridge what I call the AI chasm. And this is a term that I read, I read an article in TechCrunch a couple of years ago and I was talking about this in the tech world. And this is the idea that if you're a startup, it's sort of easy potentially to do a cool demo of a machine learning system, but it's a very different thing to have a scalable deployment and meaningful human AI interactions at a large scale. And if it's hard in the tech world, I think it's extremely hard in healthcare to bridge that chasm. And so that's what we've been trying to do since we published the paper in 2018. So what are the different ways you need to do that? Well, one of the things is working on the technical maturity of the algorithm. And as a non machine learning expert, I think it's fair to say that most doctors don't realize that the piece of experimental code that is reported in a research paper is not really something that can be deployed at scale in a piece of software around the world. And so the team at Google have been working to essentially rewrite the algorithm so that it can run with a fraction of the computing power in a fraction of the time and make it into a cloud-based API. And the cool thing was that we did a live demo of that in Wired Health in March 2019. And we actually got a patient with AMD, we got their scan and live on stage, we sent it to the cloud, we ran the algorithm on it, we logged in with a different laptop and showed the algorithm giving an output.  The other thing that I think it's important to emphasize is the need for better clinical validation. And this is a paper that came out in the Lancet Digital Health that I'm a co-author on in September, 2019. And it was a systematic review and meta-analysis of essentially deep learning papers in medical imaging. And really the take-home message is that the quality of those papers is very, very poor in terms of clinical validation standards. And only a tiny percentage of papers are of a good clinical standard. And so we are very aware of this. And so we've been taking, I think, a very measured approach to validating our algorithm. And so one of the things we've been doing now is trying to validate it in OCT scans taken from eye clinics all around the world. So we want to make sure that an algorithm that we've developed in London works as well in Ghana or Brazil or the US or Asia or any other countries around the world. And it turns out that there's lots of complexities to that. Now, the other thing with implementation is that it's not enough just to have an AI system or a machine learning system in isolation. It has to be able to fit into pathways. There has to be billing within the healthcare system to be able to support the machine learning system. And so Moorfield is where I'm a consultant is building a new hospital in King's Cross in London beginning in 2021. And we're trying to anticipate now what flexibility we'll need in our clinical pathways so we can take advantage of these breakthroughs in machine learning technology as they come through. The other thing is that we're increasingly seeing OCT being used in the community and community optometry settings. And so we're trying to think about how we can link the community optometry clinics with the hospital system and then put a machine learning system on top of that. And of course that's the considerable challenges associated with that. And then there are in fact OCT devices which are being built for home use and a number of companies are in advanced stage in developing those. So ultimately we see a chance with this technology to bring world-class expertise into the community and into people's homes. Now, just to say that was the main thing I want to talk about. I'm gonna tell you a little bit more briefly about the follow-up study from the Moorfields DeepMind Collaboration. And then lastly, I'll tell you much more quickly about some of the other studies that we're doing. But just to say, we're really proud that our work has been patient centered from the start. This is a patient of mine called Elaine Manna at Moorfields. And Elaine lost her sight from wet macular degeneration more than 10 years ago before there was any good treatments. And in 2012, she started to lose her sight in her good eye and she got an urgent appointment to see an ophthalmologist but the appointment was like six weeks later. And imagine if you were a situation where you're at home and you're losing sight in your good eye and you're told that there's a treatment now available but you need to wait six weeks for it. Well, if that was my mother, I would want her to be seen and treated within six days and not within six weeks. And so that was the motivation for the first collaboration I've just told you about but it also got us thinking about something else which is this idea that patients with eye diseases are really worried about their good eye being affected. And so they may be coping when they've lost sight in one eye but if the good eye is affected that's when they really get stressed about it. And so we began to think could we use machine learning to predict ahead of time the involvement of the good eye relying on the fact that if patients are being treated for their first eye, they typically would have scans done on both eyes for a period of time. So essentially it means that we have a situation where we have lots of normal data or data before a catastrophic ocular event happens that we can use to do a prediction task with deep learning. So if we could do that, then maybe we can protect the good eye with kind of novel treatments and maybe that means we stop the event from happening or we can mitigate its effects once it does happen. So in May, 2020, we published a second paper in Nature Medicine and just to highlight that the joint first author on this paper is a PhD student of mine who's at UCL is an optometrist called Rena Chopra and really was the brains behind a lot of this paper with the amazing team at DeepMind and Google and UCL and others. So what we did with this was this idea, the setup was that you have patients with wet AMD, exudative AMD in their first eye, they're receiving injections into this eye and having regular follow-up visits and at the same time, they're having scans done in their first eye. Now, we created essentially a prediction model that was an ensemble with number five here which is an end-to-end prediction model based just on the raw OCT scans plus also in number two, three and four here, a prediction model that leverages the segmentation outputs from the system. Put them together in an ensemble and then essentially the output of the model was predicting what we called imminent conversion which was development of wet AMD within six months, the site threat and form of the disease. Now, I'm gonna go through this quickly but just to say that in terms of our data set, it was big by AMD terms but maybe small by machine learning terms. It was about 96,000 OCT volume scans and in our test set we had about 5,000 scans and we wanted to predict imminent conversion. Now, the main figure from the paper really is this one, the main and essentially we've got the blue curve and we've got the orange curve and the orange curve looks better than the blue curve but actually the blue curve is the one that's clinically meaningful. That's actually predicting the conversion point and for that we had an AUC of 0.745 and of course that's not spectacular when you compare it to the sort of 0.99 AUCs that you see in a lot of papers but the fact is that the AUC is not clinically meaningful. Really what you have to do as you all know is select operating points on that ROC curve to determine your sensitivities and specificities and those are the clinically meaningful metrics and so we chose what we call a liberal operating point and then more importantly a conservative operating point. So our intended use case for this was that if we could predict imminent conversion then maybe these patients would be able to get a treatment such as an injection into their eye to prevent conversion. Now for that use case we wanted to optimize for specificity because we didn't want to have, we didn't want to be injecting people false positives essentially we don't want to be giving them an invasive treatment if they're not going to converge. So in that scenario we had a with a 90% specificity we had about a 35% sensitivity. So we're not so much worried about the false negatives because we're monitoring those patients pretty regularly anyhow. Now the other thing was that we wanted to benchmark it against human performance and this was actually not a task that human experts had ever done before and they did better than we expected but the fact is that they were very variable as you would expect compared to a machine learning algorithm. Just to highlight the subgroup analysis and this is mainly for an off-talmic audience but the key point is that of the 5,000 OCT scans in the test set the only about 4% of them actually developed wet AMD within that six month time period which sort of highlights the challenge of the task that we set ourselves. I'll just skip ahead through some of those. Even this is just to show that even in the highest risk group it was only 6.7% developed imminent conversion. So where I think this is going is the reason that I'm excited about this is because in 2021 we would like to use we would like to begin a clinical trial which uses this algorithm or some improved version of this algorithm to identify high risk patients for fellow I conversion to recruit them into a trial. Now, I think that that's going to be a very powerful use case for machine learning because it mitigates against a lot of the issues with deep learning in terms of its brittleness, its fragility, et cetera when it encounters data that is different from what it's been trained on. So in a clinical trial setting we can control what type of OCT device we can control the quality of the images and we have a lot of other controls that sort of allow us to maximize the likelihood of getting a meaningful answer from the system. And so I think that's one of the first use cases that we're going to see in the real world in the near future. Now, I know that I've talked a little bit too long some of the things and we've got about 15 minutes left but to include some questions but just in the next five to seven minutes I just want to tell you about some of our other spin-off projects and what building from what we've learned from all of this work that we've done with DeepMind in Google. So the first thing is at more fields we've come to realize in a granular form exactly how much imaging data that we're generating. And so just from one or two different vendors we're generating about 1.2 million of pelvic images per year at more fields. And we've greatly expanded our research database now and we've also moved it to the cloud. So we have a more fields research cloud and essentially we've got a BigQuery database and we've got a lot of these scans in the cloud. And one of the exciting things that we're doing then is that we've got funding at the end of 2019 from Health Data Research UK to share all that we've learned about the aggregation and curation of aftalmic data with other NHS trusts. And so the idea would be to try and create a national bio resource for aftalmic data. And then this bio resource would be available to both industry and academia all around the world to get access to this data to hopefully lead to patient benefit in the future. So that's something maybe I'd love to talk about at some point in the future. The other thing that I'd like to talk about our learnings is we have learned a lot about how to approach the ethics and the governance of using clinical data. That has allowed us to do a spin-off project which involves using the eye as a window to the rest of the body. So this is of course an old idea, but since about 2018, we've come to learn that using deep learning you can potentially see a lot more things looking at retinal photographs. So with this image we can say, this is a woman just from deep learning and no retinal specialist or no ophthalmologist can do that very well.  She's not a smoker. She's not diabetic. Her body mass index is around 25 and her blood pressure is 150 over 85 approximately. Now, this is kind of mind-blowing for ophthalmologists that this is possible. Now, if you're skeptical, you may say, well, what's the point? It's easier ways to tell if it's a man or a woman or the age of a patient than applying machine learning to a retinal photograph or to measure someone's blood pressure. You just put a blood pressure cuff on or you measure it. But I think what we've come to learn is that if we can get the appropriate data sets and apply the appropriate expertise to them, maybe we can uncover some interesting stuff. And so one of the things we've done using our experience now around governance is we've linked about 3 million OCT scans plus 3 million paired retinal photographs with a national database in the UK called Hospital Episodes Statistics. And it took us about two and a half years to get the permissions for this. And essentially that means that we know every patient who's had a retinal scan at Morefields that has gone on to develop a heart attack or has gone on to develop any kind of dementia or a range of other diseases. And in fact, the numbers we have are two to three orders of magnitude bigger than a lot of other cohort studies like UK Biobank study, for example. Now, if you're interested in learning a little bit more about this, this was featured in The Economist in December 2019. And interestingly, this was featured at a point where we didn't have, and we still don't have actually any breakthrough results from this work, because we've just started it. But the emphasis in the article in The Economist, and this is a tweet from the person who wrote it, how Hodson was our attention to detail around the use of health data and the governance of health data. And this is what has allowed us to get to this point. And then lastly, just in my last two or three minutes, I just want to say some interesting, maybe stimulate some conversation, which is that I feel that there could be an emerging ecosystem for clinical AI. And so I think that much of our work has been with industry for the last five years. In the last two or three years, we're developing really good links with academia. But I'm really interested in actual healthcare professionals beginning to explore machine learning, and particularly those without specialized expertise. And so this is inspired by an article I read in The New York Times in 2017 about automated deep learning platforms that are now available from many companies. And these are essentially drag and drop interfaces for the most part that allow people without coding experience to develop a deep learning classifier. If you've got a thousand pictures of cats and a thousand pictures of dogs, you can upload them and you can develop a model to be able to distinguish between the two. So we got excited about this and we got some five publicly available medical image data sets, skin lesions, chest x-rays, retinal photographs, OCTs. And essentially members of my research group without coding experience, were able to train classifiers and get pretty good results within days. And that was tremendously exciting for us, particularly because we've been kind of like looking and all at some of our machine learning collaborators over the last few years and kind of jealous of their capabilities in some sense. And we published this in the Lancet Digital Health in September 2019. And one of my heroes, Eric Topol, said, this will be regarded as a classic paper when AI becomes part of every medical school curriculum. Because whatever you think about whether this kit is actually useful in the real world, this is certainly I think a good tool for clinicians to be able to learn about how to do these things and what are the mistakes you can make and where you need to be careful in a variety of ways. Now an interesting thing that I can't help telling you is that the editor of the Lancet Digital Health said this in the revision process. She said, one point that we hope is that you will be able to address is to emphasize that these tools will not necessarily replace AI experts.  And I'm excited about the technology, but I'm realistic. So I don't think that healthcare is going to be solved in the next five years. And I don't think that artificial general intelligence is going to be in solved in the next five years. So I think that there's always going to be work for AI experts, at least for the foreseeable future and the same for doctors. So why am I excited about this? I think it could be a bicycle for the mind. And this is an advertisement that I love from August 1980 about the Apple II computer. And Steve Jobs said, personal computers can be like a bicycle for the mind. Now what's interesting is when you read this, you forget actually that back in the 70s the idea of a personal computer was not widely accepted.  Like essentially at the time, IBM made mainframe computers, digital made mini computers, and they were used by the military and they're used by universities. But they said, well, why would anybody want a personal computer in their home? They can't really do very much. They're very limited. But actually Steve Jobs and Steve Wozniak and others had the vision that if you give these things, even if they're crude, people will come up with lots and lots of applications. And I think the same could happen as we begin to automate some of the machine learning tools that we begin to have. So when I'm talking to clinicians, I say, if not you, then who? So I say, if you're interested, go and download some public image data sets and play around with these things. Now, of course, before the audience start throwing things at me, let me say just there's lots of caveats. So the first thing I say to clinicians is, don't use images from your own institution unless you have the appropriate approvals. And the second thing is that I say is, nobody is suggesting that these could be used for direct patient care anytime soon because if we have to be cautious about the clinical validation of bespoke state of the art machine learning models, we need to be even more cautious if it was an automated system. But I think that this will be exciting because I think clinical researchers can look for a range of ideas, begin to dip their toe in the water and find some interesting things, do a proof of concept, and then if there's potential, work with real experts to be able to actually develop better systems. And Eddie Korat is the clinician for my team that is leading that work. My last two slides just to say, I think of themology is there's a real potential to reinvent the eye examination for the 21st century using machine learning. And if you'd like to read a little bit more about it, there's a short essay that I wrote with one of my heroes, Eric Topol in the Lancet in December, 2019. And we need to move towards implementation, AI assisted clinical trials, AI assisted scientific discovery and I think democratization and industrialization of the technology. That's it for me. This is my email address. If anybody wants to get in touch, I always try to answer. Thank you very much. Thank you so much, Piers, for this excellent talk. That explains like the realities of bringing machine learning to relevant clinical problems into the clinic. It was very exciting for us. So let me see whether they are raised hands already. Yes, Sukhi from the network, Sukhi Lee has a question. Please go ahead. Hello, hi, Piers. Thank you, just gave a very interesting presentation. My question is about your model evaluation. So you said, as you mentioned, like not all the true labels are actually true. So there are some mistakes. So how do you measure and address this problem? So I think there's no easy answer. I think that what I've learned is that for deep learning projects with medical images, the major blocker is the governance and the permissions, but the other major blocker is you need to have meticulous attention to detail for labeling the data. And you have to have a proper strategy and protocol in place to label the data. So typically we would get automated labels from the electronic health record, but those are gonna be garbage a lot of the time. We would actually manually review all of the labels and we would typically at least have two graders, one of whom is a junior grader, one of whom is a senior grader. They would have a protocol that determines and then in areas where they have discrepancy, we would have some other person who adjudicates. So you have to have a systematic approach to labeling those. And even with that approach in place, oftentimes the gold standard that you have is gonna be still ambiguous and this is just a challenge for the field. Yeah, so do you also take like a consistent protocol for this kind of labeling? Some, what kind of protocol? Consistent, like for all the data you have. Yeah, so yeah, we like I get my group to prepare protocols where they describe two or three typical cases of each label, they identify edge cases for each label. We also look at the reproducibility of our labels to see if we're consistent with each other. It takes a lot of attention to detail to get that right. Okay, thank you. Thank you. Do you have an estimate of how transferable to other populations outside the UK these models are? I mean, that's the million dollar question. It's about the generalizability of machine learning models. So I don't even really know how well it generalizes within the UK yet and we have to do the study. So our big thing that we're interested in is will the model generalize from a hospital based setting within the UK to a community setting where the disease prevalence is very different? And then will it generalize to other countries? And we are doing, I briefly mentioned it, but we're doing a global validation study where we're trying to test that if we have a patient that's an African patient with diabetes who comes in London, will it work as well if the same patient was in Ghana, for example? And I hope that we'll have some results of that in the next year. I feel quietly confident that we'll do okay because of the fact that London has a very diverse population in terms of ethnic group and in terms of disease severity, but we need to prove it. And until we prove it, we cannot assume it will generalize. I'm trying again. Leslie, does this work now? Yes, I see you are muted, please. Did any of my question go through? No, not at all, it's not on my side at least. Okay, I wanted to say thank you for your talk. It was very interesting to hear this all from your perspective. My question is about when you mentioned that you had substantial hurdles to getting access and approval to work with the data, you mentioned that you were also very intentional on making your efforts known to patients by having it on the website and having information available in your office. And I was curious if you were aware of any patient reactions to your work, if there were any? So the patients, to my knowledge, have been unanimously supportive of the work. One of the things that I didn't say is that when we launched, when we announced the work, we also brought the support of the Royal College of Ophthalmologists plus all the major eye disease charities plus our patient groups. And we've had a number of talks to the public and to patient groups where I've talked about it. And so when I spoke to the Macular Society, which is the main disease charity in the UK for Macular Degeneration, I spoke to more than 200 patients on one occasion. And at the end of my talk, the chief executive said, what do people feel about this? Because this is sharing your anonymized OCT scans with Google. And she said, hands up those people in the room who would be happy for this to happen.  And the chief executive asked her, what's your concern? And she said, I'd be happy as long as my scans are anonymous. And of course, we do anonymize the scans. And so I think that the patients, in my experience, have been very, very supportive. I think that there are members of the public who I think quite rightly ask some questions around sharing of NHS data with the company, like Google, and making sure that people's privacy is respected and a range of other questions around that. And if I was in the same position, I would ask the same questions. That's great, thank you. And thank you from all of us peers. We sent you a round of virtual applause for this excellent talk. I hope it's the start of a beginning of a dialogue between us and you. So this was wonderful. Thank you very much. It's my pleasure. Thank you, Kristen. You're welcome.