Pertisha ntrabuna siteni nwaia dee Abia state capital on Friday, October 6, a stroke out the people's democratic party's election petition against the governor of the state, Alex Oti, of the Labour Party. Now, the three-man panel led by Honourable Justice, each today are doada in our three-hours judgement. This means the three grounds. Alex Oti's qualification to participate in the 2020 governorship election. Alex Oti's inability to score the required number of votes to qualify in to be declared the winner of the 2020 polls and for which PDP approached the tribuna. Now, in our judgement, Gwada Desmi is a PDP petition for lack of merit.


How's everyone feeling? Happy holy? Right? I've been greeted with that so many times today. I woke up not even knowing it and feel really excited to be wearing bright colors for you today. Who's excited to learn about artificial intelligence? Raise your hand. So everyone's excited. Who's not excited to learn about artificial intelligence? Great. I want to first introduce myself and we're going to see if this video decides to play. Great. We're just going to have to deal with that play pause. Hello. My name is Allie Miller. That is a very large photo of me. But I came here from San Francisco, California. Has anyone been to San Francisco, Silicon Valley? Nice. That's actually a lot of you. I am very active on social media. I encourage you all to take photos of slides that you really enjoy, slides that you want to take home back to your team. My username is Allie K. Miller on pretty much every platform. A bit about me, I am a previous lead product manager at IBM Watson running product development for our computer vision work. And we'll be starting a brand new job at Amazon working in artificial intelligence in two days. So I'm very excited to be here. I'm going to fly back, immediately get started, and can't wait to take everything that I learned from this room back. I studied computer science as well as psychology at Dartmouth and also have my master's in business from Wharton. And so that's how I approach AI. I think about it from design, from technology, and from business, and try and combine them all. So we're going to, there we go. Amazing. I do want to give just a nice foundational starter in artificial intelligence. Who's worked in artificial intelligence? OK, not that many. Who's read an article about artificial intelligence? Who feels like it's overhyped? Too much noise going on. Not that many. If I asked the same question in San Francisco, everyone's arm would go up. So a bit about artificial intelligence and how we define it. AI is really just the idea of a computer performing as if it's a human. So when you think about a human, we have vision, touch. We have ears that can hear. And that's exactly what AI is trying to do. It's trying to mimic what a human would do. In the world of visual recognition, that might be something like color recognition or depth perception. And I'll be giving you a few examples throughout this presentation of use cases that I've worked on. I've worked with over 200 clients in 10 different industries from manufacturing to agriculture to retail to health care. And we'll be talking today about a few of those that I've seen. Artificial intelligence is kind of this big, big term. The subset of that that we'll really be focusing on is machine learning. And I'm going to demystify machine learning a little bit and make it less magical. All it is is giving the computer a lot of examples of something. It learns that as a pattern, as a model. You show it a brand new something and it decides how it compares against the million things that you've given it before. So one example, you've given it lots of emails that you've gotten from customers. And you've said, these are really angry emails. Everyone hates us.  And then you give it nice emails. Everyone loves us. We did so well. And you teach the computer these two different things. It creates a model. You give it a brand new email that it has never seen before and it compares against the two. And it's able to say something like, I'm 83% confident that this is an angry email. And it's up to you to decide what to do with that. This is gonna be really fun. I'll sing my own background music as it's going. All right, so now we know a little bit about what artificial intelligence is, what machine learning is. And I want to teach you, in a very, very quick manner, seven steps of really how to build AI and how to consider it. So I'm gonna be walking through the following seven, problem, scope, team, et cetera. And here's the great news. 80% of what I'm about to say is exactly how to build a non-AI product. So if you've built anything in your entire life and raised your hands if you've built something amazing, you're already 80% a master of building an AI product. And so I'll try and call out a couple of the differences so that as you're looking at each of these different categories, you'll be able to take home what the differences are and figure out how to map that to actually building an AI product. All right, none of this slide is new. Every single one is exactly what you already know. So first step, whenever we're talking about design thinking, agile work, right, it's always what is the problem we're trying to solve. And I'm gonna tell you right now, the worst thing that you can do for your organization or team is believe that AI is cool and that you just wanna use AI. That is the worst thing you can possibly do. It is always starting with a problem and then deciding whether AI or non-AI is the best way of solving that problem. So always, always start with the user pain point. Doesn't matter if it's AI or not. The second thing, what is the outcome? What is the result that you're craving? And it could be numbers-based. Generally it is when you're looking at a spreadsheet and you decide that a number that you're unhappy with you want to improve. An example is I worked with a retail company and they were actually having a lot of store theft issues and their store theft in one city was much higher than every other city. So the outcome that they wanted was to reduce theft by 10%. They decided to use AI for that. Another example might be in manufacturing. And so you have a lot of items coming off the assembly line. Some of them are damaged. You don't wanna sell those. So an outcome that you might be aiming for is improving exactly what product comes off the line and reducing quality assurance time by, say, 15%. So deciding that outcome ahead of time and the last, again, nothing new is meeting your users. I do wanna ask who works in user experience? Who's a UX person? Okay, design, general design. Okay, how about software engineer? Wow! You guys should all work with me. Okay, software engineer, what about business leader? Couple hands, what am I missing? Anyone that I didn't get? Great. All right, so the last would be meeting those users and speaking to them. And that's really where the design comes into play, developing those user personas. I think we might have lost it. I'm happy to make it all up from here. We already know the next six categories. Just died for a second. Great, I will talk less. All right, so problem, that is step one. We're already done. Item two is scope. Number one, again, not AI specific, is there a timeline that you absolutely must adhere to? That will usually drive constraints that you need to play with in. For example, if you're in agriculture, do you want to get something finished by the time the next crop season rolls around? If it's retail, and you know that a lot of people buy items on Saturday, do you want to release on a Tuesday to make sure that it's up and ready for them? So, thinking about timeline, what constraints do you have to build within? In general, and we're talking about resources and budget, those are kind of combined. AI is not necessarily more expensive than non-AI, but one thing to consider is that upfront cost might be higher as you're training people, as you're taking time away from that normal development cycle. So thinking ahead about what these budgets might be, if your budget is higher, do you want to bring in an outside group? If your budget is lower, do you want to teach in-house? So thinking about where that flexibility has to come into play. And always, I know it's blocking right now, add a buffer. I'm sure you guys have seen that. Whenever you're coming up with a budget, add 20%, be safe. All right, team, here is where we really start to get into the differences between AI and non-AI. And I see a lot of phones out, that's really good. Number one, machine learning is a skill. It's not a random talent that I woke up and was like, oh, now I know machine learning. It is very much something that you need to learn. And so whether you're hiring outside, which again might be more expensive, or teaching machine learning in-house, you absolutely need to have that expertise on your team. Generally, as I've worked with several teams, the ratio tends to be one machine learning engineer to about three to five full-stack software engineers. The more software engineers needed, the more difficult it is. The difficult the architecture, infrastructure needed. So I've seen ratios of one to three, one to five. Diversity and bias, this is a huge topic happening in Silicon Valley right now, as we're building artificial intelligence products. One of the worst things that you can do is have your entire team be 25-year-old white males who all went to the same school, who all work at the same company, who all worked at the same company before that. So as you're thinking about how to build these teams, think about things like age, gender, what city, what country, and a lot of pieces around this might be where you want to release the product. If the product is 100% focused on Bangalore, you should absolutely have people from Bangalore on that team. If the product is for the US, you should have Americans on that product, right? And as we think about more global, larger products, that is why it is so important to have diverse teams. For example, the last team that I was on at IBM, which was building multimodal artificial intelligence, our software team alone was from five different cities. So these are things that we have to think through, right? How do we bring remote teams together to be able to enable that diversity to help fight bias?  It's not just engineers, it's designers, it's product owners, business, even things like finance, legal, marketing, sales, they should all be in the room right at the beginning building that product and knowing what is going into it. Data, this is another big one, super different between AI and non-AI, because frankly, when you're not building an AI product, you might not actually need data going into it. You might be testing hypotheses getting out there and testing, but it is very different to build an AI product. You need to have this constant pipeline of data coming in. There are two questions you should ask yourself about where you're getting this data from. Let's say that this is a popular use case on YouTube that a lot of AI people are talking about. Let's say you're driving a car and there are security cameras on the street and you just need to be able to read a license plate and maybe every time you pass a city boundary that you wanna be able to see what that license plate is. Pretty easy, right? But as you think about this, you're thinking what if a car is dirty and I can't see the license plate? What if they go 20 miles an hour? What if they go 90 miles an hour? And my camera can't catch that fast enough. What if there's a motorcycle blocking the front of the license plate? So these are all things you need to think about to build a big enough data repository to be able to know what you're about to see. So one, what is the source of your data? In the example that I just gave about traffic, maybe it's those traffic cameras, right? If all you had was someone standing in front of a car like this taking the photo, that would look nothing like what you're getting from those security cameras and from those traffic cameras so your model would not perform well. Second question you need to ask is it owned? Is it proprietary? Do only you have access to this pipeline? Because if someone else has access to it, they're gonna be able to build your exact product in probably half the time that you just did. So when you can, think about when you can bring in a unique pipeline, something that only I own. So maybe you sign a contract that gives you exclusive rights to the traffic cameras. Maybe you send out a team of 10 people to take a million photos each and that's so much work that no one else would do it. So start to think about really how it can build a unique pipeline. Second is good data, right? So sourcing, that's like where do we actually get the photos from, cleaning up the data. So if you're thinking about that traffic camera, a lot of the time it's taking photos, there's not a car there, an animal could be walking by, right? And that might be noise, right? Like photos that you don't need. That might be noise that you actually need. Because as a car is coming through you wanna know when it's a car and when it's a human and when it's whatever. But if there's noise like a bird is in front of the camera and is blocking it, maybe that's stuff that you wanna clean out. So thinking through who is actually gonna be annotating this data? Who's running quality assurance? How are we making sure that these million photos are perfectly labeled? Because if there's even one small error that could expand when you're actually building your models. The last one is generally the more the better, right? We always want good data. We want data that mirrors what we're actually gonna be seeing. That's why cell phone pictures wouldn't mimic traffic cams. But you're always going to need more than you think. And depending on what type of model you're building, whether it's that email system that I was talking about in the beginning, right? Angry versus happy, maybe you only need like 30 emails of each. If you're talking about traffic cameras that have dogs and humans and it's day and it's night and it's foggy and it's not, you might need eight million photos and you won't know right when you start the project. So more is always better. Who knows the term edge cases? Is that a new term? Okay, edge cases just means something you didn't really think about in the beginning. So again, maybe you never thought that something could block the camera, but that's a brand new edge case that comes up and so you're gonna wanna constantly have that in your model. We are halfway done. Who's excited? Now you get to sleep. All right, but also how are we halfway done if there are seven? That's because surprise, I have a UX bonus. Because this is a design conference and I wanna make sure that you guys are getting the information you need. So here is exactly how you need to think about user experience for building AI products. First question, is it AI? Right, and obviously if you're building an AI product you're using AI, but does it actually matter if you tell the end user whether it's AI or not? Here's an example, right? We called an Uber yesterday to head into town. I wasn't told by Uber, oh, we recommended this amazing ride for you using this machine learning algorithm. It's the perfect ride. It's the cheapest, the fastest, the best, the coolest. I don't care, I just need a car and I need to get to somewhere. Something like a doctor recommendation, like what medicine to take, I would rather know as a user maybe, whether a doctor thought about it, whether AI thought about it, whether the two combined thought of it. And so you always need to think about when do I need to tell the user that it's AI and when does it actually waste time or make it a worse user experience? The second is thinking about interface, right? Are users involved? Who's used like Netflix? Lots of Netflixers, nice. How many times have you like thumbed up and thumbed down something? Like everyone in this room has done it at least 50 times. This room has gotten to know how many people in it, and so that multiplying factor for all of the Netflix users, you guys are basically training Netflix, their algorithms, you're doing their job. So the most common algorithm and most common way of getting users involved is this like thumbs up, thumbs down, right? We gave you this answer to this chat box question, was that the answer you wanted or did we completely mess it up? There are less common ways of getting users involved and I'll let the genius designers in the audience think about it, but there are some common ways that are coming up. And then the last one is more transparent, always better. So let's say that Uber decided to tell everyone that they were using AI in their car recommendations. Is it really important for you to know that it's AI? Yes, no. Is it important for you to know why they picked it, the way they picked it? Yes, no. Is it important for you to see the algorithm? Yes, no. And is it important for you to see the training data or how the AI model came up with it? So there are different levels of transparency that different products require. Generally, the more like scary the recommendation is, like doctors telling you what medicine to take, more transparent is better. Things like is this a dog or a cat? Don't care, right? So think about what level of transparency your product's going to require. UX bonus is done, there are no more surprises. I'm so sorry. All right, when you're testing something, right?  We're gonna see how it performs, see what people think, see if it works. Who knows the term MVP? Pretty much everyone. All right, minimum viable product, right?  One of the hardest things about starting an AI product is that everyone wants to say no. Everyone wants to say AI is not gonna work, I have my right team, we've got the right product, I don't wanna innovate. And so you want to be able to say my first AI project out the gate was successful. So how do you prove small? And that's researching competitors, that's building the right team, and that's making sure that everyone on the team and not is aligned to what you're building. Success metrics, right? This is a term that everyone's heard of, but there are AI metrics and non-AI metrics. So like I mentioned before about store theft, those are common, you guys have heard of those. But AI metrics, right? How well is that model performing? How many thumbs up, thumbs down? Are we getting when a user is giving us that feedback? And I'll give you kind of a rule of thumb that I like to use. Generally when we would launch a product on my past teams, we would spend about four weeks, get to about 60 to 80, 85% accuracy, launch that, test, test, test, more data, more data, and then within the next four to eight weeks, get to about 80 to 99% accuracy. So that's why it's so important to start small. And then the last, which I kind of gave a heads up to, don't aim for perfection. AI is so new, the worst thing you could do is say, when I launch this, it will be perfect, it will be 100% accurate, it'll be better than humans, it'll be the best, everyone's gonna love me. Aim for 70% accuracy, aim for 60, just try and get something out there and see if it actually works for what you're trying to build. So after you test what is the exact thing, optimize, right? Get that feedback loop going. How are we understanding from the market what works and what doesn't? How are we understanding from the team of whether people like it or not? Are we building more AI projects or not? The biggest thing that you need to do is treat mistakes with care. A lot of AI projects right now, when I said that there's imperfection, when you're wrong, make sure that you're building that back into your algorithm, right? So if it was deciding between a dog and a cat and it gets a picture of a cat and it's 99% confident that it's a dog, bring that back in as trading data, teach it that it's a cat, and keep optimizing. Something that people might have heard of is continuous feedback loop, that's when every single new image coming in is updating the model. You also might have heard of like batch updates, maybe it's at the end of every single day, at the end of every single week or month, you're doing a full retrain, and it totally depends on cost and how your team's aligned. Data lifecycle, a big topic in Silicon Valley is like data is king. It used to be that content is king, now it's data is king. We wanna have that constant stream, so if I was talking about those traffic cameras and you decided to have people manually go out and take those photos, if you don't have that happening all the time, your data stream is gonna dry up, someone new is gonna come in, someone's gonna take over that product, make it better, and they'll have set up some sort of constant stream. So you can't just bring it in and let it die. You always have to be thinking of how to refresh it. Can it be automated? If you're manually sending people out, that's gonna be really hard to keep it automating and you're gonna always have that cost. So how can we automate that flow? And the last is don't silo it. You always wanna have it when AI makes a decision that it then does something else, right? If we decide that an email is angry, I'm not just gonna sit there and be like, that's amazing, it's angry, AI got it right. Now I can go on with my job, no. What you wanna do is take those angry emails and get it dealt with faster, right? If someone's happy, I don't need to reply within two seconds saying, oh my God, thank you so much. Thanks for your nice email. No, you wanna deal with the person whose car is on fire. You wanna deal with the person whose crops just died. You wanna deal with the person whose manufacturing line is broken. So figuring out what to do with those angry emails or the results of really any AI model and deciding how to triage it is really important. We're on our last step. How are we doing on time? Good? Great. I wanted to leave ample time for questions. Last step is scale. And a lot of people, when they think of scale, they think about more money, right? All I want is more money. How do we grow it? How do we go to more cities, more countries? How do we get more users, right? Scaling AI is not always about more money. It could be about a more accurate model. It could be about making something more robust, less brittle, so that competition can't just come in and steal your idea. So things like are there new features that you should be thinking about? Should you combine computer vision with natural language? Should you combine facial recognition with emotional sentiment so that I can tell, sorry, like you've been smiling most of the time, right? So do I want to know that you are more engaged than someone who was on their phone? Thinking about how to add those new features will only make your product better. And again, always using more features to solve a problem. Combining tech, that's like the next big topic. Everyone's talking about how do I combine IoT, Internet of Things with AI? And make sure that everything's running on the edge so that we're not sending everything up to the cloud. And so that is a really popular topic that's happening right now. How do we bring more things together, hardware plus software, to be able to build something even better? And the last topic of scale is to not be lazy. You gotta check in, see how it's going. You can't just put AI in the corner and say, I've done my job, I did AI, everyone said to do AI, I did it, and now I get to quit, right? Check in every few, I say every few months. In the beginning, it'll be every day, then every couple of weeks, then every few months, at no point should it be every year or every few years. So thinking about, is it still valuable, right? A year ago, someone might have built something that no longer makes sense today. And that could be it needs new data, needs new features, it's not robust enough, someone came in and beat us. But thinking about how it can constantly be improved. And the last one is that all of these big tech companies and all of you as well, are building new products all the time. And so making sure that you're always having at least one person on your team, keep a pulse on what is new in the market, so that you're always using the best and brightest tech out there. So we finished all seven steps, cough, cough, eight steps. And this is the summary. So if you have been sitting on your phone the entire time and you just wanna make sure you have everything, this is that final summary slide. So again, thinking about problem. Who are you helping? Why are you helping them? And making sure that everyone's on board about that solving that one problem. Scope, what are the constraints? Thinking about team, resources, budget. When you're thinking about team, how do you align everyone to get the job done? How do you keep it diverse? How do you keep it multifunctional? Data, we call this the fuel of the rocket engine. Engine won't take off unless there's fuel and unless you're constantly giving it fuel. So where is that data coming from? What is the source? Is it unique? Is it yours? How long does it take? Can it be automated? Testing, will it work? I need to prove that it will work. Start small, start imperfect. Optimizing, how can we make it better? How can we grow it so that it's something that is valuable and robust? I know I've said the word robust about six times, I'm sorry. And lastly, how do we scale it? How can we grow this project? Both for more revenue, more users, or frankly, just a better AI product. Those are all the steps.  And I'm so grateful. I've never been to India before, Canada Gotila. So we're gonna have to go in English for the questions, but I really appreciate it. You guys have been so engaged and I welcome 15 minutes of questions, which is exactly what I wanted. This is the only AI session, so I wanted to make sure that you guys got your questions answered. But thank you very much for your time. Hello. Hi. Thanks, that was a good talk. I come from a healthcare sector, work with a product that we are building in. Can you put it a little closer? Embracing AI. Perfect. The question I have for you and this being an Agile conference is most often than not, we want to release the product every two weeks, which is a sprint for us. And what I hear from the guys on the AI part of my team, AI doesn't work that way, Rajiv. It's not gonna be two weeks. We need more than four to eight weeks for us to train the model, get the attrances and blah, blah, blah. And then we can make it into a release cycle. So you can go and let the dev team or the software team do two sprints while we prime this up and then maybe we'll converge. How do you take care of that in whatever experience you had? First of all, blah, blah, blah. Sounds really important. Here's what I would say. And it's a flaw in what they believe is a two-week sprint. And I know we have some of the other speakers in here, so I'll duke it out. In my mind, every two weeks does not mean that a new product has to be released. So I think that the process that you were talking about around four weeks, training models, what I was saying about accuracy right after four weeks, maybe you'll get somewhere around 70% accuracy, that's around building. Optimization can absolutely happen every two weeks. On the build side, what I would say is breaking down these steps, how to make sure that data pipeline is set and maybe that's a two-week sprint. Then when you're talking about how do we clean it up? Maybe that's another week and then the next week can be QA on the pipeline. So design sprints and software sprints within AI should absolutely be two weeks. Your AI cool guys should not be like off in some other room with the door closed saying, Rajiv, I'll talk to you in two months. But every two weeks, they don't need to be delivering a full-on product. The last thing, and just to make sure, I know a lot of you have not yet worked in AI, training a model takes an hour, right? Once you have the actual data and once you've decided here's how to annotate it and here's what we need, the actual hit a button that says train and have it be done depending on how big your pipeline is, could be somewhere between a couple minutes or a couple hours. In no way should training the model take two weeks. I would say, to go back to them and say, AI cool guys, we need you at the table, you're not super cool, you're part of this, we are one team, and you need to make sure that something is delivered every two weeks, even in the build phase. Thanks, Ali, that was a great, great talk and gave a good overview of what AI is and so for the software engineers in the room, one of the technical practices that we rely on heavily is test-driven development. Do build a test before you build the piece. How does that work in training data? So a couple pieces to unfold. So the question was how do we make sure that we're testing appropriately? Do we kind of come up with a test ahead of time and what are the processes for maybe testing intelligently? Is that a good ish summary? Couple pieces to unpack in that. One is the bias piece and one thing that a lot of people believe is AI is unbiased as long as it completely mimics what a human would decide. And one example, right, Shane's already nodding no because he knows that that's wrong but a lot of people think that that's how you do it. So what I would say, and this is kind of two-stage, so foundational, what you should always do in that annotation phase, let's say that you had dog versus cat and you have 1,000 images of each one, set aside 20% of your annotated data to be able to test, right?  Set aside 20% that is already trained, train the model on the 80%, train it, use that 20% as test, right? Like not all at once, one by one or 5% at a time and see how well the model performs. If you realize that after going through 100% of your data that you're 30% accurate, that is abysmal and you should absolutely be 10xing your data. So that's foundational, always set aside annotated data to be able to test against your algorithms and figure out how well it performs. The second piece which is kind of this mimicking humans which I just said you want to be able to test against is that humans don't always pick the right thing. One of my favorite examples is in the courtroom, I believe this was a US study, in the courtroom judges decide more favorably, they let people off more after lunch, right?  If your appointment, if your court date is at 2pm, what's up? Yeah, that's awesome.  So that is something where a human's decision is not always what you want to mimic. And so one way to build into that is having multiple humans see what the output of the algorithm is to say whether it's thumbs up or thumbs down, and frankly that's why Netflix so often will ask you for recommendations because they're doing it to millions of people, if that makes sense. My question is particularly from the predictive policing if we talk about. Say that one more time. Policing, predictive policing the federal department. Okay. So what is your approach around the systematic biases which comes because of the algorithms, because there is a limitation. Whatever data you provide, it's only that how the machine is gonna react. Which is the case whether it's for federal government or not. Yes, so cats and dogs, you can still generalize that this is the particular breed. What do you do about the felonies that have been committed by different diversity? I don't want to sound racist, but if there's a white man or there's a black man, how does that work in this case? Yeah, absolutely. There's a term called the risk of imperfection. So in the dog-cat world, if you're wrong about whether it's a dog or a cat, depending on what you're trying to do with that data, it's not a huge deal, right? Because a human will probably, like if you're trying to split up dogs and cats into the right kennels or daycare, that seems insane, why would you send a cat to daycare? But you know what I mean. The fact that it's wrong, a human's gonna see it and be like, I'll fix it. The risk of imperfection in things like should I arrest that person is much higher. So step one, when you're getting your whole team together, and this is with executives, finance, legal, especially, everyone, is deciding the risk of imperfection. So that use case where I kind of generalize saying you can launch with 80% accuracy, if it's a police and you have a body camera and you're deciding whether to arrest the person in front of you, 80% accuracy is not gonna cut it. Or you can have the AI decide and always give it to a human no matter what the AI says. So one thing that we do with our clients a lot is deciding what are the MVP metrics that we have to hit, and usually that's a conversation around risk of imperfection. The second piece is, and this is to Shane's point, around kind of gathering the testing plan ahead of time. Getting baseline metrics for how human would perform in that exact same situation is really important. So if a policeman realizes and picks the person and recognizes the person 50% of the time and an AI is at 85, suddenly 85% sounds fantastic. The protection against it, again, is always giving it back to a human to have a human in the loop, so that we're making sure that the best, safe AI is getting out there. Yeah, so, hi, one question. We like this table a lot, I wanna make sure. Just one question, I think it was similar to what Rajeev asked. So we are also a team, we are building AI solutions and often the data scientist plans certain experiments and we also struggle a lot while estimation and during the sprint it can't be planned or people say that it requires research, can't be time bound and so we moved from sprint to kanban. Because in kanban you don't have fixed time frame as opposed to sprint, but was that a right approach or was that an escape? Your guys' questions are so funny, I love it. I'm not gonna tell you whether it was the right or wrong decision. But typically what methodology being followed in because everyone building AI would be having the same challenges when you plan experiments we don't know exactly how much efforts are required or how much data analysis is required. So when you dig into that, then only small thing becomes bigger. So how to handle that and give the more predict, deliver is in more productive manner? Here are two things that I can tell you. One is that my software engineers, right, like my most recent job is running a team of 30 engineers, they hated estimations. They're like, Ali, why do we do this? It's always wrong. I can't even get it within 10%. Everything that we're doing is new. I have no idea how to estimate this. So I can tell you that the pain points that your team is feeling are felt by a lot of AI teams. The second is what to do with that pain and what to do with that problem. There are some teams that decide to only build out AI projects or sprint planning for 80% of their time allotted, right? I was talking about buffers in budgets. People do buffers in time estimates as well. But again, that only gets you so far. So I know teams who have not moved to Kanban and keep it at those estimates and just only estimate for 80% of their time and they just get better and better at estimating. That's one option. And it could be that you are just more mature and that you guys have moved to Kanban. I can't say whether or not that's the perfect decision because I haven't worked with your team, but come work with me at my new job and we can figure it out. No. Thank you. Sure. Yeah. Are you under arrest right now? Is that something that we should know? Not exactly. So that was quite an expressive talk for artificial intelligence and I liked it. So my question might sound a bit science fiction-y or black-mirror-y.  Like I was watching other day, Joe Rubin's talk with Elon Musk and he was quite concerned. Do you, like a young professional in this field, are you concerned? So Elon Musk is a very interesting example. Slight brag moment. I met with him in Dubai to talk about AI and I know, right? Who knew? He talks about how are we gonna send people to Mars? We're gonna do it in two years. Like his timeframe for a lot of things is easily one-tenth of anyone's. So I'm gonna put Elon Musk to the side, amazing as he is and say that he is, he's gonna be like a smaller opinion. That doesn't mean that he's wrong. I'll give you one example. Google, the head of Google Assistant and the founder of Siri made a bet it was a year and a half ago on when we would reach AI, artificial general intelligence. Founder of Siri said it's over 50 years. Head of Google Assistant said 10 to 20. Open AI, anyone heard of open AI? Very popular, right? If you had asked them a year ago when they thought AGI was gonna occur, they would have said over 50 years. If you asked them today, they've changed their mind and said 10 years. And they're the ones working on it, right? They have some of the best, brightest PhDs working on this. My answer to your question, which was am I worried? So question one is when will it actually happen? And the answer is like who knows? Best and brightest minds completely disagree by decades and generations. Where I start to get worried is on the policing of it and the regulation around it and can law keep up with artificial intelligence? We start to see things like self-driving cars. Are we ready as a society for that? And Elon Musk has said it'll take at least about 35 years for every car on the road to be replaced by self-driving cars, just based on how long cars last and how much they'll be in technology analysis. So my biggest worry is actually on the law side and that's one reason why I and why I encourage you all to participate in law and ethics talks. So I go to a lot of government meetups. I've addressed the European Commission about AI and the future of workforce and the future of education. So the way that I feel okay about it is by getting involved in those conversations and making sure that safety and policy is happening right now and not waiting until it's too late. Oh, hi. Yeah, the microphone doesn't help because it's from everywhere. No problem. So you said that you'd worked on about 100 AI projects still now? Yeah, maybe it's like 250. Okay, okay. So can you just give us a couple more examples of some interesting AI applications? Sure. Other than the ones you've mentioned during the talk? One of my favorite examples, because it's hilarious, is a cognitive nose that IBM originally built just kind of as a fun research project and they were testing it on Pesto and seeing if we could pick up all the sense and it's now being used to track air pollution in Beijing, right? So even things that are kind of like fun, researchy and can move into great use cases. Another one that I love was the measurement of pigs and trying to figure out whether pigs were ready to be slaughtered and looking at cameras to be able to predict how big the pig was so that you didn't have to individually weigh every single pig and using computer vision. Last one is damage analysis is incredible so we had one about cell phones coming off of product lines and using an automatic camera to be able to see whether there was a scratch that was a millimeter long. And so knowing whether that phone had to be fixed or whether the phone could be immediately thrown out. Yeah, banking is an awesome use case. One of the best is around pre-fraud detection so if you're using, you know, I know exactly, what's your name? Say it one more time. Vivek. So I know how Vivek has been spending, all he buys are t-shirts, right? Vivek is all about t-shirts and now all of a sudden he's bought a car. Not that weird, because people only buy cars once every 20 years. But if all of a sudden he's buying 1,000 max, that would really trigger something. And so fraud prevention is one of the biggest use cases in banking, two more that I'll quickly say is ATM vestibule safety. Trying to pre-predict whether someone is going to detonate a bomb inside of an ATM vestibule. So using video vision to figure out whether that would happen. And face recognition at ATMs is kind of what everyone tries to talk about so you don't have to type in your pen or scan your card. We've run out of time now so you need to take the question. Yeah, I'll be standing outside and I'll be wearing this exact outfit so you can come find me. Thanks a lot. Thank you so much.


Good evening. I'm Mark Uptegrove, the president and CEO of the LBJ Foundation, and I want to welcome you to a brand new season of programming from the LBJ Presidential Library. It's not exactly what we had planned. We had hoped by now to welcome you back into the library for in-person programming, but as with so many things, our plans changed with the resurgence of COVID-19. Still, we're as committed as ever to bringing you the very best in public programming, and we'll continue to offer our beloved Evening With Series and other programs virtually until we can safely gather again. In the next month, we'll feature two programs exclusively for our friends of the LBJ Library members. A continued conversation with Jason Stanford, one of the authors of the recent book Forget the Alamo over the lunch hour on September 15th, and then on the evening of October 6th, we'll welcome you to an evening with renowned Dr. Anthony Fauci, chief medical advisor to the president. You won't want to miss these programs, so if you haven't already done so, this is a good time to renew your membership online at lbjfriends.org. We have a great start to the season of programming. Tonight's guest is Chris Wallace, anchor of Fox News Sunday. Throughout his five decades in broadcasting, he has interviewed numerous U.S. and world leaders, including seven American presidents, and won every broadcast news award for his reporting, including three Emmy Awards and the Peabody Award. Tonight, he'll talk to me about his new book, Countdown Bin Laden, the untold story of the 247-day hunt to bring the mastermind of 9-11 to justice. And now, please join me in welcoming Chris Wallace. Well, Chris Wallace, welcome and congratulations on Countdown Bin Laden. Thank you. I hope you liked it. Well, I loved it, and that is a sincere statement from this interviewer, and I can tell you that I also loved our last conversation and your last book, which was Countdown 1945, the extraordinary story of the atomic bomb and the 116 days that changed the world. This book recounts the 247 days that it took the United States to capture and kill Osama Bin Laden, the mastermind behind the 9-11 attacks. So what led you to writing this book? Well, part of it was a direct result of 1945. I loved writing about Truman and all the people in his world and the decision to finally to use the nuclear bomb, the atomic bomb, to try to end World War Two by dropping it on Hiroshima, the frustration with the book. And I guess this says something that really deep down I'm more of a reporter than a historian was that all the key players, all the people that I wanted to ask, why'd you do this? And what were you thinking? We're all dead. And so I was spending a lot of time reading previous history books, looking at diaries, looking at memoirs, looking at letters, and it was rich. There's no question about it. There was plenty to use to get the interior conversations. What was really going on behind the scenes in the decision to drop the bomb? But I thought, wouldn't it be wonderful to be able to do this, to find a history thriller, and all of the people are still here? And I don't know, a series of conversations. I came up with the hunt for Bin Laden. And in the course of researching the book, I spoke to almost everyone. I spoke to Leon Panetta, the head of the CIA, which came up with the original lead on the compound at Abadabad, a lot of the top people, and also not only him, but also his chief of staff, Jeremy Bash, his deputy director, Michael Morrell. We got the first time that the head of the Pakistan Afghan Department, a fellow we call Gary, has ever told a story, has ever talked to an outsider. All the people in the White House, with one exception, which I'll get to, all the people in the military, including Admiral William McRaven, the head of Joint Special Operations Command, and two of the SEALs who were on the mission, including the man, Robert O'Neill, who actually killed Bin Laden. The only interview that I wanted that I didn't get was with Obama.  He had just written his memoir, and he has a chapter about Bin Laden. And his press guy said, you know, he's going to let that chapter speak for itself. So we took stuff from that chapter, but I also got a lot of the interior sense of what Obama was doing and thinking, because I had spoken to his national security advisor, Tom Donilon, and his chief counterterrorism advisor, John Brennan. And the other great advantage of this book is there were a lot of stuff written right after 9-11 when everything was still very classified. And this was now 10 years later. And so people from Panetta, to McRaven, to Donilon, whatever, were able to tell more than they had ever told in their own memoirs of the history books that were written in the year or two after the takedown of Bin Laden. Your book recounts the 247 days from the CIA's lead on Osama Bin Laden's whereabout to the mission on his compound. But it took us 10 years, nearly 10 years, between 9-11, the attacks on 9-11, and that mission to bring down Osama Bin Laden. Chris, what took so long? How did Osama Bin Laden evade capture for almost a decade? Well, it was a tremendous operational security and discipline on the part of al-Qaeda and considerable operational security in the part of Bin Laden, who in the early years kept moving in the tribal territory, we believe, the tribal territory between Afghanistan and Pakistan. And as they started rolling up people inside al-Qaeda, they would question them. First of all, they didn't know how to get the lead that would get to Bin Laden. And some thought, and they captured some members of Bin Laden's family. They led nowhere. They captured some officials. They led nowhere. And officials in al-Qaeda, one of the leads or one of the avenues of investigation that they kept pursuing was couriers, because they believed that Bin Laden would be too careful to ever directly be in contact. And he still was running al-Qaeda even while he was on the run. But they thought he must have a courier system where he was able to deliver in a non-electronic way, a way that couldn't be intercepted messages from wherever he was to where al-Qaeda central was. And so, but they didn't know whether that was going to work either. And what happened was over the years, they kept pursuing the courier angle. And for instance, Khalid Sheikh Mohammed, who was the actual specific architect, not the mastermind, but the architect of 9-11, they finally capture him in Pakistan and they take him to a prison and they waterboard him 183 times until he used the CIA terminology. They got him in a compliant state. And then they asked him about they had gotten the name of a courier. Again, they're, you know, it's like organized crime, which I guess this is. They're trying to use building blocks that gotten the name the none de guerre, if you will, of a courier named Abu Ahmed al-Qaeda. And they asked him about him and he said, well, yes, he was a member, but he wasn't Bin Laden's courier and he had left al-Qaeda after 9-11. What Khalid Sheikh Mohammed did not know was the entire prison was tapped, was wire tapped, which is being surveilled. When he gets back to his cell and he's in a cell with other members of al-Qaeda, he says, and they're able to hear over listening devices, don't say anything about the courier. So the fact that he had played down al-Qaeda and now goes in and tells his colleagues, be careful about al-Qaeda. Again, it wasn't a great lead, but it was helpful. And that led eventually, they're surveilling, trying to be quick about it. They're surveilling another terrorist who's talking to somebody on the line. And the guy is having a conversation. They were surveilling this other guy, not not him al-Qaeda. But it turns out it is al-Qaeda on the other line. They're able to get a cell phone for him. They're able to identify his real name, Ibrahim Saeed. And they are Ibrahim Saeed Ahmed. And they were able to trace the cell phone to the rather large city of Peshawar. And from Peshawar, they're able to identify by GPS where the cell phone is. It's in a white Suzuki Jimny. And they're able to track that back to a compound in Ababad. It just took time. There were a lot of false leads. And I will tell you, every time they would get a lead, they'd be, oh, this is it, it wouldn't be yet. So when they even got the Abadabad compound and they told the Navy SEALs and they told people in the White House, the first reaction wasn't, oh, my gosh, we got it. The first reaction was, yeah, we've seen this this movie before. This comes after intensive interrogations, including, as you mentioned earlier, waterboarding. Based on what you see here emerging, does waterboarding work? Did that method of interrogation work in deriving information that led to our finding out where some of Abadabad was? Well, that's, you know, that's a controversy. And obviously, some people will say you could have gotten the information without waterboarding. Colleague Sheikh Mohammed, 183 times, we'll never know. But they did get a lead from doing that. It certainly, you know, it was only one small step. They still had to after they had this name, they had to be able to figure out who the real person was and then get the cell phone and then track the cell phone to a to a Suzuki and then track the Suzuki back to Abadabad. You know, the the answer is it depends. People in the CIA would say, absolutely, that waterboarding didn't produce the lead, but it helped to produce the lead. There are other people who said they could have done it without it. There are a number of heroes that emerge in your story, two of whom you've mentioned. I want to talk about both of them. One is William McRaven, the at the time, the commander of US Special Forces. Talk about Bill McRaven and how he factors into this story. You know, Obama at one point said, if they were going to make a movie of this story, McRaven would be the lead and he would play himself. He is Captain America. He is just, you know, you guys know him in Texas. He, of course, was the chancellor of the University of Texas System. I have to say, I think he's one of the great Americans alive today. I had known him before I did this book. I got to know him a lot better in the course of doing this book. He was extremely generous with his time. He, you know, he came moved from to Texas, grew up there. His father was in the military. He decided to become a Navy SEAL. At some point, he went to the Naval Post Graduate School in Monterey, California and literally wrote the book, which ended up becoming the Bible on these kinds of missions. And his theory was that if you had a small group for a very short specific period of time, that had a speed of the attack, utterly rehearsed, you know, took people by surprise that a smaller group could overwhelm a bigger group. And he actually took case studies of about a dozen cases from the glider rescue of Mussolini during World War Two on down to the Entebbe raid, where the Israelis freed hostages in Uganda. And this became the Bible. And he was the head of the Joint Special Operations Command, which many was in charge of the Green Berets and the Delta Force and the SEALs and was running operations out of Afghanistan. And they would do raids sometimes a dozen a night. So when he's first presented and he gets into this very late, the original lead comes to the CIA in August. He's not brought in until the end of January of the following year, 2011. He says, you know, we do this kind of thing all the time. The big difference between this raid and any other is the fact that they are going to have to fly one hundred and sixty two miles from the Afghan border into deep into Pakistan to this town of Abadabad, which was the home of the Pakistani West Point. And it's a resort. And then there's a military base there. The West the West Point is there. There's a police force. There are a couple of hundred thousand people there. So getting in past Pakistani air defenses, raiding the compound and then getting out is what makes this different from the kinds of raids they did in Afghanistan every night. You know, I know Bill McRaven, but I did not realize that his father, also a military man, had graced the box of Wheaties in his time. And so the apple does not fall far from the tree. If there's anyone who should be on the box of Wheaties, it's it's Bill McRaven for the reasons you suggested. But there's a wonderful scene, Chris of Admiral McRaven talking to the Navy SEALs just before they embark on the mission. And he knows at that point that he's not just their commander. He's a coach revving them up to get into the game. And he he gives a speech, talk about what he says to those Navy SEALs about to go on that dangerous mission. You know, it's interesting. I just want to say one thing before I tell the story. You know, one of the things I was concerned about when I started to write this book and research it was, of course, there was a movie made a year or two after called Zero Dark Thirty, which focuses on a particular woman played by Jessica Chastain, whose role is greatly exaggerated. There wasn't a woman named. Well, her name wasn't Maya, but that's the code name that they had in the CIA for her role is not nearly as big, although she was still a key player and a fascinating character. But one of the things I realized as I researched the book is that. The reality, the true story is so much more interesting and frankly, in some ways, so much more theatrical than the movie was. And this is a perfect example. So he's has he decides they go through they're in Jalalabad, right on the eastern border of Afghanistan, just over the border, you're going to get into Pakistan and begin the mission. And he's meeting with the two dozen SEALs that are going to carry out the mission. And they've gone through all of the logistics and he kind of realizes I've got to give him a speech. I've got to give him a talk, as you say, by a coach.  And a lot of them, he used to play pick up basketball games with them and used to throw elbows and stuff. And, you know, with these SEALs and they greatly respected them. But they also got a smile on their face when he starts talking about basketball. And he says, my favorite, one of my favorite movies is Hoosiers. And do you remember the scene? And anybody who's a sports fan, I think has seen Hoosiers and loves it. And he says, remember the scene when the small town team, small Indiana town, they come to the Indianapolis to play the state championship game against a local team. And they're playing it in the big state gymnasium. And he says, when the kids come in, they're all overwhelmed by the size of this gym. They've just played in these little country gyms in rural Indiana. And so he says to one guy, the smallest guy on the team, get up on the shoulders of the biggest guy and here's a tape measure. And they go where they do the tape measure. And he says, how high is the basket? He goes 10 feet. And they says to another guy, walk off the the size of the basketball court. And it's 94 feet. And the point is, it's exactly the same size. The stands are much bigger. The arena is much bigger. But the court is exactly the same size as the court back in their little town. And he says, this mission is exactly what you guys have done every day. The stage is bigger. The stakes are bigger. But if you just do what you do every day, when you go out on these regular missions, you will be just fine. And Robert O'Neill, who is, I guess, our main character among the SEALs, the man who ends up killing bin Laden, says that he was born to give that speech. And he said, I would have gone out right at that moment and begun the raid. I was the perfect speech to put in context what they had ahead of them and and that just stick to your knitting, stick to your game and you'll be fine. And we haven't heard the last of tape measures as we'll discuss in a moment. I'll get back to that in a second. But the among those two dozen Navy SEALs is Robert O'Neill, who led the mission and ultimately is the one who took out Osama bin Laden. Talk about Rob O'Neill, Chris. Well, Robert O'Neill was born in Montana, average kid. He his parents were divorced and his father, Tom, decided at some point that he really wanted to help bring him up even in this broken home. And he, for instance, O'Neill was a big basketball fan. He took him to a court and they and tried to teach him how to throw shoot free throws. And at one point you had to shoot 20 free throws in a row before you could end the practice. His record was over 100. And then he introduced him to a SEAL, not thinking he was going to become a SEAL. But he ended up going up into a mountain with this this guy in a very steep, tough climb to see. I think it was L and and they didn't shoot any. They just looked at them and the guy was impressed. And he said, you know, you're in pretty good shape. You could be a SEAL. One problem because he was in Landlock, Minnesota, rather, Montana. O'Neill didn't know how to swim. So he had to go learn to swim. And fortunately, there was a swimmer, a collegiate swimmer from Notre Dame who took him and the local pool, took him under his wing and taught him to swim. He went to SEAL training and the buds and ended up becoming a part of SEAL Team Six. And he was, you know, a warrior and was a little older than a lot of the guys. He was into his 30s and had become a team leader. So sometimes he wouldn't go out on the missions. He would organize the missions and stay back. But he was one of the people when they were forming the group that would be part of these two dozen SEALs that would take on. Some could say that the most important, the most dangerous mission in the history of the SEALs, he was on the team. And there is another unlikely hero in the raid, a dog named Cairo. What role did Cairo play? Well, Cairo was, you know, they would bring dogs on missions because the dogs could do things sometimes the SEALs couldn't. First of all, they could smell explosives and they could chase down people running away that the SEALs couldn't. And Cairo was a Belgian, I hope I pronounced this right, Malinois, who had actually been shot in combat. And when he was wounded in combat, he was as if a SEAL had been wounded. He had a handler, a fellow named Will Chesney, and they had a helicopter come to the helicopter. And they took him to a combat hospital that would have been used for humans to stitch him back up. And he had a time of recuperation. And then he comes on the mission to and his job when Will Chesney obviously handling him was to patrol the perimeter because this was a very crowded town. They were going to have these two helicopters come in. One of them ends up crashing, as you remember, into the wall. That had a hard landing. But the the tail of the helicopter, of this Black Hawk, ends up perched up on the on the wall of a compound inside the compound. And, you know, they make a lot of noise and neighbors start coming and then police start coming. And part of the way they try to fend them off is the threat, you know, that was posed by by Cairo, who, you know, is is out there in the perimeter trying to keep the people away, whether it's first responders or just nosy, nosy Pakistanis, while the main group is inside raiding the main house. The the the most gripping part of this book, our audience will not be surprised to know, is your description of the actual mission itself. It is an it's worth the the price of the book alone. Can you just give us a condensed version of the mission, Chris? What happened? Thank you for saying that, Mark, because, you know, what I try to try to do in 45 and what I try to do in Countdown Bin Laden is to write a history thriller. You know, so often books are written. Well, we know this has happened. How did it happen? Why did it happen? No, I wanted it to be written from the point of view of people in real time in the room when they get the first lead. You know, we know about this compound. One of the things in the book is I never tell you whether Bin Laden is in the compound because they didn't know whether Bin Laden is the compound. When when Obama decides to he's going to launch the raid, he decides that on April 28, three days before they actually carry out the raid, various people in the National Security Council meeting in the Situation Room are saying, well, I think it's a 60 percent chance he's there. I think it's an 80 percent chance. And when it finally comes to to Obama, he says, you know what, guys, it's basically a 50 50 proposition. He's either there or he isn't. But that's no more in the president's mind than 50 50. He's even going to be there. So you have these two Black Hawk helicopters. And we told you in great detail how they prepare. They set up a replica of a compound and a CIA secret base in North Carolina. And then they go out to to Nevada, out into the desert. And they carry out another raid. And this was part of McRaven's theory. If you're going to if you're going to do a raid, you have to practice every element of it. So when they do the final dress rehearsal, they literally get the helicopters take off, fly 162 miles, trying to evade radar by the mountains in Nevada, come out. And they're trying to get the amount of time from when they come out of the mountains until they hit the compound. They know that there's going to be a certain noise signature. And as soon as that they want that at first, it's two minutes. Then it's 90 seconds. They finally get it down to 60 seconds that people in the compound would be able to hear the sound of the helicopters before the helicopters get there. So they're these two helicopters.  They finally end up over a Bada Bada Chalk One goes in and they're going to hover over the compound and people are going to repel out by rope lines into the compound. And then the man, that's the one that loses lift and crashes into the wall and has a very hard landing.  Is it just gone completely nuts or not? And McRaven like a pilot when you're in big shop and you're on an airliner and he comes on and he sounds like he doesn't have a care in the world says, well, we got one chopper down, but we're fine and we're proceeding with the mission and everybody's part, you know, goes out of their throat and back into where it's supposed to be in their chest. They go and attack the main house where they believe that bin Laden is there. He's in the main house. And meanwhile, Chalk Two goes to the perimeter, drops off the people that are going to be on the perimeter and the dog, Cairo, and then was supposed to go and above the main house and drop a group of people off who were going to repel down to the ceiling of this three story house. So they'd be coming down at bin Laden at the same time the group in the courtyard was coming up. But when they realized that something was wrong in the courtyard and we can talk about it afterwards, that made the planes unable to hold their position, they dropped O'Neill and his group outside the perimeter. They now have to get inside the fence and all of them go up from from the bottom of the ground floor to go up to the third floor to see who's who's there, who's the person living in the best quarters of the house and whether it's bin Laden or some other criminals, some other al-Qaeda terrorist or somebody's crazy aunt. And they find out when they get up there, they look and Rob O'Neill is pretty sure that it's bin Laden. He shoots him in the head twice. They eventually have to get out of the compound. Talk about that part of the mission. They think they have Osama bin Laden. He is dead. They put him in a bag, but now the mission is not over. They have to get out of the compound and out of the country because the Pakistani government does not know that this raid is happening. So talk about that part of the mission. Well, there are a couple of things they had always said. This was a three and a half hour. Let me say one thing first. And there are several things in this book that are new, that have never been reported before. And one of them was they all thought the SEALs did, and particularly O'Neill, that this was a suicide mission. They absolutely believe that this was as he put it to me, O'Neill did. It was a one way ticket that they were going to get there. But if it was bin Laden, that one, there would be explosives all around the compound. Two, that the house itself would be booby trapped. Three, that he would have body guards who would go down and, you know, take out any, any raiders. And four, that they probably would have an escape tunnel and that bin Laden might be able to get out in that 60 seconds between when he heard the chopper noise and when he got out. So they were amazed. They are still alive. It was always going to be an hour and a half there, a half an hour on the ground and an hour and a half back by helicopter. But after they killed bin Laden, they go to the second floor to what was bin Laden's office and they find a treasure trove of terrorist material, more than they have found in the entire 10 years since the war. They find computers. They find hard drives. They find thousands of pages of documents. They find bin Laden's personal journal. They also find huge freeze dried packets of opium and which I guess they were using to sell. But in any case, they find all the stuff and they make a request to headquarters, can we have more time? But obviously they're trying to balance. We don't want to leave this treasure trove behind. We may be able to exploit it and stop missions and roll up Al Qaeda. On the other hand, every more minute we're there, there's a chance that there'll be more people, Pakistanis that will come and stop. Anyway, they probably stayed around 35 or 40 minutes instead of 30. They get out. There is one helicopter, the Black Hawk that had brought Chalk 2 that hadn't crashed and they put bin Laden and all the material because that's what they've got to get out most and some of the seals. They get them out. But in the course of planning for this, they had put a spare helicopter inside Afghanistan. And as soon as one crash, McRaven said that this other Chinook needed to come was about a half an hour away, needed to come and wait just outside the compound. So they're able to get all the rest of the people on that helicopter and to get out two things they had to do before they left. One, they had to get because there were a lot of children. There were three wives of bin Laden were there. They have to get all of them out safely. And that says a lot about the mindset of US military. And, you know, we saw this in Afghanistan that where people are risking their lives to protect children, here they were delaying their departure to make sure everybody was safe. And then they had to blow up Chalk 1, the Black Hawk, which was a stealth helicopter with all kinds of absolutely new technology to evade radar. This is the first time it had ever been used in combat. And they had to blow that up while not endangering any of the other people. Anyway, they get back in the in the helicopters, the one Chinook, one Black Hawk, they head back for the border an hour and a half. And, you know, there's plenty of concern about now the Pakistanis know they've made a lot of noise. Can the Pakistanis scramble their jets and take these helicopters down? One of the other things that I don't know that it had been widely reported is that as part of his plan, in case they should get into a firefight, McRaven had what he called his guerrilla package. And they were they were in the air flying in circles just inside the Afghan border. They didn't come across, but he had fighter jets and he had AC 130 gunships. And if it became clear that they were going to be in a firefight with Pakistani fighters, he was going to send all of these planes, this guerrilla package across the border to protect the SEAL team coming back. In fact, they didn't need them. They get back safely.  And that's when the tape measure comes back into play. Yeah, so they finally get back to Jalalabad. They land, they take the body bag, carrying what they believe to be bin Laden into the compound, into the hangar. And McRaven, because he's going to be have to be the one who reports back to Panetta, who is, you know, in a formal sense, but not really in charge of this mission. This is done by the CIA, not the Pentagon, so that if everything had gone wrong, they would have been able to deny. No, we weren't in a bonobot. I don't know what you're talking about. So he's going to have to report back to Panetta at the CIA. He goes in and he personally, McRaven, opens up the body bag, unzips it and pulls the body out of the body bag.  I asked people, what had what had bin Laden looked like at this point? His face had been largely blown apart. I mean, it was not. It wasn't like there was some discrete coal. He had had three shots in his head because when he landed, McRaven put a rather O'Neill put a third shot into him just to make sure he was dead. And so he's looking at it.  And but he knows one other thing about bin Laden, and that is that he's big. He's six four. And he suddenly realizes McRaven for all of the planning that he had done. And he had done an enormous amount of planning. He doesn't have a tape measure. So he says to one of the SEALs, he says, hey, how tall are you? The guy says he's six two. He says, I want you to lie down next to the corpse. And the guy kind of looks at him like, what? And but he's on the craven. He's the head of joint special operations command. So he lies down next to the corpse and he and he fees that the corpse stretched out is about two inches taller longer than the SEAL is. So he figures, well, this has got to be bin Laden. And he reports he ends up reporting all of this back to the president. And the president, by this point, is pretty sure it's him. So there's a little bit of a lightning of this extremely tense mood. But he does say to McRaven, wait a minute, you had $60 million for a helicopter that we just had to blow up, but you didn't have $10 for a tape measure. And a weeks later, McRaven meets. This actually says so much about both of them. McRaven meets Obama in the Oval Office and Obama, you know, congratulates him and tells him, you know, I want you to express all of this to the team. And then he goes back behind his desk in the Oval Office and he pulls out a plaque and on the plaque is a gold plated tape measure. You know, wish you'd have this McRaven, incidentally, ends up giving it to the unit because he feels this isn't for me. This is for the unit, you know, this present, this gift from the president of the United States. And so some weeks later, he's at his home in Fort Bragg in North Carolina. There's a knock on the door and there's a military officer and he comes and he gives McRaven. He says, we hear that you gave the tape measure from the president away. He says, yeah, look, I thought it belonged to the unit and he wants to make it clear I didn't care about it. So I just gave it away. He says, well, the president would like you to have one for your family to keep. And he hands him another plaque with a tape measure. It's an amazing moment. You quote Bill McRaven as saying of Barack Obama, he was the ultimate team player and the smartest man in the room who made the incredibly bold decision to do the mission. How does this mission reflect the legacy of Barack Obama? Well, I think it I think it does and in a lot of ways. First of all, one of the things that impressed me is the number of meetings they had just in March and April of 2011. Now, remember that after Panetta gets the lead in August, he goes to the Oval Office and briefs Obama in September of 2010. They have meetings throughout the fall. He has a meeting with him just before he heads to Hawaii for his Christmas vacation in December. But as it gets really closer and really serious, there are 17 meetings in March and April. Now, the president isn't in everyone, but there are different levels. There are deputies meetings, which are that sort of the people under the principles like the Secretary of State. There are principles meetings, which are the Secretary of State, the Secretary of Defense, the CIA director, but not the president. And then there are National Security Council meetings, which is when the president is at the end of the table in the sit room. And each of the meetings is held very carefully and, you know, building blocks and reviewing what they know and what they don't know and answered. Let's answer this next question. And so Obama could really set a process in motion. And I will tell you, Tom Donilon, who by the spring of 20 of 2011 is his National Security Advisor. They take this they absolutely maintained a meticulous process, which is hard because this is so secret that a lot of people were never read in. Hillary Clinton, the Secretary of State, wasn't read into the mission. As I tell you to say, they got the first lead in August. Defense Secretary Gates isn't read in until December. Hillary Clinton, because he's the Secretary of State, isn't read in until March, a little over a month before the mission takes place. And they were told, you can't tell any of your staff. They were so scared that there would be a leak. This was the best lead on bin Laden since Torah Bora in 2001 when he escaped from the mountains in eastern Afghanistan. And they were so scared there'd be something. Well, they seemed to have a lead in bin Laden, and then he'd be gone like that. So the the principles were not allowed to tell any staffer. She wasn't allowed to tell the Deputy Secretary. She went to a meeting at the White House, Hillary Clinton. She had to make up a story as to why she was going. She couldn't say, I'm going to have a meeting about bin Laden, even to the people right around her. And having said all of that, when they finally get to the final decision meeting on April 28th, they go around the table. Biden says, we don't have enough information. We're going to really mess up our relationship with Pakistan. I don't think you should go now. Let's see if we can get more information. Gates, these are the first two people he talks to on the right side of the table in the Situation Room. Bob Gates, the Secretary of Defense. And this is very interesting. Donaldson said history was in the room that day because everybody came at this based on their experience. Well, Bob Gates had been an advisor to an executive assistant to the CIA director back in 1980 during the around hostage crisis when they carried out Operation Eagle Claw. Remember the rescue mission under Jimmy Carter to rescue the hostages? And they ended up at a spot in the desert in Iran and there were transport planes there and helicopters there. And they ended up having a terrible collision and killing eight people. He had never gotten over that. And his feeling was something always goes wrong in these missions. I think if we want to have a drone to take out this one guy who walks around, they call him the pacer. That's fine, but it's too risky.  The chairman of the Joint Chiefs, Mullin, Leon Panetta, Hillary Clinton, who said it was a 51-49 proposition. Obama says, look, it's 50-50. I'm going to make a decision and I'll let you know in the morning. You know, talk about a great go to commercial. And he sits by himself upstairs in the family quarters in the treaty room, which was his office with a basketball game with Kobe Bryant playing at night, thinking about it, going over everything and makes the decision to do the right. And I will tell you that when the next morning, on Friday the 29th of April, he calls Donald in to the White House to tell him before he heads out on a trip he had to take. They weren't, he wasn't at all sure he was going to approve the mission when Donald calls Panetta. He's not at all sure. And when Panetta calls McRaven in Bagram before he goes to Jalalabad and says the mission is a go, McRaven wasn't at all sure he was going to say yes and thought, as you say, that it was an incredibly gutsy decision. It bears mentioning Chris that this happened, this was staged in Afghanistan when it was under U.S. control. That has changed. Back in July, Joe Biden said that there really wasn't a chance of the Taliban coming back. He said the Taliban overrunning everything and owning the whole country is highly unlikely. He was wrong. You covered this as a journalist when it was happening that the stunning takeover of the Taliban of Afghanistan and you will look back on it as an historian, you are both. So as a journalist, why was Joe Biden so wrong and as an historian, how will history look upon this moment? Well, that's interesting. You know, you got to remember when Joe Biden came in in 2009, there was a big discussion because we were in the process of losing the war in Afghanistan then. If you remember, President Obama had a series of meetings with top advisors. People started saying it was paralysis by analysis because he had so many meetings as to whether or not to search troops. He ended up surging tens of thousands of troops to Afghanistan. Biden was against it. And this is before they had taken down Bin Laden. But his feeling was this idea of this big mission to kind of transform the country and turn it into a Jeffersonian democracy and transform the role of women in Afghanistan was a mistake that we did have a role there, but it could be much smaller as a simple counterterrorism role. And he got beaten by, I mean, he got, he was overruled by the president who gets to make the final decision. He even said at one point to Obama, you're getting rolled by the Pentagon here, but he always wanted out. So if you remember in the campaign in 2020, when he's running for president, Biden says, I wanna end the forever wars. Of course, Trump had said the same thing. And when he comes in, he keeps up. Remember Trump had made the deal with the Taliban was planning to pull out by May 1st. Biden makes a deal with the Taliban. We're gonna pull out, it'll be a few more months. It'll be September 11th, but we're gonna get out. And I think there's a few things. I think some of the intelligence was wrong. Look, there were very few people who said the Taliban was gonna take over in 11 days, which is basically what they did. But I think a lot of it was also the fact that Biden had wanted out of Afghanistan for more than a decade. And now he was in charge. He was the man making the decision. And that was the decision he wanted to make. So what do we learn from this ignominious chapter in American history? The failed takeover and democratization of the Taliban by Afghanistan? Well, I think that the decision that most of the American people had made, which is why they voted for Trump in 2016 and why they voted for Biden in 2020. And there were other people running who said, no, we need to stay in. And in both cases, they said we need to get out. Is that people felt that had a different view than we had up till 2016 about our national security interests in Afghanistan. That where, you know, the original thought was we need to stay there. We need to have a presence. We need to solidify a government and a fighting force. Their feeling was that in a strict definition of our national security interests, what we really need to do is to make sure that they can't strike the US homeland. I remember having a conversation after the President Biden decided in April that he was gonna pull out with a top White House official. And this officials, I asked, I said, you know, at some point the Taliban may take over. And if they take over, things are gonna turn very bad for the millions of Afghan women and Afghan girls who have come to lead different lives in the last 20 years. And this official said, that's true. But ultimately sending Americans to fight and die for Afghan women is not in our national security, which is kind of cold-blooded and calculated, but I'm not sure that it's wrong. And, you know, this official said, there are a lot of terrible things that happened around the world and we can't police them and we can't stop them. Our function in Afghanistan is to prevent an attack on the US homeland. And frankly, and I talked to a top person in the Pentagon this last week who said Al Qaeda in the Arabian Peninsula, Al Qaeda in Yemen is more capable of striking the US homeland than any Al Qaeda unit. Not to say Al Qaeda is gone. The President Biden was dead wrong about that, but that it's not as up to a real threat to the US as Al Qaeda and the Arabian Peninsula is right now. So, you know, people are gonna disagree about this. There are a couple of things that I feel about Countdown Bin Laden and why I think it's important, particularly now when a lot of people have such regrets and feel so sad about the way this war has ended, our America's longest war. One, we did accomplish our prime objective there. We did bring the mastermind of 9-11 to justice and we did roll up Al Qaeda.  Two, as screwed up, and I think that's not an unfair thing to say as our way of getting out of Afghanistan has been. The Bin Laden raid is a case study of how the Intel community and the political community around the president and the military community under McRaven worked together flawlessly hand in hand to pull off a great mission, a risky mission, a dangerous mission, a mission where they didn't know what they were gonna find in the compound and they did it. And then thirdly, it's just a great story. It's a story that I think Americans can rightly take pride in, the ingenuity, the persistence, the stick-to-it-iveness, the bravery, the courage, and the daring of Americans and at a time when a lot of people may not feel that way right now about our government and the decisions that we made. This is a pretty good case study of how well we can do things sometimes when we get it right. Well, I enjoyed every word of Countdown Bin Laden and I have great admiration for its author, Chris Wallace. Chris, thank you so much for joining us today. Mark, thank you. Thank you for joining us. Signed book-plated copies of Countdown Bin Laden are available for sale at lbjstore.com. I hope you'll join us for our next programs, continued conversations with Jason Stanford on September 15th at 1130 Central and an evening with Dr. Anthony Fauci on October 6th at 7 p.m. Registration is available online. See you next time.