All right. Thank you. Good afternoon, everybody. I hope everyone had a good lunch, certainly a good morning, a lot of thoughtful and provocative talks. So thank you all for continuing to be here. My name is Paul Mainz-Helsen and currently I'm a data scientist in residence at Montaigne Ventures, which is an early-stage VC fund here in India, and I've spent most of my professional life as a data scientist. But in the past year as an investor, I've had the time and space to step back a little bit from the day-to-day work of data science and think about things at a little bit of a higher level. And I've also gotten to see and talk to a lot of startups doing really fascinating and promising work in AI and ML and analytics. And I think the thing that characterizes the best data science projects I've seen is the way a team approaches the foundation of its data. Good quality data and a well-defined source of data, I think, are the critical factors to success. And that's what I want to talk to you about today. And the thesis I'm going to present is that quality is less to do with data itself. It's not a list of metrics or characteristics that you can rank. It's not an intrinsic feature to the data. It's more to do with how we look at it, how we work to understand the data we have, and how it came into its existence, especially. And so the quality comes from that symbiosis between the appropriateness of methods and the data that you have. And I'm borrowing from the statistician George Box here. I originally pointed out that all models are wrong, but some are useful. And I'm saying that something very similar applies to data. And whether it's good data and being about how you approach it. And I'll start with a story that motivates my message. During World War II, the US government put together and set up a small organization of statisticians and mathematicians in Manhattan. It called the Statistical Research Group. And the SRG worked on a variety of problems related to the war effort. Among these was the problem that the Air Force was facing. Taking off from England, bombers were flying deep into Northern Europe and into Europe to bomb strategic targets in Germany. They were also getting shut down, and the Air Force was losing a lot of planes. So the Air Force, they decided, leadership decided that they needed to up-armor their planes. They also realized, as with most problems we face, I think, that they faced a trade-off. The more armor they put on the plane, the heavier it would get. Which meant that that would decrease the distance that the planes could fly. They could reach a fewer set of targets, and they'd be closer to the line of running out of fuel. So they decided they had an optimization problem, and they turned to the SRG to help them with it. And they said, look, we've been tracking all the planes that are coming back in and landing and that are flying these routes. And we've created this data set of where they've been shot and where they've been hit, where they're taking damage. Take it, please, and model it so that we can understand where we're getting hit. And we'll focus our armor there. And there's a statistician on the team named Abraham Wald. And he took on this problem, and he looked at it, and he thought about it for a bit. And then he came back and he said, he told the Air Force, here on the engines, this is where we're not getting hit. This is where we don't seem to be taking damage, and that's where we need to apply armor. Because, you see, he realized that the data he was seeing was a sample of the world. And there was key data missing, namely the planes that were taking off, getting hit, and crashing, and not making it back to England. And so on those planes, we don't see them. Those are not a part of our data set. And so we can make the inference that missingness corresponds to fatal damage to planes. And instead of focusing on armor, our armor on where we're taking most fire, we focus it on where the fire is doing the greatest damage. And I think this characterizes story for me characterizes what I want to talk about. And to convey my message today, I want to kind of walk through four stories from my own experience. And I hope to persuade you that in data science work that you should be caring very deeply about what we'll call the data generating process. To care enough about it, to do the work that doesn't always get the glory and doesn't always feel like it's on the cutting edge. To understand where your data is coming from. And also to do the work of communicating this. Because I think both are necessary for real success.  And these are not nicely wrapped stories. They're mostly open-ended. There's no easy, obvious, clearly right answer. This is not a SAT or JEE exam. Because I think that's how life is. Mostly I'll be walking through a kind of a sample of experiences of discovery as I've experienced them.  In scenario A, I apply X method. And scenario B, I apply Y method. But rather to kind of walk away and think, huh, I see how I'm facing or might be facing something similar, something analogous in my own work that deserves further thought, that deserves attention. Or maybe even just sort of as a process of intellectual simulation that you see something about the examples I give. And you can simulate in your own mind something that I seem to have overlooked or how a line of approach might take you the distance towards a better solution. So what is the data generating process? One of my professors and statistics in graduate school used to describe it this way. To convey kind of, it's all the things that take data into the world and from the world and into your data set. And there are three general things that I like to think about that define it a little more concisely or a little more specifically. The first one, which we often overlook, and I think that's a little bit about what, you know, Chris's talk earlier this morning, which I thought was quite good, talked about it. And probably things we learn in statistics class about sampling strategies or, you know, how marbles are selected from a jar. They're actually really important but often overlooked. Statistical model of a data, the statistical generated data generating process is often what we're trying to model when we set up a regression, for example. We're explaining some effect or some phenomena in line, you know, using some other set. We're explaining a Y with a X matrix. And then finally there's this data collection process, which is often not what statisticians are talking about. It's often more what we should be thinking about as, say, software engineers or developers or builders. It's the routes and technical procedures by which data reach a database. And that's data engineering. So my first story picks up the thread from the statistical research group, except we fast forward to India in 2015. And I think that we often overlook issues with our data generating process, not because they're incredibly subtle or hard to spot, we have to be brilliant. Instead we overlook them because we're so excited about building solutions and finding a direct answer to our primary business problem. In 2015 I was vice president of data science at housing.com in Mumbai. And we were excited about building an excellent real estate experience so that instead of wasting your weekend, you know, crossing a city in traffic stuck just to see a flat that it turns out was obviously not right for you, you should be able to find your next home from the comfort of your home. And a core part of that effort was bringing as much real estate inventory online as possible. In the summer of 2015, our ops team reached a million properties and that was a huge success for us registered online. And up to the time I joined the data science lab had very little insight, really no insight into that process, that ops process and very little attention for it, given all the other things we were thinking about. For example, we were really thinking a lot and very excited about overhauling our recommender system. And we are intrigued by, really intrigued by all the ways that conventional recommender systems for e-commerce or say media, all on Netflix weren't right for real estate. And that's what had our mind share and that's where our data scientists were keen to start, except we remembered that we needed to start with thorough data exploration. And our exploratory work led us to looking at our data in this kind of way. Each of these plots shows two curves for a city. In blue is the kind of properties that our users were searching for. And in green were the properties that we had in our inventory for that city. And in looking at these we realized we had a problem. See, in cities like Mumbai and Delhi you can see there's considerable area under the blue curve under what the demand curve that didn't follow within our supply curve. Which means we had users looking for kinds of flats that weren't in our inventory, that weren't on our site. And so it didn't really matter how sophisticated our recommender system would get if we didn't have the right inventory to recommend. And these plots introduced a question for us that we needed to answer about our data. Is it representative of the market? Because if it is, then our users are looking for flats that don't exist. They're not available. And we need to educate them to that. Like you're looking for a particular kind of flat in Bandar West in Mumbai and it's just not there, it's good for you to know that. So you don't waste time. On the other hand if our data isn't representative then there's something wrong with our data collection process and that deserves our attention as well. So what is our data, what was our data collection process? Well in each city and locality we hire data collector teams to go out and find properties to register and bring on the site. And those collectors had been incentivized based on quantity. Get as many properties on the site as possible. And demographically our data collectors tend to be young, male, often bachelors. And they were great at the job we'd given them, getting inventory on the site. But then we realized that potentially we hadn't been precise enough in our direction. See in their rush to get inventory on the site they were going to brokers who they were familiar with. It was like a path of least resistance. And that means they tended to be going to flats like this on the left. Large, lower end, it's more affordable brokers who were kind of just renting out in volume. And they were very happy to just put 12 flats on the site except those 12 flats are all basically the same. So there's very little value add from a user's perspective for looking through 12 flats that look exactly the same and have the same layout. And our collectors weren't nearly as much going to flats like this, higher and more expensive, lower... Because the brokers there were not so comfortable, right? They don't deal with large volume and the two worlds just don't really overlap. So when we realized this, we were like, well, we can actually change our data generating process. We can say let's just build a feed. Our data collectors already have an app that they use to collect. So instead of just having them go and be incentivized based on raw numbers, we can actually create just a simple API that says, here's what we're looking for this month. Feed it to the app and then incentivize the brokers to do the extra work to break out of their comfort zone to do the things that can get us a more diverse set of inventory. So our data generating process can often be influenced by social factors, demographic factors, things that we just wouldn't normally think about as a computer scientist, but actually end up being incredibly important. And we also had data generating process problems on the demand side at housing. See, we were ensuring that we collected as much data as possible about each flat, and then we tried to make the search and filtering process for users as clean and delightful a user experience as possible. To do that, we needed to track and learn about how our users interacted with our next flat. Basically that meant using clickstream data across the user's product journey, all the things they're doing. So what does that look like? Basically users get on the site and they take a couple of actions and they sort of sequentially look through different properties. And we're learning something about the kind of properties they're looking at, but we're making inferences based on the sequence in which they do that.  They'll say, closer to job, I'll pay more for less. If I'm further away, I want to pay less, I'll get more. And so there's this event stream that you can kind of imagine. So most of you are probably familiar with clickstream data, but in case not, your standard stuff that pretty much every website is tracking is things like, okay, the page, a timestamp, some kind of user, either an IP address or if you're logged in, a user ID, the URL, things like that. But services like Google Analytics and others that kind of out of the box set this up will only take you so far. If you want to really understand and see more granular data about sequences, where people are, which filters they're selecting, et cetera, end up getting into quite a more complex event data modeling. And our product analytics team, as part of the broader data group, spent a lot of time working on this because they realized this is our data-generating process, and it's really important, so it's worth the time. And they went to a lot of work mapping out the flow that the site facilitated so we could track it and learn about it over time. And this would be fine if we had a very stable product, except we didn't because while housing was known for good data science at the time, I like to think we were, housing was also known at the time for really great design and the design team, what they cared about most was how the product felt and which means they were willing to do all kinds of work to improve how it felt, how it flowed, how it was used. And our front-end team, which had to implement all of that, cared about performance, they cared about making it fast, making sure that information is compressed or flowing smoothly. And so what we realized was that all the tracking we set up, when workflows are changing, when the site's changing constantly, those aren't, because the teams, the front-end teams and the design teams weren't really thinking about the data they were generating, they weren't necessarily always taking the effort to go back in and remap or work with the data team to understand if this event is no longer even possible on the site, a drop in that event doesn't mean that people aren't doing it anymore, it means they can't do it anymore. And basically the lesson here is that when you have a data-generating process that's changing very quickly, it's very hard to make inferences from it. You lose the longitudinal value of your data. And what it means is that this is a technological process by which data is getting into your database. And if you're not paying attention to how that's set up, then you're going to get data that's not appropriate for the questions you're asking of it. Now the two stories I've told from housing so far, both illustrate how people, it might be business partners, it might be ops teams, it might be front-end developers, are somehow affecting an influencer data-generating process and the data we work with. And in my next story, I kind of want to disrupt the tidy dichotomy between data and the world on the one hand and the data science team on the other. Many of you have heard the phrase that data is like the new oil, right? And at housing we like to say, no, it's not really like oil, it's more like soil, because oil is a non-renewable resource, right? Once you use it, it's done. But data is renewable, you can use it, you can use it for multiple purposes and it can even increase in value over time. But the issue is that the things we do to it affect the data, it generates new data, we're affecting that. And then over time there's new people coming onto the team and there's new ways data is being used and we still need to understand the things we did in the past to affect how our data has come into being. So I'll talk a bit about our story. So after I left housing, I was Chief Data Officer at a startup, a fintech startup in Mumbai called PaySense. PaySense does mobile loans through an app and we were working on a problem that's very common, probably for everyone in this room, you're all kind of familiar with it as a standard data science problem, how to decide to give a person a loan. And a lot of times we jump into using, building a model and we might use a linear regression which performs pretty well, right, that Chris was talking about. But what I want to talk about is these features that actually are going, before we get to the model, where the data is coming from. And I'll talk about, for instance, a feature that we decided was important, average monthly bank balance. And average monthly bank balance, you know, is this thing that represents a deeper process. So most people who are applying for a loan, their financial lives are cyclical. Every month they make some money and every month they spend some money. And what we're interested in knowing is what's the top and what's the bottom and what's the curve. So really we're interested in knowing something like this. Normalized across day of the month, is where their balance tends to be. And then we're going to reduce that, right, to just one number, monthly average balance. And you might think, well, that doesn't capture enough information about this curve. You might want to know, like, monthly average range or variance or max or min or something like that. And yeah, you can add lots of features in there. But we just need to understand the underlying process. And then we realize that, like, this looks pretty pretty. It looks like we've got enough data and we've kind of drawn a nice curve and we kind of understand what's happening. But actually a lot of times the data looks more like this. So we've not normalized anymore. Now we're saying we have three months of data for observations of a person's bank balance. And we can kind of see, okay, so in June and April, we sort of see this incline. It looks like it might be representative. And we kind of see a similar slope in May, but it's lifted and it's somehow shifted rightward for some reason. We can think about why this is happening, but the point is that it's not a neat curve and we need to figure out how to interpolate and do other things. So we also need to ask ourselves, where is this data coming from? Let's not jump into just, like, coming up with the right curve or coming up with the right number. So where we're getting it from, where we got it from, was SMSs, financial SMSs. When you do a debit transaction, when you do a credit transaction, you get an SMS and it contains a reference to your bank balance. Okay, but not everyone receives the same rate of SMSs. Not everyone, some people delete them. There's lots of different processes. So let's just look at, like, a histogram of how many SMSs we have for a given user. And we kind of see, oh, there's kind of a bit of a curve. So we expect our users mostly to have around 20 to 50 SMSs. And then we might ask ourselves, well, let's describe it in a different way. So we can also look at how much time. Because you can have 1,000 messages that came in a week. And that's really not conveying enough information to you. You care about over time. So let's take the earliest date observation we see in the last date. And we look at the span of time there. And we can also draw a curve for that. And we kind of see, oh, there's these peaks. Well, by the way, these peaks had to do early on in the process with the fact that we realized, first we were saying, let's take 30 days. And then deciding, well, one month is not going to be enough anyway. So let's take 60 days. And we kind of see that show up in the data. It has nothing to do with people's finance. It has nothing to do with how they do transactions. It has everything to do with how we set up our SDK. And you can continue to kind of ask yourself, how are you modeling this? What is the underlying SMS, why do we expect people to have all their bank balance transactions being recorded on their phone? So now we can say, let's take the count of dates so like the number of dates covered. But just like 1,000 messages in one week is not sufficient. If we have an observation, and then two months later we have another observation, but like maybe only two observations in between. If you look at it just in terms of the count of dates between the two, it looks like a lot of dates. But actually if you look at the count of unique dates, it's quite low. You have insufficient data. So if you plot this together just very simply, right, we can kind of see. Ideally, our users would follow along this line. If you have 40 days of messages, you should have at least like around 40 or high 30s of unique dates. But of course that's not how all people are. Some have large spans of time, but not a lot of good coverage. And we can also kind of look at the density of messages within those time spans. All of this is just to sort of say that all of this work kind of needs to be done before we just jump in with a raw algorithm or like an equation. And when we come up with that average balance, we're not just creating an average from observations, right? Because if we have five balances across a month, you don't average over five because there's all the dates for which you don't have observations. So you have to interpolate it. You have to impute missing data. So what do you do? Do you forward fill? Do you back fill? So let's say I have an observation on the fifth of the month and it has 1,000 rupees. And then I see another bank balance on the tenth of the month and it's 2,000 rupees. Now, do I say all the days between those? Do I use 1,000 rupees? Or do I use 2,000 rupees? Or do I split the difference? It makes a difference because time spent at a different level of bank balance affects the credit that people are going to be able to get and your evaluation of whether they can repay. That average bank balance is not just going to be used in the decision to give them a loan or not. It's going to be used in the decision by a different model of how much to give them a loan. A different model of when to market to them. Another model that's going to be asking how likely they are to be engaging in fraud. And other data scientists are using it. So we generated that data that then just becomes data in our database. It's in the monthly average balance table or whatever table it's in. But we affected its data generating process. The decisions we made affected how the data ends up in our database. Okay, so let's continue with the story of giving a person a loan and we take another feature. And a lot of the stories we've looked at so far, we've kind of thought about a single data generating process that kind of covers all of our users. But this last story shows how sometimes figuring out our data quality involves recognizing where we have multiple data generating processes and where we have some work to do to tell the difference. So our next feature we might think about is monthly salary.  And well, in the application process, we rely on them to tell us, to report on their salary. But obviously we know that we can't just take data at face value. There's a problem there. And so there's another process that you can follow which is kind of logistically complex of sending a courier and collecting a paper that has a salary slip or an offer letter. And then you kind of have to convert that into data. And that's complex and expensive as well. So we kind of want to know is, can we validate salary more efficiently and more cleverly using our data? So we say, again, turn to the SMSs. Because we observe that some of our SMSs seem to be telling us that when a person receives a credit into their account associated with their salary, then we've kind of observed a fairly trustworthy, a little bit better or additional to the report they've given us. And sometimes, by the way, we're really confident that that's a true SMS. Sometimes we think, okay, this is not, it doesn't look like, we're not sure it's a salary, but we feel like it is, right, here in red. It doesn't say salary, but we're pretty sure this is an employee of eBay. And that they've received this salary this month. So we can say, well, let's use this and map it up against what people are reporting. And so if we do that, and we'll just log transform it so our distribution looks a little cleaner, and we realize that, yeah, this kind of represents what we were hoping. People, there is some match. There is some mapping between the two. But we kind of can also suspect what our intuition would tell us is that there's a lot more people telling us they make more than their SMS says than there are vice versa. And what we need to understand is what else is going on here or which ones can we trust and which ones not. So let's think about how this data is being generated. Where is it coming from? What is it? Well, we can go all the way back to the beginning and we can say, well, how is salary itself generated? What's the data generating process there? Well, it's actually kind of a process of negotiation, right? And then you're like getting way off track and you're kind of thinking about like qualifications and how a person gets a salary. But it's important to at least think through that, okay? That comes up with the salary that you agree to with your company. And that's usually a humanly meaningful number. So you never agree to a salary of 37,237. We like round numbers. We like simple things. But when we get our salary, that's not exactly what we agreed to. Because the finance department gets their hands on our check and they do all kinds of deductions and things like that, right? And so that's a data generating process for what actually gets deposited into our account. And we can kind of follow that through when we say, okay, that's our actual salary. And then we have our user report. How does that get generated? What's the generating process there? Well, a user is not looking at what their finance department gives them. Usually they don't even look at their payslip. They're applying for the loan like say in the mall or as they're buying for something. And so they're thinking of their humanly meaningful number. They know their CTC is say 40,000 a month. They don't know that this tax and this debit and this blah, blah, blah. So they're going to report that. So to a certain extent, okay, so that's the user report. And then we know that some users are going to exaggerate their salary. That means they're just going to bump it up a little bit because honestly they're going to get this bonus next month or when you calculate it out, they're going to get an increase within three months. So that's really safe. And then there's some that are dishonest. They're just telling us a lie. They're making 25, but they say 60 because they think they'll get a better loan. Meanwhile, on the other side, we have our SMSs. We know that our salary is generated by a financial process and then we're getting a notification. And that's generated by an SMS, whether the person deleted their SMS, all the things that we talked about before with balance. And then also there's this process where we're not even sure all the time where we say it's a salary SMS, whether it actually is. Okay, so thinking about this, we can kind of ask ourselves, well, we still know we have two sources of information. What a person tells us and what their SMS tells us and how do we look at the difference. So we might take their income and subtract what we've identified in the salary and we draw a distribution for what the difference looks like and how big does it tend to be. And it's kind of nice that it's fairly closely approximate to zero, so it seems to be a good number of people who are somehow being honest as we have a match. But what we can... And you might try and run a regression to kind of come up with what the difference should be. What's the bias? Because then you can kind of be a little bit smarter in the salary. But you can also ask yourself, do we actually have multiple data generating processes here? Should we be estimating a bias that a user has as if it was a deterministic effect, as if it's just a coefficient and a regression? And I don't think so. I think each person has a bias and that's not uniform across the population. So maybe we'd want to be modeling it that way. But we can kind of take what's called a mixed distributions approach and we can say, all right, well, we actually have our two populations here. One is an error distribution where people just are off a little bit. It's narrow and then bias distribution which covers when people exaggerate. So it turns out that, yeah, we can kind of expect around 61% of our users to fall into the error distribution. And we can assign each one based on, say, a likelihood ratio test. And there's good estimation models where you can kind of choose the parameters of the distributions themselves, model those as distributions. But the point is that now we can sort of say, for some people we're going to choose their true salary and for some we're going to adjust them based on SMSes. So the point is that there's, if we apply just one uniform probability model to all our data, if we think about what itself is generating the information we're looking at. I think this is a tweet from many years ago, but I think it really represents well what a lot of data science work is. It's this question of we're counting and we're figuring out what's our true population, what's the relevant population for the problem we're looking at and can we reproduce it? I'll talk about that later. So I also want to kind of quickly talk about that second part of this because it's not just about knowing this or thinking about it or working through it. It's also about how do you, why is it so important to actually talk to people about data generating processes, especially your business partners, especially your product managers? And I think it comes down to something, to my background, like most of my research work was done in cognitive science and applying to school and to presentation to how people make decisions. And there's this thing called theory of mind, which has to do with how we understand what's in another person's head. And there's also this thing that comes from that called the curse of knowledge, which represents the problem all humans have with knowing what they know, knowing what someone else knows, and like not telescoping or telegraphing what we know onto that person. It's hard to remember that we know things that other people don't know. That sounds weird. You can kind of look and sort of see that humans aren't born with this capability. It kind of is generated over time. And so this is an experiment that's often done with young children. And basically they're shown this series of events. Basically there's two young, there's two little, two children. Sally has a basket and Anne has a box. And Sally has a marble and she puts the marble in her basket. And then she goes out for a walk. She leaves the room. And Anne takes the marble out of the basket and puts it into the box. She moves it while Sally's out. And then Sally comes back and she wants to play with her marble. Where will she look? Now we all know that she's going to look in the basket because that's where she put it. But children of a particular age will ask, will say she's going to look at the box because they know it's in the box and they just kind of assume that Sally knows it's in the box as well. We stopped doing this when we're children but there's still a lot of it that sort of stays with us. It's still very hard to understand. Another way this shows up is in our language for example. So I'm just going to do a quick experiment. So let's say, if I say highly likely actually can someone help me? Can you help me? Just stand and memorize the numbers I tell you. So if I say highly likely raise your hand in the room if you think I mean it's about 50% chance. Nobody, right? Maybe two or three people. Now let's jump to 65. Anyone? 75. All right, we're getting there. 85. Good, good. 99. Okay, so we're right around 85 or 90. Remember that for highly likely, yeah? What about probably? 50? Okay, a lot more. 60? It's kind of going down actually. Okay, so say 55, yeah? You remembering these numbers? Good. All right. Probably not. 50. 35. 25. 15. That's weird. We're kind of skewed on probably not. Somehow it's stronger than probably. All right, let's say 25, yeah? And then finally, highly unlikely. Let's skip jump. We're not going to start at 50. Let's start at 30. 30? 25? 18? 15? Sorry. Minus from 100. Yeah, sorry. No, that's right. 10. 5. 3. 2. 5. 5, right? Okay. Well, actually, there's been work that's done and looked at this. And it's pretty cool. It's a nice visualization. The point is, if I say highly likely or probably not, we don't really know what I'm saying. I'm trying to be more statistical, right? I'm trying to be more database. But there's quite a range on what people believe we're saying. And this is what happens with our data. Our business people get really excited. They're like, data science and AI. And we've collected all this data. And what's the answer?  And I know everything. It's not really true. But we have to do the work to help them understand. What are the limitations? What are the appropriate questions that they should be asking of our data? Because it's kind of like turning the world into data as this process that's both really wonderful and letting us be more concise and more precise and more accurate. But also loses all the things that Avi was talking about this morning, right? About all the things that we pack into our knowledge that we forget to explain. So, right? We have people here and on the left, like the world we know. When we look at a representation of data like this, we're filling our minds with all kinds of knowledge, like all kinds of context that helps us understand why they like these sports or what have you. And we can kind of, as we move over, we get less baggage, less bias, less things that we're not explicitly incorporating to the model. We're also getting more abstract. We're losing information. My point is just that data is not just data. It contains a lot of stuff. And that communicating it, what's right to ask, what the quality of it is for our given purposes or questions, is something we have to work with, with the business partner and the product partner. Not just something we assume that they know as well as we know. And my final, so that's my point, is that quality is not an intrinsic attribute of data. It's a description of the appropriateness of the question we're asking of the data. Data is always going to be incomplete. It's always going to be problematic and flawed. We don't just come up with a binary, yes, it's good data or not. It's all about, well, we can ask this kind of question. We can use this kind of method. And that's the best we can do. Or we can use it for a completely different purpose that it's more appropriate for. And the final thing is how do we know we're using it in a good way? And how do we know we're going through this process in a good way of communicating our day-generating process? And I think coming from computational social science, in the last five to 10 years, there's been a lot of work that's been done in understanding the importance of reproducible research. The fact that we shouldn't just believe a paper because it was published, we need to understand what was happening behind it, what were the methods being used, but reproducible is actually just the first step. So here's a sort of a two-by-two that I like to think about. We can think about our data, in our left column we have the same data and different data, and then you think about our code or infrastructure or our algorithm, and we can have the same and different. And when we have the same data, what that means is as a data scientist, I do a problem, I use some data, I use some code, and I come to a result. Now, I might have messed it up somehow or overlooked something. I need someone else to be able to go in and do it again and come to the same result before we can really believe in it. But that's not as robust as it could be. If we use different data, that's when it becomes replicable. And if we use different code, that's when it becomes robust. It means, yeah, you use this, use this linear regression, but if I use logistic correction, I shouldn't have completely different outcomes. There should be some resonance, some similarity, some consistency. And finally, when we're saying, like, we can use different data and we can use a different approach, then we think, okay, now our model is generalizable. We're really learning something about the world that we can believe. And I think this is the kind of thing I like to think about how we kind of measure ourselves in terms of the appropriateness of what we're looking at. And that's it. Thank you. I don't know if we have time for... Sorry, you can sit down. Questions? Hello. Questions? We have time for a couple of questions. Good, because I think I ran over. But I will be around. I will be around. And also doing a... Hello? Okay, yeah. There's a VOF on data engineering. We're going to talk more about actual methods and techniques and, like, technologies that you can use to kind of do these things, all the work of data engineering. Tomorrow, I'm pretty excited about that. So if you're interested in this topic please do come by that. And anyway, I'll be around if you have questions or thoughts or just want to chat. Cool. Thank you.


Hi everybody, we're back at GTC 2024. We're here in San Jose, San Jose Convention Center downtown. They had still rocking at GTC. This is Dave Vellante, John Furrier's here, and Martin Yip is in the house. He looks after EC2 and networking product marketing at AWS. Martin, thanks for coming on theCUBE. It's great to be here. Thank you for having me. Yeah, we got the little pop-up cube. We're running and we're gunning, and it's kind of like, you know, unconferences, this is like the uncube. So we just kind of sort of wing it a little bit. But of course we know AWS. We had you guys on and following you for years and years and years. What do you make of this conference? I mean, we're entering a new era, it feels like. We absolutely are. It's an amazing conference. Love to see all the innovation. Love hearing how Jensen sees the future in his keynote, and just loving talking with everyone, networking. This is the first GTC that's been in person since the pandemic. So it's great to meet and see everyone again. There's a lot of, you know, talk. I would even say confusion. Well, NVIDIA has GPUs, but AWS has GPUs too. And yeah, huh? So you guys have been making silicon for a long, long time. It gives you competitive advantage. You started with Annapurna. Even before you bought Annapurna, we saw Graviton, we saw Tranium, we see Inferentia. So you guys have been at this for a while. What are you doing specifically with NVIDIA? And then let's talk about what you guys are doing in silicon. Yeah, we are deepening our partnership with NVIDIA. We have a great partnership with NVIDIA. We were the first to bring GPUs to the cloud quite a while ago, 13 years, I believe. And we are bringing a lot of their new GPUs to the cloud. One of the big announcements is that we are working with them on something called Project SEBA, which is building a supercomputer for them in the cloud using their GB Blackwell, Grace Blackwell architecture, bringing that platform to AWS for the use of NVIDIA's R&D. So Blackwell was the big star of the Monday keynote. Of course, Jensen was sort of tongue in cheek when he was saying sorry, Grace Hopper. We love you, but this guy is big. So it's quite amazing to me to see the pace of innovation here. I mean, it was just like last week, I felt like everyone was trying to get their hands on the previous generation. You guys were shouted out with some other cloud providers as in line to get the first orders. So how should we think about the progression of, I mean, we know for years we watched Moore's Law and we saw the innovation there. We're at a much steeper innovation curve in terms of performance. It's insane. Of course, energy consumption is a really big consideration. How are you guys thinking about all the dynamics and the complexity of this new era? Well, it's absolutely great what NVIDIA is doing. If you saw some of the performance numbers that Jensen shared, he was showing more than 2x gains in performance over the Hopper architecture, as well as lowering the energy efficiency, the lowering the energy usage of the GPUs while increasing performance. So that's just amazing. I think it's going to unlock a lot of new use cases. We're going to see an explosion in new applications and customers figure out how generative AI is going to fit into customer experiences, whether it's building new models, making it easier for customers to do something like create an image or summarize text or something else that we haven't thought of yet. It's going to be interesting the next couple of years. We wrote a piece several years ago talking about AWS's secret weapon, Nitro and then sort of Graviton and then these other silicon chips that we talked about. I wonder if I could ask you about Nitro.  And we've had a number of deep dives and very impressive. And it really is a competitive weapon in that it allows you to both lower your costs, bring in other silicon suppliers, bring in your own silicon and continuing to drive costs down. Now we enter the AI era, is Nitro ready? Nitro absolutely is ready. And in fact, Project SEBA is going to be in part bringing the best of Nvidia with the best of AWS, including things like Nitro, EFA, key management system. So it is absolutely going to be a part of that Project SEBA super computer that I talked about. In addition to lowering costs, bringing more choices to customers, it also helps with performance quite a bit in terms of our ability to offload virtualization functions onto dedicated hardware, ultimately freeing up resources as well as increasing performance for things like networking and storage. So it's going to be tremendous. Security is one aspect that you didn't talk about. And one of the things that Nitro enables is, we have this capability called Nitro Enclays that allows you to have trusted execution environments. And that's going to be huge in terms of, when we think of GenAI, we are going to think about security as well and making sure that the customer's data is secure. You can obviously have security with encryption at rest and in motion. But then when you're computing on the data, you can't encrypt it. So then therefore you need this trusted execution environment. And that's where Nitro is going to play in all this. And classic virtualization, too many people can get to the data and even AWS can't get to the data in this situation. So that's definitely a huge breakthrough. So what's different? I mean, the cloud obviously changed the way that we thought about IT, took away all that undifferentiated heavy lifting that became a famous phrase. How much of that applies to the AI era, the GPU era? And what's different about the era in terms of the workload, the characteristics of the workload and how are you guys and your customers adjusting? Yeah, no, the benefits of the cloud, the main reason why people move from on-premises to cloud is very much the same reason why customers want to access the GPUs in the cloud as well, which is the scalability, the reliability, the security and the availability, the global availability. The way we build our data centers is very different from other cloud providers. But that leads to the reliability again that I talk about and the availability. No, I think what is different is just, you could see very quickly what the impact of generative AI is going to be, whether it's through building and training your own model to deploying it, to using applications that have generative AI capabilities built in. It's very clear to see what some of the benefits are very quickly. How would you characterize AWS's compute strategy? I would say it's obviously a lot of optionality. There's not a vendor that you really don't work with. Obviously, Nitro and Graviton, et cetera, ARM-based designs, using Intel, using AMD, you've got NVIDIA, et cetera, et cetera. But how would, help people understand your strategy? How would you characterize it? Yeah, it's really about offering the broadest and deepest choice for customers. It's really up to what the customers need for their application workloads. Some workloads are compute intensive, others are memory or storage intensive. And others require an accelerator like a GPU. So it's really about giving customers the choice to get the level of performance and at the cost and security that they need. Do you think AI can help us choose the right processor? Absolutely. I mean, I'm kind of tongue in cheek question, but I'm actually kind of serious. Absolutely, and one of the things that we have built recently is a queue, we have something called queue and we have an instant selection tool that will absolutely help customers pick the right instance for them. We have over 750 instances now, which is a lot. But then based on a customer's workload and the characteristics of the workload, we can absolutely help them pick the right instance that will fit their needs. And then we have compute optimizer, which is kind of, as your workload is running, we will assess and analyze how you're using the infrastructure and make recommendations to you based on what your workload needs are again and help you optimize, again, what the infrastructure you need is. So Amazon's got a very easy to understand, even though it's very rich and capable and detailed AI stack, starting with the infrastructure at the bottom where that's where the silicon is and then I think the middle layer is AI tools. So that's where bedrock would be in there and then the top of the stack is the applications, which would be queue. And as well as Code Whisper and many other AI tools as well. Code Whisper is the coding assistant. Yes. Okay, and so in thinking about where all that goes and thinking about compute and all the different optionality of silicon you have, I'm curious as to what you're seeing from customers in terms of training and inference. I asked Adam Salipski, actually John asked him, I was listening in to the recording, Adam about this, about training versus inference and he said, you know, we got a lot of training to do and that's happening in the cloud but more and more people are talking about inference. Is it different infrastructure, the same infrastructure? Sometimes it depends, how are your customers sort of adjudicating between training and inference and what types of chips there or compute they're choosing? Yeah, I mean the characteristics and the needs of training and inference are quite different. We have instances, our P-series are mostly targeted towards training workloads and then we have our G-series as well as Inferentia which is more for inference. So P-series and training for training, G-series and Inferentia for inference and the characteristics is quite different. With training you need of course powerful processors and powerful compute as well as very fast network. You need large clusters of compute, aggregate compute with inference. It's more about near real-time responses so you need really low latency. You know, it would really not be a good experience if someone asked a question or had a query and it took minutes or even hours to get a response, right, you want it in real-time so you need that low latency there. And just, bedrock is all about LLM optionality and I presume that's, it's horses for courses from the EC2 standpoint, right? It's the right tool for the right job but are you discerning any patterns now that you've, you know, GA'd bedrock? In terms of matching the LLM characteristics with the EC2 instance and the choices that you have there, is there clarity there amongst customers or is there still a lot of experimentation? I think there's still a lot of experimentation. I think Adam always says, you know, there's no model to rule them all so really you have to figure out what you want to do and then try out different models to see which one works best for your needs. So if you're creating an image, it's very different for an image generation model versus a translation model versus something that will write a essay for you. All right, give us the final word here. You know, you guys, a customer obsessed, so give me the customer angle and thinking about the customers that you talk to. You know, what do they need in this AI era and how is AWS helping them? Yeah, you know, I think it's still really, really early in this AI era, especially in AI. Customers are still trying to figure out what they need, honestly and they're coming to us to ask the questions. The biggest thing is try things out, make sure to not just trial the infrastructure but also talk to the experts to help you deploy it. Gen AI is conceptually simple but under the hood it's quite complex and you need to work with the right experts also to help you. But the big thing is that it's still really early on and customers are still trying to figure things out and we're here to help. Yeah, so two thirds are the data from our partners from ETR shows that two thirds of the customers say they want to get ROI inside of 12 months. That's very aggressive. I don't know, I said the last question but do you think 2024 is going to be the year of AI ROI or do you think is maybe it's going to get pushed into latter part of this year or maybe into next year? I think it's going to be very much like the internet boom that we saw in the early 2000s where there'll be a mad rush of companies trying to figure out what this internet's about. Some of them won't make it but for those who do it'll be an amazing thing and regardless the industry is going to change because of it and we're going to see a lot of great applications, a lot of new use cases, a lot of new companies pop up to support it. So you want to experiment, the cloud's a great place to do that Martin. Thanks so much for coming on theCUBE. Appreciate it. Great to have you. All right, keep it right there everybody. This is Dave Vellante for John Furrier. We're live, we're not live actually but we will be on demand. GTC 2024, you're watching theCUBE.


I'm going to embark upon that very dangerous little space in time. General Schwartz has left his previous location. He is en route to this location. The amount of time that I have to talk to fill the space is imprecise. But those of you who know me know that I rarely run out of words. So there's little risk here in terms of filling the space. The risk is in having relevant content. This is consistent with my entire career in the national security business, of course. I want to welcome all of you to the 2012 edition of the CSIS Military Strategy Forum. Those of you in the room, those of you who are joining us on the web. I'm David Burto. I'm the director of the international security program at the Center for Strategic and International Studies and as such, I have purview over the Military Strategy Forum. I do want to extend to you the regrets of our president and CEO, Dr. John Hamry, who is unable to be here today to do the lead-in because of a prior commitment, but he asked me to assure you that he'd love to be here and to introduce this event himself. I would ask those of you in the room to make sure you silence your cell phones and other noise-making devices so as to have no interruption, if you will. And I appreciate that. Those of you who are on the web, I suppose, are free to do whatever you want. CSIS, as many of you know, has been presenting this forum for years now. And we're delighted to be able to thank our sponsor, Rolls-Royce, for their generous support that underwrites this series. They've been with us all along and they've been a tremendous partner in making sure that we can continue to bring these discussions to you every year. Today, though, marks a new page in the way we're approaching the Military Strategy Forum. In addition to the continuing series of senior military leaders that we've had here all along, we also recognize that the Defense Department and the U.S. National Security Forum face a future with fewer resources, but not fewer challenges. And so we at CSIS are expanding our military strategy forum, not only to continue to provide that opportunity for public insight and commentary, but also to address the challenges that are presented by the drawdown of the budget and the associated strategic challenges that come along with that. I think you all are familiar with those. We are only days away from the release of the Defense Department and the entire U.S. government budget. We all know that we've had some previews of that initially with the release of the strategic guidance by the President and the Secretary of Defense and the Chairman back on January 4th, followed up by the announcements from the Secretary and the Chairman two Thursdays ago of some insight into what's in the budget. But it was really just enough to sort of wet your appetite and make you look forward to next Monday or perhaps to the leaks over the weekend to know the specific details, if you will. And as is often the case with the budget, we'll get some details, but there'll be as many questions raised as our answers. So we're going to have to wait for the justification material to roll out, et cetera. So today's speaker, I think, both reflects the long tradition of our speakers in this forum, as well as serving as the initial speaker in our budget and strategy series. I don't think you can expect from him a lot of additional details about what's going to be in the budget next week, but I think what you will be able to get and particularly what your questions may be able to focus on is insight into how they're thinking about it as they put this together. And that's really the critical value, I think, of that opportunity at this point in time. We all know that the Defense Department has made it crystal clear that they've cut as far as they can go. There are no more cuts to be taken, $487 billion over the 10-year period mandated by the Budget Control Act of 2011, but no planning for or numbers for or preparation for the additional cuts that will be mandated effective January 2, 2013 by the sequestration element of the Budget Control Act. I don't think, as you know, if you've been to some of our other events, we don't anticipate any movement on that. I'm sorry, we anticipate a lot of movement, but no resolution on that issue until after the election. So I think it's fair to say that there will be a plenty of debate and discussion about change in sequestration, providing relief to DOD, providing further punishment to DOD, or a host of other options along the way. But those will largely be political illusions as we go through the manifestations of the coming year. Ultimately, we'll reach agreement more than likely after the election when we know who it is that's going to be responsible for dealing with the outcome of that agreement. Nonetheless, we're going to continue to watch that very carefully over time. We will have at CSIS a week from yesterday on the 15th of February a session looking at the industrial base implications of the defense budget as it comes out. I wish I knew what we were going to say. We're going to have to work pretty hard between Monday and Wednesday to lay those out, but make sure that everybody in this room is on the invitation list for that session once we've got it. I'm going to yield the microphone shortly to Dr. Tony Cortesman. He will introduce the general and will moderate this discussion this morning. I believe that General Schwartz has a hard stop, though, at 1 o'clock. So the time we've lost at the front end, we will, unfortunately, not make up at the back end. I'm hoping I'm wrong that they've changed their minds as we've gone through this schedule and the reality that we face. So I appreciate you bearing with that as we go, if you will. Dr. Cortesman, of course, is our Arleigh Burke chair at CSIS. He's been a long friend, ally, opponent, and thorn in the side of not only the Air Force, but the entire Department of Defense and the National Security Establishment. I am reminded every day of when he used to work for the other guys and make my life miserable, but I tend to forget that in the day-to-day operations. We've got two minutes to go. I don't really have two minutes worth.  But there's really no point in my occupying that time and space, if you will. I want to thank you all again for being here. I want to thank Rolls-Royce for their continuing sponsorship. I want to welcome you to our event. Thanks. This is the point where I usually say, OK, now you can finish your dessert, except I don't have one. Ladies and gentlemen, some of you who have been in Washington for a few years may realize that every meeting in the hill does not end promptly on time. But thank you very much. The general's been kind enough to say that he will give us a full hour, which should give us time for him to make his presentation and get to the questions. I do not think he needs a long introduction from me, he has a background in special forces in transport command and dealing with long-range strike. Four years as chief of staff, that's about as many credentials as anyone can have. But I do want to give a brief commercial for the Air Force, if you'll excuse one. That is, it is the first service to put on the web at its website, both an explanation of how its plans fit into the new strategy and how it will be dealing with the budget. I know the general is going to get into a lot of these details in his speech, but I really thought this was a very impressive act on the part of the Air Force, one you should be aware of and one you can easily find on the website. In general, why don't we get to the substance of things. Ladies and gentlemen, again, thank you for waiting. Much appreciated and I apologize for being late. Two congressional delegations had extensive interests in the proposal, the budget dropped next week and so I had to pay due deference. But thank you for waiting here. It is a pleasure to be here and, again, our kids serve, our airmen, soldiers, sailors, Marines, Coast Guardsmen, serve with tremendous distinction and I would argue that we had a landmark year in 2011. It's a year that began here just with the advent of the Arab Awakening. And in a few months, emerged into the occurrences in Tunis and then subsequently in Cairo. And in a few days, actually, President Mubarak stepped down from his nearly 30 year reign. And after observing these events unfold across North Africa and the Levant and then the Arabian Peninsula, we were called upon. The Air Force and the Joint Team called into action on March 19. To help enforce the UN sanctioned no-fly zone over Libya. And we should not forget that the Air Force and the armed forces were already participating in surge operations some 5,500 miles away with a humanitarian relief effort given the concurrent disasters that occurred in Japan at the same time. And performing magnificently, Airmen, among others, were responsible for evacuating 7,500 American citizens and delivering 60% some 5 million pounds of relief supplies, providing the vital intelligence surveillance and reconnaissance of the incident location from our remotely piloted aircraft and ultimately providing much needed measure of comfort for those who suffered so dearly from those disasters. And when you consider all that, the Air Force remained fully engaged in Afghanistan and Iraq in the latter days of the Iraq mission and commenced yet another theater level operation all at the same time. And I might just add that if you recall, the president was also traveling on state visits in South America at the time, which consumes not a trivial amount of aluminum. And so this was our definition of March Madness. Can I? Chris, are you in the room? Can I get a lot of throttelossians, do you mind? I've been fighting a little bit of a cold. Need that, please. So all considered, I think these concurrent operations constitute, as I said, our version of March Madness and I think demonstrate the Air Force's ability to provide full spectrum air power that ranges across intercontinental distances, Japan to Libya to Iraq and Afghanistan, with unmatched speed and certainly a range of capability that spans the operational continuum. And it reaffirms, I think, the Air Force's capacity to surge in a matter of hours to meet the wide-ranging air power requirements, something that our joint teammates and our international partners have come to rely upon. But given the future security environment, that continues to unfold with greater ambiguity and along clearly with the budget pressures that I just discussed at length with two delegations on the Hill, it's appropriate to ask ourselves, what options must we have for meeting these future challenges? And so the department's been engaged for the better part of the last six months in a comprehensive reevaluation of our national interests and the role that the US military will play in achieving these interests. And in the new defense strategic guidance, we recognize the broad contours of the national security interests that will endure, but that many specific geographic and technological and security dimensions of our globalized world will clearly continue to evolve. And so from a geopolitical perspective, the US is rebalancing its resources and our focus toward the Indo and the Asia Pacific, where many economic and diplomatic opportunities have become increasingly vital to our core national interests. The nation clearly will sustain its hard-earned foothold in and around the primary locus of violent extremism in the broader Middle East and in South Asia and will tailor our commitments in Europe, strengthening the transatlantic alliance in the 20th century that clearly prevented the Cold War from becoming hot, and which as we venture now into clearly well into this century will remain, I think, a key partnership for ensuring stability in an uncertain world. So to prepare, we as an Air Force had to make hard choices, both to reshape and to resize our Air Force. And we realized that no matter how tremendous our service men and women are, simple physics will limit the number of places that a smaller Air Force can perform, certainly concurrently. And so with preservation of a highly responsive and effective force is our paramount consideration. In other words, a ready force, not a hollow one. We determined that careful reductions in equipment and personnel were necessary. We essentially traded some size for sustained quality so that although smaller, we will still be an unmatched Air Force with agility, flexibility, and the readiness to engage in rapid succession across the full range of threats and contingencies. In short, this is about maintaining our status, I think, fairly earned, and some of the international partners here can comment on this as the best Air Force on the planet and certainly the one that's the most feared. So the new strategy also calls for rebalancing our surface forces from a land-intensive focus to a broader, more maritime, strategic posture. But we must not forget, ladies and gentlemen, that what covers 100% of the land and the maritime is air and space. And so from a capabilities point of view, the new defense strategy emphasizes air power as fundamental to its major priorities, such as deterring and defeating aggression, projecting power in anti-access and area denial environments, preventing the spread of weapons of mass destruction, conducting space and cyber operations, and maintaining the preponderance of the nation's nuclear deterrent. And toward a broader, cross-domain posture, initiatives such as Air Sea Battle will ensure an enhanced and more strategically oriented partnership between the nation's air and sea services. We will assure America's freedom of navigation and action wherever we have national interests and ensure that the nation has access to the global commons, in particular, the vital lines of communications and transit that are beyond any national jurisdiction. In providing the nation's ability to project global power, we also ensure the nation's global access for diplomatic and development access and our ability to conduct non-military missions, such as humanitarian and disaster relief efforts. Therefore, the Air Force, with its global responsibilities, remains committed to providing the nation with four core capabilities, and they are enduring capabilities. Air and space control, global intelligence surveillance and reconnaissance, global mobility and global strike, plus the unique ability to provide high capacity command and control of air, space, and cyber systems integrating these capabilities across the spectrum of operations. While air power will enable us to repeat scenarios similar to the enforcement of the no-fly zone over Libya, that is Operation Unified Protector, it is not necessarily the template for all future conflicts. But it did facilitate the nation's contribution for seven months in Libya at approximately the operating cost of one week in Afghanistan and with no loss of coalition lives. And all the while, we retained the ability to withdraw quickly as quickly as the coalition was assembled. This operation demonstrated, I think, in a very compelling fashion, air power's versatility, its rapidity, its reversibility, all attributes that the new defense strategic guidance suggests are the future for our armed forces. But during this time of belt tightening, we had to make some very difficult choices in order to remain superb, even as we become leaner and shed some capacity. And considering that our budget-excluding contingency funding has been flat since 2004, with a 12% real decline since 2009, we fully expect that we will be required to make yet additional tough calls. In the broadest sense, we pursued a strategy of balancing risks to include the deliberate acceptance of additional risks in order to align with the new strategies requirements for the joint force being able to conduct one large scale combined arms campaign in one region, while denying the objectives of or imposing unacceptable costs on an opportunistic aggressor in a second region. And in order to achieve maximum savings and an acceptable level of risk, the Air Force has identified in the fiscal year 2013 budget submission, which we'll make next week, formally, 200 fighter mobility ISR aircraft for divestiture in fiscal 2013 toward a total of 286 aircraft through the program period. And as a general principle, we favored multi-role systems over those that are more specialized and emphasize more efficient common configuration of those systems that are retained, such as our modified C5Ms, the upgraded C17s, F22s, F15s, Cs and Es and, of course, F16s. Retiring, how are you? Got something for me here? Retiring entire aircraft types where possible generally has the benefit of enabling elimination of entire support and training infrastructures, thus producing greater efficiency. But if fleet retirement was not viable or economical, then we evaluated options for eliminating the least capable or the highest cost aircraft in terms of cost of operations and sustainment. In total, we estimate that divestiture will likely save a bit under $9 billion over the five-year defense plan. And specifically, this divestiture includes retiring or reclassifying 123 fighters, 102 A10s and 21 F16s, amounting to about a seven-squadron reduction. This leaves 54 combat-coded fighter squadrons that maintain with acceptable risk the requisite level of fighter capability and capacity while providing a bridge, an important bridge, to the delivery of the fifth-generation F-35 fighter. On the mobility side, we plan on divestiture of some 130 aircraft that exceed requirements due to the nation's new driven strategic guidance that reduces the ground forces. This enables retiring 27 C-5As because of their historically lower mission-capable rates relative to 223 C-17s and 52 C-5Ms that will remain. Also divesting the next C-27 force structure of 38 aircraft in favor of the more versatile C-130 platform, but retiring 65 of the oldest C-130s to streamline operations and maintenance activities. So in addition to retaining 275 C-17 and C-5M strategic airlifters, we will continue to operate 318 C-138 and J-Model aircraft. And as a bridge to the initial delivery starting in FY16 of the new KC-46 tanker, our area of fueling feet will be 453 aircraft strong after the planned retirement in 13 of 20 KC-35 aircraft. Finally, we will adjust the numbers in the ISR fleet, beginning with the divestiture of 18 Block 30 RQ-4 Global Hawk aircraft in favor of the more mature and proven U2 sensors and aircraft. This specific initiative, that is, the trade space between U2 and the Global Hawk Block 30, generates savings in the neighborhood of $2.5 billion over the program period. We will also divest 11 C-26s, RC-26s, and retire a single E8 J-STARS aircraft that is damaged beyond economical repair. And in order to ensure maximum versatility and effectiveness of current and future weapon systems, our modernization strategy will focus on those areas that we deem most critical. For example, we have minimized reductions, and in some cases even increased our investments and our top acquisition priorities, such as the KC-46 tanker, our variant of the Joint Strike fighter, the Long Range Strike family of systems, the GPS-3 satellite system, and, of course, remotely piloted aircraft. Overall, that has meant slowing, in some cases, even terminating, lower priority programs like the C-130 avionics modernization program, as well as the defense weather satellite system. The other critical element of force structure, of course, is people. Adjustments from which naturally flow changes driven by the aircraft numbers I just described. Therefore, personnel reductions, however painful, will be necessary. And they are on the order of 10,000 guard, reserve, and active airmen. But the rub here is not only in sheer numbers. It is also in maintaining the right and proper active to reserve component ratio. And there is no doubt, none, absolutely none, that our investment in the reserve components have been and will remain smart investments. Through the creative use of active guard and reserve associations, we have integrated our collective capabilities in all major Air Force mission areas. And the return on investment there has quite simply been invaluable, and surely it will remain so. As a total force, we are now more ready, more sustainable, and more capable of meeting many surge and rotational requirements, such as those that have evolved over the last decade. But through two decades of military end strengths and force structure reductions, the active reserve mix has shifted. In 1990, the reserve component represented 25% of the total force end strength. Today, it is 35%. And in the same period, the reserve component ownership of iron increased from 23% to 28%. In relation to requirements, these numbers represent the reality that the active component has been reduced to the point at which capacity cannot be reduced further without harmful effects to the benefits that I just mentioned, readiness, sustainability, and the ability to surge and rotate at a sustainable tempo. Additionally, there are some functions for which the entire total force relies on the active component to address predominantly. For example, recruiting, training, experiencing, and equipping of the future force. So in order to sustain the total force capabilities and effectiveness, we must restore and maintain an appropriate active and reserve balance that is consistent with current realities and likely future trends. It therefore follows that the Air National Guard and the Air Force Reserve will necessarily be involved in all the analysis and decisions involving and affecting the total force. Together, we made tough calls, admittedly not with complete unanimity, to ensure that the total force remains viable to fulfill the surge and rotational requirements of the new strategic guidance, to ensure that the active component retain the recruiting, the training, and experiential base to sustain the total force, and to ensure that the reserve component remains relevant and engaged in both enduring and evolving missions. I stand by what chief of the Air Force Reserve, Charlie Stenner, and the chief of the Air National Guard, Bud Wyatt, and I wrote last week in the Air Force Times op-ed piece that the active component, garden reserve, have worked in will continue to work closely together charting the total force's future. And even more important than force structure alone, of course, is our readiness. Already, we have assumed risk that we deemed acceptable and manageable with a strategy for a weapon system and facility sustainment and modernization that avoids a hollow force. A $487 billion defense spending reduction over 10 years is something for which the department has spent a extensive amount of time and effort. And it is best reflected in the new strategic guidance and certainly the budget that will be formally released next week. But we have no illusions that the road ahead is going to be easy. And as I discovered this morning on the hill, but I do think it's manageable if we all deal with this in an unemotional fashion. I would add that I think further reductions, for example, as a result of sequestration, in my view, are untenable, that such reductions would clearly send us back to the strategic drawing board, particularly if the cuts are executed across the board and without respect for the defense strategy we just unveiled. And so in the end, as Secretary Panetta has been saying since he took office, any further salami slicing of the budget beyond the $500 billion in cuts for which we've been planning will have severe impacts on our ability to maintain our force structure, our readiness, and ultimately, our combat effectiveness. So in any foreseeable circumstance, the road ahead won't be easy. I harken back to observing the leadership of the Air Force during the Reagan buildup. I was a relative youngster then. And this is a different setting, but we're up to it just as they were. And as an Air Force, we are focused on maintaining an agile, a flexible, and effective force that remains prepared to defend American interests across a range of contingencies. And it is worth noting that although we have withdrawn ground forces from Iraq and continue to work toward a drawdown in similar fashion in Afghanistan, we should expect that the Air Force and probably the Navy presence will remain to a significant degree long after significant US ground forces presence has diminished in the region. In fact, historically, as land forces withdraw from active combat, the relative requirement for air power typically increases. And so when our unmatched joint team fought a brilliant and decisive campaign, an Operation Desert Storm, for example, the land forces returned home weeks and months later. Meanwhile, America's airmen, that is, of any service affiliation, continued flying and operations Northern Watch and Southern Watch over Iraq for a few dozen years afterwards, thereby maintaining America's toehold until ground troops were ordered back into Iraq for Operation Iraqi Freedom in 2003. The decades-long combat operation in the skies over Iraq demonstrates that American air power is certainly not a birthright. Rather, it is the result of dedicated and unrelenting work of America's airmen. And so although we achieved the magnificent milestone on December 17th, the first time in 20 years that the Air Force did not fly a mission over some portion of Iraq, some US airmen will remain in Iraq and Afghanistan in order to assist our partners with their request for air power training and for other operational support. You can expect highest level of energy and commitment and professionalism as we proceed now into this new geostrategic environment. You can expect airmen who will continue to innovate, adapt, and to do whatever is necessary to safeguard the hard fought gains and lessons of the past 10 years in counterinsurgency operations. Even while we remain vigilant for full spectrum threats on the horizon and even amidst the intensifying budget pressures and resource constraints that we face. Ladies and gentlemen, again, I appreciate your time today. I appreciate your patience in waiting for my arrival. And also for CSIS, a magnificent organization in this town for many years for hosting this event. Many thanks for your continuing support of our airmen, of our joint teammates, and very importantly, our families. Thank you very much. Ladies and gentlemen, I'm going to apologize in advance that we can't answer every question. Let me again note that the Air Force has answered at least some of them in the two white papers that it has on its website. But there are a number of questions that people have asked in several different forms that focus on the key issues that the general has just raised. I think one of them is, could you define in, I guess, more precise terms, what changes the focus on the air seat battle will mean for the Air Force? It occurs at three levels. The first level is rather than ad hoc, episodic, or occasional collaboration between the two services that have a global perspective and operate, as I indicated earlier, in the commons and elsewhere, it will, what air seat battle has done and will do is establish and maintain a regular, normalized level of collaboration and interaction between the two services. Not the exception, but the rule. At the operational level, the question is not for the future, but the question is with the inventories we currently possess, how can the two services leverage each other in order to better perform the missions that we are assigned? And there are, you might argue a far-fetched example, but all of you know that essentially there are two stealth platforms in the Department of Defense portfolio. One is the B-2 and the other is the submarine fleet. Now, has anybody ever given any thought to whether these two stealth platforms can reinforce one another's mission effectiveness? Maybe that's fanciful, but I would argue had we not pursued this line of effort with the Navy, such a question would never get any attention at all. And thirdly, at the materiel level, clearly there are opportunities for us in the Navy to do things that are prudent, economic. And perhaps the Global Hawk BAMS collaboration is a very good example. Why base these two assets at different locations? Why should we use two different depots? And perhaps, you know, why should we have two different training pipelines? This is the material aspect of IRC battle that I think also offers substantial promise. I know you've already touched on the happy word sequestration. There is a question here in several of them. Have you examined what would happen if you really were suddenly confronted with either sequestration or major new cuts in terms of specific impacts on the Air Force? Clearly, what we have done with this program proposal is reduce capacity and we took care to avoid to the extent that we could reducing capability. In other words, we have less depth, but not a less versatile force. It is absolutely clear that if we go further, we will not only be required to reduce capacity, but capabilities, we will be able to do fewer things, not just fewer- $30 per megawatt are. Additionally, the reality is that the strategy that we spent several months putting together is the prelude to the program and the logic behind the choices that we made would have to be reopened. And so my shorthand for sequestration is surgery performed by a plumber. Extremely high risk. And something I hope that there is a way ahead to avoid. I wonder if you could provide perhaps a clearer picture of what an emphasis on Asia means in terms of changes in the Air Force, posture in Asia, in Conus, an extent to which moving forces from Europe might affect this. Clearly, the tilt toward the Asia Pacific does a couple of things. It clearly emphasizes what I would refer to as our lily pads in the Asia Pacific region. Those installations that we have access to in host nations of traditional allies and potential other coalition partners that might evolve over time. So the footprint that America has in the Western Pacific most recently reflected by the Marine reallocation, relocation in part to Australia reflects this larger effort that clearly is underway and needs to be sustained for all of the services. The Navy is proposing to home port several assets in Singapore. We clearly are keeping an eye on our major operational platforms as well, Kwam, Kadena, Milan, Japan, South Korea, so on. With respect to systems, Tony I think clearly the vast distances of the Asia Pacific underwrite the logic behind our pursuit of a long-range strike family of systems. And as it very visibly appears in the new strategy and in something I think that clearly is important to the Air Force and the larger joint team as we go forward. There are two aircraft that people have asked a lot of questions about I dimly have heard of. One is the F-22 and the other is the F-35. The F-22 questions range from I guess accident investigation down to the status and deployment of the force. The F-35 questions began at the IOC date and go onwards and I just wonder if you could bring us up to date. 186 F-22s. It is the current fifth generation platform in our fleet. Here too we are looking for versatility while it is the premier air-to-air platform on the planet, the premier air superiority machine. We through an improvement program seek to increase its multi-role capability to do both suppression of enemy air defenses and destruction of enemy air defense missions. We have experienced an issue with the onboard oxygen system on the airplane over the last number of months. An extensive study, an effort, engineering effort and one done by the Air Force Scientific Advisory Board has just concluded and the report will be available shortly. And through that effort we have not identified a specific engineering fault but a number of ways that we can assure that adequate oxygen in a very high performance airplane that operates over a very extensive altitude band protects the operators and maintains their physiological capacity to rock and roll. In terms of the F-22 inquiry by the DOD Inspector General, quite simply this is something that actually is fairly routine. The DOD IG has a mandate to assure that accident investigations comply with service instructions in our case 51503. And that the findings of a particular accident and investigation reflect back on the evidence that was collected. And again, this is not specific to this weapon system they simply chose to pursue this inquiry and naturally we will support that completely. With respect to F-35, F-35 ladies and gentlemen this is a future tactical aviation for the United States Air Force, for the Marine Corps in large measure for the Navy and for at least 10 and perhaps more of our international partners. Yeah, we're committed to the F-35. At the same time it's not at any price. And so we clearly have expectations on the part of the manufacturer to help us in a budget constrained environment buy more airplanes. So if airplanes are cheaper, we buy more. If they're more expensive, we buy less. This is the reality. There ain't no more money. And so the emphasis here is on the government side to perform exquisite program management to be transparent with our industry partners and on the industry side, we would expect to have the production lines normalized to reduce the time it takes from discovery of anomalies and tests until they're incorporated in the active production line to well less than two years and of course to control cost. We all succeed together if we follow that basic formula. You mentioned our withdrawal from Iraq. There is a question too about what is our posture and going to be, what is our level of cooperation with our Gulf allies going to be in the Gulf now that we have withdrawn from Iraq and we face a growing challenge from Iran? We have about 50 airmen that are currently a part of the Office of Security Cooperation in Iraq and certainly the Air Force is prepared to expand that contribution of trainers and to assist our partners in Iraq, the Air Force who we know well. In fact, General Anwar Sun is in pilot training at Waffeland Air Force Base as we speak and will graduate in March. And General Anwar and I are gonna go down and give remarks at the graduation ceremony together. We want them to be an Air Force that can serve Iraqi's national security interests and maintain sovereignty of Iraqi airspace, among other missions. And so clearly we're prepared as soon as the two governments agree to provide training and assistance to the Iraqi Air Force in any form that they choose and as you're aware, they have already come to the table for procurement of F-16 Block 50 class airplanes as a beginning. And Iran? Iran clearly is a, a national security interest for many nations, certainly ours. Their pursuit of nuclear capability is a strategic concern and the potential for misbehavior in the Straits of Hormuz is clearly another strategic concern. And if called upon, I have no doubt that the armed forces of the United States will deal with whatever contingencies might unfold there. We have two sets of questions about new systems. One is the timetable, the nature of the new bomber. Someone from industry might be asking this question, whether you can have time for a competitive fly-off or solar source. I don't know if you'd care to comment on that. Sure, sure. Here's the deal. The independent variable for the long-range strike family assistance in the bomber in particular is cost. And so we are going to make our best effort not to design, over-design an airplane. We're going to deliver a bird that provides the essential capabilities that we need, improve it over time, no doubt. But we are not intent on delivering a capability that is extravagant, that is access to our absolute need. And the idea here is to produce an airplane in the mid-20s that would begin delivering in the mid-20s that would connect both with intelligence surveillance and reconnaissance capabilities, electronic warfare capabilities, other sensors off-board capabilities that will allow us to produce a penetrating platform that is sufficiently affordable that we can buy in numbers. We are not going to do the B2 again. That is not in the cards. There are similar questions which really come down to the future of unmanned aerial vehicles. And I noticed that in your white paper, the increase in the number of UAVs is roughly equivalent to the number of combat aircraft you've cut during the same period. But this is a pretty open-ended set of questions. Remotely piloted aircraft have a, in my view, a permanent place in our Air Force. And we have postured ourselves accordingly, both in terms of the force structure itself, establishing a career path for remotely piloted aircraft operators and support personnel and so on. But at least in the nearer term, certainly the next 10 years, 20 perhaps, maybe more, we should not take the example of Iraq and Afghanistan which were essentially benign airspace to suggest that remotely piloted aircraft are any time, any place machines. They are not. Contested airspace is a different place for remotely piloted aircraft. And so it's my conviction that while the balance clearly is shifting toward remotely piloted aircraft, that there will continue to be at least my estimate is for the next 30 years, a place for manned tactical aircraft. And I would ask you candidly, would you be comfortable with a nuclear-laden remotely piloted aircraft? I wouldn't be. And this is just but one example of why we need not to be too doctrinaire or too ideological about this. We need to look at the missions, we need to understand the environments, we need to respect the technology that exists today and how it may advance and they make the best of all that. There's another set of broad questions which relate to what are the strategies and priorities in this kind of budget environment in two areas. One is information technology and the other is space systems. MIT and cyber specifically clearly is a growth area and collectively we are struggling to get our arms around this. The nascent US Cyber Command, Keith Alexander's organization, our 24th Air Force, counterpart organizations and the other services are deliberately working ahead. There are a multitude of policy and jurisdictional and statutory issues here as well but from an Air Force point of view we are increasingly dependent on our networks to execute the missions that were assigned, the way they need to be executed and so defending those networks is a profoundly important mission for our cyber operators. There are places for more offensive use of cyber and in those areas where it would support traditional Air Force missions we are pursuing that kind of capability as well. What am I talking about? Clearly you could try to mitigate the risk associated let's say with an SA-20 kinetically or perhaps you could do it non-kinetically with electrons. In any case that's the focus of our Air Force in the IT area or the cyber area. With respect to space, we're actually in pretty good shape there. We are embarking on the recapitalization of our major space constellations. We finally got the first advanced extremely high frequency satellite into its geosync orbit which by the way was a real act of rocket science. As you may know, the kick motor failed and over nine months we maneuvered this thing with thrusters that offer about a half a pound of thrust about the same as blowing on the piece of paper. In over nine months our space operators got that $2 billion satellite into the operating orbit, not a trivial achievement. Sivers, I know it's got history but there's one on board, another to be launched in the not too distant future. We just launched the fourth wideband global system to succeed the discus constellation of satellites. The bottom line and GPS three is moving as we speak. And so the nature of space, we are not predominant there. It's becoming both congested and contested but we will continue to support the various missions we have there with a very robust space capability. One of the key elements of the strategy is to improve our cooperation with regional and coalition partners in terms of interoperability, planning and the rest. No wonder if you could highlight the Air Force priorities. It goes from the tactical to the strategic but in the broadest sense, we're not talking about mature partners here. Although certainly we have long-term relationships underwritten by foreign military sales, activities, training, professional military education, liaison, many levels and so on. But with respect to nascent air forces in particular, we have a capability not just to talk about platforms and instruction and operating specific machines but more broadly what is it that's required to operate an air force, to operate an air field, pavement expertise and air traffic control and security of the air drone and air field operations. These are skills that one can transfer that a nascent air force requires. And that is one area that we are not reducing. We have that capacity and we'll continue to maintain that broader constellation of liaison with skills that, and maintenance logistics and so on, that will produce more competent air forces as partners. The one thing we were unable to do in this program was to sustain dedicated force structures associated with building partner capacity. Specifically the light lift and the light strike initiatives. There simply wasn't enough head space to pursue those as dedicated air force for structure. I have a unsigned offer to reduce your budget problems by paying $150 for an A-10. There are some other A-10 questions as well. Sure, here's the deal. If you accept the philosophy that in a smaller force versatility becomes all the more important, that those assets that are more multi-role are, have greater value. And so the reduction in A-10s, and by the way, there's still 236 A-10s remaining in the force structure. It's not like they're going away. That we opted to move toward a more multi-role mix emphasizing F-16s. So the A-10 is a noble airplane, but one should not accept the hype that the air force is walking away from close air support because we're reducing the A-10 force structure. Nothing could be further from the truth. And believe me, our youngsters on the ground in Afghanistan know that a weapon delivered from a B-1 is as meaningful as a 30 millimeter delivered from an A-10, properly done. Well, I know that you have been very generous with your time and that you're pressed by it. I haven't seen any gestures from your staff that they are looking mildly disapprovingly. We're doing fine. Question really, today DOD is recommending to Congress that women may be able to broaden their roles in combat. And what is the impact this is going to have on the air force? Modest because we already, 99% of our skills are open to females as it stands today. Those that are limited right now are primarily the battlefield airmen skills that are involved in close combat kind of situations. But we will be opening, for example, our air liaison officer opportunities at the battalion level to females. That is an example of one area that the new department policy will permit. The bottom line, though, for our air force, and I think this is true certainly for the other services, is you gotta go where the talent is. And 20% of our air force is female, and that's where the talent is. In another completely different area, do you see the gradual emergence of something approaching a peer threat from the Chinese air force? They're striving to arrive at a broad-based, multiple mission capability in airspace and cyberspace. So the short answer is it's clear to me that the Chinese are in pursuit of that objective. And it suggests to me that we need to be cognizant of that in response to that reality with respect to our own capabilities. You know, we have a host of highly technical questions, but, General, I think you've been extraordinarily generous with your time. Let me again note that a lot of what the general has said is supplemented in these two air force documents, airforce.mil, and may I ask all of you to suggest or rather thank the general in the usual manner.