Now medical aid from the United Arab Emirates and Thailand arrived in India in first day to help the country fight a creepling wave of coronavirus infections. A plane carried ventilators and oxygen musks arrived from the UAE in New Delhi while the Indian Air Force airlifted three oxygen cylinders donated by the Thai government from Bangkok. The world's second most populous nation is in deep crisis with hospitals and morgues overwhelmed. India reported 379,257 new infections and 3,645 new deaths yesterday.


Hello, everyone, and welcome to this week's Product School Webinar. Thanks for joining us today. Just in case you didn't know, Product School teaches product management, coding, data analytics, digital marketing, UX design, and product leadership courses online and our 16 campuses worldwide. On top of that, every week we offer some amazing local product management events and host online webinars, live streams, and ask me anything sessions. Head over to productschool.com after this webinar to check them out. Hello, and welcome to How to Measure the Right Things. This is Drew Dillon, and I've prepared this document for this presentation with. Hello, and welcome to How to Measure the Right Things. This is Drew Dillon, and I've prepared this presentation for Product School. Today we're going to be talking about web applications, mobile applications, how to make sure that you're collecting all the right data that you need to become a data-driven organization. This is me, Drew Dillon. I'm a freelance product leader currently. In previous roles, I've been Chief Product Officer at a company called Schedulo that does mobile workforce management, VP of product at a company called Anyperk, which is a YC company that does employee perks, discounts, and rewards, and then prior to that, Director of Product at Yammer, which is where I learned a lot of what you're going to be hearing about today. First, we're going to talk a little bit about the character of data, what data can do for us, what it can't do, so that we can understand a little bit more about what we're working with before we decide to figure out how we're going to collect it. Now, when I talk about data, what I'm really talking about is this projection that we see when companies go from a Series A startup from three individuals working out of a garage, all the way to become these data juggernauts out in the public markets, collecting and using this data to inform product decisions, build better AI to better adapt the product towards users, and really just understand it more and more about their users. What I'm not talking about is big data. When you hear the concept of big data, what people typically are talking about is data that's been collected over a number of years, put into a database. It's not really well understood what that data can do and how it's been collected. That's one of the biggest things you need to think about when you're looking at data. When somebody hands you a database full of tracking data, the way to think about that is you can't really understand the data because a lot of opinions go into gathering data. Do I measure this or measure that? So data that you get is basically a set of opinions that you can't really trust and you can't be sure that the opinions are going to be ones that you would have made on your own. So when we're talking about data, what we really want to talk about is actionable data. Most likely when you're talking about web and mobile applications, this is data that you've collected in the context of the user using your application and any metadata around that. When they logged in, what browser they were using, what mobile phone they were using, not any of the actions that actually took place on your site to drive further usage and engagement. What we won't be talking about is what's known as shadow profiles. So shadow profiles in the social media context are typically data sources outside of your application that can be used to gather data about your users. We're not going to be going too deep into that today, but I think there's a lot of ethical soul searching that the product community should be thinking about when we talk about shadow profiles and these large scale data brokers. Now, when you look at data, the way to think about it best is actually as a history book. Data can't tell you exactly what's going to happen in the future, but it can tell you what happened in the past. By collecting the data about your users, you're storing that information of what happened in the hopes that that fits some kind of pattern that you can use to apply to the future and drive better and more useful engagements with your users. We were thinking about that history book. Here's Archduke, Franz Ferdinand. You can think about basically, you can look at the history of World War I and say that the murder of Archduke Franz Ferdinand actually led and caused to the outcome of World War I. And with a certain amount of tunnel vision, you could look back in the history books and say, okay, killing a world leader might actually lead to World Wars. Now, the challenge with both of those statements is they're both backward looking. If we were able to go back and reverse the assassination of Archduke Franz Ferdinand, we might know for a fact that that would have prevented World War I or not, except we would have had to run that experiment both times. In order for science to work, it has to be run kind of in a clean environment. You can look back in the past and say, okay, assassination very often leads to a world war with a certain amount of tunnel vision that might be true, but you can't really know for sure looking into the future whether that will happen again because you again didn't have a clean environment.  A lot of other things going on obviously in the context of World War I. Now, what you've looked back and seen is a correlation, and that correlation needs to be tested in the future to make sure that it actually was causal, that those two things were really related in the way that you thought they were. Here's a funny website. It's called Spurious Correlations. You can see basically graphs and things that match up reasonably well. In this case, it's divorce rates in Maine against the per capita consumption of margarine. So if you really want to help couples in Maine, this graph would tell you that you should cut back on the margarine. But similarly to assassination and World Wars, they might look very close together, but you can't really know for sure that the two are really related unless you are able to test in an isolated environment. So with that being said, what can data tell us? Well, data can tell you what areas of the product are getting the most use and how. People vote with their clicks if they are looking at a certain page more than other parts of the application, chances are that's a useful and valuable page. They can tell you how you're doing against key performance indicators and that's what we're going to be talking about for the majority of this presentation, these KPI that drive your business forward. They can tell you the outcome of an experiment. If you can actually really isolate two environments and test them against each other in a scientific way, your data can actually tell you whether one was good or bad against the KPI that you care about. This is one kind of magical.  Ordinarily, I would ask this as a question, but I'll give it to you right here. Something that happens a lot in applications you'll find is people complain about search and I'm sure you're now recounting every app that you use where search isn't very good. The fascinating thing about data is it can actually tell you whether or not search is any good. The way it can do that is basically whether searching leads to higher engagement on those other KPI that you care about. So if you spend a week and a half improving your search, you ship out a search experiment and it seems like everybody's searching more but people are coming back to use the app less, then search probably isn't very good. Now if you improve search and improving search thus leads people to coming back to the app more, well then your search is pretty good. So it's whether or not a feature leads to more repeat usage, not necessarily just of that feature, but again going back to those KPI. Of the things that you care about does using that thing lead to more of that? And if so, then it's probably a good thing and it's worth more investigation. And there's a lot of things data can't tell us related to what we just talked about. Can't tell you what to do next. Data is not necessarily 100% predictive. It's not inventive. It doesn't your databases, you know, pop up and start going in research and competitors and all the different products in the space and figuring out what they need to do in order to be useful. So that is still product manager intuition. They can't 100% prove calls or relationships between past events as we just discussed. And they can't tell you when to make larger bets. Data can't hint at this. That's kind of a subtle science. But it can't tell you that the thing that you're working on is a waste of time. And it can't necessarily tell you whether you've hit relative maximum, whether you really worked this area to the point where it's really completely optimized and you need to rethink the entire thing. You just have to take that on gut. You can see hints of this in the data, but you can't necessarily know for a fact that it's time to go build a brand new app or make a totally new bet on what your product needs to be doing. So given that, what should we be measuring with that knowledge? How should we be thinking about what we're going to measure and the kinds of KPI that we're going to use to drive the business forward? First off is growth. I think that's something you'll hear from pretty much every startup at this point is everybody has some kind of growth team or some kind of growth initiative where they're trying to do this, boost users and customers. When you have those users and customers, you need to know, of course, how many are coming in, but also where do they come from? So you can continue to expand those channels. Everybody cares about this from every B2B app to every consumer app out there. Needs more eyeballs in order to be successful. Next, after that, once you have those people is actually retention. Are they coming back in 48 hours? Are they coming back in a week and they're coming back in a month? Different apps are going to care about different retention intervals. So if you're going to buy a luxury car, they don't necessarily care that you come back to the luxury automobile shop every week. But they do want to know that when you do come back, that you're going to come back to your Mercedes or to your Lamborghini dealer.  And then finally, invites. The way that you measure invites is a number called a K factor. K is basically the viral coefficient. So every user in your app creates X number or K number of additional users. So if one user comes in, they invite two of their friends and both of those people sign up, then that person has a K factor of two or that invite mechanism has a K factor of two. So every one user turns into two converted users. That's an extremely viral app. What you really need is just basically K factor could be greater than one to know that your app is viral. That invites are actually creating more users that are converting into more users. So your app will continue to spread and grow virally. So that's growth. Everybody cares about growth. After that, I tend to break apps down into two simple categories. One is a transactional app. This is an app that has a specific goal that it's marching you towards. And then an engaging app. An engaging app wants you around. They want you to come back. They want to use the app more. They want you to take more actions within the app. And we'll see why in a second. So transactional apps are really about conversion. They're really about looking at this funnel. So let's assume two million users hit this site. And after a couple of steps, 250,000 make it to the purchase step and actually purchase an item. The total conversion rate across this funnel is about 12.5%. So transactional app is really looking at the gaps. So whether it's this one million opportunity between step one and step two, or even out and beyond to the left here, the entire internet or the entire market and capability of your site. So that's your whole TAN, what you're pitching your VCs on, all the way down to the purchase. And when you get to this purchase, potentially understanding loopbacks. So coming back from this purchase to coming back to the site or coming back from this purchase and making you, again, viral where you're pushing out referral codes and things like that to your friends. So transactional apps are really looking about funnel optimization. How do we get as many people from point A all the way down to that actual purchase at the very end? Engaging apps are really intended to keep you there. So engaging apps care about the sign in, page views. Time on site isn't a great metric, but what they really care about is the user coming back and the user coming back has inherent value. So once you understand the number of times people coming back, that retention or engagement curve, then it comes down to the specific interactions. Which interactions drive more engagement and which interactions actually will go out and engage more other users. So what are the two most important interactions on Instagram? Things to think about here are posting, the original person who posts. You could think about most social apps, follow a trend that's called the 99 one rule, which says that 1% of people are going to create the majority of the content, about 9% are going to respond and interact with the people who create that content and 90% are gonna be pure lurkers. And an engaging app, that's okay because your pure lurkers are adding value. They're adding value and signal back to the 1% and the 9% who are actually creating and generating content and getting more eyeballs back into the app. As we're at a transactional app, that 90% would be basically useless for you to drive the actual bottom line of the final purchase. Successful apps ultimately have to prioritize one. You have to prioritize either the funnel or the engagement as your core loop. That's not to say that transactional apps don't have loops in them. They do want you to come back. They do want you to refer other people. Similarly, engaging apps have tons of funnels. Just because you're coming back and using a social game every day, they still want you to go purchase coins and purchase the currency and spend more money on the app. But they believe ultimately they have to believe that you coming back more and more is gonna continue generating more and more revenue. So the big trade-off, the way to think about the difference between the two is actually in the case of let's say Amazon. So Amazon has you coming back.  Now, Amazon might make the trade saying that you are never gonna purchase as many socks or the profit margin against those socks is never gonna be as good as the one that we're getting against that big screen TV. So Amazon might detect that I'm this user and might go down one of these two paths and actually force me into the TV path, whereas an engaging app is going to keep me around. It's gonna want me to continue taking more actions. And that the reason ultimately is that you can determine effectively the lifetime value of a user. And you can say which is going to lead to the higher lifetime value for a transactional app. In a transactional app, they have that very well-quantified and know exactly how much this user is gonna be to me. And if I can get them more efficiently to that lifetime value, it might not be worth having them come back every day. That actually might not be a valuable transaction for me. So it's really about how you think about and prioritize between the funnels and the loop itself. Where, and to think about the counter example here, clash of clans, if I'm coming back every day, I keep playing the game, chances are they're going to get me with the coins or one of the other psychological tricks that they have. So they might run an experiment where it puts a big coin purchasing thing right up in the front of the app. But if that actually causes me to go away and not come to the app as much, even if I spend less money over the phone, so it's just really that trade-off between metrics. It's pretty rare that you actually find a trade-off between your own metrics. Once you set your core KPI and once you decide whether you're transactional or engaging, but that's how they would prioritize. That's what the trade-offs would look like from one to the other. So we'll go through a handful of apps here.  So what are these apps? Would you consider them transactional or engaging? Obviously Amazon just gave that one away. So hopefully you'll get them pretty quick. That one, next one, that one can be kind of key. Last one, sports betting. That's a really a lot. Australia is a big hotbed for sports betting. So Amazon you probably guessed is a transactional app. They do care about referrals. They do care about me coming back but only in as much as it adds to that customer lifetime with that lifetime value. A subscription fee and engagement probably shows that I care and will continue to renew. And then sports betting, sports betting I would put some more in the transactional. They probably do want me to come back and bet more but addiction probably handles some of that. The interesting kind of corollary to sports betting would actually be more like games of chance like online poker, which I would say actually might be a little bit more engaging than transactional. What we've been driving at here is understanding between core metrics, the activities I believe to lead to long-term success and those are your growth metrics. Those are your kind of core funnel and engagement metrics versus feature metrics. Feature metrics are metrics around a given capability of the site or mobile app. So search, number of times that people search. That is probably not a metric that ultimately adds to the bottom line. It can indicate that other people might be doing more stuff that might lead to more value but if you see that let's say you run an experiment and fewer people come back after searching as we discussed before but they're searching more, you can look at it and say, well, my search, number of searches have gone up. That's a feature metric but my overall engagement has gone down. That's a core metric. Feature metrics always lose the core metrics and that's what you have to continually think about as you're building new capabilities. What not to measure? Clicks, clicks aren't always positive. Sometimes clicking around can signal user distress or confusion. There are a lot of A-B testing solutions and general analytics type solutions that will actually give you clicks as a top level metric. So just a note of caution there. Page views, I think anytime you go to one of those sites that has 15 different slides and you have to keep clicking every single time to get to each new page, that's what they're trying to do. They're trying to drive impressions of ads on their pages. So when you think about that awful experience of clicking through 15 different pages full of ads, that's really the kind of, that's what page views can lead you to do. Time on site, it's not terrible data but I think similarly to clicks it's one that people prioritize a lot within analytics services because it's easy to measure and to show kind of immediate value. It's not objectively positive similarly. It's also positive that, or it's also possible that time on site will actually be because the person walked away from their computer and left it sitting idle or something like that. The last thing to think about as you're stepping into this data-driven world, chances are your data is wrong. As we mentioned at the outset you have to form an opinion about data and that opinion itself can be wrong. The level of precision, you have JavaScript plugins and things like that collecting data. Meanwhile, those are sitting in somebody's internet explorer browser where they can counter bugs and miscounts and things like that. This is one of those things that people use to argue against using data entirely is saying that the data is incorrect or imprecise. That's irrelevant and it's also true that the data is always gonna be imprecise in some way or some form. The most important thing then to look for is actually trend data. So it's not critical that you understand exactly what this value of P is here but more that you understand where that curve is going. Because the trend will tell you whether your app is growing or your app is shrinking, whether you're adding more users significantly or you have a bunch of churn on the back end. So do your best to get as precise data as possible but when you have it, don't stress over a thousand users here or there. Really look at which way your data is trending and use that to form most of your decisions. So for further reading on this, this did come from a blog post that actually came from a Quora question. You could find that full blog post here at the bit.ly link, bit.ly forward slash DD product, data-driven product. Feel free to connect with me on LinkedIn, ask me follow-up questions on Twitter, medium. I also do Quora, that's where I, you know, the original post came from and where I do answer semi-regularly. So please stay in touch and thank you very much.


Good afternoon, everybody. Welcome to the Ford School. I'm Michael Barr. I'm the Joan and Sanford Wildein of the Gerald R. Ford School of Public Policy here at the University of Michigan. I'm delighted to see all of you here this afternoon for this very special Veterans Day policy talks at the Ford School, a conversation on national security, service, and policy with our distinguished guests, the Honorable James Clapper, Lieutenant General Michael Nagata, a U.S. Representative, Alyssa Slotkin, who will join us when she emerges from the snow, and moderating the conversation as our own Towsley Foundation policymaker-in-residence, Javed Ali. On Veterans Day, we reflect with gratitude on the sacrifices of those who have served the United States in uniform. It is also an opportunity for civilians to consider the responsibilities we have to strengthen our communities and institutions through public service. Our school's namesake, President Gerald R. Ford, was a Navy veteran serving as an instructor at the Navy Preflate School in North Carolina and later in combat on the USS Monterey in the Pacific Theater during World War II. His legacy of service continues to guide our work here at the Ford School today. Our speakers today certainly embody this commitment to service, each having spent their careers, both military and civilian, protecting and defending the national security of the United States. And those careers have been distinguished. The collective experience on this stage is weighty. To give you a few highlights, Lieutenant General Nagata, during his 38 years of active duty in the U.S. Army, has served as commander of U.S. Special Operations Command Central and was heavily involved in the first two years of combat operations against the Islamic State in Iraq, Syria, and elsewhere. Until 2013, he had served as deputy director for special operations and counterterrorism within the Joint Chiefs. James Clapper has served over 50 years in roles in the U.S. intelligence community, culminating as director of national intelligence during the Obama administration. He also served 34 years in the military. Among his many roles, he was chief of Air Force Intelligence during Operation Desert Storm and served two combat tours in Southeast Asia. His last active duty assignment was as director of the Defense Intelligence Agency. And after retiring as Lieutenant General in 1995, he had served as director of the National Geospatial Intelligence Agency. Although Representative Slotkin is not yet here, in the interest of time, I'm going to introduce her as well. Representative Slotkin was elected to represent Michigan's 8th Congressional District, where she has previously worked at the CIA as a Middle East analyst and served three tours in Iraq alongside the U.S. military. She also served in senior positions in the Department of Defense, including as the acting assistant secretary of defense for international security affairs. This is just a very brief highlight of what our speakers have accomplished. We're thrilled to have such a distinguished group here on Veterans Day to discuss the importance of public service and current issues in national security and policy. I also want to deeply recognize and thank the Towsley Foundation and the family members represented here. Established in 2002, the Towsley Foundation Policymaker and Residence Program allows the Ford School to bring individuals with significant national and international policymaking experience to our campus. Javid Ali, our moderator today, is one such distinguished policymaker. Having served with distinction at the National Security Council, the FBI, and other agencies working on counterterrorism policy. Javid is responsible for putting this event together today. So please join me in thanking the Towsley Foundation for allowing us to bring policy practitioners to the Ford School and enriching the conversations that we have here. Special thanks go out to Lynn and Stuart White, who are sitting right here, members of the Towsley family, and to Del Dunbar, who's the Towsley Foundation President for being here today. Finally, a word on format. We're going to have some time for questions. Our normal format is to do those all at the end. Given that Representative Slotkin is joining us in the middle, we're probably going to go a little bit to the audience and then a little bit back to the panel. John and our wonderful students are going to help us ask those questions. So if you have questions as the dialogue is going on, there'll be a staff member coming around with cards. Please write your questions on the cards. Edward Elliott and Norana Nadkumar, apologies for my pronunciation, are going to sift through your questions and present them to the panel for answering. For those of you watching online, particularly in this snowy day, that may be a lot of you, please tweet your questions using the hashtag policy talks, and they'll be integrated into our conversation as well. Again, welcome and please join me in thanking the Towsley Foundation and our wonderful panelists. Dean Barr, thank you for that kind introduction. And again, for those of you who don't know me, my name is Jabid Ali. I'm completely honored and humbled to be here, not only for this day today, but for the second go round in the Towsley position. So Dean Barr, thanks to you and Paul Lance for bringing me back again. And thanks to the Towsley Foundation and the family as well for having me back again. It truly is an honor for me to be here. Certainly also want to thank all of you for showing up. And as Dean Barr said, this very cold and kind of gloomy day in November, I'm not really sure what happened with Mother Nature and General Nagata and I were joking as we were planning this event that wouldn't be funny if we had done this in February or January to see kind of how Mother Nature would have sort of had an impact. And here we are November 11th, and we have a lot of six inch blizzard on our hands, right? So thanks to you for showing up. But also as Dean Barr said, this is also an event designed to honor the service and pay tribute to the veterans, not only on the panel, but in the room and at the university, I think if the numbers are correct, there are 390 veterans on the campus right now, either former or active duty. So thanks to all of you for your service to the to the country. So with that, why don't we get started with the panel and what we have our series of questions for Generals Clapper and Nagata and Representative Slotkin. Once she gets here, we'll probably break for a little bit for audience questions to let Representative Slotkin get settled in and then we'll turn back to audience questions too. But let me first start with General Clapper and General Nagata first. As the title for today is a discussion on national security policy, national security policy and service, but I want to start with the service aspect first because I think it is important to highlight that as we're here on Veterans Day. So just for for each of you a quick few minutes of observations about your own careers and how what service has meant to you throughout all your distinguished years for the United States. Well, thanks to you other than thanks for your recent Michigan and Ford School for hosting this and particularly on this day. And I think speaking for Mike, we generally appreciate it. And you're being here. So for me, it was a fairly simple thing because is this anybody here? You're in the back? Okay. My dad was army officer. For 28 years he served in World War Two during Korea and and in Vietnam. So for me, it was kind of a natural, almost a natural thing to kind of fall in his footsteps. And he instilled in me a sense of duty and service to the country. So I really didn't seriously consider when I became old enough to make decisions like that, doing anything else other than what what I did. So it's really more a testament to my dad and the example that he set for me. Well, I will also start by thanking the Ford School for giving myself and General Clapper the opportunity to be here and Happy Veterans Day to all of you. The and most importantly, thanks for giving me an excuse to escape from Washington DC for a couple of days. It may be snowing here, but but I'd rather deal with the snow here than go back. But my wife's waiting for me. So I have to go back more seriously. Service as I look back on my own career, I've come to the realization that what I how I defined that word service when I first joined the military in 1981 is not not the same thing as I've is not the same way in which I view that word today. I think if I'm able to remember what I thought about that word many, many years ago, it probably was a one dimensional characterization that isn't untrue. It's just wholly inadequate. My view I think at the time was it was simply subordinating myself to other people. I would be a servant of something hopefully better and larger than myself. That's not untrue. But my view of that word now is is is much larger. It still contains that notion of subordinating myself to others, others authorities, others direction that still is a part of government service if you're going to have a successful career. But it's also subordinating what I may wish for personally, or what my family and loved ones may need. For a purpose that is more important than me or even my family. Now that's a that's a really hard pill to swallow.  But I've had to condition myself. And many other military colleagues have had to do the same that, you know, that particularly those who have been on the endless deployment train since 9 11. I haven't seen very much of my family. I've not been there for them. There there are all kinds of problems, opportunities, even a few tragedies that I haven't been nowhere near my family, and they could not depend on me they could not rely on me. But I kept doing it. Why? Because somewhere deep in my heart, I believe there was something that was more important. So that is what service has meant to me. Great. Thank you. The next question is sort of looking at not only from the perspective of your own distinguished careers. And I also forgot to mention that I worked for General Clapper when he was the director of National Intelligence, and I worked with and for General Nagata, when I was at the National Counterterrorism Center, and then at the White House. And I couldn't be more honored again to share the stage with people I had that personal relationship with as well. But I'm getting a little bit more of a substantive line of questioning. Based on your own careers and now how that you've been out and what you see going on in the world, what are your top two or three national security priorities for the United States? And what do you think we should be doing to tackle them? Well, at risk of generating controversy here, I think transcend it, national security issue, in fact, international security issue is climate change.  A little known factoid is of great concern to me is the availability of arable soil on the globe to support the number of people we're projected to have. And arable soil is disappearing at a faster rate than regenerating just one factoid. So this to me transcends all the other things we worry about Russia, China, terrorism, whatever, because we don't get our act together and do something about the climate on the whole planets in jeopardy, which I think transcends all the other issues. Just to continue the controversy here. The second issue I'm concerned about is more of a spiritual one than scientific or physical one. And that is what ran corporation who's published a book recently cleverly an aptly called truth decay. And that is the degradation of reliance on empirical fact data, objective analysis, and those institutions that support truth, like academics, like science, like law enforcement, like intelligence, all of which, and on and journalism, all of which to one degree or another depend on on truth and its practice. So those to me are to one physical, scientific and the other more spiritual, and more nebulous climate change and and the assault on truth. Thank you, General Clapper. Generally got it. Well, I agree with everything. General Clapper just said, I'll add two things. One is general and one is specific. The general observation is something that I probably can't empirically prove. But I have so marinated in what I'm about to describe in my various travels around the world, particularly in the last 20 years. But I would also argue that one about describe applies to the United States as much as it applies to any place I've been internationally. And that is the what I have come to believe is a ubiquitous universal global erosion of confidence between populations everywhere, and the governments that purport to govern them. It is eroding everywhere. It is, I would argue, it's eroding inside the United States, not as bad in the United States as it is some other places I've been. But the this is having staggering consequences. As people everywhere come to have less and less faith in their government, they start turning to alternatives, which leads me to my specific concern. I spent almost all my career in the world of counterterrorism. So I'm a little biased on this subject. So just bear with me. I think it's this part is empirically provable. There are more terrorists today around the world than there were on 9 11. Why? There are in my humble opinion, there are many explanations for this that converge to make this reality true. This terrible reality true. But I think the single most important one is the general observation I just shared with you. As people lose faith in the people that claim to lead them claim to government govern them claim to care for them. One obvious alternative are people who advocate the use of violence to overthrow the political regime that is delivering so much disappointment. And that explains why the arc of terrorism, unfortunately around the world continues to go up. Generally, gotta let me just pick up on that point.  So is there anything you can say about from your time in the special operations community?  Yes, that's a great question. I believe that we are going to relive what we once experienced after the death of Osama bin Laden. But let me stipulate first. The the the operations that led to both bin Laden's demise, as well as Baghdadi's demise, were two things that I don't think get enough credit in some of the public commentary I've seen about this. First of all, much is made. And there's a part of me that really likes this of the amazing special operators that went on either of those targets, and they deserve enormous credit. But if anybody thinks they did this by themselves, you've seen too many movies. There were pilots who had to take them to the target. There were intelligence collectors that labored for over a decade to find bin Laden's whereabouts in the case of Baghdadi, who labored for five years to find this person's whereabouts, the analysts that had to pour over unbelievable amounts of imagery and intelligence reports and tried to separate the wheat from the chaff law enforcement officers, diplomats, etc. This was an extraordinary example of the entire national security establishments of the US government working as a cohesive team for a protracted period of time. So whenever I hear as too often I hear in Washington DC that the whole government's falling apart, well, the government has problems, no doubt. But this was a magnificent accomplishment that generated enormous tactical and operational success. Now getting back to what I said a moment ago about how its arc will be similar. In the immediate aftermath of the death of bin Laden, enormous damage was done to Al Qaeda. They suffered enormous reputational damage. They had to go through a lot of leadership turmoil before they finally settled on Zawa Heery.  But that was the high watermark of strategic value. The day, the night he died was the high watermark of strategic value. That value has gradually diminished with every subsequent year.  My answer now is unfortunately the value is negligible. The value was perishable. In my opinion, we are going to go through exactly the same thing with Baghdadi. It is having significant effects on ISIS, but the effects will diminish with time. ISIS will recover. And we need to remember that the leaders of ISIS in large measure were formerly members of Al Qaeda in Iraq. These are battle hardened, experienced, savvy operators, no matter how much I may despise them. And they've been long assuming that Baghdadi would eventually be captured or killed. So it was not surprising to me that they found a replacement for him very quickly. And finally, as if this weren't depressing enough, it's important to remember that ISIS has a global network. They're not just in Iraq and Syria. They have a global network. And ISIS has only existed for five years. Al Qaeda has existed for more than two decades. Here's the sobering truth. ISIS global network today is substantially larger after only five years of growth than the Al Qaeda network ever became. And they worked on it for two decades. That should be a sobering realization for anybody who thinks that we're well on our way to strategic victory against terrorism. General Clapper, let me draw you into the counterterrorism conversation too. And when I was working for General Clapper, we used to have some pretty tough conversations about how counterterrorism needs to be a little bit less of a priority for the intelligence community. So we could focus on some of these other topics. But now, General Clapper, that you know, you've seen this kind of shift a little bit away from counterterrorism to some of these other national security priorities. How do you think that will look in the future going forward? Well, when I left the government, the biggest single claimant target claimant in terms of money that we spent for any target at all that the intelligence community confronts was counterterrorism. So the new administration and I think rightfully so has placed great emphasis on our two peer competitors, Russia and notably China, which I think is entirely appropriate. That's not to imply that we weren't paying a lot of attention to China and Russia before in the last administration. But what this will mean is presumably more people and more resources devoted to those two key targets. I think short term, Russia is our greatest threat because of their intense efforts to undermine us politically. Long term China clearly is our greatest concern, I believe for lots of reasons, not the least of which is their economic power and the impressive and disturbing military modernization program they they embarked on. So I think the change in strategies change in change in priority by this administration was was appropriate. That is in no way to minimize or marginalize the importance of counterterrorism. In some ways, the success the counterterrorism program has enjoyed is perhaps worked in its disfavor because we've gotten used to the fact that, well, we go after him, at least to this point, we go after him at distant places and not it's much better to do it there than it is to do it here. All right, so I know that we're still waiting for Representative Slotkin and as Dean Barr mentioned, we'll probably sort of turn to the questions from the audience to see if we have any right now and then hopefully when Representative Slotkin comes in, we can draw some of these insights from her as well. So, Edward or Karuna? Hello, thank you all for being here today. My name is Edward Elliott. I'm a Masters of Public Affairs student here at the University of Michigan and before coming here, I helped set up and run a think tank called British Foreign Policy Group in the UK for a few years. So my question is with regards to Macron's comments last week about NATO, we call it brain dead. So I'm not going to ask about whether you still see value in NATO, but specifically on that issue of kind of having a strategic aim and whether in your view NATO has kind of lost a bit of that strategic aim and if so, what do you see the solution to that being? Who do you want to take that? Sir, why don't you go ahead and generalize? Well, I think you're referring to Macron's characterization of NATO as brain dead. Is that what you're referring to? Well, I don't I would take respectful disagreement with that. I don't think that's the case at all. NATO is, you know, one of a series and probably the most important alliance that we formed in the aftermath of World War Two as a bulwark against the Soviet Union. It is, in my view, regained importance as a bulwark against resurgent Russia. I think NATO has made a lot of progress. It is obviously grown in membership since its original founding, I think 12 or 16 members now is up to 29. And I can tell you how the European countries felt feel about it or at least felt about it when I was last in a position in Iraq with them. They considered extremely important. And of course, central to that is Article five, which, you know, an attack on ones and attack against all. And it's a good thing to remember that the only time in the history of NATO that that's ever been invoked was on our behalf after 9 11, where they considered the attack on us as an attack on them. That's the only time Article five has ever been invoked. I can tell you that the countries, particularly the new members, like the Baltics, for example, attach great importance to the deterrence value of NATO and its strength as a classic example of where the sum is greater than the parts. I know your question was specifically about NATO and I'll try to hew to that, but I'll make a general observation first. I think the answer to the question about NATO turns on a larger question that I will ask and then try to answer for myself, my point of view, how important are our alliances in terms of our own prosperity and our own security, American prosperity, American American security? In my opinion, ever since World War Two, alliances have been fundamental to ensuring our prosperity and our security. I personally believe I'm not an economist, but I believe this anyways, that the fact that we have a I think $17 trillion economy right now, we got problems, no doubt when it comes to our economics, but we have a $17 trillion economy. A significant portion of that comes from our tradition of investing in and being, in many cases, quite generous towards our allies and our alliances. In other words, it has riddounded to our benefit. It's expensive, it's difficult, it requires constant attention and energy, but we have materially benefited in both in terms of prosperity and security. So assuming that's right, and I can imagine there are people who disagree with these assertions, but I believe it's right, and so long as I believe it's right, then alliances like NATO, which are well established, have well developed and mature processes, traditions, mechanisms by which we work together, however imperfectly, there's a well worn road and process by which we share intelligence, by which we conduct military training and preparedness, by which we conduct actual operations together. We should be very, very careful, not just the United States, but all NATO members should be very careful about heaping scorn on something that has been so beneficial. And even if that's not true, even if the gains have only been marginal, until somebody's got a better solution, we should be very careful about doing things that harm it. Because if we don't have something better to replace it with, why are we letting go of something that we don't have a decent replacement for? Thank you, and thank you both again for being here. My name is Karuna Nankumar and I'm currently a junior in the Ford BA undergrad program. I spent some time this summer working for the State Department and hope to pursue a career in diplomacy and international security. Going towards audience questions, to what extent does the DNI focus on US domestic terrorism? And what do you think can be done within the next decade to combat the rise of a domestic extremism? So the question is what is what is the role of the DNI in combating pure domestic terrorism? And where do you think it could go in the future? Well, the Office of Director of National Intelligence, which was created by the signing and the law of what's called the Intelligence Reform and Terrorism Prevention Act, which President Bush signed on December 17 2004. And it is intended to be to promote collaboration, integration and coordination across the intelligence community. And not just for the issue of counterterrorism with a foreign domestic but for all intelligence issues. So one of the important instrument arms of the Office of Director of National Intelligence to do that is the National Counterterrorism Center. And you have two veterans from it here on sit on either side of me. And that also created the aftermath of 9 11, which is provided a focus on on counterterrorism, bringing to bear the resources of the entire intelligence community, notably the CIA and the FBI. As and one of its responsibilities is to promote dialogue and intelligence sharing at the state local tribal and private sector levels. And they do that I think with great distinction and improved as time has gone on. So I think that's a great capability that works male seven by 24 and in relative silence and unsung to keep this nation safe and secure. Thank you for the question. This is one of my favorite topics. The end as as a general clapper already mentioned, this is something that I was the director of strategy at the National Counterterrorism Center. So as a as a strategic question, the I'm on record as that director, while I was still on active duty as asserting in several public remarks, that we are seeing a significant growth in all forms of violent extremism inside the United States. This is an oversimplification, but there's there's one group that is the there are the al-Qaeda ISIS inspired types of terrorists, and then there's everything else.  Not all of them are violent, but but a sufficient number of them are violent that we they deserve the label domestic terrorists. So we're seeing this growth in both parts of the domestic terrorism arena. And while there are intelligence requirements and intelligence challenges, particularly when the need for intelligence are intentioned with civil liberties, privacy rights, etc. But I think our biggest problem, our biggest challenge, and in many ways, our most important challenge is something that we're only gradually moving towards right now. I would I wish I hope that the United States as a government and as a people start becoming more energy energetic about a debate, a conversation. And it's a classic debate. It's a classic conversation. How do we reconcile these very important values, privacy rights, constitutional rights, societal norms, you know, there, there, there are certain lifestyles, certain traditions in America's culture that are very important. And we should tread on those very carefully versus the need to ensure the safety of our own people against violent extremists. There is a conversation going on, both in government and in civil society, to some degree between government and civil society. But I would argue all those conversations are inadequate right now. They're inadequate in volume. They're in their inadequate in content. And I would argue civil society leaders and government leaders should become more energetic in fostering this debate, this dialogue, so that we find what I would personally argue America Americans and Americans government were designed to create effective compromise, find the middle ground between these important values, but also the need to guarantee public safety. Where is the middle ground, particularly in an era where the problem is growing? This problem is not waiting for us to solve it. We wait long enough we're going to have a catastrophe. We should not wait that long. It should not take a domestic 9-11 to cause us to fix ourselves either as a government or as a society. There's an opportunity now, but the conversation today in my view does exist. It's not adequate. So the next question is from a member of the audience called Gordon Rooney, who's an NPA at the Ford School. So the question is, how do you see China's Belt and Road Initiative affecting ever-tall U.S. national security interests? So the question is, how do either of you see China's Belt and Road Initiative affecting U.S. national security? Well, it's potentially hugely important. This is it's often compared to the Marshall Plan. And the Chinese investing somewhere in the neighborhood of seven times the size of the Marshall Plan for the recovery of Europe. And so the Chinese are using this Belt and Road Initiative as a way of gaining inroads into other countries, notably in Africa and South Asia. And they're spending a lot of money at it. Now, the downside for them is that this is perceived as exploitative on their part, where they are trying to normally gain resources, natural resources out from a country for and return for some infrastructure project. And in more cases than not, they bring their own source of labor, which minimizes the, you know, the economic impact on the country in question. So there's a lot of downside features to the one Belt and Road, but to be clear, this is an instrument, I'll call it that, that the Chinese are using to gain inroads, access and influence in many other countries in the world, notably in Africa and South Asia. I personally encountered this problem in the world of counterterrorism because there are places in Africa, places in the Middle East, places in Central and South Asia, where we're feeling the effects of this competitive offering by the Chinese. And the, you know, the Belt and Road Initiative is a global initiative. We need to remember that. This is not small potatoes. This is a global strategy that China is implementing. But what I've seen too much of is local actors, whether they're governmental or non-governmental, so people in a foreign ministry, people in the defense ministry, or or people in education, people in business basically come to this conclusion. I'm oversimplifying for the, for the sake of brevity, you know, a dissatisfying or disappointing deal with the Chinese now is better than an awesome American deal that I have no idea when I'm going to get it. We're, we're, we're rather ponderous, we're rather slow and we're increasingly, it appears to me, internally conflicted about how much we care about the economies or the education or the, you know, there's a tendency to, and this goes back many years now, you know, this is their backyard. They need to clean up their own mess. Okay, but if they're incapable, if they just aren't willing and somebody comes and offers them a dissatisfying deal, that's still better than a potentially magnificent deal from the Americans that's never, that never arrives. And so we're ceding, whether we intend to or not, we're ceding that opportunity and at the end of the day, we're ceding that influence to other actors. And just to pick up on that point, do you think that's not only a question or an opportunity that China is taking advantage of, but also Russia and other countries? Absolutely, absolutely. I mean, you know, particularly those populations that are, that feel they're in desperate straits and many of these populations do feel they're in desperate straits. You know, I've actually heard, I've actually had people telling me, listen, General Nagata, I'm sure that what the Americans could do for me would be spectacular, but I can't wait, you know, I don't know if you're ever going to provide that to me. These other people, they'll give me something now. I got to do what I got to do. Corona. So one question that has been on my mind and was also raised by a couple audience members is looking towards the upcoming election. Which one? Presidential, upcoming presidential election, my apologies. I kind of guess. And considering our most recent presidential election, what changes in the intelligence community need to be made to ensure that interference from Russia is avoided as much as possible? Well, the intelligence community needs to keep on, keep on, do what it's been doing through the election of 2016 and the midterms of 2018. And you can bet the Russians are going to be back in 2020. Why not? They enjoyed huge success in interfering in our election process in 2016. And, you know, they had several objectives, but the first one which they achieved to a fairly well was to so doubt, discord and distrust in this country. And they're going to continue to do that. Putin's calculus, in my opinion, is that when he does that, he helps to weaken us. And since in his mind everything is zero sum, that thereby strengthens him. So the Russians have long done this. They've, you know, long practitioners of interfering in people's elections, theirs and other people. And we have records that lead going back to at least the 60s where they tried to interfere or somehow meddle in our elections, but never ever on the scale as they did in 2016. And of course, the big difference was their very sophisticated and broad use of social media. So we can look for that to continue. And by the way, Russia won't be the only ones interfering in 2020 because others have gone to school on what the Russians did and they're going to be drawing from the very same playbook as have the Russians. So you can you can expect the Chinese and the Iranians and probably even the North Koreans and others who have their own individual policy acts as a grind who are going to try to interfere in our election. So the national security apparatus in in Toto's got to it's got to be very vigilant about that and point out what's going on. I have to also make make a point here about we put a lot of emphasis on cyber security, you know, securing the voter apparatus voter voter registration roles, the mechanisms for compiling and reporting votes. That's all very important. But you have to remember that's all done on a decentralized basis because voting apparatus in this country is done at the state and local level. That's all important. But the bigger problem for me is what I'll call intellectual or information security. And the way the Russians exploited divisiveness and the polarization in this country, particularly via social media. And, you know, how do you get people to question what they see reading here on the Internet? That is a much tougher problem. Well, I can't possibly speak to the intelligence portion of this as eloquently or completely as general clapper can, but I will make this observation. Actually, this goes back to something that a very famous special operations commander many years ago said about my own community. And he said famously, we want all of our people, all of our special forces operators, we want us all to think like bank robbers. It's just we cannot rob our own banks. So I'm going to try to follow that guidance for a moment. I'm going to try to think like a Russian bank robber. I have nothing but a target rich environment so long as Americans are constantly screaming at each other on TV and on the Internet. And whenever I see that I'm going to identify who's the most purple in the face screaming at his antagonist or her antagonist. That is a susceptible target because when you're angry, you're not thinking. That's my target. And to the degree to which Americans keep screaming at each other instead of talking to each other, it is nothing but a target rich environment for all the things the Russians are planning to do and other actors besides Edward. OK, so the next question I promise is actually from an audience member and not for myself. It's from Mike Mendez. But it's so what if so do you think that Brexit will have or is having any effect on intelligence sharing between the US and the UK, but also the US and the EU? No, I don't. I think by and large the through all the Brexit drama that intelligence sharing, well with the UK, but the larger alliance of the so called five eyes allowance, which is UK, Canada, Australia, New Zealand and US. And I think despite the vicissitudes of things like Brexit that that will continue. I do think, though, that it's if things happen here and the UK does withdraw from the EU. First question you have to ask yourself is why would any country vote consciously to lower their GDP? And that's kind of what's happening with with with the UK. It's going to be very disruptive in Europe, I believe, and I think internally disruptive for the for the UK. Just an observation. This is kind of a blinding flash of the obvious, but that's my specialty since I'm a general officer. And that is regardless of whether or not it's had any significant effect on intelligence sharing, one profound effect it has had is it's injected an enormous amount of strategic uncertainty into the European Union, consequently into NATO. You know, everybody has to hedge their bets a little bit about where are where are UK colleagues going to land here. Uncertainty is never a good thing, particularly in a strategic alliance. It's it's it's it's toxic to the effectiveness of any alliance or any union in case of the EU. So it has debilitating effects, whether or not it's hindered intelligence sharing to push you both a little bit further on that. So so if the you know the UK leaving the EU, it will also leave your poll with that have no effect on UK US relation from a national security perspective. So we're suggesting leave NATO not NATO will there be a sort of your poll additional effect with the your poll relationship between the United States and law enforcement information. I doubt it. I doubt it.  And I don't see that changing. I think General Nagata makes a great point, though, about the the the strategic uncertainty it creates perhaps maybe in a military sense. And that's a great unknown. Corona. All right, so our next question concerns how some members of our community have come to express their criticism of US military activity in Vietnam, Iraq and elsewhere, as well as the role of the intelligence community. What do you regard as some of the key mistakes that the US has made in this domain? And how have you dealt with these issues personally and how can similar problems be avoided in the future? It's about Vietnam, Vietnam, Iraq and a sort of intelligence mistakes kind of lessons. This is obviously for me. Either you or I served two tours in Southeast Asia as my war. I went to Vietnam very early 1965 and 66. And now I got to plug my book. If you read the book, you will find in there that I finished I did a year there in Vietnam and was the most miserable year of my life, both personally and professionally looking back on it. And I grew very, very disillusioned with the war in Vietnam. I almost got out of the Air Force as a matter of fact, after that. And as luck would have it, I was mentored by some people and ended up staying in the Air Force for another 30 years. I went back for a second tour. And I flew missions, reconnaissance missions out of Thailand over mostly Laos, a couple in Cambodia. And I always felt that during that tour, which was much more professionally rewarding for me, that I was engaged in a particular form of intelligence which was designed to be as precise as possible about targeting. In other words, saving lives by being very discreet about the location of valid enemy targets. Again, I've explained all this in excruciating detail in my book, which is out in paperback by the way. So it's done well. Now, as far as Iraq is concerned, my fingerprints were on the infamous National Intelligence Estimate of October 2002. I then served as a second time. I was an agency director. I was director of what's now known as the National Geospatial Intelligence Agency. And the result of that, and this is the finding of weapons, the presence of weapons of mass destruction in Iraq. And that arose from a combination of factors. One, some process errors that the intelligence community committed, which we've learned from, and I think will avoid a similar recurrence. It certainly had huge impact on me personally when I served in more senior capacities, particularly as DNI. And the other was, of course, a willing recipient by the administration at the time who wanted that news. So it was a combination of those too. I won't go into a lot of detail other than to say that one of the things that the intelligence community has done is to more prominently allow dissent. And I will tell you that President Obama, on more than one occasion, spoke to those of us in intelligence by saying I would be very surprised, in fact, suspicious if all of you agreed on everything all the time, which we never did. So an important thing there was there, and there were dissenters to that national intelligence estimate at the time. And NIE is the kind of the apex intelligence product normally produced for a policymaking community, notably policymaker number one, the president. And one of the things that we've done since then is to more prominently portray whenever there is dissent. So decision makers, policy makers are aware of the dissenters and the reason for dissent. And we've done all kinds of other things like outside readers, alternatives, what if we're wrong, that sort of thing. So we've done a lot of things to try to forestall a recurrence of what happened to us in 2002. Well, I'm going to rise in my defense of my IC colleagues, but truth and advertising, I've served several assignments inside the intelligence community as a special operations officer. So I've lived in both worlds. First of all, I wish both as a government and frankly, as a population, that Americans would do a more careful job of discriminating mistakes from malfeasance. Those are completely different things. There's nobody in this room that hasn't made a terrible mistake somewhere in their life, including me. I've made a bunch of terrible mistakes. I don't think I've been guilty of malfeasance. Now, of course, I'm trying to look at myself and that's always imperfect, but I don't think I've ever seen the wrong, the ethically morally repugnant thing and told myself that's precisely what I'm going to do. I'm pretty sure I've never done that, and that's probably true of all of you as well. But it becomes awfully convenient sometimes for people to conflate the two and they're not the same thing. Secondly, and this is rather obvious, there's actually a journalism saying about this, right, if it bleeds, it leads. Tragedies and errors that come from either mistakes or malfeasance, particularly if they come from malfeasance, you will read about that for days. It will endlessly loop on whichever your favorite news channel is because it's juicy, it's entertaining, it gets people riled up. But here's what you don't see, how many times an Intel collector has risked his or her life to get intelligence that ends up saving someone or securing the success of a mission. Here's another thing you never see, a service member throwing his body in front of direct fire from an enemy force to save a local native person. Why did he even know to do that? Because he had the intelligence that told him where the fire was coming from and he deliberately exposed himself to that fire, sacrificing in many cases his own life to safeguard people who aren't even American because that was the mission to secure the population. And then finally, just the unbelievable labor and as I said, having served a couple of times in the IC, there is no more sleepless community in the American government than the people who work intelligence. I mean, I'd like to consider myself sleep deprived on more than one occasion. But I pale by comparison to some of my intelligence community colleagues. I don't know when those people sleep, including this individual right here. He's making up for lost time now. But I don't know that he ever slept. Now, that may sound like a small thing. But that that's also time, energy and affection you're not putting on your own loved ones on your own family, as I alluded to earlier. When's the last time that played endlessly on your pick your favorite TV show? Never is the answer. Representative Slotkin, thank you for joining us. So I'm so sorry to be so late. I'm so sorry. We had some veteran commitments we just could not miss. And the snow is just made. I apologize. This is obviously not the outfit I hope to wear next to these former generals, but it's Michigan. So welcome to Michigan. Well, thank you again for being here. And let me just kind of get a couple questions that I'd asked generals. I got an clapper to you and then we'll maybe have some time for another round of audience questions too. But just can you just share a couple insights from your own very distinguished career of public service and kind of lessons learned or what got you started on the road you took and to where you are now. So what does public service mean to you? Sure. Well, I'm part of the what I guess we call the 9 11 generation. I happened to be on my second day of grad school at Columbia when 9 11 happened. So I was in the city. And I went from working on international development to making a hard turn into national security was recruited by CIA right out of Columbia. And within a year of joining up at Langley was on my first tour in Iraq alongside the military. And for me, I think the training that comes from being in the intelligence community, obviously, and the military is the kind of training that serves you the rest of your life whatever career you choose to go into. And it's about discipline. It's about a mission focus. And it's about service to something higher than yourself. And what's been interesting is, you know, after being the CIA and working at the Pentagon, and a lot of my colleagues departed the Pentagon around the same time when President Trump was inaugurated. And they felt, you know, this sort of missing piece when they left. And I certainly my husband's 30 years in the army. And when he retired, there's this missing piece. And that is the mission focus. And what I find is colleagues who are in this line of work, whatever they go and do afterwards, they need that mission. And if they don't have it, they're like, what's the point? And that is a wonderful tradition to have in just a citizenry. And I see it all the time now. And I couldn't feel more strongly. I'm actually a proponent of required national service of 18 months of national service military, if you want it, but then civilian service, if not.  All right, thank you for that. And I also want to say that Representative Slotkin and I used to see each other in you pick a three letter agency, we were in meetings together or fighting over analysis or you name it. So also shared, you know, parts of my career with you as well. So I'm glad that you're here. But the second question I asked was so on top of the service question, what are your top two or three national security priorities looking forward for the country? Sure. So I'm on the arm services committee and the Homeland Security Committee. So we think a lot about sort of current and future threats. And the the thing that we talk about a lot in the arm services committee is is two big things. One that we need to make sure our military is prepared for, frankly, 21st century threats. And, you know, we all talk about everyone will throw out different names of cyber and artificial intelligence and hypersonics and space and all of these areas that are kind of where the military is going in terms of tools. But what we have found is that in addition to dealing with the newness of all that, we just don't have the doctrine down to know how to deal with those tools. So for instance, and for the Michiganders, we know that last year, we had a totally accidental gas or fire at a gas facility in the middle of a polar vortex. And if you remember Governor Whitmer went on TV and asked all of us, we all had to turn down our heat below 65 degrees, which by the way, everyone was complaining about. But when it's 65 degrees in Michigan we're in short. So I don't know exactly what happened. But we that was totally accidental. And we luckily had backup. And so we were fine. But if that had been an intentional attack, a cyber attack from a non state actor based on China somewhere. And we had had a polar vortex and we didn't have the redundancy in our infrastructure. And suddenly 26 elderly people died in their homes froze to death. 26 American citizens dead. What is the appropriate proportional response to that kind of attack on our country? What do we do if the Chinese government is denying they have any knowledge of that attack? Do we go and respond? Well, we're the United States of America. We don't freeze elderly people in their homes, right? That whole doctrine of proportionality, which is a defining central point for our military, gets confused and thrown out of whack when you think about you know, applying that to the new tools of the future. So doctrine, sort of the 21st century doctrine, and then we just have to acknowledge particularly to Chinese, their their innovation cycle is so much faster. We've built this amazing military, but it comes with this extremely lumbering bureaucratic system. And so we may have a great idea. But turning that great idea with the newest technology into a fielded technology that we can all use and take advantage of, the Chinese have us any day of the week on that. And there's a lot of good reasons like laws and principles and values. But that's just the truth. So what does it mean? I think a lot about how China is catching up to us in many areas and poised if we don't make some changes to advance beyond us. Great. Thank you. So hopefully we have a couple more audience questions. Edward? Yes, so I'm just grouping together a few questions from the audience about careers and career advice. So the questions are how can you make these different careers whether it's sort of military or intelligence or political, how can you make those attractive, particularly in in current times or where the present discourse of making public service a suspect hauling or in light of several of the challenges that we've discussed today. Representative Slacken, you want to take that one for us? Sure. So I guess, first of all, I really appreciate that you got the British guy to ask the questions because sounds great. Everything sounds great. He's in my class as well. So it's very smart choice, super smart choice. So I guess I almost sort of question the premise that while it's true, we've had a rough couple of years as civil servants, especially I would say public servants in general. I don't think we've had a better two months in the past 30 years as civil servants than we have in the past two months because civil servant after civil servant, intelligence officer, foreign service officer, ambassador, military officer have come forward in the past two months, put their absolute jobs on the line to speak truth to power in a very apolitical straightforward straight shooter kind of way. And I think what the country has seen in the past two months has done more to boost the image and the frankly heroism of a civil servant than anything that's happened in the past decades. And you know, we're going to have to do quite a bit when this administration is over, whether it's over in a year or it's over in five years to really do a huge callback of our civil servants, of our ambassadors, to offer them the opportunity to come back. And then we're going to have to make up for some of the recruiting challenges that we have had. We've had the smallest number of young people taking the Foreign Service exam this year than in a decade. That's not by mistake. But I do think that what we're about to go into, at least in Congress, the next three weeks are all public servants coming forward and doing public testimony. I think that that is going to do wonders for what it actually means to be a civil servant. It's a very complex question. But I will try to be brief about my answer. I think that more American leaders should level with the next generation of Americans that may be aspiring to have a career in national security that they will be developed and that we're going to make the investments necessary to develop them into being at the risk of inadvertently offending somebody who's currently in government or recently departed government, including myself, so that they become better, more proficient, and more effective leaders than the current generation is demonstrating. And I don't say that to condemn the current generation. The current generation has its problems. But the challenges the world is confronting the United States with are not static. They are growing in both complexity and number and difficulty. All three are increasing. So if we simply replace the current generation with replicants who are only as good as the current generation are, the United States will fail. We will strategically fail. The quality of the workforce and most importantly, the leadership that workforce has to rise. And it's going to have to perpetually arise so long as the three things I've just asserted about complexity, difficulty, and significance continue to rise. And it begs this question, is that the investment we as a government and we as a society are making to ensure that the trajectory of future American professionals in the national security arena will be better than the current generation. If we do, then more people will join. And people will have both long and satisfying careers. If we don't, then we fail. General Clapper. Well, if I understood the question, it's what is it that must be done to attract people to come to public service? And in the end, despite all the changes, General, I've got to describe, I think the fundamental fact is that it's for people who believe in this country and what it stands for are values. And serving that country, not an individual, serving that country in the spirit of public service, of being part of something much bigger than yourself, keeping the nation safe and secure. I always regarded that as a sacred public trust. And that is a spiritual thing. And I think that still should be the essence of the appeal of a career in government or specifically in national security. It isn't about the money. If money is the preeminent thing on people's mind, then they shouldn't come to the government. That's not the place to make money. But there are other rewards and satisfactions from performing service on behalf of your country. Colonel, maybe one more question, and then I'll wrap it up with a final one for me. All right, OK, sounds good. So our last question then, and I'm sorry I can't follow the accent, but I'll do my best. The next audience question observes that our past counterterrorism strategies have seemed almost reflexive and defensive. And wonders what strategic shifts are necessary to not only prevent terrorist attacks, but achieve some sort of strategic victory in the longer term. I'm going to take the first whack at that. In my last position, I was asked to be the, my office was asked to be the organization that would integrate more than 14 different agencies to conceive the national counterterrorism strategy that was signed on October of last year. Then I led that effort when I was at the White House, right? Yes, you did. Yes, you did. Can finish the job, though, right? That's job. It was my top cover.  I'm going to give you a conditional answer here. The language in the strategy asserts that the United States government must become far more proficient than it has ever been before in what I will imperfectly call non-kinetic counterterrorism, preventing people from becoming terrorists, denying them their ability to travel, denying them access to resources, effectively contesting their activity on the internet, yada, yada. Notice nothing I just said requires a 500-pound bomb. So that's the direction. That's presidential direction and the strategy. That's the good news. The incomplete news is how well will we implement that direction? Will it be funded adequately? Will it receive the kind of durable policy support that is necessary so that these things can overcome occasional missteps, accidents, mishaps, and failures? I left government before I knew the answer to that, and I still don't know the answer to that. I am hoping that we give it our best to implement this direction well. The jury is out. General Clapper. Well, this is maybe kind of cosmic or high-level observation here. But in my view, we're always going to be in a state of suppressing terrorism until such time as the conditions that give rise to terrorism are attended to. And by that I mean, particularly in places like the North Africa or the Middle East, the economic conditions there. The large, ungoverned spaces, nations of wash and weapons, porous borders, weak governance, and most importantly, a large cohort of frustrated young unemployed males who are ripe to be recruited. So those basic conditions have to be attended to, otherwise, we're going to continue to be in essentially the whack-a-mole mode that we've been in and need to be in, by the way, because they pose direct threats to us. But the fundamental thing is, how do we get at the conditions that give rise to terrorism? Representative Slotkin. Yeah, I think that's right. It's root causes. And it's sort of a combination of the two answers to me. You can't get at the roots of terrorism without getting at the root causes for why people are unhappy. And those people tend to be in places that are ungoverned or under-governed. And there's usually very clear reasons. The state isn't strong. Maybe there's significant corruption. That theme around the world is amazing. But the truth is, in order to deal with the root causes of terrorism, I mean, it's actually by design, not a military operation. It's not a military work responsibility. It's the State Department. It's USAID. It's Treasury. It's our commercial service. It's everything else. And one of the most powerful things that I remember experiencing was watching then Secretary of Defense Gates sit up at Congress and advocate for the State Department in USAID budget. It was like radical, you know? And the Defense Department obviously is a huge proportion of our budget and is funded every year without a problem. But we're always scraping at the bottom of the barrel for USAID and for State Department. So to me, I would personally find it interesting if whoever our next president is said to whoever they're gonna hire as the Secretary of Defense, I need you to see your job as helping to get resources for the other side of the government so that you don't have to send our young men and women off the fight as frequently. I think that would be powerful to have an upfront conversation and do what Secretary Gates did for the State Department at the time. Here, here. Thank you. All right, so that'll close off the audience questions, but thanks again to everyone who contributed to the question and good job Edward and Karuna for sifting through those and getting us some really good ones. So let me just ask one final question to all the panelists. So this picks up on something that Representative Slotkin, you touched on as sort of a national security priority, sort of preparing the military for the 21st century and I'm looking at what this new landscape is gonna look like. But just more broadly, since we are here on Veterans Day, what do you think are the things the military, the U.S. military needs to be sort of looking at in the future to get ready either for the next fight or the next sort of peace dividend that hopefully settles over the country for a long period of time? Oh, sorry. Sure. So I think maybe now that because I'm an elected official and I don't sit at the Pentagon as frequently, I've spent more time in Silicon Valley and getting inside that loop that makes it reasonable and interesting and attractive for the private sector to want to do work for the Defense Department on new and emerging technologies. I think they wanna work with us in spirit, but the Defense Department system is so slow and so lumbering and so cumbersome and so confusing from the outside that a reasonable entrepreneur would say, hey, I'm interested, I wanna help my country, but I need to keep my organization afloat, I need capital, I need to move, I need markets. And those cultures need to be bridged and in order for us to take the best and brightest from places like Silicon Valley or our engineers here in Michigan and put that to use for the Defense Department in a way that's a win-win for everyone. I think that's a big one for me. And then we just need to, as General Legato was saying, we just need to accept that there's a different type of military officer that we're gonna need to, right? We just sent our special assistant in my office. She went off to be an officer candidate school in the Navy to be a Cryptologic Warfare officer. We probably need a few more of those. My husband was a pilot in the military, in the helicopter pilot and we love our hardware in the military, but we need people with facility on software and we need to go recruit those people and understand that they may need a different pitch than maybe the officers of previous generations needed. And we need to come up with a human capital strategy to match the needs of the 21st century and we don't have that as I know of as of yet. Great, thank you, General Legato. Okay, I'm gonna try to roll through a couple of things quickly. First of all, I can't resist sharing the story because I was actually witness to a conversation between a senior general officer and one of the heads of the automakers here in Detroit. This is many years ago, but it was an unforgettable conversation. This general officer asked this very important chief executive officer, hey listen, I've never seen your company offer to compete any time we're trying to solicit bids for the next generation of vehicles for the United States Army. How come? How come you never show up? The answer was unforgettable. I consider myself and everyone who works in this company to be an American patriot. We would love, we would be proud to have our vehicles as part of the military's next generation of vehicles. And then he paused and he said, but working with the United States government and the Department of Defense is impossible. Now, having said that, that's a serious problem. It gets to what Alyssa was talking about. We make it so freaking hard for technology companies and everybody else to offer their goods, their services, their technology to us that somebody else will buy it. And it might not be somebody we like. Second, and this is probably my most important offering. The United States military has to get better at training and educating for complexity and uncertainty. When I first joined the military, there was a school solution for every training scenario. And you were graded on whether or not you got to that school solution. We need to jettison much of that in the United States military. And we need to inculcate a tradition in the United States military that does exist in some parts of the military, but it's nowhere near as pervasive as it needs to be. This is an incredibly complex, multi-dimensional security challenge. There is no solution. There is no answer. And we're actually gonna evaluate you not on how well you solve the problem. We're gonna evaluate you on how relentless you are in trying to solve it. You're never gonna quit. You're never gonna get frustrated.  That in my judgment needs to be inculcated far more broadly in the Department of Defense. And then finally, leaders need to become more risk tolerant. And this is more and more true the higher you go, not just in the military chain of command, but in the government. We actually become increasingly risk intolerant as you go higher and higher. And it's not irrational, particularly when you get a policymaker realm, they know how they'll be pilloried on every evening news show if they become connected with some perceived failure. But the problem with that is the age-old human truth. How do you learn? You learn by screwing up. That's how you learn. How did you learn how to ride a bicycle? You learn how to ride a bicycle by falling off your bicycle. It's like, I'll end by just sharing one of my favorite stories. I'm sure all of you heard it in school. It's what Thomas Edison was quoted over and over again saying after he invented the light bulb, people showering with praise or inventing the light bulb. He always replied the same way. And we need to inculcate this in the US government. I did not invent the light bulb. I invented 900 ways not to have a light bulb. The only thing you people remember is the last experiment that succeeded. The only thing I remember is if I hadn't failed 900 times, you wouldn't have a light bulb. General Clapper. Well, clearly I'm the geezer. I'm the table here. I often found myself longing for the Halcyon days of the Cold War. We had one enemy, the Soviet Union. It was very simple. And the Soviet Union is very slow moving and predictable. And I think a lot of the frustration you hear expressed here about the way things operate are actually vestiges of the Cold War because we could take our sweet time and bring on new weapon systems or whatever changes we had to make. We had lots of time. Now we're in a very diverse threat situation like I've never experienced. To me, the watchword for the government institutionally and those in it personally is agility, adaptability, and diversity. Great, thank you. Well, I think that's getting to the close of the event. So big round of applause for our panelists today. Thank you.


So you're not really sure how to organize your UVs. Not a problem. Let's say you're working on a character like this, and make sure that anything that can be textured is removed. Now in one of the last videos, somebody asked, how do you know what parts of your character can be textured and what parts cannot? And basically the general rule I follow for this is anything that doesn't stick out a lot can usually just be a texture. For example, originally when I made this character, before I knew how to use substance painter, I actually modeled in the leotard that this character wears underneath all of her clothes. But after getting into procedural texturing, my texture guy basically told me, listen, anything that is remotely like a bodysuit or a bikini or a swimsuit or scuba diving gear or any skin tight thing can almost always just be a texture. And he proved that to me when he was texturing the new Unreal 5 custom mannequin for the Jiggle Physics tutorial. The original 3D character had pretty low topology and was completely flat. But with good texturing, she looked like she had a lot of detail wearing a bodysuit. So this time when I'm making the character, I'm deleting the actual 3D leotard because that can just be a texture. Likewise, the ornamental designs on the armor can also just be textured. And technically the feathers on our arm and on the crest of her hat can be textured as well. For example, if we look at substance painter, if we just have a flat quad face with basic texture and transparency techniques, we can make it look like it's a full fledged ornament even though it's just a flat face. But for the purposes of this tutorial, we're just gonna leave these ornaments modeled in. And some of the objects like the boots and the gloves are clearly supposed to be mirrored. And then there's other random aspects of your character that are asymmetrical. So how should you go about organizing that? Well, first, just lay out all the UVs for each part of your character the way you normally would. And then once you've got everything flattened out, on the top right under UV, you will see something called Average Island Scale. Hit that, and what this does is make sure that all of your maps are sized in proportion to each other. Assuming you have all the proportions right, make sure everything that's gonna be mirrored fits into a nice square on the top and keep it all on one side. Then, start organizing everything else to fit underneath it. And that way, when you press U and mirror all the elements, we get the whole map covered. Hope that helps. And as always, hope you have a fantastic day and I'll see you around.


In those days, Coxe, not very many Bournemouth players got international call-ups, and I remember you getting your first Trinidad and Tobago call-up, and I also remember Mel Maching probably not being too happy about that because he was going to be losing you for games. Is it fair to say that he wasn't overly keen on you going away? Yeah, I think at the time, I don't think it was taken seriously personally. Having an international call-up, going to play for Trinidad and Tobago, I don't believe that it had taken people all seriousness. I think if it was getting a call-up for England, that was a little bit different. But we were travelling on the other side of the world to play lots of World Cup qualifiers. And as a player, you need to be picking your wits against the best players, whether it be here or whether it be away. So for me, it was something I needed to do, but I know he wasn't particularly happy for him. And again, Mel hadn't had... He had to look at how he was going to get the best out of his squad, and he wants all his best players there, I guess, to get a result on Saturday. It's a rumour only, it's a rumour only. Can you confirm or deny that he threw the call-up papers in the bid and didn't tell you? You know what? I can confirm. Goodness, Vee.