Everyone, welcome to today's hearing. The technology and AI fight for the 21st century operations, the Department of Defense. Since the start of this Congress, the Armed Services Committee has held nine hearings dedicated to driving change in DOD through adoption of new technology and pivoting to an innovation-first posture. No one obviously expects the Pentagon, the largest bureaucracy in the world to turn on a dime. It's an aircraft carrier, not a little Corvette. But I don't think any member believes the department is moving with the speed, force, or unity of action necessary to get our warfighters what they need. We've established that national security in this century is contingent on rapidly embracing new technology, experimenting, iterating, and pivoting from a platform-centric combat to network-centric warfare. And yet, there doesn't appear to be as any adversary as formidable as the change-resistant culture in DOD. My hope for this hearing is not only to hear about your plans for the coming fiscal year, but more so to understand the precise obstacles and barriers to reform that you've seen, that you've encountered on a day-to-day basis. And as I've said before in this room, for those of us who attend the Reagan National Security Forum every year, you can almost copy and paste the speech that the Secretary of Defense gives there. It's all about, we're gonna kill the Valley of Death and we talk about it and the same people have the same conferences and then it doesn't actually get any better. So, but differently, why it seems to this committee that a year later, the implementation of policies we've passed in the directives on AI, zero trust and software integration from your bosses seem to have made little progress. And I assume we all agree that speed is essential, then I hope that this hearing fosters an honest conversation about what's holding you back, what you're doing to go faster. But we're very pleased to have before us John Sherman, the Department of Defense's Chief Information Officer, Dr. Craig Martell, the inaugural Chief Digital and Artificial Intelligence Officer and Lieutenant General Bob Skinner, the Director of the Defense Information Systems Agency and the Commander of Joint Forces Headquarters, Department of Defense Information Network. Generally get the award for longest title after your name. And I will note that this is Dr. Martell's final appearance before the committee having announced last week that he set to depart his position on April 15th. And before hearing from the witnesses, I yield to the ranking member for any comments you may have. Thank you, Mr. Chairman. And thank you for your bipartisan focus on how we make the department more innovative, agile and capable of adopting the latest technology and software and working with the private sector to remain the most innovative military in the world. I hope that today's hearing we can explore not just what the department is doing, but what more flexibility is needed and how we can have greater agility. The reality is we have a challenge of harmonizing this ecosystem to bring it up to the speed of modern innovation while operating in our era of great power competition. In particular, efforts such as securing the defense industrial base, implementing zero trust architectures, accelerating the cloud transition and roll out of a robust identity credentialing and access management capability are fundamental. Two weeks ago, this subcommittee had a very interesting hearing on software development and software acquisition. And a month ago, the committee held a hearing on accelerating the pace of innovation across the department. As we approach the end of this term, I guess what would be helpful to me is a few, two or three very concrete recommendations, particularly of what the committee can do or needs to do legislatively to help with making the department more innovative. Is it at this point that Congress has done everything it needs to do and it's more a matter of implementation? Or do you see a role for the committee to continue to do more legislatively? And if so, what would those recommendations be? Thank you, Mr. Chairman, and I yield back. Mr. Chairman, you are recognized for five minutes. Good morning, Chairman Gallagher, Ranking Member Khanna and distinguished members of the subcommittee. Thank you for the opportunity to testify before you today. I'm glad to be here with Dr. Martel, as well as General Skinner, as you noted, Director of DISA and Commander of Joint Force Headquarters, Doden. Together, Lieutenant General Skinner and I lead a team that provides direction, oversight, and technical expertise to secure and modernize the department's information technology, enhanced warfighting command control and communications for C3, manage the DOD's use of electromagnetic spectrum, and cultivate a digital workforce. We look forward to sharing the progress on the department's digital transformation efforts and discussing our key priorities for fiscal year 2025. As we have seen in Asia and Europe, the cyber threats we face today are evolving and we must keep pace to both support the nation's warfighters and protect key national security capabilities. Protection of our networks and the networks of the defense industrial base is critical. We are laser focused on zero trust implementation and earlier this year we expanded eligibility for the defense industrial base cyber security program that will significantly enhance the cyber security posture for these companies. Cloud computing and software modernization remain central to our IT modernization efforts. When I testified last year, the department was just beginning the enterprise cloud journey and I'm happy to report significant and successful progress. Through the joint warfighting cloud capability, or JWCC, DISA has successfully awarded more than 47 task orders over the last year and over 50 more in the pipeline right now. We also publish DOD guidance to streamline cloud contracting and reduce contract sprawl across the department. In today's environment, and as you all focused in on your hearing last Wednesday, it is critical more than ever that we provide DOD personnel with secure and resilient software when and where they need it. We recognize the urgency of this issue and are working hard to ensure we are successful. Our software modernization strategy and ecosystem of more than 55 software factories are transforming the way DOD develops and delivers this force multiplier. This requires changes to our processes, policies, workforce and technology. Accelerating the authority to operate or ATO and strengthening reciprocity are absolutely key to this effort. Just as important as the software is ability to operate in any environment, our adversaries have spent decades investing in capabilities to make the best use of electromagnetic spectrum and that is critical to defend the nation. However, we also understand the increasing commercial demand for spectrum and are working with the White House, Department of Commerce and other inter-agency partners to explore ways to address increasing federal and commercial demand for spectrum access without compromising national security. 5G is also critical to both DOD and civilian industry. Last fall, my office became responsible for this mission and we are accelerating the deployment of 5G on military installations, advancing enterprise capabilities and addressing resource requirements. Each of these missions and others are critical to our war fighters and would be impossible without the right people. This past year, we laid the foundations to enable DOD to grow a dynamic and innovative workforce needed to succeed in the 21st century. Our implementation of the DOD cyber workforce strategy is ongoing to ensure we work with industry to recruit and retain the right people with the right skills for the right jobs. As people are our greatest resource, I understand their interaction with the department's IT infrastructure, weapons systems and business systems directly affect the mission and morale of each war fighter, civilian and contractor. To enhance user experience, we established a customer experience office to harness resources and data to tackle this multifaceted challenge. Thank you for your consistent and dedicated support and for the opportunity to testify this morning and I look forward to answering your questions. Thank you. Thank you, Dr. Martell. Thank you all. Chairman Gallagher, Ranking Member Kana and distinguished members of the subcommittee, I thank you for allowing me to testify here today and to share the work the CDAO has been doing. I will read through this, but I want to start by answering the direct question, which is what have we been doing and how can we make it faster? We've spent the last year building what I would call a virtuous cycle. That virtuous cycle is delivering value, creating new demand and then iterating. And so I'll talk through the tools that we've been using to do that, but I think to answer the direct question at the beginning, we have to continue that virtuous cycle and I think that's going to, is what's going to increase the acceleration and delivering value to the war fighter. The CDAO is accelerating defense-wide adoption of data analytics and AI so that the DOD can make better decisions faster from the boardroom to the battlefield. We create sustainable change at scale through two functions, leading and overseeing, and this is our principal staff assistant function, mostly through policy guidance and oversight, but also actively delivering capabilities across the full range of the hierarchy of needs. So the hierarchy of needs is how we analyze what needs to get done when. This enables change in speed, change at both speed and scale, speed by delivering apps and products directly to the end user quickly, think both war fighters and running the business of the DOD, and scale by delivering platform and services to allow builders at the edge, the owners of the problem to best solve their problem. So think about it this way, we're a centralized organization. If our job is to build all of the AI of the department that does not scale, we'll build onesy twosy solutions and we are tackling very large individual solutions, Jatsie too, for example, we can talk about Harbinger and other things that we've been working on. And that's really important for the centralized org with the expertise to drive those issues. But it's also really important to create the tools, policies, processes and best practices so that the folks at the edge can actually deliver value when they need that value. So they can deliver AI when they need that AI. They're the ones who understand the problem. They're the ones who are gonna understand the solution. It shouldn't be centralized. It shouldn't be only centralized. It has to be distributed as well. So we've been tackling both of these fronts simultaneously. The first allows for winning the fight tonight. The second allows for sustainability of those wins. And we do this, as I said, through the hierarchy of needs. The hierarchy of needs for us is that you gotta get the data right. So the lowest layer, the foundational layer is do we have the right data? Is the data accessible? Is the data understandable? And can people use that data to build apps that provide solutions? So think about a separation between data and apps. The next level above that is simply analytics and metrics that we spent a great deal of time this year getting people to move from effort-based metrics to outcome-based metrics. And that's allowed for the running of the department in a way that we haven't seen before. So every monthly meeting with the deputy, there's a dashboard with the metrics for that principal staff assistant and how well they're doing. And gathering the data for that has been significant. And on the top of that is AI, because without metrics to know how well you're doing, because remember, AI statistics at scale, you measure the past to predict the future. You have to know how well you're predicting the future. To do that, you need the metrics to measure yourself against it. And you need the quality data. So this hierarchy of needs combined with our agile approach is how we're gonna drive this sustainable change through our virtuous cycle. That's a bunch of buzzwords, but what do I mean by an agile approach? We've had major success on CJAD C2 this year, in particular by having data owners and industrial software engineers sitting right next to warfighters, literally in the room. And as those warfighters need something new, the data owners go find that data and the software engineers change the software and it's delivered within days. This sort of iteration, this sort of agile iteration is what's gonna continue to drive that virtuous cycle. So we deliver by learning by doing, shipping fast and iterating quickly with warfighters. And I said, developers sitting side by side, this is done on real networks with real data to learn and deliver for real warfighters fast. And we couldn't have done it without working closely with our industry partners. And we've used a lot of the tools that you've given us to be able to build contracts fast and to procure their services very quickly. So CDAO looks forward to working closely with the subcommittee on these issues and others as we scale DOD's current and future use of data analytics and AI for national security. Thank you. Thank you, General Skinner. Good morning, Chairman Gallagher, ranking member Kana and distinguished members of the subcommittee. I am honored to be here today and represent the approximately 19,000 personnel who support the missions of the Defense Information Systems Agency and the Joint Force Headquarters Department of Defense Information Networks. I'm also honored to sit alongside Dr. Craig Martel and one of my two bosses, the Honorable John Sherman, a key ally and partner in the campaign to innovate, modernize, secure and defend the department's networks, systems and data to achieve and maintain information superiority. Each day we are energized and focused on ensuring the joint force is in a position of advantage against any nation or group that desires to harm us or our allies as we set the globe in the cyber domain. Along with our key partners within the federal government, allies, industry, research and academia, we continue to leverage lessons learned from the ongoing conflicts in Ukraine and Gaza, the nefarious activities of our pacing threat, the People's Republic of China and global cyber events to strengthen our digital technologies and forces. Through the support of this committee, we have made significant progress over the last year and look forward to highlighting our future plans. Resiliency, agility, survivability and velocity are key tenants in describing our initiatives starting with our department's flagship joint warfighting cloud capability contract which was awarded in December of 2022 and provides access to multiple global cloud fabrics that ensure our warfighters can conduct operations anywhere in the world. Along with the 47 task orders Honorable Sherman mentioned, we have also successfully deployed an initial overseas cloud capability in support of IndoPaycom missions. Recently, we also kicked off a pilot, the Joint Operational Edge, that will deploy hybrid cloud capabilities to remote locations across the globe. As discussed in the hearing you held earlier this month, it is critical that we ensure the DOD personnel have secure and resilient software regardless of where they are located. We continue to develop agile and modern software development methodologies with templates, tools and automation in our software factory integration platform which is available to the department. Equally important to operational software availability is our perimeter defenses. We have two key pilots ongoing with commercial companies that are increasing our ability to see and counter-adverser activity which are increasing each and every day. Along with these activities, we are lockstep with the department's zero trust strategy through our ThunderDome initiative. In 2023, we deployed ThunderDome to 15 sites on classified and unclassified cyber terrain and will accelerate deployment to more than 60 sites this year. ThunderDome, which is part of our next generation network and data environment, will provide defense agencies and combatant commands with improved user experiences while also increasing cybersecurity by knowing who is accessing the network and data, limiting individuals to only that data they're authorized to access, enhanced data analytics through artificial intelligence, and increased segmentation like firebreaks in houses that prevent fires from spreading. While we continue to make significant advances, our work is not done. Leaning forward, I believe our new strategic plan, outlining the agency's fiscal year 24 to 29 goals, sets the North Star to enable best value capabilities to our department and our war fighters. A final area to highlight is our commitment to no-fail missions, such as the National Leadership Command capability. Within our strategy, this will deploy an integrated multiple level, secure voice and video communications and conferencing capability to provide direct support to senior leaders, including the president, secretary of defense, the chairman and the nuclear command and control community. None of these initiatives are possible without a bold, innovative and critically thinking workforce. In line with DOD's cyber workforce strategy, DISA has released our workforce 2025 strategy that outlines key objectives to recruit, retain and professionally develop our team to ensure the right personnel with the right skill sets are in the right positions. Our overall readiness, strength and resilience and war fighter success relies on the strong support that this subcommittee has provided for many years. I am grateful for your support and the opportunity to testify this morning. I look forward to your questions. Thank you. Thank you. We will now move to questions. Dr. Martel, Indo-Pakam has pieced together the Joint Fires Network in the absence of a Jazz C2 solution or operational capability. How would you elaborate on how your organization, what you've done to contribute to the JFN? Absolutely. And by extension, Indo-Pakam command and control, I guess put differently, how do you see the joint operating system and CDO's, Jazz C2 efforts fitting into the JFN? Yeah, thank you for the question, Mr. Chairman Gallagher. We work hand in hand with Indo-Pakam, with Arne and Indo-Pakam, with research and experimentation and Indo-Pakam on JFN. The underlying data layer, which drives the right data to JFN to be able to make the decisions, to be able to close chains, comes from the CDO. Some of it was theirs already, but they needed a whole bunch more. We have a team that sits with them, and when they need more data, we help find that data, we help deliver that data, and we help make sure that data's there at speed and scale, so that they can use it on a regular basis. So they were a key part of our last guide experimentation, and it was at Indo-Pakam, and the success and some failures in that experimentation was due to that partnership. I'm happy to go into deeper about exactly what worked and what didn't work in the closed session, but we're tightly aligned. Can you elaborate, take a step back, just elaborate for us lay people on the difference between the joint operating system and the data integration layer? I would say the joint operating system is part of the data integration layer, so think about the data integration layer as two major components. A catalog to help you find where data is, and APIs that allow that data to be served. That's really what it is, and that data has to obviously sit on a cloud somewhere, so the work with CIO is really important there. So the data's in the right cloud place. The data has an API that allows an application builder to access that data when needed, and it's discoverable, and not only is it discoverable but when it's discovered it's understood. So there is metadata that describes what that data means, how that data is used, and how that might be ingested into an application that's being built. JOS does specific things that we had originally contracted them for, and they are now part of that data integration layer. They have data that they're providing, that Andrew has data that they're providing, that's part of, that has an API layer and is discoverable through that centralized catalog. So you wanna think about the data integration layer as a multi-vendor heterogeneous entity that really is, the data that's needed is discoverable and accessible, and so that's what we're driving forward. And then in PB25, it says that the JOS is moving to production. I like the idea of CDO using all sorts of acquisitions authority to move rapidly, taking commercial technology like the JOS through prototyping and into production, but could you elaborate more on how you see the JOS in production? I may have to take that for the record or for the closed session because I don't actually know specifically what we can talk about what it delivers. But so they deliver, and my team can tell me if I'm allowed to now, but they delivered specific information that was necessary for the Joint Fires Network. It's their hardware that's distributed, it's Andrew's hardware that's distributed across the IndoPaycom Theater that allows that data to flow easily, and it's discoverable through our catalog and through our sets of APIs. I think I didn't answer the question more specifically than I did before. And you're leaving soon. What, as you look back, what are the metrics that you would judge your tenure on? Massive increase in demand for getting it right. We just did a presentation across the whole department about the current state of Jatsy too and the successes that we had. A year ago, there were a lot of, I don't know what this means, I don't know what you mean by getting the data right, I don't know what you mean by having the data accessible to the warfighter at the right time in the right place. That's now well understood. See Jatsy too, I was very pleasantly surprised. There's no longer an ambiguity, at least within our org about what See Jatsy too is. It isn't a thing, it isn't a system, it isn't even a destination. It's a set of behavioral patterns and the underlying data flows to support those behavioral patterns. I mean See Jatsy too is command and control for the 21st century. You do command and control in the 21st century by the right flow of data. The demand for that data flowing correctly has not quite exponentially, but geometrically increased, I would say over the last year and a half. And if there's a win that I'm strongly willing to claim, it's that. And in the seven seconds I have left, how would you respond just to the articles about the poor command climate surveys? Yeah, I think that's great. I mean that's a great question and a really important one. We did a very hard task. We took four organizations, we merged them to get forth with very distinct cultures, and we had to break a lot of expectations in doing that. There were lots of people who wanted to do particular things and we said, no, we're gonna look for economies of scale and we're gonna do these things instead. That's a monumental task for any merger and acquisition and it's going to upset some people. We've worked really hard over the last year to work with some industrial partners on getting our culture right, on making sure that our teams are being heard, that communication flows better. We did sort of bias execution, maybe to the expense of communication with our people in the beginning and I own that. I'm looking forward to next month when the new pulse surveys are released because I feel confident that they'll say good things. Thank you. Mr. Connor's recognized for our mess. Thank you, Mr. Chair. Dr. Martel, I see that you worked at Dropbox and Lyft before your service to our country. Can you describe some of the collaboration with Silicon Valley technology companies that you have and how that can improve? We couldn't have done the job without a tight partnership with key companies. Palantir is one, Andrew is another. Databricks is another. Most of the, almost everything we do, and then all of the cloud providers, so Google, Amazon, Oracle and Microsoft, thank you, John. Please keep listing companies in my district. That's just joke. I think every Silicon Valley company's in your district right there. So we couldn't have done it without them. We spent a great deal of time talking to the leaders of those companies to get buy-in for the store that I'm telling you now. What I tried to do was take best practices from Silicon Valley, which is really, the data is more important than the app. The data has to proceed the app that's on top of it. It's not more important, but it has to proceed. Quality data has to proceed the apps that are on top of it. Quality data has to proceed AI. It's logically inconsistent otherwise. And so I vetted this view with all of my colleagues in Silicon Valley. I had many conversations, not just with the big companies, but also with lots of startups, to make sure that they were on board and that they would be willing to engage as we built out this multi-vendor, heterogeneous data integration layer, that would allow for a separation of data from apps. Thank you. Are there things we can do to improve the collaboration or do you think it's in a pretty good place? Well, I think we still have some work to tackle. I think DIU, the Defense Innovation Union, is doing a really good job at tackling those. Doug Beck is a really strong hire, and so I'm really glad he's on board. You know, I've seen an increase in the way we've been interacting with small businesses, for example. Because of the marketplace we've built through trade winds, we're now up to 65 non-traditional company contracts. We're up to 75 small business contracts, and many of these are done in 30 to 60 days. And that's because of the authorities that you all have given us. So we're seeing some growth, we're seeing some inertia increase, but we need that to continue. I mean, I think the biggest takeaway, and actually, if I'm candid, the thing that I feel best about is at the end of my tenure, people are demanding more of what we're trying to do, and faster. 100% agree, faster has to happen. 100% agree. But I think it's now clear what has to happen faster, which is getting the data right, having the data flow correctly, being able to integrate with companies as quickly as possible, and it shouldn't be something that we build. It has to be multivendor heterogeneous. Dr. Martell, I also see that you have a computer science PhD degree from the University of Pennsylvania. Now, many people rightfully have been concerned about anti-Semitism, and there's no place for that on college campuses, but some of the rhetoric in this building has gotten pretty excessive. One member of Congress said, I want to start defunding these universities in the rot in higher education. Could you talk about what it would mean for our national security and our ability to have a lead in AI if we just started defunding MIT, University of Pennsylvania, Harvard, and many of these universities? Thank you for that landmine question. Ranking member Kana. I think it's a tough call. The environment of a university in order to be effective has to allow for everyone to be able to think freely. But I do agree that if we don't continue to fund STEM the way we need to fund STEM and continue to fund the technology in the way that we need to fund the technology at not just the top universities, but at all universities, then that's going to put us behind. So continuing to fund has to be the case. I yield back. General Skinner, are you a gamer? No, sir, I'm not. No Madden, no, like EA Sports, Call of Duty. My son is, but I'm not. I'm not either. But it seems as though a lot of our service members are. I mean, I've seen a lot of these USOs. They're building all these great gaming complexes. And I took interest in a report that we were feeding some of the Starcraft II game models into an integration with chat GPT. And this piece of AI surpasses humans, US military chat GPT outperforms in war scenario planning. And I guess what they did in this case was they told chat GPT to function as an assistant to a military commander in this engagement. So I imagine like an AI Dwight Schrute assistant to the regional manager, but it turns out in changing conflict dynamics, the military assistant turned out to be quite capable. So Dr. Martell, you seem to have some familiarity with this circumstance. What can we learn from the integration of gaming models, AI and military strategy? So I'll talk less about gaming because I'm not a gamer, but about how we might effectively use tools like generative AI like chat GPT. We've been working really hard to figure out where and when generative AI is gonna be useful and where and when it's gonna be dangerous. The danger is it's extremely difficult. It takes a very high cognitive load to validate the output of this model. And so there's a very large demand signal for AI to replace experts and allow novices to replace experts. That's where I think it's dangerous. Where I think it's gonna be most effective is helping experts be better experts or helping someone who knows their job well be better at their job that they know well. I don't know, Dr. Martell. We're all in the front end of this wave, but I find a lot of novices showing capability as experts when they're able to access these large language models. So if I can answer, I think the reason is it's extremely difficult to validate the output, right? So as long as there's a way, I'm totally on board as long as there's a way to easily check the output of the model because hallucination hasn't gone away yet. There's lots of hope that hallucination will go away. There's some research that says it won't ever go away. That's an empirical open question I think we need to really continue to pay attention to. But most importantly, if it's a difficult to validate output, then it's gonna be, then I'm very uncomfortable with it. Yeah, I mean, I've even used just in my modest way, like Claude to audit the hallucinations of Chad GPT and vice versa. I can see the young people behind you nodding in the affirmative. So sometimes that can be a check. And I think you're right on the outputs. I think you're dead there, but I think it's dead right. I think it's interesting though to think about this in terms of an assistant. And then you think about that in the air domain and the space domain. But I wanna get to the inputs as well because you made a really good point in your testimony about the quality of the data dictating kind of the ceiling on this enterprise. And I envision a circumstance where we're in this room marking up the NDAA and there's a big fight among all the lobbyists for the defense contractors about who owns the data, right? And just as we see in the large language models, the New York Times and these entities saying, well, you trained on our data and our stuff. And so we have some ownership interest in the work product that comes out. What, as you depart government service onto something else, what advice can you give the subcommittee about how to have as much of that data open source and acceptable so we don't have a circumstance where Lockheed Martin is saying, well, we have to protect our source code on the F-35 and back to that circumstance. So I think the right tactic there is to separate stovepipe solutions, which really are data all the way up to the end user into two layers, a data layer and an application layer. And then create two separate marketplaces. So the app marketplace makes perfect sense. You guys get it. Who's going to open my word doc? Is it gonna be Google or is it gonna be Microsoft or some third party? That's the app layer. But then the actual word processing doc is the data layer. But you can also build a marketplace in the data layer. If Lockheed Martin had invested a great deal of IP and work to building out that data layer, well, maybe we pay them for access to that data and figure out interesting contracting ways where they can actually make money selling that data, not just to themselves but to other app builders, but require that data to be accessible. So I really think that's- That's a really important point. I hope we're able to get back to it at some point because what I worry about is they'll create the cost of that data as so cost prohibitive so as to vertically integrate all the features of the contract. And Mr. Sherman, who testified about contract sprawl earlier, I think is acknowledging that we're gonna have to confront that. And I think it's a tricky wicket. I think that's right. Thank you, Chairman, I yield back. Thank you, Mr. Chairman. Mr. Sherman, I co-led the legislation to establish 98th and New National Mental Health Hotline. And in the first year and a half, calls are up over 50%, texts are up over 1,000%, literally saving thousands of lives. The commandant called me last year to share what he's heard from Marines about what a difference it's made in their lives and their units. I know the leadership of the department is working on reducing our shocking number of suicides. And one straightforward way to help is to simply make sure that 988 is dialable from all DOD phones. It's a legal requirement that you can dial 911, but not yet 988. So what can we do to make sure we get there? Congressman First, thank you for sponsoring that legislation. As Secretary Olson has said, mental health is health. And this is imperative. So within the Department of Defense, we have validated its use in the Pentagon, Fort Meier. We even called it from my office to make sure, because I'm gonna be honest, the first few times we were doing this, to make sure you don't have to put a nine in front of it or something like that to make sure it could work. So working with General Skinner with his DISA hat on and also personnel and readiness, making sure that that guidance is out there, we're gonna continue to validate. And I wanna make sure I'm clear on this. This is gonna be an ongoing process for whether it's at Fort Cavassos or Camp Pendleton or Edwards Air Force Base, wherever a service member or civilian is that needs to be able to get to that number, that we're gonna have to continually validate this. So this is something we've been talking about just recently here, that this is gonna be an ongoing process to make sure that 988 is reachable without having to put one one in front of it or nine or things that I would have had to do at Fort Stewart calling from the day room down at 1588. So sir, this is something that has got our attention for large and small installations alike to make sure we can validate that. Once the troops know this number by heart, and we've done a lot to make sure it's publicized on bases, it's gonna be important to have it accessible from places like Okinawa and Longstool as well. So I hope you'll keep your horizons for us. Dr. Martel, thank you very much for your time with the department. We all clearly agree that we need to invest in AI capabilities for the future. And we also understand, as you said, that AI is dependent on data. It's kind of insane to me that American taxpayers have spent more money on the J35 than any defense system in history. And yet we download its data from many missions and literally throw it out because we don't have a place to store it. When is this gonna stop? I'm getting all the great questions, aren't I? That is a great question and I appreciate it. We call that data exhaust, where valuable data is being blown out the tailpipe and there's an example of it. I don't feel that I have the expertise to answer that question when it's gonna stop, but I can say that we have to continue the demand for quality data to deliver solutions. And I believe that virtuous cycle is gonna then start creating the demand to understand how that data can be most effective. Well, I would just point out that if my laptop or my phone told me that I was out of space, I could solve that problem in a couple hours. So I don't expect DoD to move that fast, but the fact that this has taken literally years and you still can't give us an answer, I think is a real problem. I'll have to take the rest of that as a question for the record and get back to you. I'm not sufficiently familiar with it. Okay, well, it would be very helpful to have a timeline. And then of course, once we simply retain that data, you've addressed this and some of the other questions members have asked, how are we gonna make it safely available, not just to the primes, right? But to some of the leading private sector companies and AI innovation. So I think there's two parts to that, the safely and the available. From the available perspective, the technology is there. It's the right APIs that have the right limitations of accessibility on them. But it's more about the contracting to make sure that the folks who have produced the data, I mean, the data is fundamentally the government's data, but then a lot of work goes into massaging that data to make it accessible. And that is IP that should be paid for, in my opinion. But we have to make the contracts clear about what that payment looks like and how it must maintain its accessibility. That's a really big set of work that we have to do. On the safely accessible part, I defer to my colleagues on how we can use Zero Trust to make sure that only the right people are accessing that. Well, look, I would just point out that because of civil military fusion in China, China's doing this today, right? I mean, they have their top AI companies working on their military problems. So this is another thing where we've got to have the urgency to get this done. Quick question on workforce recruitment, Mr. Sherman, how are we gonna find someone to replace Dr. Martel who brings the level of academic expertise and private sector experience to the job? Well, we're certainly gonna miss Dr. Martel. He's been a huge combat multiplier. I think Dr. Rada Plum, his designated replacement, it's pretty darn amazing too. And I think she's gonna pick up the baton where he's leaving it there, sir. All right, thank you, Mr. Chairman.  That puts me kind of in my comfort zone that somebody's coming in behind you with some expertise to keep widening the road that you've paved.  So I'm sure the expansiveness of that is growing, yeah? You lead the data integration layer for CJAT-C2, correct? This is likely, in my opinion, the most critical position in that department. Given its centrality in moving and exchanging data between and within each services, this likely requires not just coordination but direction to the various stakeholders. Can you explain what directive authority you've been afforded in building the data integration layer? Can I? That's a great question. In terms of the authorities, I can tell you what we've been doing and then I can take as a question for the record what authorities we've been doing them under. Okay. But right now we've been building out the prototype of what it would mean for the hardware to support the flow of data across combatant commands. So combat, for example, so combatant commands can have a unified picture of what's going on in the world. We're doing that as the key learning exercise and we do it every 90 days through these guide exercises as the key learning exercise to understand what combatant and through wargaming what combatant commanders would need to see and what all of the components under the combatant commanders would need to see, would need to exchange and how data would need to flow in order for it to go from swivel chair and PowerPoint and email to digital data flows as that information goes across the combatant commands and within the combatant commands. So the next step which we should be doing within the next three to six months is building out a set of requirements so that industry can help join. I mean, we've been doing this with key industrial partners, mostly Palantir and Andral, but it needs to be, it needs to, and the technology that we've built, we're leaving behind, it's there, it's available, that's what we call it a minimum viable capability, it's viable. We then need to build out clear requirements that allow other industrial partners to join in and to expand that data integration layer and then also to expand those capabilities. It sounds like something, we might need to tweak this a little bit as you're starting to transition to something that you wish you would have had that the incoming can have, that's something you need to address with this committee, I would say. Absolutely, and I would prefer to take it as a question for the record and get you an absolutely clean definition of what we need. Mr. Chairman, are we doing two rounds or just one? Okay, I could use about four or five. All right, I'm Dr. Marder. Can you explain what would be the consequences of affording your position with some sort of directive authority over the services and their development of CJSC2 solutions? I'm not a fan of that and I'm not a fan of a hard authority there and I may be saying the wrong words because I haven't quite got up in the two years I've been here and all the bureaucratic lingo I should. I think the services understand in general, let me just say in general, the center should provide oversight and policy and best practices, the edge knows their problems best. So solving the problems from the center and imposing it upon the edge, I think is dangerous. It's gonna create one size fits all solutions that don't. You should write that down and put it on a plaque so we can hang it in every room in this building. I'll take that as an activity for the record, so. Okay. That was a joke, we're not really gonna do that, guys. And the services understand what they need better than OSD is gonna understand what they need but there does have to be authority about the interface. So data's gonna have to go down into the services and the right kind of data's gonna have to flow up out of the services. So where we've been spending our energy with the services say project convergence or with overmatch is figuring out how would that data flow? We still have the policy questions or the workflow questions about what data should flow. So I actually see what we've been doing to be clear is putting the tech before the policy. The tech to allow the data to flow is gonna force the right questions. What data should flow? We can't answer title 10 and title 50 combinations at priori, we can't just start with that. We have to watch the data flow and then there's gonna be an increased demand for data to flow and the right data to flow and then we can say well now that's a policy issue that we can tackle or that's a small version of the policy issue for just that one piece of data. That's the way that the change is gonna happen. The change is not gonna happen, my very strong opinion. The change is not gonna happen by some large a priori view of philosophical view of the way the world should be and then trying to implement that. Okay thanks sir. Thank you Cormac. Thank you Mr. Chair. I find it fascinating. We just had our first AI task force meeting just two floors away from us. Literally right in line with us and there's so much focus on this technology right now. It's just unbelievable from every aspect of whether it be cybersecurity, defense, mechanical applications, labor force, education, law and what do we consider illegal and how do you verify facts and then of course our security. This is such a comprehensive issue right now that everybody's focused on right now. I'm just gonna kind of dive on some more broad scope so you can give me some more insight to how you're approaching it. Dr. Martell, much like the computer power and data storage capacity, artificial intelligence advances at an increased rate right now, obviously exponential rate. What challenges does this rate of acceleration, in other words, as we're trying to keep up legislatively and regulatory wise, what does this acceleration create for integrating data, artificial intelligence and digital solutions for the DOD and how it applies, like just trying to keep up with this pace? So it is remarkable and I will say in my 30 year career this is the most exciting time to be an AI expert. It's the science has been really remarkable even just the last five years but if you go back 15 years it's been amazing as well. I don't know if the science supports the marketing though so I think one is to be... Skeptical is the wrong word. I'm very bullish on this technology but I think it's important to be skeptical about or to verify, trust but verify, if I can quote someone, the claims of the marketing, so that's one. Number two, I think it's extremely important to not see this as a monolithic technology which is how it's sold. Get this thing, if we have it we win and if they have it we lose. I think that's fundamentally flawed. It is neither a panacea nor is it a Pandora's box. So if we have it, it doesn't mean everything, every problem we want solved will be solved and if they have it, it doesn't mean every fear that we have will come to pass. We always have to think about it on a use case by use case basis as we always had with any technology. So the reassurance is it's just a technology, it will work in some use cases, it won't work in other use cases, we have to have quality data and evaluation metrics to be able to say it's working now or it's not working now. So I get extremely frustrated with my colleagues in industry who pitch it as magic. I actually think that's the case often but then when you ask them about the use cases, they have a clear set of small use cases that they know how to measure and are effective. Information retrieval via RAG or first draft generation or a number of things which we all feel fairly confident are working well. But the sort of pitch that it's gonna reason for us, it's gonna solve our problems, we're gonna be able to do away with brains and use this instead, that's just way over hyped. Now last year, I think that hype was greater, I think we've all learned in the last year where that hype is starting to come down and that's making me very happy. So you mentioned a couple of things, claims of marketing and also maybe, I would dispute one thing that you said because we talk about them having it, us not having it. When the country literally talks about taking over an island that produces basically 100% of our AI chips and we don't have a backup, that's kind of a denial one from the other and it's a different than what you were discussing. I understand. And I'm 100%, the underlying hardware, for example the CHIPS Act is the right direction to go because we need to be, that I think is a national security issue, the ability to generate the underlying infrastructure and the underlying data that's gonna allow this to be successful. I just meant the models themselves. We have a gap between the science and the marketing. And one of the things our organization has been doing in Task Force Lima is trying to rationalize that gap. We're building what we're calling a maturity model very similar to the autonomous driving maturity model. So level one through level five autonomous driving, that's a really useful model because people have claimed level five, but objectively speaking, we're really at level three with a couple folks doing some level four stuff. As an ER doc, I'm all for kids not driving in cars, figuring out how to do it safely, but I will say from a military aspect in my military experience, I'm very concerned about the kill chain and what it could mean to us when we have disruptive Chinese technologies that are literally competing with us at this cybersecurity level that really worries me the most. I'm gonna lie. Sir, if I could respond, I'm 100% in agreement. So I'm not saying we don't have to worry about specific use cases. We do have to worry about specific use cases. We just shouldn't worry that it's monolithic. So in particular use cases, we should do what we always do, evaluate what our adversaries can do, evaluate what we can do, invest in ways that we can do it better and invest in ways in which we can maybe contribute to our adversaries not doing it so well. I just, my only caution was, don't think about it as a scary monolithic technology, think about particular use cases and then do the right evaluation of where they are and where we are and invest accordingly. Thank you, Mr. Chairman. I knew it was gonna be a good day when I walked in and I saw so many Air Force folks here, being a former Air Force. Dr. Martell, what concerns me about a conflict with a near peer and obviously talking about China and Taiwan is, and it would concern everybody in the committee and everybody on all the witnesses is the tremendous potential loss of life. Another thing that really bothers me is when you're looking at war games where there's some scenarios where we lose in just a very short time frame, a thousand fighter aircraft, that's devastating and to avoid that kind of a catastrophic loss, I think we need to employ more unmanned aircraft, most notably, of course, autonomous platforms for not great strike purposes for system as well. To do this, the DOE needs to not only invest in hardware but of course software, probably even more importantly, the software. So that exits in the commercial space but companies like Amazon don't have the requirement to operate in GPS denied environments like the military. So what the question I wanna ask you is, what do you think the significance of autonomous unmanned air platforms will have on helping us win a future near peer fight and are we doing enough to get where we need to be? I completely agree with you, I think they're gonna be extremely important and I think we have to continue to drive that forward. One of the things the CDAO is doing is supporting the Replicator Initiative by creating a sandbox, if you will, where companies can come using our data to evaluate the success of their unmanned aircraft software with respect to the scenarios that we think are gonna be effective. So we're strongly supporting this, part of what we call AI scaffolding that we're building out, part of this way in which we're trying to help scale the department by giving tools to folks closer to the edge, is this Replicator, this Replicator Sandbox. And so we'll provide data to that, we'll provide scenarios. I'm pretty sure I'll double check that we're gonna provide some software for that and then the folks who are part of the Replicator Initiative will come and try out their software, their prediction, their detection on our data. So we think it's really important and we're strongly supporting it. Can you talk a little bit more about that? You mentioned the AI scaffolding. So as I mentioned in my opening remarks, we really see our job as twofold. One is immediate help to the warfighter and building out tools, processes, policies, best practices that will allow other folks, the ones closer to the problem, to easily build the solution to the problem because a centralized team cannot build everything, right? That, I guess we talked about before, that creates a one-size-fits-all solution. Part of our support to be able to scale is what we're calling AI scaffolding. So AI scaffolding includes things like the ability to do data labeling as a service because knowing what you wanna detect means humans have to label that data. That's very difficult. Most teams closer to the edge won't know how to do that, so we wanna provide contracting and expertise to allow them to do that. Data transformation is a service. The data that has to come into your problem is gonna be in a thousand different formats. The amount of work to transform a PDF, a Word doc, et cetera, et cetera, et cetera, into structured data that you need is pretty massive and that'll take up for the folks on the edge all of their time. We wanna be able to provide services like that. The building of the model, we strongly believe that should be, because our IP is the data, the government's IP is the data, we strongly believe that the building of the model should be contracted to industry, but on the other side, and we'll provide those tools, those contacts, those ability to build initial models yourself, that's part of our AI scaffolding. The final model should probably be built by absolute experts in this field. And then on the right side of the model building, we think very hard about model monitoring. So one of I think the biggest issues that industry just started tackling, I'd say five years ago, five to 10 years ago, and government's behind on is after we ship a model, does it continue to bring value? Well, remember AI is statistics at scale. We gather data from the past to predict the future. In a war fighting scenario, the future changes. It doesn't look like the past. So a model trained on pre-war fighting scenario is going to start to degrade as the world changes in a fight. How do we measure that degradation? How do we retrain to get that model back up to where we need it to be? So that's the model monitoring piece and I wish we had gotten further on that in the two years I've been here, but that's gonna be my biggest charge for Dr. Plum is really focus on how do we model and measure the value of models over time? Right now, and it was this way in industry five years ago, they're shipped and we forget. They shipped and we continue to believe. We don't forget, we just continue to believe. So our contracting, for example, should include model monitoring. The way we deliver the model should include the ability to monitor it. And so we've done a lot of thinking about this. We need to move out faster on actually executing on it. Thank you, Mr. Chairman, I'll go back. Great. We will now go into the classified session, which will be upstairs in two, three, three, seven. Rayburn, I know there were a variety of questions that were not answered because of classification, so we look forward to getting into those and we'll see you up there in a few minutes. With that, the unclassified portion of the journey.


What's up everybody? Jason here for jasperjasecasebrakes.com. 2023 hip parade, high end 10 box football mixer just sold out this time. It's number two random teams. And again guys, five VIP boxes, five one of a kind. And to my knowledge guys, we kind of went through the sell sheet on the hip parades website, did not hit any of these and all the other ones that popped up. So should have something really, really nice on this side. And again, it's five boxes of 2023 VIP series three, treasure hunt and then five boxes of one of a kind edition series one 2023 football. Now it's just between two customers guys, Michael and Chris. Good luck to the both of you guys. I hope I get something for you guys, both. And then of course, all teams in, there you go. Dice roll it. It is a five and a five, 10 times. Good luck. One, two, three, four, five, six, seven, eight, nine, 10. There we go. Five, five, 10. 10, Michael down to Chris. And 10 times here now on the list of teams. One, two, three, four, five, six, seven, eight, nine, 10. 10 times, 10, 10. Titans down to the Carolina Panthers. Let's see how it played out. So Mike, we have the Titans. Chris with the Giants. Michael with the Cardinals and Bucks. Chris with the Dolphins. Michael with the Steelers. Chris with the Broncos and Seahawks. Michael with the Bears. Chris with the Texans, Chargers, Lions. Michael with the Cowboys and the Commanders. Chris with the Los Angeles Rams. My Eagles, Jaguars, Jets, Colts. Michael with the Falcons. Chris with the Ravens. Michael with the Browns, Saints. Chris with the Chiefs, 49ers. Patriots. And then Michael with the Packers. And then Chris with the Raiders, Bills. Michael with the Vikings. And then Chris with the Bengals, Panthers. And now again, like I mentioned before, guys, in the first break, group of Jets didn't have the last couple of years. And like I said, actually the person that's been running it actually hasn't updated anything in a while. So hopefully he's okay. But for that fall, it's collegiate. So it's officially, there is no checklist for that. NFL-wise, through the distributor, it's gonna go to the team that he's currently on, teams play for Longs, college rules. So of course at Tom Brady, since he's not in the league anymore, we'll be going to the, to the Patriots. But obviously there's no guarantee that's in here, but just like I said. Just like I did in the first one. All right, well, any trades between you guys? All good, okay, just making sure. Let's do it though. Oh, well, actually, I will still allow it for you. Pittsburgh, do you want to trade Pittsburgh, Michael? I'll still give it for you guys, since it's all you two owners. Michael says he'll keep. Okay. All good in the neighborhood. All right, here we go guys, just have some fun. Oh wait, hold on, another one? Browser Prankles. Okay, that's a joke. All right, here we go guys, good luck. All right, so we'll do the one of a kind again first, just because like I said, I mean VIP technically is a higher end product. And then we'll hand it off with the VIP. Good luck guys, here we go. Nice way to end a Saturday night. Been doing a lot of tops Chrome, basically 20 cases, 10 of each. My fingers kind of love this product today. So here we go. First one. Wow, keeping the Steelers. How about a George Pickens NFL Shield? One of one. And I don't think this was on the sell sheet. At least not that I've seen it on the one of a kind. Was it? Let me make sure, let me pull it up. I mean, it's crazy that this isn't even on the sell sheet. Just to let you know how loaded this is, no. I don't think it's on the sell sheet. Nope. Wow, not even on the sell sheet. TJ Watt, one of one was on the sell sheet. Well, there you go. Pickens NFL Shield, one of one from Phoenix football. And try to trade, well, not try to trade. Carl tried to trade for them. And it's a Trayvon Walker Immaculate RPA, Nike Swoosh, one of one, for the Jacksonville Jaguars. Last one, Mojo Chris, nice. Jaguars should be a lot better this year on offense and hopefully a little bit better on defense. Number one overall pick right there. Nice, two nice rookies right there from the last year's drop class. Whoa, look at that. You know what this means. You just pulled a big hit from hip parade. We'll send you a $100 gift code if you just prove it that you have it. 2021, clearly Donner's Peyton Manning, one of one downtown. That is beautiful. And that is he graded a PSA nine. Look at that, one of one. And that is a hit right there. And there you go. Peyton Manning, Indianapolis Colts going to Chris. So actually let's take this to this. And just show hip parade that that's your big hit Chris. So then they'll send you a hundred bucks and two news on their website for hip parade. 66 as well. Well, we found the big hit in the one on one. Let's see what's hiding in the VIP now after these two. Now we're going to roll there. There could still be some other big one on ones in here. That one is the big sell sheet, I should say. Looks like from a peony rewards. It is a one of one, Josh Jacobs for the Redis. 2021 Midnight Signatures. Vegas Raiders going to Chris, look at Josh Jacobs. Big 10 running back there. Although I think he's not reporting the training camp, right? Hasn't signed that extension and hasn't signed his franchise tag, I think, unless he did. I don't say it went back, but he kind of renegotiated it in the contract. Still one year deal though. Last one here. Ooh, it's from, this is actually black gold. Michael Irving, one of one from black gold football. Back in what, 2014, 2015? Yeah. Nice for the Cowboys. Michael, there you go. Nice one. Got two hits right here, buddy. Pretty solid. Let's go to the big boy stuff here. Pete, Pete, Pete, Pete. A lot of Jalen Hertz patch autograph. Two-color patch and autograph to 49. That is from 2022 Immaculate Football. Going to my Eagles and Chris. Next one. So I don't think they would put another big Jalen Hertz in here. At least not in this case. So I think maybe the Jalen Hertz sell sheet hit might be gone. Potentially. Got a 2018 Panini one. Platinum. Terrell Davis. Nice. Five out of five. Grated a nine, five, 10 for the Denver Broncos and Chris. Solid. Three out of three Joe Montania. That is for the San Francisco 49ers. Going to Chris. Nice. It's the victory there against the Falcons. Stats right there. All right, Dolce & Miles, two more. Then we got a two out of three Dalvin Cook. That looks like an RPA from 2017. Yeah, this is a little hats though. For the Minnesota Vikings. Going to Michael. Obviously not on the Vikings anymore. And potentially maybe a jet. I think he's visiting the Jets on Sunday. Is this a dud case or something? Let's see if I missed anything. I don't think I missed anything though, do I? Oh no. No, no, no. It's gotta be here, right? Unless they made a case with that one. Either way though, there's been some really, really nice stuff though out of this VIP case. Usually. There is one in here. Oh, it's in here. Patrick Mahomes. 2018 Panini 1. Jersey Patch Autograph to 49. Graded at PSA 9 too. 14 out of 49 for the Kansas City Chiefs. Kansas City. Chris, there you go buddy, you got it. Add a little Mahomes to your collection right there. Nice. Big hair right there, buddy. So there you go. So if we ever do get any more of this Series 3 VIP, although it is sold out, we hit the Patrick Mahomes. So out of the cell sheet hits guys, we hit the, we hit the Panini 101. We hit the Patrick Mahomes. And of course amongst others, a lot of really, really, really nice hits. So there you go. And now I'm able to get hits out there for both of you guys. So that was kind of the cool thing for me. So I appreciate it guys.  That was the break. 2023 hip-rate high-end football mixer number two.


Thanks for showing up early in the morning for the first talk after breakfast and at least I'm not between you and lunch today. This is the talk where I have to be very careful because I'm sort of biting the hand that feeds me. But it is my favorite talk that I'm giving this Mises you. And it's one that I think would be of interest to pretty much everybody here because you're all in the middle of this higher ed thing or you're about to be. Some of you are about to graduate from high school or thinking about college and university and so forth or grad school. And so it's an interesting time in higher ed. A lot of changes going on, not all for the better. Some for the good. I think there's some interesting and hopeful things going on as well. If you take a look at the slide here. So you'll notice all the red lines here, which are the categories of goods and services that have gone up faster than general inflation over the last 22 years. And what do you notice about the ones that are highlighted here? What do they have in common? Yeah. So we've got housing. We've got childcare, nursery school, medical care services, college textbooks and college tuition and hospital services. So these are all goods and services that government has heavily subsidized or otherwise involved itself heavily in compared to the goods that have gotten relatively cheaper, things like cell phone services, household furnishings, clothing and computer software, toys, that sort of thing. So I talked yesterday about medical care and I mentioned some of the problems with government involvement in medical care. And here we have another case where government has involved itself very heavily, ostensibly to help students by making education cheaper. That's the narrative. We'll see we're going to help students out by subsidizing higher ed, make it easier for people to go to college. And then what do we have? Higher costs. And so these are the three crises that I'd like to talk about briefly here today. First of all, college does not do what it advertises. You're told everybody needs to go to college because it will put you in a better position to get a good job. And in fact, we see that college often does not do that. Second, the government subsidies, which are massive, are backfiring. And then third, we have a problem of stifled academic discourse, which you're probably personally aware of. So let's look at the first one first. So what is it that college does? I mean, when I went into higher education, I had this idea that what I was doing is I was instilling knowledge in the heads of students that was going to make them better off when they eventually went on to careers and that they would remember everything that I had said and that they would use this to great effect in their lives. And I guess I still think that to some extent, but being a little older and more jaded and cynical, I guess, I'm starting to investigate some other possibilities. And by the time I retire, the timing will be just about right where I'll be so fully cynical that I'll be ready to leave. But what is it that college is doing? Is it instilling human capital, as we refer to this knowledge and skill, or is it maybe a signal of the student's work ethic or their disposition toward higher order thinking? Or is it just basically an opportunity to party for four years or five or six years? Is it a consumption good for people who are higher or middle income, who can afford to go to school for that period of time, often on the taxpayer dime or their parents or some other source of funds, not usually paying for it immediately. Maybe they borrow a lot of money with student loans and they use this to enjoy life for a while. So the narrative that has been used to advertise for college education for a long time is that if you get a college degree, then you're going to be, you're going to have the ticket to at least a middle class income, if not better, and if you look at the probability of being in an educational group for a given income level, it does seem that that's what's going on. I mean, if you have an income of $150,000 or more, then the chances are that you are a college graduate. The vast majority of people with those incomes do have college degrees. And then many of the rest have some college, at least, under their belt, whereas lower incomes on the left end are much more likely to have only a high school diploma or maybe not have even graduated from high school. And so this is the story. You need to go to college because that'll put you in a higher income group and that's what we would like to have, most of us. And so that's the story. Unfortunately, what this is doing, this kind of statistic, is it's lumping in people who went to college way back when with students who are alumni who have just graduated in the last few years. And it's just possible that college has changed over that period of time. So again, this, by the way, this chart or table as well as some of the other data that I'll show you are from Richard Vedder's excellent recent book, Restoring the Promise, which is an analysis of the problems with higher ed. And I recently wrote a review of Vedder's book in the quarterly journal of Austrian economics, which you can take a look at if you want to know more about what he's been saying. And he's been saying this for decades. But basically, if you look at the median income by educational attainment, you can say, well, if you've got a bachelor's degree or master's doctor's professional degree, then you're going to be middle income at least. And if you've got, say, just a high school diploma, median income is going to be a lot lower. So again, that's the story. And it looks, again, if you look at the entire range of college graduates, it looks like the return on investment is high. I mean, associate's degrees, bachelor's degrees seem to do better than the stock market seem to do better than a lot of other alternative investments. However, we are seeing some disturbing developments in the last one or two decades, at least, one of which is that the education requirements of the jobs that people are actually doing often are less than the educational attainment that the person actually has. So in other words, you've got people with college degrees that are doing jobs that don't really require a college degree. You've got people with four-year degrees that are waiting tables or they're baristas or whatever, and did you really need a college degree in order to figure out how to make a latte? And so in a sense, we're over-investing in college education, preparing people for careers that didn't really require that. Now, why is that happening? If you're hiring for Starbucks and you're looking at a variety of applications and you see some people that didn't finish high school and other people that have a four-year degree, you're going to use that information as a way to kind of gauge the level of, I don't know, competence or the drive that that person has. It takes a lot to finish a four-year degree, and so you figure, well, I've got multiple applications in front of me. Maybe I'll pick the person that is evidently the one with the most drive here. So that's partly what's driving this. But we've got a lot of people who are college graduates who are taking jobs that they're really over-qualified in terms of their formal education. So you've got, for example, this is comparing 1970 to 2010, you have taxi drivers and chauffeurs who in 1970 very rarely had a college degree. And by 2010, 15% of taxi drivers and chauffeurs had a college degree. Obviously not one of those occupations where, in respect to taxi drivers, I mean, I'm very glad we have people to do that, but you don't need a four-year degree to do it. Same thing with a lot of other occupations like shipping and receiving clerks, carpenters, bank tellers, firefighters, and so forth. So what is it that college education is really doing? Is it the instilling of knowledge that's desperately needed for the job that you're going to wind up doing later on? Maybe not. So let's think about college education as a signaling process. So you've got the population here, and you've got people who are blue types and people who are red types, and I don't mean Democrat-Republican, I just mean people who have motivation, perseverance, and intelligence. I suppose the blue dots are people with a lot of motivation, perseverance, and intelligence, and the red dots are people that don't have a lot of those characteristics. So you've got that general population, and we've created this kind of filter called a university, and you can take yourself through that filter, and the filter will filter out people who don't have motivation, perseverance, and intelligence. And so if you pass the filter, then you're able to say to an employer, look, I've got this sheepskin here that says I'm motivated, persevering, and intelligent. It almost doesn't matter what the degree is. If you've got that degree, it shows that you've got those underlying characteristics. Now, it may matter to some extent. If you get a four-year degree in chemical engineering, you're probably demonstrating more motivation, perseverance, and intelligence than if you get a degree in, I'm going to say out of trouble, something else. But as an employer, you recognize this. Now, this is why Wall Street firms like to hire physics majors, not because physics is particularly relevant to the financial industry, but because if you've got a physics degree, everybody knows that's tough. And you must have a lot of motivation, perseverance, and intelligence. You succeed in getting a physics degree, especially if you get a decent GPA on your transcript. So you've passed the filter. And then there are people who drop out. Was it Bill Gates or was it Steve Jobs? I can't remember which one dropped out of Harvard. So dropping out is not necessarily a bad thing to do, right? If you've got better opportunities, by all means. Take advantage of those opportunities. But you've got some people that fail or drop out and others that just say, I'm not going to even go through the filter. And so you get a mix of people who did not go to college who either didn't have motivation, perseverance, and intelligence or they do, but they decided not to put themselves through that filter process, save themselves a lot of student loans and so forth. There are plenty of motivated, persevering, and intelligent people who decide not to go to college. But if you're the employer and you're looking at these two groups of people, some of which graduated, others who did not, you're more likely to get somebody who's motivated, persevering, and intelligent if they've passed the filter. So it's just a probabilities thing. Now, what happens if you get government involved in this filtering process and government says, well, we're going to tax the general population, then we're going to subsidize people who decide to go through the filter? Well, now you get a lot of people who want to go through the filter because it looks relatively less expensive. And so you get the filter expands.  They shove a lot more students through because each student arriving or each potential student arriving in the admissions office applying to the college now has a suitcase full of government cash associated with them. The only way the college or university can get that cash is by accepting the student and keeping the student there retention, keeping the student there through four years to collect the maximum amount of cash from the taxpayer. And so the motivations of the college or university are such that they want to expand this process and the filter becomes, as you might think, more porous. More people are passing through because the college or university has an incentive to pass them through the filter. So Brian Kaplan wrote a book recently called Case Against Education where he made a point similar to this. So he says, once workers have been ranked, giving everyone extra years of education is socially wasteful. Furthermore, since the status quo supported by hundreds of billions of dollars of subsidies, we're probably underusing alternative certification methods like apprenticeships, testing, boot camps, and so forth. Signaling explains why students are far more concerned about grades than actual learning. So again, if you've got a four-year degree, you're saying, I'm signaling, I've got motivation, perseverance, and intelligence. That's a less effective signal today because, as I said, the filters become more porous and it's passing people through because the university can collect cash along the way from these government subsidies. And so you get a number of people who've passed the filter who actually aren't motivated in persevering in intelligence. I should come up with an acronym, MPI or something. Anyway, so what does the student want? They want an easy A. They want to pass the filter. They don't really care about gathering knowledge along the way. They want to pass the filter. They want the A. They don't want professors necessarily who teach a lot of job skills, explains why cheating.  And he says, signaling explains why students readily forget course material the day after the final exam. Once you've got the good signal in your transcript, you can usually safely forget whatever you learned. And so he makes this argument, I won't go through all of this, but he says basically we need to have less education because we have dumped a lot of people into this filtering process that personally would be better off if we didn't have so many people trying to get through the filter. So this is part of the problem that we have so heavily subsidized education that we are getting a lot of students going through the process that are uninterested in the knowledge that college ostensibly can transfer to students. I think it also helps us understand why we get a lot of plagiarism. There is an epidemic of plagiarism in the United States, and I think in some other countries as well. It's a manifestation of that incentive to cheat. You're trying to mimic someone else who actually did the work. And I've seen enough of this kind of problem grading papers that it, and I've been teaching for 24, 25 years, and it's more of a problem than it used to be. Part of that is because it's easier to cheat in some ways. You've got a lot more stuff that's available. I mean, there's dozens of websites that are like cheating assistance sites, and it's easy for students to buy a paper and submit it. It's also easier for professors to catch that with tools like Turnitin and that kind of thing, but it's still a massive problem. And so what's the incentive for administrators at a college or university? They have the incentive to just vigorously prosecute plagiarism cases, knowing that if they really do take this seriously and impose all the penalties that are in the book, the student handbook or whatever, on cheating that that might mean that the student leaves and takes their government subsidies with them, maybe goes to some other college or university. I mean, is that really what an administrator wants to do? I mean, they're going to talk a good game a lot of times. They're going to say, well, we're going to root out cheating and we're going to make sure everybody prosecutes this to the fullest extent and so forth, and we're going to talk to professors about how they need to root out cheating. But when the rubber meets the road, is that what they're really going to do? And furthermore, the faculty member recognizes that if they see cheating, it's going to be a hard slog through the bureaucratic process to make sure that they don't get themselves in trouble from some kind of false accusation or get the university in trouble and have some dean upset with them. I mean, it's the incentive for a faculty member is to just turn a blind eye and say, well, I didn't really see that and I'm just not going to bother with it. It's just too much trouble. It's going to take me hours to deal with the cheating case and it'll take me just a few minutes if I ignore it. And so really one of the driving factors, I mean, speaking for myself, one of the driving factors is just a sense of justice that I hate to see people mimicking hardworking students and getting credit for something they didn't do, but there are faculty that don't have that kind of, maybe they're more jaded than I am and they don't see the necessity of going after those cases.  You're not going to get that by just rigorously going after cheating and ticking off students who maybe have been passed through class after class having engaged in this cheating time and again and now all of a sudden they run up against this person that's going to go after it with a vengeance. That's not the way for a graduate teaching assistant to get any points that they care about. So, Jonathan Newman wrote a great little piece for Mises Weyer several years ago called Four Reasons Why College Degrees Are Becoming Useless. It's a great little article recommended highly and he says that some of the most prestigious flagship universities test results indicate the average graduate shows little or no improvement in critical thinking over four years. I have to say that's not surprising. At Yale College where 62% of grades are in the A range, proposals to curb grade inflation are in doubt following student protests and faculty concern. This is not from Newman's article, this is from another one on the massive problem we've got with grade inflation. Again, this is an incentive issue, right? So, the faculty member knows with any class if I give a student an A, that student's not going to complain.  There's something that maybe I get a complaint. Maybe I get a bad teaching evaluation. Maybe whatever the problem might be and it's easier for a faculty member to just say I'm just going to give that student an A and not have to worry about any kind of problem. I know that I'm not going to get a complaint. So, we get grade inflation, especially in an environment where students seem to have an increasing propensity to complain about faculty. I'll maybe get to that later. So, we see a lot of this kind of grade inflation. One national survey showed that 41% of students had grade point averages of A minus or higher in 2009 compared to just 7% in 1969. Another study from this year says that the increase in graduation rates over the last three decades is driven by grade inflation. Why is it that we see college completion rates improving? Is it because students are getting better? Is it because admissions departments are being better at kind of filtering out the students that would not succeed all the way through four years? Is that why we're seeing improving graduation rates? Or is it simply because faculty are giving more and more A's even though the student performance is not commensurate with those high grades? And that appeared actually in an AEA journal. It's gotten quite a bit of press as far as I could tell. This is from 1940 to 2012. Notice the red, that is the percentage of A's. Notice the F's haven't changed all that much. What has changed is the number of C's and to a lesser extent B's and D's. So we've got a lot of students who are getting A's who otherwise would have gotten a lower grade in previous decades. And students are spending less time on their work. So it's not like they're getting higher grades because they're working harder and they're learning more and so they're doing better on the test. No, actually they're spending less time studying. They're spending less time devoted to their education. So this study shows that from 1961 to 2003 the number of hours per week that an average student spent on studying on their academic work fell from about 40 hours to under 30 hours a week. And I'd suggest that since 2003 that has probably fallen further. I would be shocked if some of my students were spending 37 hours a week on their academic work based on what I'm seeing. Alright, second crisis, government subsidies are backfiring. As we know, we've got a lot of student loan debt. $1.75 trillion in student loan debt, an average of about $29,000 per borrower. 92% of that is federal, the remainder is private. There's a perception that a lot of student debt is held by people who are lower income. And there is proportional income that may be that there's a lot of student debt held by lower income individuals. But a lot of it is held by higher income people. I mean think about somebody who goes to law school, med school, dental school or whatever and they've got a higher income maybe, but they've also got a lot of student debt not only from their undergraduate education but from their graduate education. So 60% of student loan debt is actually held by people in the top 40% of household income. What impact has this had on colleges? Well, increasing subsidies and I mean, subsidies of student loans that make it easier for people to borrow for a college have led to higher tuition. So we're seeing colleges basically raising tuition in lockstep with the availability of funds for students. So is it really the students that are benefitting from these subsidized student loans or is it the colleges and universities who are now raking in more cash and able to raise their tuition because of these kinds of programs? This is a ratio of average tuition and fees for all four-year degree granting institutions compared to median household income. And I think Vedder's got something along these lines in his book as well but this shows basically that college has become more expensive relative to household income over time, especially in the last couple of decades. College is, if anything, less affordable even though governments have poured more and more money into education or maybe because governments have poured more and more money into education. And at the same time, we've got students that are graduating with loads of student debt and it has become a serious problem for a lot of people. Now a lot of the problems are for people that maybe borrowed money for two or three years of college and for whatever reason they dropped out of college and they don't have the college degree that signals I'm hardworking, persevering, and intelligent but they do have two to three years' worth of student loan debt that now will need to be repaid. So you don't have the income boost but you do have the student loan debt. And this is a pretty serious problem where you get some delinquencies on student loan debt and you can see the, let's see, it's the red line there that starts at about, what is that, about 6% or 7% here in 2004 and is up to, looks like about 12% or so by 2017. The single largest delinquent type of loan by 2017 whereas other types of loans have had declined and their delinquency rates in the wake of the 2008-2009 recession student loans jumped afterward. So the Bennett hypothesis named for William Bennett says that increases in financial aid in recent years have enabled colleges and universities blithely to raise their tuitions. And I cut out a lot of the slides that I had here with the data on the studies that have basically backed up the Bennett hypothesis over the years. I can share those sources later if you're interested but basically this is what we're seeing. And where is the money going? Is it going to improve the academic experience for students? Well, not exactly. Princeton University spent $136 million on a student dorm with leaded glass windows and a cavernous oak dining hall with a cost of about $300,000 per bed, okay? That's where your money's going. Since 2000, this is from Richard Vedder's book, actually from Wall Street Journal article from Vedder. Since 2000, NYU has provided $90 million in loans, many of them zero interest and forgivable to administrators and faculty to buy houses and summer homes on Fire Island in the Hamptons. I need to go teach at NYU. Former Ohio State President Gordon Geirn earned nearly $2 million in compensation last year while living in a 9600 square foot Tudor mansion. The Columbus Camelot includes $673,000 in art decor and a $532 shower curtain in a guest bathroom. Ohio State also paid roughly $23,000 per month for his soirees and half a million for him to travel the country on a private jet. The University of California system employs about 2,358 administrative staff only in the president's office. Vedder says 30% of the adult population has college degrees. The Department of Labor tells us that only 20% or so jobs are requiring college degrees and went through some of this before, but basically we're making college less affordable as you see with that ratio between tuition and household income. And the money is flowing toward the colleges and universities and some of this is to provide expensive dorms, some of this is for recreational facilities for students, some of this is for an administrative staff that is expanding dramatically. I mean, again, I've been in higher ed for about 25 years and middle management in higher ed has just exploded. I mean, the deanlets and the administrators and the vice assistant backup dean or something.  I know they're doing something. I mean, they go to their offices and they appear to be working, but somehow higher education managed to get more work out of students before a lot of this was the norm. All right, final crisis. Stifled academic discourse. So you may have heard of a few of these cases and more seem to be occurring all the time. Evergreen State College in Washington student protests hijacked classrooms and administration. Protesters took over the administration offices last month and have disrupted classes as well. The chaos at Evergreen resulted in anonymous threats of mass murder resulting in the campus being closed for three days. This is from Neumann's article, Jonathan Neumann's article that I mentioned earlier. A 2017 study of college students says that students, some students say, shouting down speakers and using violence is sometimes acceptable. We heard some of this from Tom DiLorenzo's talk. I think it was yesterday where he was discussing the other forms of socialism and he mentioned one of these cases. So what happens if you want to bring an alternative viewpoint in front of your students and have them think about it and say, okay, well, there's this view, but then there's also this one and this one and this one. We want you to be able to evaluate them and think about them. Even if you are predisposed to disagree and even if you come away from the discussion still disagreeing, you at least know more about what that other side thinks and that's generally helpful. You need to know what different views are saying. And yet the increasing tendency seems to be, well, if you even allow this other viewpoint to be heard, you're committing some kind of injustice. So many students who may have a more mature viewpoint and say, well, I can tolerate hearing something I disagree with if I can learn a little more about that point of view and be able to maybe argue more effectively against it next time, these students are squelched. They keep their thoughts to themselves. Maybe we're losing that healthy self-evaluation sometimes. If you hear an alternative viewpoint, maybe you were predisposed against it, but you hear the argument from someone who espouses that view and your mind has changed. Does it really do our society good when people are not confronted with views they disagree with and they're allowed to some erroneous thought to just fester in their heads and then maybe they act on that and do something that's socially quite damaging because they've never had to think through the view to its fullest extent. So we're getting this stifling of academic discourse. In one study of 1,500 U.S. undergraduates, fewer than half of the respondents believed that the First Amendment protects hate speech and hate speech has become a larger and larger category, it seems. A public university invites a very controversial speaker to an on-campus event. The speaker is known for making offensive and hurtful statements. A student group opposed to the speaker uses violence to prevent the speaker from speaking. Do you agree or disagree that the student group's actions are acceptable? 19% of the respondents stated that they agreed with the student group's actions. Now if I walk into a classroom and I'm thinking, should I mention something controversial today and I'm thinking one out of five students sitting there in front of me believes it's okay to use violence to stop views that they disagree with, that's gonna stifle academic discourse to some extent. You gotta be a really unusual sort of person to say, yeah, I'm willing to have students beat up on me after class or something, that's fine. I even have to be careful who I invite to campus. If I invite someone to campus who's got an opposing view, then I can be subjected to repercussions. So you've seen some of these things before. They're not unusual stories, unfortunately. This is something that I saw just last month. A study from a group called, let's see, the Chally Institute for Global Innovation and Growth. If a professor says something that students find offensive, should that professor be reported to the university? Now I'm less concerned here about the fact that there seems to be a disparity between what left-leaning students think about this and what right-leaning students think about this. I'm more concerned with the fact that pretty much everybody here is saying a majority of conservative students or conservative-leaning students are saying, yeah, we should report the faculty member to the administration for saying something that students find offensive. And what students are offended by is a longer and longer list. And I don't even know everything that's on the list. I'm trying to keep up, but it's harder to keep up with what students may find offensive. And so that, again, will cause faculty to self-censor. Maybe they're thinking about bringing up something in class that might be an interesting point for debate, but they play it safe. I like my job. I don't really want to rock the boat here. I don't want to wind up in the provost's office having to explain why I said X or Y or Z or what, you know. So, again, a lot of faculty will just behave that way. University bureaucracies are becoming more and more powerful and able to make life miserable for faculty, and they will sometimes take advantage of that. Part of this is because they've got a certain group of students that will say, you know, yeah, go get them. You know, fire that professor because that professor said something I disagree with. And even if the accusations are baseless, even if they're absurd on their face, they can still put the faculty member through long and stressful, traumatic kinds of experiences. I had a colleague of mine years ago who was put through something that she ended up with major medical problems as a result of this. Even though, you know, she ended up keeping her job and she was exonerated and so forth, but it was still such a stressful process that it damaged her health. So you've seen recent cases. Peter Bacassian, Jordan Peterson, Ilya Shapiro, even though in some cases like Shapiro was told, yeah, you can keep your job just, you know, be aware that you can't say certain things or you're going to wind up in trouble. And so sometimes even just the threat of sitting under the sword of Damocles of, you know, when's this going to come down on my head is squelching of academic debate. Murray Rothbard said some great things in ethics of liberty about free speech and I think they're worth paying attention to. He said, look, basically it's not like an absolute right of free speech. If I'm standing here and I'm speaking to you, it's because the Mises Institute said, yeah, we want you to stand here and give this talk. If I stood up here and I said, terrible things about Austrian economics, they'd say, well, you know, that's not really why we brought you here. We wanted you to explain what Austrian economics is about and not go on some Keynesian rant or something. I would expect that I would not be invited back. I haven't lost my right to free speech because of that. You know, I've got the right to decide who speaks at my dining room table.  So Rothbard says there's no such thing as a separate right to free speech. There is a property right. I have a property right to my dining room table. Any institution that's got a podium and a lectern has a property right to this space and they can use it however they want. But if a university says we're giving faculty the right to free speech, we're allowing faculty to say whatever they see is appropriate in their classroom and then the university says, well, we didn't mean that. Then there are some, you know, we're seeing a constriction of what was once a broader range of thought that could be taught from classrooms. And I don't think this serves students well. I don't think it serves our society well. I've got some, well, yeah, I can share these with you later. I don't expect you to write all this down right now. But some of the stuff that I've said, I've got some resources I can share with you later if you're interested. I think I'm out of time, though. Thank you very much. I appreciate it.