Good afternoon. We're going to go ahead and get started here. We're going to do this quick introduction. Originally we were supposed to have four of us and the other two hightailed it back to DC for some reason, so we're trying to figure that out. But I'm Jerry Dixon. I'm currently a director of analysis for Team Cumberie, a small security research team. Previous to that I was the executive director for the National Cybersecurity Division at Homeland. Basically that in ran USR for a period of time, so I wanted to lose a recovering DHS alumni. Basically we're going to touch on the cyber commission and I'm going to turn it over to Mark to introduce himself. Thank you Jerry. And we were going to do cardboard cutouts for the missing ones here. And just kind of put their little faces up here so if you took pictures because it's 2D it would look like we're all here. Speak up. How about if we do that? Is that better? Alright. So anyway, Mark Sox is my name. I run the Internet Storm Center. How many of you have that as your homepage? Awesome. Thank you. I see there's one reader out here. This is a good thing. I'm also a former DHS was there when we first launched. Prior to that I was 20 years in the military, retired back in 2001. I spent a couple of years at the White House working in the National Security Council. Early in the current administration helped create Homeland. Lasted about six months and decided I really needed to go off into the private sector which is where I've been for the last five years. What we're going to do is let me go through about 20 minutes or so of a little background just so you get where we're coming from with the commission. And then we'll open it up for Q&A. We can get a lot more going there because we really would like to solicit some of your ideas. In Washington, there's of course been a lot of bubbling over the years about whether cyber should be a part of any administration. Should the president himself be briefed on cyber issues? Is this presidential?  Every little thing that's going on they're in there talking. Well, I can tell you having been in that world working inside the White House, it's not like that, not even close. In fact, one time we were able to get into the Oval Office, that's back when ASN.1 was a big deal. Do you remember that in early 2002? We lasted about five minutes.  And he looked at us and said, so what can I do about it? And we like, well, Mr. President, you're supposed to tell us what you're going to do about it. And that's the point where we were then escorted back out of the Oval Office and never went back in again.  So this is a challenge for us, is how do you take our world and translate it into something that the executive of the United States of America can actually work with? And in fact, it's a challenge for just about anybody who has an agenda. Last summer, there was a series of unfolding events in Washington. Many of you have been reading in the press. You've seen the large number of attacks that have occurred against the Defense Department, Department of Commerce, State Department and countless others, as well as defense contractors, others in the industrial base that helped support the nation. Those intrusions finally rose to the presidential level. And there was a briefing to the president and he got very concerned about it, as you can imagine. The Congress was also briefed. A lot of concern there. There are a number of think tanks around Washington, D.C., one of which is CSIS, so I'll talk about them in just a moment, also concerned. So by the end of the summer, there was a groundswell of concern in Washington that something needed to truly be done about cybersecurity. We can no longer just keep talking about it, keep waiting on somebody else to fix it that we really needed to move forward. So CSIS, and this is an independent organization, it's a think tank, the Center for Strategic and International Studies, and they've long had a relationship with the federal government and with the private sector, proposed the creation of a panel, a blue ribbon commission, if you want to think of it that way, that would develop recommendations not for the current administration, not for the current time, but for the next administration. And if you think about this time last year, we did not know where we were going. The primaries had not started yet. We pretty much knew on the Democrat side it was going to be either Hillary Clinton or Obama, one or the other, but nobody really knew until just a few months ago. On the Republican side, it was wide open. There was lots of contention on the Republican side, no clear front runner. And that's where we started down this road, knowing that there was probably two on the Democrat side, but five or six on the Republican side, no idea who we're going to wind up with. So we used to always say, you know, we would brief this to Mr. or Mrs. President or Madam President once it's over with, because we again had no concept.  We have always been wanting to keep this easy, just four or five things. In fact, if we can get it down to two or three, that might even be better. Things that we want to give to the President of the United States are actually the transition team right after the election is over and we know who the President is going to be that says here are the things you can do immediately, almost like a playbook for the United States of America to increase cybersecurity. Let me tell you a little bit, though, about what we are not just to clear the air. There was concern when the Commission was created that we would be like the 9-11 Commission. In other words, our purpose in life would be investigative. We were there to find fault. We were to go back, you know, through the current administration, go back to the 90s, figure out why did all this bad stuff happen. That is not what we're doing. So just rule that one out. We're also not part of a separate but parallel effort known as the President's Comprehensive National Cyber Security Initiative. This is what you'll hear of just the short term as the cyber initiative that you'll hear that floating around. This was a, and still is, a classified project, although many parts of it have been made public. There's been press articles out since last September. And in fact, Secretary Chertoff at RSA back a few months ago gave a fairly detailed discussion about what's in the cyber initiative. So if you want to know more about that, there's a lot of information now that's been made public. But that's not what we are. We're different. We're the CSIS Commission. However, many of the commissioners, and I'll show you how we were made up in just a moment, are in fact providing input to the various portions of the cyber initiative. We are very much trying to keep these things separate. But we do know that when the same minds begin thinking about the problem together, we're going to have a lot of overlap. And in fact, once we get done, when our recommendations are handed over to the next president, many parts of these recommendations may say, keep doing what this initiative is doing. Don't stop. Just keep right on going. Or it may be that we've got some other things that the initiative is not in there, new starts, new things to think about. One of the first things that we did realize, though, that cybersecurity is a national security problem. There is a clear and present danger, folks. We as a country face a turning point here. I think all of you are aware of this. We have built an incredible capability. You know, we are the fathers of this global phenomenon of highly connected world that we live in. And so there's a bit of a challenge that we're facing now is how do we take this from being just the nuts and bolts world that we live in, the very technical community, and elevate it up to a strategic level important problem. Just as important as nuclear warfare, weapons of mass destruction, fighting hunger, all the other things that we face is a very strategic and challenging national problem. So what we have to do then is how do we elevate? How do we take cybersecurity? Something that's seen as our world and just kind of pushed off to the back burners and bring it up to where it can compete and fight for allocation of resources, money, manpower, smart people like yourselves that can work together. And I think in fact that's our biggest challenge is how do we do that? So a little bit about our group. Bipartisan from the start, we've tried to do that. We were as close as we can be 50-50 as to who represents the Democrat side versus the Republican side. But honestly our political beliefs have been set aside. We're just trying to do this for the nation, not for any particular party. We've got two congressmen. I'll give you their names and backgrounds in just a second that are working with us. And then behind us sits over 100 and actually we're approaching close to 200 people now who have volunteered people who want to help out that are assisting the various working groups with the drafting of the specific language that's going into the recommendations. And in fact if any of you are interested in assisting we would love to have your help. We are pretty much at a point where most of the recommendations now have been gelled out. We're now trying to take hundreds of pages of thought and cram them down into just a few pages, get it to some nice, concise recommendations. Where we will really need your help is in about a month we should have the first draft ready for public review. Please, if you could download it take a look at it, give us some criticism back so that we can refine and make this worthwhile. It's not our recommendations to the president, it really is yours. We've done the hard work and now we will need your assistance in making this refined as we get it ready. So here's our leadership. We've got Congressman Langevin, he's from Rhode Island, along with Congressman McCall from Texas. Both of them serve in the House Committee on Homeland Security in the subcommittee on emerging threats to cybersecurity and science and technology. So this is the specific group in the House that worries about cyber things. As many of you may know the Department of Homeland Security has a lot of oversight. There's over 80 different committees and subcommittees that have something to do with Department of Homeland Security. And in fact there are many that feel that cybersecurity is theirs as well. This group has it specifically in their title. And in fact they're the ones that have been holding the series of hearings over the past couple of years to find out more about what's happening in Homeland. Supporting them are two people from the private sector, Scott Charney over at Microsoft and retired General Harry Radighe, who in his last incarnation was the retired three-star Air Force General and was the commander of DISA. He commanded there for five years, one of the longest DISA commanders. Both of them with a lot of expertise and background and are assisting accordingly. Let me go back one here. If you look at the bottom you'll see the website there CSIS.org slash tech slash cyber. I'll put that back up again at the end of the slideshow. That's where we have more details, who the other membership are. I won't bore you with that on the slides. We've got some brief bios so you can see who else is participating there. We started this, like I said, back in October. Big press release, had a kickoff session shortly into November. Since then we created a series of working groups and those working groups have been giving many back briefs back to the larger group. We found this is better than trying to have 35 people all working as a big cluster. We've also had outside expertise, two really good sessions. We've got a few more that are scheduled throughout the summer. We've had intelligence community, we've had foreign representatives come in, Defense Department and other branches of the federal government have come in and worked with us, kind of giving their take on what they see, what the problems are, and the issues. We have several more, like I said, that are scheduled throughout the summer and then we begin to bring this to a wrap in September. Let me just put the commission aside for a second and take a quick diversion into why we're doing this and how challenging it is for us. Let me talk a little bit about how this works with policy and you'll get a better perspective of what we're up against. If you remember back to 10th grade, remember Mrs. Johnson, your civics teacher, and Mrs. Johnson was explaining to you about how government works and we've got the separation of powers into three branches in your constitution. Remember all that good stuff? Yeah, okay. That's what you have to think about here for just a second. I know many of you probably dumped it after you left 10th grade, but this is what we're up against. We have the executive branch. The executive branch can give executive orders. They can sign decision directives. There are presidential signing on the bottom of bills. They can add things in there. It's very interesting what all the power that sits in the executive branch, of course, that's more than just the president. That's also your department's agencies and others. We've got sitting over in the legislative world, the power of Congress to appropriate, to enact laws, to do all the things that they do, and we have the regulators. Never forget them. I work for a very large regulated company, Verizon, and we are very concerned, as you can imagine, about the regulatory world. Many of you work in a regulated world, so you're having that to balance you. Plus, we have standards, things like what NIST and many others are putting out that the federal government has to adhere to, and then don't forget what the judges do as they interpret the laws and try and twist things hither and yon. All three of these are affecting all of us. Fortunately, though, we can rule out a lot of the policies that the federal government worries about. There's a lot of things that are internal, like staffing and funding and allocations, but unfortunately, cybersecurity, that's one of those areas that affects all of us, and it doesn't matter what the government comes up with. Any policy that has to do with cyberspace is going to affect us, and I should be to be fair, it's not just U.S. citizens, it affects people beyond the United States in terms of the policies that they're developing. Many of you, if you're not working in the federal government, you're still subject to FSMA. If you're working in any type of organization that holds federal data, if you're storing it, managing it, otherwise you're underneath FSMA, you've got standards that apply to you, et cetera. So it's quite a bit of impact. If you want to speak, you have a microphone right there. Just sit down and speak, Mr. Dixon. I like to stand when I talk anyways.   Go ahead, Jerry. So just to give you an insight into how, because I had to live through this process for the better part of eight years, there's what's called a policy coordinating committee, and when you're working to, when we were establishing USR and Mark knows, very familiar back in the early days of DHS and trying to lay, you know, what is the jurisdiction of USR? What authorities is it going to have and what's going to drive the mission forward and actually define that mission? So what a policy coordinating committee, which is actually ran out of the White House, and it can actually be chaired by anyone from the National Security Council, the Homeland Security Council, or the Office of Management Budget. I don't know if the executive branch or the federal agencies, OMB typically ends up being the chair or somebody from the Homeland Security Council if it relates to DHS. Well, something as simple as coming up with standard incident categories, incident reporting guidelines. Many of you are probably familiar with the data breach from the VA, for instance. There was a whole discussion on trying to set up reporting related to privacy breaches. This process is a very long drawn-out process in trying to get consensus from Department of Justice to state to DHS. And basically, you end up going through multiple iterations. Well, to give you an idea of some of the timeframes, just to set the incident categories and reporting requirements within the federal government that took a year to get it approved and finally signed off on by all the departments. And there must have been probably 48 different rewrites of the actual concept of operations for USR. So it's not something that the government can turn on a dime. I'm sure most of you are aware that it's like turning a battleship, but I want to kind of give you a little bit of insight into that. Thank you, Jerry. How you doing? You demand. So when we look at some of the ways we go here, we can have a private sector solution or we can have a public sector solution. I think we all would like to have private sector solutions to everything in terms of cyberspace. Let the government go back to just doing what the government does well. But there are consequences. There's pros. There's cons on both sides. There's competition in the private sector. Unfortunately, competition doesn't always allow for the best solution. It's the one that competes best or the one that they can make the most sales on. Government unfortunately has a lot of advocacy groups that are out there trying to elevate their potential ideas. They're little pet rocks. They also compete with each other. In the end, the best approach, which many of us are very aware of, is some type of public-private partnership, some way of working together with cyberspace. You throw in that additional international dimension. As all of you know, a bit does not know whether it's sitting in the U.S. or Canada or China or Germany or wherever it is. And at nearly the speed of light, these bits of information flow around the planet. So whatever we come up with here in the U.S. affects the entire planet. And that's one of these little lessons that we continue to try and push to our counterparts in Washington. So there's many places we can look for policy, and all these places are going to be competing that we come up with. Anything the commission develops competes with all these other things that are out there. We have to worry about consumer protection. We have to worry about diplomacy. We've got to worry about all the other things that are happening. There are some interesting proposals, but we are not talking about this in the commission because we want to go presidential. Proposals like taking computer code itself and writing laws into it, making it computer code making things illegal. Very interesting idea. There's very clear black and white with no room for negotiation if your computer won't let you do certain things because the law says you can't do it. One example of this and emerging problem is the supply chain. As we are globalizing and as we're getting components and parts that are coming from all different countries besides the United States, how do we fix that? How do we make sure that what we're getting is in fact secure? We can let technology fix it. We can let politics fix it. We can legalize it. In other words, pass laws to make it illegal or we can just let industry worry about itself. You can see all four of these approaches compete with each other. There's no one particular approach that's better. Let me show you a few examples of the big high level things that we think about and then I'm going to get back into the commission. For example, at the national level we don't have a response plan. Right now if the United States came under attack, much like Estonia did this time last year with the Russian involvement, we do not have a plan. Something we can turn to that says here is how the United States responds to that. Now that's incredible. How do we get to that point where we don't have that oversight? We have it in the nuclear world. We have it in countless other places, but we don't have it in cyberspace. So that's one of these police things that we're looking at. Well, we're getting a lot of noise from next door, but that's cool. Maybe we'll get some noise in here in just a second. We're also looking at global governance. You all know that nobody runs the Internet. It's a big get along. There's no president of the Internet. There's no company that runs the Internet. That's names and numbers, but there's no real running of the Internet. And in fact, what should happen presidentially should there be an effect on the Internet? Does the United States take it back over? Do we seize control? These are kind of the interesting areas. Likewise, research funding. Research dollars, as you might realize in cyberspace, have been continuously going down. What a shame, because so much of what's built this nation has been built on the backbone of research and has gotten us to where we are today, is to dwindle. For yourself as individuals, privacy, big deal, for those of you who are sitting in here in the talk earlier, you know that that is absolutely fundamental to how do we make this work for ourselves. We have to keep that in mind. Security, everybody defines security a little bit differently, anonymity, and how do we protect your desire to be anonymous in cyberspace, balance that with law enforcement, counterintelligence, and others who want to know exactly who you are thinking. I'm not going to go through all these, but we could just keep going on and on and on with all the different things we can think about. But again, we have to, we have this challenge. We've got to boil it down to four or five things. You can walk into the Oval Office, talk to the President about. Could you imagine what that would be like to actually sit down with the President and say, Mr. President, I want you to think about the following thing or I need you to do this, this, or something else, and try and put it at that level against the war in Iraq, against illegal immigration, against the economy, against rising gas prices. All the things the President is really worried about, the things that make him lose sleep at night. How do you walk in and say, but sir, there's this thing called cyberspace? That's our challenge. This is where we're going to really need some help from you. And I look forward to the comments as we open up the microphones here in just a moment. So let me wrap up what we're doing and then we can get into some good Q&A unless Jerry interrupts me again, actually I do have one additional thing but I'll wait till you're done. All right, then you'll wait until you're on turn. In fact, there's a microphone down there if you want to ask me a question. I'm afraid of the response. Yeah, you should be. So our first round, we've divided this up into phases. Phase one is into history books now. Phase two is where we currently are and phase three is coming up next month. Let me just show you a little bit about what's happened in the past. That first phase we took all of us and then what has the federal government done over the last 15, 20 years with respect to cyberspace? Do we have any policies that have worked? Do we have any that were failures? What do you think we found? One big bucket that had a lot in it. That was the failure bucket. We've also looked at what are the threats and I don't mean necessarily nation states or threat actors, you know, criminals, insiders, things like that but what truly are the threats? For example, the lack of trained technology professionals that have led a threat to the security of our nation is a supply system, a threat. Is globalization a threat? Those are the kind of things we're looking at beyond just the obvious, is this particular country or is this particular company in some cases a threat to the security of the United States. We also tried to get a handle on the complexity of which there are many because of all the overlapping corners of the world, who the actors are, who the players and the different parts of departments and agencies and then some of the key infrastructures like telecommunication and electric because those are your two areas that underpin everything about our economy. Without the power sector, without electricity, we wouldn't be here. Lights would be out, we'd be out on the streets. Without telecommunications, we wouldn't be here. There's no way that this building would run. The SCADA systems wouldn't work. Nothing would happen. So without those two, nothing moves forward but all of them depend on cybersecurity as do the many of the other sectors. So with all that behind us, what do we develop recommendations based on what we know is wrong? We have one group that's been working on federal organizations. Should we start over? Do we rebuild the parts of homeland security that do cyber? Do we have a cyber czar, for example? This is an idea that's been bubbling up and up. Should there be somebody like the surgeon general, you know, have a cyber general that's in charge of everything for cyberspace? Some have said, yes, that'd be great. But nobody runs the Internet. There is no elected king of the Internet. There's no potentate out there. We seem to all get along. So shouldn't we have a model like that in the federal government in terms of the security for the United States? So two really good competing thoughts there. We've also looked at the authorities, a group working on that. What legal mechanisms are standing there? If our nation comes under attack, if we have a situation like Estonia last year but multiplied to the size of the United States, what parties do we have to respond? What can the private sector do? What does government do? Which organization actually picks up the sword? Or does anybody pick it up? Or do we just build this big firewall around the United States and just close ourselves off? Is that the response? I think we've got a long ways to go before we get to an answer there. Also, acquisitions, spending, how much money do we put into it? And then a group I've been working with, and this has been absolutely fascinating, is the partnership. We have some organizations out there, the InfraGuard, the ECTF, others that exist already, but none of them are providing the partnership that we really need where the federal government, rather than just governing the United States, is seen as almost like a business partner because their networks are just as fragile as your networks. And together, we need to be working to improve the security of our nation. I've actually just walked through all these, so you can read those later on your own. So the things we've looked at, the last one I'm going to, and we're going to move over to Q&A here in just a second, is we've got to be strategic. We've got to be able to talk to the president. We can't come up with a list of little things like we need to fix this particular piece of software, or we need to use this type of email, those are pebbles. If we go down that road, we've lost. We've got to have big strategic initiatives, something that might take years to undertake. Things like, how do we change the way we do the educational model of our children? Starting in first, second, third grades, how do we start injecting into them science and technology? The love of doing these things so that as they grow up, they can come back and be our future leaders. How do we get moms and dads more secure, small businesses, all that kind of wrap together? Those are big strategic problems that we need to think about. Some of us have even proposed that we ought to lay out a roadmap for completely rebuilding cyberspace. In other words, looking at the argument that if you take just the internet as a piece of it, developed in the 70s when threat models were completely different. When we could trust people, but we couldn't trust machines. Should we in fact make that a presidential initiative that the United States leads the way in building a new architecture for the 21st century to where we have a lot of trust now in the machines and we back off a little on the people side? Because essentially what's happened is we've taken a small community, a little group of farmers that all got along with each other in small town America and we've opened it up to the neighborhood and we've let the hood in and look what's happened as everything has come in. So we need to rethink that. Those are big strategic motions. Jerry, your turn for a question and then it's going to be your turn. There are two microphones. Well, there was a microphone here. I guess there's just one there. We need to know what are your recommendations? What are your ideas? What can you do to tell us? We can form a line right there. Please, Jerry. Just a quick correction. I don't want you to think that there isn't a national response plan. There actually is one. It's a national response plan framework and within there there's what's called emergency support functions and actually the cyber piece of it rolled into what is called the communications support function which companies like Verizon and others work with what's called the national communication system or actually make up what's called the national communication system. The challenge is that once you start digging down further in there if you look at other sections, other emergency support functions, there's a little bit more detailed plans on how to react or respond to particular events. There's not a single group within the U.S., within government where one agency I should say that can respond to multiple states that might be affected by a major cyber attack that's affecting delivery of public services or what have you. So that's kind of the challenge is how do you address that issue? You know, maybe there needs to be a state level and some response capability to one of the states that feeds into a higher tier at the federal level to help respond on the ground. So those are some of the kind of things out there but the key thing there is I wanted to make sure that there is national response plan out there there's a lot of work that needs to be done in that space. We can respond to the nukes but if our networks come under attack, who's in charge? And that's back to the key point. So U.S. SIRT is basically the focal point for cyber related issues at least within the federal agencies and at the executive department. When you start getting into sector specific areas that's where like the Nuclear Regulatory Commission would step in or if it's something affected with energy which would also include the energy department. There's a group what's called the National Cyber Response Coordinating Group. It's basically bringing together all the major departments that have expertise in specific areas so like FAA or the Transportation Administration. But again there needs to be better detailed plans that come into place. I don't know how many of you read the after-actions from Cyberstorm. Those are some of the things that on the commission that we're taking a hard look at at Cyberstorm 1 and Cyberstorm 2 and trying to glean the lessons learned from that so that we can make sound recommendations to the next administration. Yes sir. You mentioned that when this all started there weren't any clear front runners from either party. Now that we have two it's going to probably be either A or B. Are there any gentlemen in the back says don't forget Ralph Nader. Are there any plans or intentions to tailor this design towards one particular individual over the other. Is it going to be the same thing regardless of who gets selected or what we're trying to do again is to be party neutral. In other words the president of the United States represents the entire country. Different parties elect through the political process they're nominees and of course you have two big front runners Republican and Democrat but like was mentioned in the back there's more than just those two. The odds of somebody else getting selected are pretty slim. It's going to be one of the other of those we don't know which but our job is not really to worry about who gets selected. Our job is to think presidential and how do we turn over a roadmap a plan just something that's presidential regardless of which party ultimately goes into the White House. It seems as though the background of whichever individual would play into how it's presented to them their own personal experience with technology or Luckily we don't have a lot to worry about there. So I didn't know even the way it was presented or made clear to the individual. What's unclear right now is we don't know who the staffs are. If you look at just the two leading candidates Obama and McCain neither of them are really techie types both Obama a little bit younger is more connected to technology just generationally the McCain is but you don't see either of them out there doing the things that we do in an online highly technical world. So then it becomes important as to who are their appointees who are the staffs who knows. That gets determined way after the election. And a key thing to remember is that even when we produce these recommendations I mean they could totally ignore them. We don't have any kind of power oversight. The good news is that since it's being sponsored by Congressman Langeman and Congressman McCall they're going to be watching and seeing what the next administration does when it comes to cyber policy. As far as a recommendation Unless you got a real quickie we've got a line back there. Go to the end of the line. There's room. Yes sir. I work for a non-profit that goes into communities, helps them set up and work together to help secure their cyber stuff. So under the world recommendation. Right now and simplification the DHS has a lot of money to communities for security different reasons. I can see where that if you would tie that into they have to have certain gates they have to meet before they can get X amount of funding that you could help direct and in fact implement. Could you give me an example of a gate? What kind of thing are you thinking of? The amount of training that they have to obtain steps in the fact of okay here you have information awareness then you go into maybe information say cyber security exercises and training. Then you get to a point where maybe you're working with other communities and maybe at the point that the next community starts helping other communities develop their own money that's being given for physical homeland security money to something that they're doing in cyberspace. Another thing as well cyberspace is one interest where maybe as they're just giving money for, you know, gratis then tie it to something so they're actually seeing something production coming out of it. The idea is wonderful. The reality of course is we're up against all these legislators that are elected to go bring money back to understand you know and so to try to get we'll try. Thank you. I appreciate it. The other piece of that is that the way the current grant process works is that a lot of the state governments, you know, the state governor, which is their own executive branch does not want the, you know, the federal department, the federal department like Himalayan Security saying how to apply or best use its federal dollars to protect their backyard so to speak. You know, there's often been discussions about maybe saying one to two percent should be put towards cyber security within that state to help improve the cyber, you know, protecting the cyber infrastructure there. It looks like, you know, highway taxes have to be our highway grants provided for X if they do Y, you know, what can you do to address the same way? You have to do X for Y. I mean, I agree with you. I'm just giving you insight into kind of the way things have been going lately is that they'll typically allot X amount of dollars to a particular state. Then that state actually has, you know, whether it's the attorney general or the state police, the National Guard, they all compete for those same dollars as well as the local counties or municipalities. So it's, so they've got a good business plan and the governor agrees and they're going to get the dollars. Let me add one more question, go ahead and sit down. I'll just answer as you walk back. Because you mentioned highways, transportation, things like that, it's very easy in that world. We have lots of standards. We have Ash Tows standards and highway building standards, things to refer to. So if I'm going to give federal dollars to build a new highway or bridge and so forth, I can build that according to a standard. When it comes to cyberspace, what's the standard? If I start allocating money, what do you build it against? Who goes out and checks it? Who validates that in fact what's being built is according to a standard. By the way, that's one of the things we're trying to get a grip on is is that presidential? Does that rise up to the executive branch to try and do something? Now obviously the president's not going to sit down by the fireplace and start writing out cybersecurity standards. That's not going to, I hope that doesn't happen. That would not be cool. But whose job is it? Is it homeland security that does it? Is it NIST that does it? Is it DOD that does it? Is it NSA? Is it us? Does it turn over to the private sector? You say write your own standards and the government then enforces the private sector standards. That's an interesting approach. Not unlike what the consumer affairs division does over in commerce when it comes to consumer protection. And the other part of that is, should there be a cybersecurity agency like you have the Federal Communications Commission? Should it be a small agency that's focused and hopefully a little bit more agile than one that's working in say a large department? So that's kind of some of the discussions that's taking place. Now Neil, we are not worried about the aliens. We are not going to ask the President to give you the keys to Area 51. Actually, you are going to ask that one. But if you want to ask something else other than about the aliens, we are welcome to listen. You got it. With all the compromises of government systems by foreign government, are we already in a cyber war? Well, let me ask you what is meant by warfare. What is a cyber war? I assume it's a foreign government attacking us. A foreign government attacking us. Who declares cyber war? The cyber present? A cyber congress? Well, I thought that's why the blue ribbon panel was created. These are great questions. I keep going. This is good stuff. It's the same thing with espionage. Espionage versus by an organized crime group or criminal group versus war. Maybe there needs to be physical damage as a result of a remote-based attack. There's a number of different things you can walk down. It's a challenging question. As Mark alluded to, back to the national response plan piece, there's actually not an overarching cyber doctrine. There is a national military strategy to deal with cyber, but there's not one that encompasses all of the government and all of the departments. That's something we're putting forward. You're asking the right questions. The word war. We have the war on terrorism. We have the war on drugs.  There's a lot of stuff like that. We've still got a ways to go. Is it fair to say that there is a cyber war? Do we want to cheapen? War is a very dangerous thing. It's like somebody in a talk wanted to take the cold war and say that this is a cyber cold war. The cold war was a very dangerous time. We almost annihilated ourselves. Nukes pointing at each other. We can still do that if we want to. I don't know that the word war is an appropriate time. Are we in conflict? Yes. Do we have differences? Are there political problems? Absolutely. Are we at a state of war? Not yet. With the state of conflicts, what about escalation? At what point does escalate? Now you're going into the questions of diplomacy. Something we are thinking of. If you've got some ideas, what do we recommend to the president? How do we do this? How do we know when escalations are taking place? What ambassadors do we draw back if somebody attacks space? Or do we? Or do we bring a company back from India? We close all the Indian call centers if the attack comes from India. These are great questions that we don't know. If you've got ideas, let us... Keep in mind, cyber cross is a lot of different titles. Title 10, DoD when we talk about war, title 50, the intelligence community and their authorities, title 18, criminal authorities. That's why you often see in government these task forces get set up so that each one of the problems set. Thank you. The end of the line is just back here if you want to step back to the end. I thought you were sending me back to the end of the line. Thank you. I could do that if you'd like. Do you feel more comfortable doing the line? It's because I didn't get up yesterday. That's what it was. After this whole DEF CON black hat cycle, one of the observations I have is that we have a process called responsible disclosure, but we really don't have a responsible response to that disclosure. You guys are handling it pretty well, but as an example, I've sat in more than one discussion where vendors are fighting it out to say, you know, we're going to ship product whatever way we want to the consumer or whatever. I know that the government produced the secure desktop initiative. NIST has actually headed up, did a great job, Air Force was involved with it. It sounds like we really need to do that with any network devices. So what is the presidential recommendation? The presidential recommendation is for NIST to be funded. Oh, you're not writing this down. I'm listening. Jerry's got the notepad. He's got a great memory. There you go. I can add it to the slide if you want me to. A set of secure standards or at least a level of learning devices off, services off by default for all network devices across America or across the federal government. Across America by default by vendors that actually sell into this country. I mean procurement is the strongest power we have. We've heard it more than once from DoD from the government. Oh, we're not going to buy a unsecured desktop. Air Force had actually taken initiative. But again, let's go. You're talking to Mr. President. Yes. Mr. President. Funding for NIST to continue with a desktop initiative type system. So use NIST for what NIST is designed to provide the funding. See that's presidential. Telling the president we need you to develop a list of all the things that are being turned off when I get my computer at Christmas. No, I'm just providing some definitions.  I'm up here and you're down there. I'm up there too. That works. No, basically just defining the definitions because as an example I sat in a presentation with SCADA. There's an argument between the vendors to try to create yet another standard that isn't tested. So use NIST as the appropriate standard body? Absolutely. I mean, Dr. Ron Ross has done a great job in a lot of the processes of 853 and things like that. But the other thing is we're seeing the same thing with the cell phones. They have to sit in the cell phone. A lot of disconnect from the vendors that actually provide more secure systems in a lot of the applications that we've seen. A lot of the exploits have come from things made by people. Again, thank presidential. NIST is absolutely correct. So I mean, basically what I'm hearing is you're asking for regulation. When we grapple with this one, you know, or market incentives. We also get into these long debates about do we have market failure when it comes to securing proprietary data, critical infrastructure. What are the different levers that can be pulled or pushed to deal with this issue? So presidential recommendation might say to the economists at Treasury or say to Commerce, you know, assign somebody to go dig into this and really figure out how can we move this ball forward. When we talk about using procurement power, you know, I've been listening to that debate for eight years and government still hasn't really gotten there. Now the federal desktop common configuration that you talked about is a step in the right direction, but they're getting each agency gets lots of mandates from the White House from various groups, and a lot of them often are unfunded, so that's where the agencies are somewhat challenged. Well that's really what I'm looking at from a funding standpoint. Okay, you're getting bumped from behind. I think you have an angry line. Thank you. Thank you, sir. Yes, sir. I have a couple questions, hopefully brief. How radical are the proposals that you're contemplating? Is this something like a great firewall of America? Is this something that's on the table for proposing to the president? The only big radical thing that we've been thinking about is to create a public-private partnership that would involve a board that's half appointed by the federal government, half appointed by the private sector with an authority. That doesn't exist right now. There's nothing like that. And so that's probably the one that's really different. The rest of them are things that you would expect. Things like, how do you establish maybe a reorganization funding like the gentleman was talking about with NIST for authorities, a lot of tweaking in those areas. We're also teeing up the idea about education and educational funding. What was that I was getting? Thank you, sir. But yes, we would love if you've got a really cool, big, completely different idea, let us know. Another question I had is, are you leaning more towards directly facing the federal government and military defense contractors or... Broad spectrum. We're thinking of the nation here. National cybersecurity. So it's you and me, as well as the federal government, trading partners and others. This is a phenomenon of our generation, something we have to fix. Our kids and grandchildren put up with what we leave behind. And so if we don't get it right now, then they're going to be trying to undo our mess and it's really kind of cool because we get to write the rules. And then as a guide student, I'd just like to say more research dollars. Awesome. Thank you. And that is on the list, by the way. My recommendation, as a professor, actually, more research dollars, but reality is we can't build this on a house of quicksand. And until we figure out what our principles are going to be, I think the first thing has to come up is the principles of things like we will be secure. I don't see that right now coming out of the federal government because different agencies have different definitions of what's important to them and the end result is we're not secure. So somewhere there has to be some principles that will be secure. Another principle is, and this works great in corporate America, eventually it filters to a decision authority. Let me hold you for just a second. If you walked into the president and said, Mr. President, we need to be more secure. Mr. President then says to you, what do you mean? What's your reply? What does it mean to be more secure in cyberspace? We need to stop shipping protocols buying, stop shipping products accepting protocols and things that have huge known gaping vulnerabilities. Well, that's just about everything that's out there. Well, wait, wait. Huge known gaping. What do you point out right now that doesn't do what you just discuss? Plain old telephone system. Wait, wait. It's a large network and it's relatively secure. One of the principles is there's no such thing as perfect security. I understand that. But define security. What is it or what is it not? Can you define it? When something does exactly what it's supposed to do and only what it's supposed to do. Fair enough. Any other definitions? I'd have to think a little longer. See the challenge? See how hard this is? I realize, but I'm not seeing the principles part coming up first. In the second part of the principles is... Just on the principle part. I mentioned cyber doctrine. You've got to have an overarching doctrine. We actually came up with some draft recommended principles, overarching principles that can be taken or modified or what have you. Back to Mark's earlier comment, we are going to be making a lot of this stuff available for public input. At that time, take a look at what we've drafted. Maybe we miss something because we're only a group of 30 plus folks. We'll have to stop and let the next person come up. The idea of principles is absolutely sound. The only other part of the principles was a single voice. If every department gets to have their interpretation... Right. We have chaos. Thank you. Some of your earlier comments about legislating what's written bad code is a really bad idea. I think the only thing the government's really good at is taxing and spending. Let's go to the spending arena. What's currently being spent at the technology director at the Homeland Security Science Technology director is ridiculously underfunded. It's less than 1% of the entire entity budget. What I look at what Carl Landworth's got at NSF is absolutely ridiculously underfunded. The current head of NSF is incompetent. Tell us how you really feel, sir. I really feel he is incompetent. He's not spending the money. If you got into the Oval Office, the recommendation is put a lot of money into fundamental research into defining the questions you've got for security. Get a governing board. You don't want a czar. You want a governing board, what you just said, of private and government people together to come to how we spend that money and deploy that money for actual fundamental new technologies and security. It's a money issue more than anything. Going down this road, one is that if you go back to just 1990, convenience starting point, roughly right before NSF and ARPANET integrated with everybody else and now come to the present, the last 18 years, just as a little group there, we've spent close to a billion dollars on things that we could call cyber in the research community, across NSF, DARPA, now Homeland Security, ARL, and NRO. What have we gotten back from that? What's the ROI for your tax dollars that have been spent on research? There's some really cool things that come out. There's also a lot of things that when the research was done, it didn't mean anything at the time. It's just sitting on the shelf. It's hidden in a paper in an IEEE journal somewhere. When it was written back in 1994 or 95, nobody understood it. Great science, bravo, here's your award for your paper. If we go back today and pull that out, there might be a nugget or two sitting there to actually solve some of 2008's problems. One of our ideas is, is there a way to create like a cold case to go back and look at all the old research, bring it forward, and what's there? Can we mine that instead of trying to? And so that's the kind of thinking, that's bold, it's different. And perhaps we can go down that road. Another point to this is that the government we've been really, when I was in government, we've been very good at being reactive to particular situations. We really need to get to that side where we're taking and being more preventive. And that's where research and things, thinking ahead for the long term solutions is the way to go. Everybody knows we only got about four minutes and then we're going to be moving over to a clinic. We'll stay with these four, and then I think we're going to be done. I want you to run your slides back the other way. Which way do you want me to go? Back to the coding. Which one? More. Number four? No, more. Keep going that way. Keep going, keep going. Until we get to the thing where you said you're going to make the little thing about coding. Coding, coding. Remember? Yeah, we're going to make it impossible. Wait, wait. I think it was the bottom. Nope. Nope, you were back. That one? There we go. There we go. Which one? Yeah, right there. All of those. Ah, very good. Okay. I will suggest to you, sir, that this is a slippery slide. Microphone. I'm not that tall. Pull it out. Pull it out. Ew. Now you can't step away from the microphone. All the rowing. I want to say that this is a slippery slope. Yes. And it's a frightening slippery slope. Do you recognize this book and this story? I can't see it from here. It says True Names. Okay. Is anybody else in this room ever read True Names? Anybody read True Names? Show of hands? Yeah. Three people. I am telling you, you need to read this. This is a slippery slope.  You may be contentioned, but the moment you start saying, let's stop subversion, you stop good people as well as bad. And the bad people are going to get around you. I spent 35 years being able to, oh no, I'm a cold warrior. You're one of the good ones, right? Kind of. Okay, okay.  Trust everything I say, son. I am good, right? In D&D terms, I'm chaotic neutral. There you go. But it's important to you understand. Is that your name? Chaotic neutral? I'm sorry, what? That's how you're all on I.M., Chaotic neutral. I don't do I.M., that's for children. The important thing is I'm really very serious about this. It's a bad path. It's not the direction you should be going.  I don't want a PlayStation. I want a computer.  I want to be able to write code and I want it to do what I want it to do. Unfortunately, what you see on the slide is present day and you're absolutely right. How do we build the future that we don't have to do this but still provide the security, privacy, anonymity, all the things we want? That's our challenge. I just want the government out. We've got three people behind you. Unless you've got a quick, what do you tell Mr. President? Goodbye.  Thanks for all the fish. Folks, we're going to have to stop right there and I think that's going to do it. We're out of time. I got a quick question. Actually, it's a request more than a recommendation. Okay. Through the many assessments and forensics investigations I've performed with state and local governments where the majority of the PII is for the people that are paying the taxes to protect this information. I would request that as part of your initiative that you support funding and standards and guidance to local and state organizations to really help keep this problem to a minimum. I'm going to stop with you. You two guys just come see us. Thanks very much, folks. See you through the con.


Hello, everyone. Thank you for joining our talk today entitled Supercharged Analytics for Prometheus Metrics with Spark, Presto, and SuperSet. So just a little bit about us before getting into the talk. So we have Rob Skellington. He is currently the CTO and co-founder at Chronosphere. Prior to Chronosphere, he was at Uber and was one of the creators of M3DB. And then he's also a contributor for Open Metrics. And then myself gets Colin. I'm currently a developer advocate at Chronosphere. And prior to that, I was a product manager at AWS. So we're just going to briefly jump into a preview of what's to come to kind of set the context. So essentially, we're going to be looking at Apache SuperSet, which is a UI that connects to arbitrary SQL backends. And in this talk, we'll be connecting to Presto, which is talking to Prometheus backend. And you can see that arbitrary metric names can be queried against once you set up everything correctly. And this allows for deeply complex analytical queries to be issued against Prometheus metrics over a very long time horizons.  So we're just going to go through the agenda real quick. So we're going to start out talking about kind of how and why you would want to use your metrics for analytics. After that, we'll go into a bit about why using your metrics data is a bit challenging with the current monitoring solutions, followed by a couple of different ways that you can solve this problem. So basically looking at what a solution would look like for this. And then we'll give a little bit of a demo followed by time for Q&A at the end. So quick overview of metrics. So metrics are essentially used mostly for monitoring and alerting purposes. As you can see, this is a pretty standard way to view your metrics, as you would like in a Grafana dashboard here. So you can see your metrics over time in a variety of different views, which you can then set alerts against. One thing, though, is that metrics are also very valuable in terms of they also have a lot of historical data. So they can also be used for analytic purposes. So now we're going to go into a few examples of how you would use your metrics for analytics. Starting with this first example with cost monitoring. So as you can see in this graph here, you have basically this would basically be an example of a cost report that you might get from your cloud provider. So zooming in a little bit more into the different diagrams here. So you can see that over time, your EC2 usage is increasing month over month. And that's a little bit different than the other services you're using, which are basically just staying steady or even decreasing. So given this pattern, you would then want to try to understand why your EC2 usage is increasing over time. So one way to do this would be to look back in time and see if any of the events or things you might have done, changes you might have made could have triggered this change. So for example, going back to June 2020, let's say you went ahead and changed all of your EC2 instances from SSD to spinning disk in the effort or in the hope that this would then lead to a decrease in usage, overall usage. So as you can see, it had the intended behavior and in July with the usage going down. However, in August it spiked up and then in September it got even higher. So you'll want to then understand why this behavior all of a sudden changed after one month. So a way to look into this a little bit further is to kind of use your metrics and do more granular analytics on those metrics. So in this example, we've zoomed in to just the EC2 usage component of the graph from the previous slide. So you can see here that the various EC2 usage amounts are broken out into four applications and that you can see also that applications see here is mainly the is the main culprit in terms of the increasing usage over time. And this is because that we know when you did switch over your EC2 instances to spinning disk, the A, B and C or A, B and D applications performed well with that change. However, application C performed poorly and needed more EC2 instances because yeah, it just did not perform as well with the spinning disk. So now that you have that insight into what was the main driver of this increase in usage of EC2 over time, you can then go back and kind of change your cost monitoring strategy and revert any of the changes that might have led to those increases. So in this example, you would then want to change application C back to SSD as you found out that it performed poorly with spinning disk. So by doing that, you are then able to go back to your intended behavior and then start seeing the intended results which is a decrease in EC2 usage over time. Thanks a lot for walking us through that Gibbs. Yeah, I wanted to walk you through another analytics case here on your metrics. So essentially there's plenty of other things we can do with metrics when we want to take a more analytical view. Of the data that they represent over time, that is a little bit more difficult to do with the monitoring and observability tools out there today. So we can do data science, perform data science operations on these to kind of like understand a little bit more at a very deep granularity on how different applications or infrastructure or product things like how your actual technology system is actually performing and kind of dissect them using more typical data science like queries. And then we can look at optimizing essentially our systems similar to kind of what Gibbs was, the use case Gibbs was looking at but with a more granular focus on specific systems deployments and kind of like discovering hidden waste which is harder to do with just a graph. And then looking at forecasting and how forecasting can be used with metrics in an analytical setting. So basically we're going to walk through a example here to extract some more probabilities and back test basically what different models have been performing perhaps under different systems that you're writing. So this example we're going to look at a high-volume real-time ad-bit platform. We're going to look at say number of ads purchased, number of ads that led to click-throughs, latency to respond to an ad request. And then we're going to, so those are the key KPIs that like would be tracking with Prometheus metrics. And then, you know, we'd also be looking at and these are really high volumes so obviously things like latency and you know high granular counts per second you would want to use with metrics. And then we're going to label these metrics with things such as the display ad type, the regions, the machine learning model used and the different input configuration parameters for each model that was used. So if we look here we can see this is kind of like a distribution on the click-through rate which is calculated by number of ad volume versus number of ad volume with purchases, sorry with click-throughs after the ads being purchased. And then, you know, you can see here basically how the different regions are performing with different display ad types. And then what you really want to be able to do is look at how the different machine learning models that were being considered to serve your traffic or AB tested performed. So you could even do shadow AB testing where you're not actually, you're only like shadowing a few percent of your request to these different machine learning models with different parameters tuned. And then using metrics on these like high volume kind of events that are happening and also using metrics to look at the different latency on how these parameters are actually performing, you know, you could kind of see what the different perhaps click-through rates would be of the different machine learning models, your kind of AB testing versus like a control set of control units. So, you know, that kind of thing is obviously possible with Prometheus metrics today. It's just over large time horizons with very high numbers of models and different variables in terms of like display ad type regions, like other kind of variables that go into different segments you want to track, to analyze all those various different segments across the various different model parameters that you're testing. That becomes a lot harder to do just on a simple graph, obviously, and SQL becomes quite, quite useful here. Then the second use case, you know, I want to walk through is essentially optimizing your systems on a more granular level than even just the breakdown of microservices. So for instance, for you can, with a SQL query, you know, enumerate every single application group by service that you're running on a say Kubernetes cluster and work out how much of the CPU quota that you've reserved or requested for the application is actually being used.  And then also like we can use, of course, these same type of analytical queries to forecast how many resources your system is going to use. And so here's an example of, you know, an organization that's forecasting a growth of requests per second by their system by end of year for 30%. However, if you look at this graph, you can see that the number of requests per second actually does not literally use more CPUs, it actually exponentially uses more CPUs. And when you break down that average CPU use by service on an even more granular level, you know, you could see essentially which service is, which application in your stack is contributing to that higher growth of CPU versus others. So you kind of know which services you're going to have to reserve more resources for, which will then let you make the decision on whether you actually need to actually make that service perform better, or do you have enough budget to just go and purchase the necessary resources and expand the cluster ahead of time to safely handle that traffic. And, you know, there's Dan Lu, who has done a lot of prior art on this. And I highly advise you reading his blog there on danlu.com, Metrics Analytics. You can really find some really amazing insights when you kind of start to be able to run these arbitrary queries. And he describes here that in one case he found a service that was wasting enough RAM to pay his entire salary for a decade. And so now Gibbs is going to walk through a bit more of the problem of why this can't really be done today. Yeah. So why is Metrics Analytics hard? So at a high level, this mostly comes down to some performance constraints with existing monitoring solutions such as Prometheus. So they just, you know, with these longer term queries that span a high cardinality set of metrics, they don't, they either just don't perform well or they will time out when they are running. And this is due to a couple of different things here with looking at Prometheus in particular. So with Prometheus, there is a max sample limit. So this means that there is a limit to the number of samples you can query. They also does have a default timeout of two minutes. So this may become a problem with your longer term queries as they do take longer to run. So you may end up timing out your query request. So, and then another constraint here is that results are typically computed in memory. So, you know, if something does take up a lot of memory or is a very large, large query, then it can lead to an out of memory or just taking up too much. And then at that point, your query will not be feasible to run. So what would you need in order to resolve these, these constraints? So when looking for a solution for this kind of one thing that you would want to look for is some, you know, some tools that can help with executing your large queries or queries of arbitrary size. So in this example, both Spark and Presto would be good tools for this, as they are built to spill data to disk, if something or if the request does not completely fit into memory. And so by doing that, then you can make sure that you don't use too much of your memory when fulfilling these requests. Another thing that you would want to look for is a way to utilize and get more insights from your data, specifically through some machine learning tooling or Spark. So in this example, Spark is actually very compatible with and it integrates with ML lib. So by, by, by using Spark, you're able to use the different machine learning capabilities within ML lib. And this will be provide even more insight into your into your metrics data. It'll also make it so that it's a lot easier to use for, you know, the end users of your data or your analytics, such as your data scientists, for example, as you can now, you know, use and query your data or your analytics through, you know, some of their more native languages, such as Java and Scala. Yeah. And then finally, what we'd want in our solution is the ability to join metrics with other data. So that not all of your monitoring data is living in metric store. So for example, here, you might want to join your metrics data with the kind of cube cluster, no information that you might have an elastic search, or even my SQL. So, so that's what we would want to have in a solution for looking at our metrics data. So now I'm going to jump into a demo here. And yeah, you can find the demo and all the working code at this repository at github.com forward slash Chronosphere IO forward slash demo dash metrics dash analytics. Essentially, what I'm going to do is this repository has a Docker compose file in it and a bunch of Docker containers, I'm going to make start, which is going to create a Presto container, super Apache Supersec container, and a whole bunch of setup scripts that is going to kick off here. And so I'm going to create basically a admin username and an admin password for Apache Superset. It's going to go and migrate all its default migrations that runs when it sets up. And then we're going to log in and connect to Presto through Apache Superset. So now that Apache Superset has done its done its startup, we are going to go in and remove the default SQLite data source, we're going to create a Presto data source. It's going to talk to the Presto container import 8080, talk to the permities connector with the default schema. Once we have created that, we then visit the SQL editor, which is actually restored the old result that I previously had here. So we're going to deep dive into looking at basically, yeah, this use case of looking at memory. And so, you know, I hit the explore button there from the from the SQL editor. And so now I'm in basically a visualizer visualization explorer. And essentially, I've chosen line chart visualization type here. And then I'm going to go in and edit the data source, which is came from the query that I read in my SQL editor. And I'm just going to remove the limit that was put on there originally, so that we can query arbitrary amounts of data. And so, yeah, we're going to look at say the past three weeks, but you can do three months, what have you, I just don't want the query to run too long because how this is a demo. And so, but of course, the power is to run really long queries, obviously. So here we're going to do a sum on the original query, which, as I showed here, was container memory RSS memory. And we're going to basically filter on the labels to make sure that we have a container name on them. So let's say a container name element at sorry, must with container name key must not be nil in the label map. And then we're going to group by the container name label. And we're going to set some really high limit. Okay, one billion to be exact. But we should make it stick. And now we're going to run our query. So, yeah, this, this query is running on Presto. And actually, if you go visit the Presto interface, you can see that it is running it across a few different parallel workers. And you can see here that's running multiple splits, because so basically it's split our query up into 70 different splits, random all on different workers. So though this is locally, you know, Docker container. So just sharing one machine here. And they and they see it process results. And we're now looking at past three weeks of data. So this basically has grouped everything by container name. So if you had even hundreds or thousands of containers going to group by the the actual deployment, rather than the individual pods in that deployment. And then of course, you know, the great power is that you can really do this over months or years or any arbitrary amount of time. And, and, you know, Presto, of course, will distribute that work. If it's on a single, you know, if you've only got a few machines and it runs out of memory, it's going to simply spill that the disk like Gibbs talked about. And it really does allow you to do some some really large arbitrary queries over a very, very significant amount of data. So yeah, and that's that's looking at over a three month period. So yeah, I hope this was useful. And I hope you get to try this at home if you're interested. But otherwise, I want to say thank you and I would love to answer some questions, if anyone has any. Thanks.