As she said, my name is Sinyaki, I'm here with Michael. We are from the CKI team. What do we do? The CKI team also known as Continuous Kernel Integration. We basically try to avoid ramping kernels developers for living. We run CI for the Linux kernel, which means that we grab a commit, we trigger a pipeline, we build the kernel, we test it and we try to report the results of what we just tested. That can't be too hard, right? Well, some time ago, we started like this. It was basically a proof of concept. We needed to show that testing the kernel on a continuous integration workflow was possible. So we sat down besides a big behemoth of course, Jenkins, because you can never go wrong by using Jenkins. But that was really useful to show us and people that testing the kernel was possible. But it involved a lot of Python and OpenShift projects and clicking and stuff that really didn't scale that well. So after showing that this was something that was possible that we could maintain and scale up testing of the kernel of the Linux kernel, we needed to revisit our problems. We started gaining more and more responsibilities. For example, with kernel developers, we needed to onboard new kernel trees. We started testing kernels from Brug and Koji, which are some builders that Red Hat and Fedora uses. We also test Git repos. We are now testing GitLab merge requests. We have a patchwork, which is an interface for mailing lists and the patches sent to the mailing lists. We also provide the gating for RPM packaging. And we are working on the kernel workflow infrastructure. I don't know if you've seen any of those talks. It's really interesting change on how Red Hat works on the kernel. But we also work with test maintainers. We help them onboard new kernel trees. We help them onboard new tests. I also configure target testings and we'll give them some feedback about how those tests are going. So we needed to be somewhere like this, where everything is automatic, everything is on track. We don't have to go clicking buttons and putting cars back on the tracks. But yeah, it's really not that easy. So we decided, I mean, we found out that we were gonna need some fancy keywords like software-sized reliability engineering and things that help us keep our machinery working. We started building our services under this main lemma that is that any component or dependency that can fail will fail. And some of them will fail even more than others. But we needed to make sure that all the failures can be retried successfully. And if there are some that cannot be retried, we need to know which ones are and try to keep them as minimum. So all the failures at first need to be prevented. Having fewer components or simpler dependencies helps us prevent important issues. But also once the failures happened, those need to be detected and we need to recover from that. What some background about what we have. Basically, we run on a lot of different infrastructures outside of our control, like different OpenShift clusters, Beaker clusters, Beaker is a hardware provisioning. So that's how we, what we use to get machines for the testing, but we also run on AWS on gitlab.com and many other platforms. To communicate all these services running on different platforms, we use a MQP cluster, which is a rapid MQ and a lot of queues to connect all the services. We also have internal microservices running on all these different clouds, like services to trigger the pipelines, to send the reports to babysit the pipelines until the finish. But we also have like really important pipeline components like the GitLab runner, which we use to run the tests, a test database where we store all the results and information that we generate, the beaker provisioning, et cetera. So the first thing is how to prevent these problems from happening at the beginning. Well, basically minimize the essential pieces. If you have less critical pieces, that means that less things can go wrong. We divided all our stack into three main categories. Essential components are the pieces that are needed for the service to run are our single points of failure and are the things that we need to keep at minimum. On the next layer, we have the necessary components. Are those things that need to run at least some time for the operations to work? Like for example, the reporter component, you don't need to have it running all the time with four nines or nine nines. We just, as long as it's running sometimes and it can pick up the work that it was generated before, that's not a problem for the operation. But we also have like optional components, which are those that provide observability and increase the reliability of the system. We cannot fail our testing if for example, the logs aggregation system is down on all the metrics or stuff like that. The first main mission we made are the message queues. We tried to decouple all the pieces of code. For that, we translated all the rest APIs into message queuing and that allows us to run all this like, to be abstracted from the place that the code is running as long as it can connect to the RabbitMQ server that turns everything into way more reliable and distributed system. Increase as I said, the service portability because you're not tied to the place that the code is running as long as it can connect to the message queue. And it allows us to automatically reprocess the failed messages after some time. We built a system on RabbitMQ, vanilla RabbitMQ that allows us to retry the messages that failed after some time. By default, if a worker returns the message to the server, that message would get on the beginning of the queue so it would get automatically reprocessed the next time, like right after it was returned and that could kind of block your services if something is going on with that particular message. In this case, the message goes to the back of the queue and gets reprocessed after some time. For example, as I said, we have this RabbitMQ cluster. It's hosted on AWS as it's one of our key components. We need to make sure that it runs with as much as availability as possible. We have a webhook bridge for those applications where we cannot decide if they have REST APIs or use message queues. We built a very reliable bridge to turn webhooks into messages in the queues. And for the retries, we are converting external message queues like UMB, which is an internal Red Hat message bus, or Fedora message bus into our own message where we can retry the messages successfully. Another example are the S3 buckets. This provides a very generic way to store artifacts. We don't need to rely on NFS volumes or Git hosting or other platforms that we were using before. S3, it's universal, it's fast, and it's really reliable. So we use AWS S3 for external files. We use OpenShift and MiniIO for internal files. It can also be used as a poor-month database. We also use it to store some files and to distribute the status across some pieces. And again, it increases a lot of the service portability because you don't need to, you don't care where it's running. Like for example, with an NFS volume, you would need to mount that volume on all the places you wanna use it with S3 as long as you have access to the server. It's all fine. For example, we use this as Ccash for caching Git repositories. So we don't clone all the kernel trees every time for pipeline artifacts and the database, as I said, for configurations on some static files. The third thing is containers. We try to turn everything into container. This allows us to forget about infrastructure. Don't worry where the things are running. Just we distribute the packages like this. We don't install packages. Just put them into an image and use the image everywhere we need to use it. Like for example, we use AWS Lambda for the web hook bridge, but we also use GitLab runners and we don't care where it's running. We can run the runners on GitLab or sorry, on Docker or Kubernetes or some disposable machines. We don't care about that. The next thing is detection. As we said, we have a lot of moving pieces. We build and test the pipeline, we have microservices, we have Chrome jobs, everything running on AWS, OpenStack, different Kubernetes clusters. So we need to log and monitor all these things running. For the logs, we tried to pipe everything to DevNull, but that didn't solve our problems. So we had to improve this. We found out that Loki was a really good solution for it. It's basically like a Prometheus, but for logs, it helps us index the logs and have easy retention policies for the logs to be rotated easily. For the metrics, Prometheus is an excellent. It's so good that we realized that every application deserves a slash metric endpoint. It allows you to monitor the internal styles of the services. You can know how a service is doing, how long is it taking. And for our Python applications, most of our code is Python. So using Python client, a Prometheus client, it's really important for GetSolution. It handles everything. You just need to define the data you wanna expose. Everything, it's visualized on Rafaana. We have dashboards like this, which allows us to see what's going on and have some alerts if something goes wrong. And at a higher level, we use Monit. Monit is a simple solution for monitoring. You can write your own shell scripts and make them fail and Monit will alert you about that. But it also has some basic, really useful checks like hosts, uptime, file systems size and capacity. But we also like, as I said, with our custom scripts, we check for bigger hosts queues, S3 bucket sizes and RabbitMQ queues size. And not acknowledged messages. This is really important to also recall all these problems because as you can see here, we can have a good track of what happened and when. And like at last, if everything explodes, you want to be the first one to know when something goes wrong. So we use S3 which catches all the problems on our code and we can have some nice tracebacks and be aware of what's going on with our applications. We use sentry.io for that and an internal instance for the internal code. All these, it's not really useful if you don't have alerting. So we have all these low-key Ruby queues and Monit things go into IRC which is our main channel of communication but also are sent to the mailing list if anyone reads that. But it's the most important thing, it's the IRC because it's where we spend all our time. But the thing is that even after you realize that something went wrong, you need to recover from that. So we have on the most basic level the worst problem, it's the network always as that cat was afraid of. We have a lot of network issues. So we added retry loops to every network access in the pipeline and on all the services. So we know that any transient network failure is retried several times. But we also used, as I said, the re-queuing of messages in RabbitMQ. There's some information on our blog about this system but basically we just re-queue all the messages indefinitely because as nothing is as motivating to fix a bug as a message full, as a queue full of messages. With all the alerting, we get alerts every time a message fails, so it's really easy to, I mean, it's great so you don't lose the failures, you can retry the failures, you can go and test locally with that message that it's making stuff fail. So it's really helpful. And the third, when all these things went wrong and didn't catch the problems on time, we have the pipeline herder, which is a bot that maybe sits the pipelines and if something failed, it knows about many failures that we know we can safely retry. So it goes and checks the logs of the jobs and it retries them for us so we don't have to click anything. If that's not enough, we have failed fallbacks. As I said, we have different clusters, we have different OpenShift clusters. So we can change the runners if something goes wrong on one of the clusters we are running, we can fall back to the others really easily with just pushing a button. So Michael is gonna tell us how we tried to do the jobs. That was a good introduction. So Njaki talked about the aspect of how we keep it running but now the system would be the most stable if we don't change it. So I need to say next slide in Njaki Duga. So we thought about how to approach actually our DevOps culture and so, and Steph Wotos also talked about this a lot now in the last couple of years. I really made those things up here but one aspect we consider pretty important is that it is as open as possible. So this is about code, about documentation, about how we work, about communication. But it also needs to be safe. So everybody is scared of certain pieces of code and we try to minimize them so that if you change something or if you have to change something, you feel safe and are more or less also eager to clean up pieces as it becomes apparent. And the third piece is that deployments need to be painless, production deployments, but also deployments locally, even if it's not really deploying but just running the code yourself just for figuring out whether it should work. What also going into staging and cannery deployments that might be hard to do if you don't have a lot of experience in a certain area of the code. So I think the next slide starts with the openness aspect. So openness in our case means, of course, open code. We have a lot of repositories for all these pieces. We tried to consolidate them where it makes sense but still it's a lot. And the open by default is actually something we try to live by. Now, there are certain pieces that we are not really comfortable putting them, for example, in GitLab.com, mostly secrets, internal documentation configuration. Our deployment yamlers are still internal. That is something that we want to change. Also the Ansible code we've learned a lot from what the Fedora infrastructure people put online and we want to do this but at the moment there is still this encrypted secrets file on there where we don't want to just have a password between our internal infrastructure and the world. So this piece is in progress. We've worked a lot on documentation and there's far more to do. Trying to find out what our audiences are. We are a service so we actually have users. Current developers that are more or less forced to work with us, we want to make that as painless as possible. But we also want to document the individual pieces. There are readme files or there should be readme files on all of those repositories. And something we've worked on is trying to inventurize how all these pieces fit together. We've shown that from CE and I will show an example in a minute. And again, we still have internal documentation. So we are not there yet. So this is, oh yeah, there's more to that point. And yeah, one of the things that work well is having documentation Fridays so that there's actually an excuse to work on documentation. So this is an example of what the inventory looks like. It tries to describe a service or a crunch up in this case, what it's good for, where you can find the code or where you can find the deployment, what it depends on, what other pieces might depend on it. And this gets rendered in our website. So on the next slide you see an example this looks like on our homepage. So that you can actually start to explore stuff you can click through. The idea is also to use that, for example, for monitoring so that you can define in this structured YAML format your monitoring or even alerting connection between deployment and code and put more metadata about how it all fits together in an abstract way, let's say, but still should make it easier to get a feeling how it all fits together. We try to be open in our daily operations. That means moving issues from Jera to gitlab.com which is also quite a bit nicer for the features we only use from Jera which are really nicely covered by gitlab.com. You have an open channel that's mostly populated by the bot that is alerting about everything but also all the discussions happen there. Everything is merge request based so there are no direct pushes and everybody can basically subscribe to the merge request or the projects and chime in. We try to formalize our feedback process a bit because sometimes discussions happen on IOC and you might not be online so you might miss them. So for more general stuff that might be a good idea or where you just feel you won't have feedback. You have an RFC process which basically means you put up a merge request to the documentation repository from the next slide as an example and you can ask for feedback. It's still, it's a feedback process explicitly. It's not, I mean, you have processes like ADR, architectural decision records where you actually ask for decisions. In this case, we are an HR team. We trust each other well enough so that you can actually just ask for feedback and we trust that whoever put this up will do the right thing afterwards. So they are the domain expert. They get feedback and then they can carry on whatever they wanna do. The next big chunk is safety and in this case, safety to change. So we want to make that as safe as possible. We want to make reviews as painless as possible that means linting and testing as much as you can. We try to linch shell code, shell in YAML code, all kinds of Python linters that we can get our hands on unit tests, markdown, linting, link checking, everything we can think of to automate and also to have less to review because if it's automatically linted, there's already automated feedback coming. Means also we have a linting script that you can just run locally and we just work hopefully independent of your explicit Python setup. You're still working on using formators. The packet team is quite a bit better or like further along with this trajectory. So using something as black for our Python code might actually take pain out of the formating aspect, let's say instead of just using Flagate, but that's the work in progress. And one other aspect for the testing is that these dependencies that we have shared libraries that are reused in other places. And so if somebody changes the library, we don't get one dependent testing pipelines. And so this is still on the list to do, but should also happen in the next couple of weeks. As I said, we split up into microservices also because I'm pretty old. I need to fit pieces into my head. So I have the tendency to simplify things. And that also makes it less scary because if you have the feeling or if you have a good understanding or at least you think you have a good understanding, it's much more fun to change code and to clean it up. And also in our experience, microservices are structured better. So there are some interfaces that you more or less need to either explicitly or implicitly define between those pieces. And that actually makes structures more clear. What's on the next slide? I've no idea actually. Oh yeah. And the last aspect is how you can actually deploy code. So we are basically, it's always nice. So we say we are an HR team and that means for us living on the edge. So if you merge something to main, it gets deployed one way or the other. And the way that works for us is that you get a review but you are responsible for merging it and for actually making sure that any fallout is handled or to revert whatever you change. So it's your code. Reviewers can give feedback but they are not responsible for whatever you want. So it's a process that's working quite well. It's very important to have some restraint at the end of the day or just before the weekend and it's a good idea to shift the deployment to the next day or to next week. And Jacqui already discussed the structure of what we have. And now for deployment, the last two types of stuff we run. So in turn the microservices and packaging components in there are slightly different in how you can deploy them. So for microservices, those are containers. We package them up. They should be policy free and code should be structured in a way that there's a non-production mode that makes running the code safe. Like for example, if you normally send emails, if you set this to false, this production is false, it shouldn't send emails. Otherwise you get current developers yelling at you. We centralized all our deployment yammers for OpenShift, all the Ansible scripts into one piece that allowed us to actually clean it up and get some structure in there. Also to get an overview, what do you want? And as everything is deployed automatically from there, if you change something, everything is redeployed. It's even potent Ansible and in OpenShift or Kubernetes yammers, so they shouldn't mess things up. But it also means that it should work. So if the deployment code is brittle, you will notice because it runs quite often. And it also keeps people from messing with the infrastructure in a manual way because basically their changes get over in the next time such a pipeline runs. Now, normally for microservice, you want to try it locally. We have an helper to build this container image. You can also just pull it down from a merge request. All our merge requests built all of those container images continuously. You get them tagged per merge request. You get them tagged per pipeline. So whatever you are interested in, you can get from GitLab, from the registry there might be faster than building it locally. Now, if you want to run those container images, that is always painful because you need to have some configuration on them to make them do what you want. Most of the pieces have a command line interface that actually allow you to check the microservice, how it processes some certain aspects. For example, for the kernel web hooks, they check merge requests, whether they have certain labels set, stuff like that. And so they have a command line interface to basically check a certain merge request. For example, you can see what they do. So if you would think of function as a service, most of our microservice are mostly event processing. So you can basically do one-shot run on some event, cloud event, something. But so yeah, it works reasonably well, but getting them up locally is a bit of a pain because you need to have this configuration and it's not something that you can easily share because there might be some production credentials in it. For the next aspect is actually how do you can, can you actually do a testing deployment of them? Now you have this non-production mode. So either you run it locally in this non-production mode or remotely on a Kubernetes cluster doesn't matter. So you can just move it there. There's a shell script that nobody runs because it's painful to run a shell script. And so the last thing we've worked on is basically making that automated by the press of a button in GitHub. So you have a merge request, you have this beautiful button that says deploy it into a merge request environment and it just spin up a comparable deployment config in an OpenShift Kubernetes to try it out. And okay, the idea is that you can find a link and then inspect the logs, for example. Same goes for production deployments. Normally it's deployed automatically when the metric is this merge, but if you want so, sometimes to find out whether something is really fixed on your code, you can also deploy a merge request into production and that will survive a redeployment of this deployment repository. Basically it takes a container image as being production and within two minutes it should just appear on OpenShift and you can actually figure out whether you solved the problem or you can press another button and another job to roll back, which should also be quite painless and within a couple of minutes. Now for the other big piece, which is the pipeline. So this is, the pipeline is always a nice work, but it's basically a lot of Python tools duct tape together by a lot of Bash, 2,000 lines of Bash, I think at the moment, which is really hard to test, but you can test the components. It's actually the way you should test them. They all have a command line interface. You pip install them and there you go. And that is actually the way these tools are also developed. For actually running the pipeline itself, we support cannery deployments. So for all the, that's on the next slide, you can run your non-merged code in a pipeline. So basically normally these pipelines are just container which is they do a pip install in the beginning of the main branch and you can override it to say like, I want to have my branch instead of the main branch. And we have a bot that will help you. And if you're nice, you can ask it a lot of complicated questions, basically saying like test this old pipeline that passed a while ago, used the same kernel revision. It should still pass, and you can modify also parameters if it's necessary for testing your changes. If you, for example, mess with the pipeline yaml, but also, for example, if you, for example, change how the beaker provisioning works, you can actually figure out whether you broke the production setup before it gets merged. Okay, and then production deployments is for the pipeline, basically you merge it, you get it as soon as it hits the Git repository, it will be used for the next pipeline that comes around. And if you need to reword code, it's a Git reword and the button press away. So that's also quite easy as well.


My name is Wally Repko. I work for the Department of Defense. I've been there now about 17 months and I came there with one purpose and one purpose only and it's trying to figure out how do we leverage all the good ideas that come from the private sector who owns the cyber domain who can help us operators in the military. Okay and what I'm going to show you today is kind of that that approach that has been presented probably 150 times in the last 17 months. It's getting a lot of traction. I'm here to show it to you now because when we get into some of these major war games I either have to have awareness to what the capabilities that you all bring to the fight or I'm going to turn to that four star admiral and give him a phone book. All right that's the challenge that we're facing today.  So how do I leverage between where I am in Washington to where all of you are? Okay how do I build that that that construct because the bottom line today is I will tell you that the government is trying to figure this out. All right now the key is we got to do it non-traditionally. We got to think differently. I am surrounded by a lot of knuckleheads. Okay they're process driven. All right but this is all about doing things quickly and smartly and I can't do it without you. So I'm going to show you what we've been trying to accomplish. This is what I deal with every day. Okay I'm trying to jam a capability into a hole where people just don't get it. You all get it but I don't have awareness to how to get to you. In fact I don't even know what capacity you're at today. That's part of the challenge. There was a gentleman that made a comment last time in the last meeting that said are you only talking to companies? Well there are people within the US government today that focus on the defense industrial base. Okay that handful of large companies, general dynamics, Lockheed, Northrop and all the respect to them but the preponderance of intellectual capital resides with some of these folks here in this area and this is my first Black Hat, my first Def Con and I think my first Black Shirt and I'm delighted to wear it but my interest is to be able to leverage each one of the communities of interest that you see up here. The academics, the research community, young bucks that are out here that are researchers in and of themselves, I need to tap into you. I know you're thinking about intellectual property thoughts, hold on, I'm well aware of the sensitivity. We talk about corporations, we're not just talking about the big guys, we're talking about small businesses, those innovative firms, three or four people that work in the basement, the venture capital community and wouldn't it be nice to have an investment team sitting there willing to help you? In fact wouldn't it be even smarter to have the government co-invest in that opportunity? Now how do we do that? I don't want this managed by the government, I want this managed by the private sector, why? Because I understand accountability, I understand deliverables, I'm all about making sure things happen. That bottom circle, all right, those are the wizards I'm referring to. I showed up in a major war game in May, 600 military and we were talking the Pacific scenario. Part of my team were the hackers and patriots, they're my advisors, why? Because when we saw something happen that I didn't understand, I would turn to that one advisor who had his network of hundreds of advisors and they would go out and say, how would you approach this? That was the value, this is all about the one on many principle. If I know one of you and you have your network, now we have the opportunity to actually work it out. Entrepreneurs, this is all about thinking differently. You're the thought leaders and finally the government departments, but I have a bunch of hard nuts I've got to deal with. I'm not talking just about people, there's a trust issue out there. I work in operations and requirements of the Air Force and I'll tell you that this day not too many cyber requirements come across my desk. That's concerning. On the other hand, I walk down to the hall to a different service, Department of the Army, Department of the Navy and you'll find that they're doing very similar things, but we're not talking to each other. That's ludicrous. If we have a limited amount of money and we're not talking to each other and the domain is owned by you all, we've got to figure out the mechanism that allows for that to happen. So this is what I did, literally. I walked into my office one day and I drew a line down the middle of a piece of paper. I called one side public sector and the other side private sector.  But teetering on that line is government, can't get away from it. Am I going too fast for anyone? This is not rocket science at all. But these are the two questions that keep coming up. Hundreds of companies of all sizes come to me and will say, Raleigh, what are your requirements? And I'll say, what can you do? What's out there? What special capability can you bring to the fight? I have got to have awareness to that in advance so I can build my playbook when a war game happens. So I need something that captures all of that. What I've got now is this donut, this neutral entity that I'm pushing. Some entity that allows us to broker, collect information, and reach out to all the knowledge nodes that you see out there, both on the left-hand side of the screen, as well as within the public sector. So what's a knowledge node? Knowledge node is you. It's the company you work for. It's the solution you may have developed. It's the process. It's some issue that's both domestic as well as international. I, however, have to be able to get to it. Now, there's a mirror image, as you see. There's knowledge nodes that sit within government. And that's a lot of the academic institutions that are out there who do a lot of good things, but there's a lot of redundancy. And I've got to have awareness to the best of breed from all those best practices and databases below. But as you grow this, I've got to have governance. I've got to have some type of board of advisors sitting on the private sector side. Now, how are they picked? I don't have a freaking clue. But I can tell you, having sit and run on publicly traded boards, that how difficult is it to identify a mix of talent from a big business to a small business, to a hacker, to an advisor, to an academic, and have them be incentivized to be part of the team for 13 or 18 months, rotate, bring in new talent, and then talk to the folks on the public sector side, individuals like myself, who have my own network. Okay? This is where the seeding and feeding of challenges ought to take place. And we need to do it through this neutral entity to kind of manage and take care of those intellectual property and other issues. Now, from an executive or from an execution perspective, I currently work for the Office of Secretary of Defense, that big blue circle. But we've got to collaborate with a lot of the folks that you saw at the table earlier, DHS, DOE, Department of Energy, Department of Justice, various states and locals, various countries. Got to have that awareness. But how difficult would it be to establish some type of smart database to make this happen? It shouldn't be that tough. But this is the question that gets everyone's goat, the money. Alright? Look where I put the package of gold. I purposely put it on the private sector side, because what we have to figure out is, how do I award you? How do I reward you? How do I incentivize you to get involved? In fact, how difficult would it be to get venture capital to put money into that pot for specific challenges? Have it be matched by various agencies that you see up on the wall? Now, I know how to track you. I can be darn sure that if I do give you money and you start to produce and do well over the next two or three years, that as a venture capitalist, I'm going to get my money back. But this is an opportunity for the government to get its money back to put into various programs, programs like training and incentivizing issues that you heard earlier. But we need creative minds to figure out what should go into this package. Right now the government is very interested in seeing a framework that works, but we've got to figure out whether the elements need to be part of this. I can tell you right now there are people in the federal government that are trying to boil the ocean. What we need to do is break up the framework into its various elements, bring in the experts, and pursue each one of the areas that are being mentioned here. So I'm going to show you what took place in a major war game back in May, and there's one that's going to take place here in October, which all of you can be involved with.  And it was taking place in an epic year, 12 years from now. We had 600 participants, all the major services, we had countries involved, et cetera. But what I was trying to prove is what's the art of the possible today? All right, so as the game goes through its various phases that you see from phase zero, phase one, in a war game, these typical war games are five days long. Okay, at the beginning everything is green, no problem. And then malware and things start to happen. Both to the network, you've got cords being cut, you've got a lot of issues taking place. While this is going on, these various teams, the red team is trying to make the blue team's life miserable, you've got white players, you've got industry cells, et cetera. Okay? But they'd show up at these war games with six companies. It's like having six companies stored into a safe game saying, hey, I've got some Antec in my back pocket, we're already. But I'll tell you, the challenges that come in this war game are ones that you deal with and we're not familiar with. So what I did was I jumped over that wall. I had a workshop and I invited 1,000 people to participate. What I had to do, however, was to make you familiar with what we do in the military. Give you an idea of what a mission is all about. And then you started to jail. Now, the issues that we brought to the war game were real. For example, if I was to impact the GPS clock, okay, think of the impact it would have on everything from phones to unmanned vehicles. Again, from an operator's perspective, these are challenging issues. Well, when I start approaching that question to 1,000 innovative and creative folks, you saw some incredible responses come back in. Okay, this is what I refer to as the art of the possible. So even though the game was going through five days, which represented 12 years from now, I proved to the DOD that there are capabilities out there that could solve the problem today, tomorrow or in three months. And the responses came in said, I'm a small company, but I can do it if I could find this type of partner. I need a couple of million bucks and I'll do it in three weeks. Three weeks, not 12 years, right? What that does for us is I will incentivize you and raise your awareness to our government leadership saying there's capability out there today. And so as you look at the flow here of us reaching out to all those knowledge nodes, not just small business, but labs and international and academia, we got all kinds of inputs. Now, there's a really important point here. This war game was classified. All right? So part of my team was to sit there and take each one of the injects, those challenges that took place each day of that week during that war game and sanitize it, write it in a manner so that I can send it out to all of you. In essence, I'm extending my reach, my operational reach to participants that have never participated in an exercise or war game in the past. And the responses were incredible. Some were a little crazy, but we got some really fascinating inputs.  Okay? That allowed us, as we are sending the questions out to the free world, to protect in essence what our vulnerabilities are. But we couldn't make the question so generic that you didn't understand what we were talking about. So we knew that if you knew how to respond to the question, in most cases, we knew that you knew what you were talking about. All right? And when we got an input that looked interesting, we then contacted you. All right? Now, when you started bringing your responses back in, we used the anonymizer to protect your input because your ideas, your creative ideas were ones that needed to be protected. So that was our approach to intellectual property. But the bottom line was, we used a 501C3, some entity down in Bozer City, Louisiana, called the Cyber Innovation Center, to be our clearinghouse. So we took a classified inject, sanitized it, went to a dot com node, transmitted it to the dot com node at this neutral clearinghouse, and they sent the requests out to the world. Within two minutes, we started getting some responses. Within five days, we had 128 inputs that came in, and they still kept coming. Now, we had to validate. And part of my role was just to prove to the Defense Department that we need to extend our reach, get out and be able to reach out to those knowledge nodes because they'll be willing to respond. Again, we haven't looked at how we incentivize, that's where you go out and say, how would you like to be incentivized? These are the questions that I'm interested in finding out from you. The bottom line is, we went through a hot wash, the four star, the secretary, Defense Department leadership saw the results. As a result, now I'm building this for the Defense Department. And we're going to be testing this in a major war game is going to take place in October called Unified Engagement. Now, this game was strategic. The game that's coming up in October is operational. It means I need an answer immediately. But unless I know what capabilities can come to the fight so I can build my playbook in advance, I'm not sure where to throw the pass. That's the challenge that the military has today. They'll show up at a game and they'll have a handful of tools, but they haven't leveraged the intellectual capital that's out there. And to me, that's concerning. So when you look at the collaboration piece in and of itself, these are all the various steps that you have to go through in today's environment. The military operator says, I got a problem, I need something. And we got to go through requirements developers, programmers, etc, etc. And that process takes a year and a half.  So how do we do all these various steps quickly? That's the thought process that I'm applying today. Now, I just as I mentioned from an approach perspective, I've got to have awareness to what you all do in advance. I need help there. What we do is we host a workshop, we open up to the public, you come in, we have breakout sessions. And before you leave that workshop, I have at least knowledge of three core competencies that you or your or your specialty area brings to the fight. Okay, that's how I build the playbook. Now when I show up at the war game a month later, when I see something that looks kind of funny, and I know that you're the solution, I throw the pass in your direction. Okay, sanitizing the war game. The next big challenge that we have right now, the government is very open towards getting away from traditional clearance management. Okay, and it's not perfect yet, but this is an opportunity for us to now reach out to you and allow you to respond to our challenges by sanitizing a classified issue, the trust issue. In order for us to maintain trust, we even in government have to always be talking to each other.  The problem is, the guys, the leaders that wear the stars on their shoulders that push the button, if they don't know anything about you, and you may be sitting 10 feet away from them, they're not going to rely on you, because they don't know you. They don't have the confidence in what you bring to the fight. So we've got to ensure that we're always communicating. They have to have awareness to what you're all about. Using a neutral clearinghouse is a new way of thinking, because that entity becomes the broker, it becomes a repository. And if it's neutral, it doesn't have a stake in the game. It's not Northrop Grumman, I'll do respect to them running that entity. It's some neutral entity like a 501 CX that allows us to do everything that we're proposing here.  At the same point, you're helping out the operator. But here's a really critical issue. The right mix. When I show up at a war game, I got to make sure I have my Jack Bauer team. Okay, they got to be the top of the game. They have to know their business. They have to know where to look. They have to be able to assist us with the game. And it's not just all Air Force or all military. It's a mix of individuals that are in this room that make this palatable. At the end, we go back and we look at the lessons learned and we throw the questions and the challenges back out to you saying, how could we have done this better? All right. That's what this is all about. That's the collaboration that we're looking for. And again, this is not for free, because I know for many of you, yes, you have a lot of patriots out there, but we're here to incentivize your attention. Okay. How do we do it? Well, I need some creative ideas there.  Well, that makes a lot of sense because there's a lot of models that work from that. But we've got to take a look at that carefully. And then we've got to continually rehearse a lot of these war games, these tabletops, so that we're proficient. These are the types of companies that I brought to the fight. Okay. So when I talk about Black Hat, I went to Jeff Moss, I said, Jeff, I know you have a lot of talent in your back pocket. When I see a challenge related to, you know, some type of malware that I have no clue what it is, I'm going to share with you. I don't need to know who you talk to. Help me get an answer. Okay. There's one little company up there called PGMP, Pretty Good Malware Protection. It's like three people out of Austin, Texas. All right. This is the type of talent that I see out here today. These are the groups, again, one on many. I go to trade association. I don't need to look at every company. I go to the leader of that association. I say, how many companies in tech America know anything about cyber? How many understand CNA, CND, CNE? How many of them are small innovative companies? Okay. They should have that answer if they don't. Basically, that's part of my concerning issue. So in conclusion, I'll be delighted to take any questions you may have, more than 85% of the critical infrastructure is managed by you all. So when the government comes in and tries to modify the acquisition cycle and says we need to do it ourselves, without talking to the private sector, there's something wrong with that discussion. Okay. We need to really think about how do we leverage the talent from the private sector, which includes big business, small business, academia, patriots, hackers, venture capital. We've got to be able to think beyond the dip, all the respect to the defense industrial base, a lot of the innovators move a heck of a lot quicker than those big companies. Trust, huge issue. How do I go beyond that? I got to have awareness. I got to be able to know who you are. I got to work with you. I got to understand what you're up against. Having capability awareness is a real issue. I will tell you that I've met with hundreds of really cool companies that come to my office and they sell a certain bell or a certain whistle. But the concern that I share with the Defense Department is this. I don't know what capacity you're at. So if I get hit in a major malware and something really happens and I tap you on the shoulder and say you're in the game and then you tell me, well, Raleigh, I'm only about 20% there. I got a problem. I got the clock ticking. I got the four star up there wondering how we're going to fix it. I got to figure out how I'm going to integrate a challenge while the hacker is doing what the hacker does well. So I've got to have the awareness of what's out there today. I believe that the neutral, you know, having some type of neutral clearinghouse, something independent has real promise. You know, I've written about this online. There's thoughts out there, but I'm wide open to suggestions on what ought to be included inside of that entity. It's not just one entity. It could be hundreds of neutral entities all talking to each other and managing all the knowledge nodes of companies and capability are out there. Okay. The key here is speed of need, not speed of thought. And this is not about me. This is about we. And we're trying to leverage that. The sanitization process was not easy for me to get the blessing from our leadership, but they knew full well that because the preponderance of expertise is out there, I needed the ability to go out there. So they provided me with sanitizers, specialists in security. Now, how difficult was that? I just asked the question. Okay. It works. These are the types of new thoughts that we're trying to get. And finally, let's stop trying to boil the ocean. Okay. This is hard work. But if we break it down into the elements, and if you're a specialist in one area versus another, well, those are the various teams that we need to do. In all due respect to the White House reports that came out that said we need to have a czar, and I got that. We've been there, right? That's up here. Now we need to build something, take action. And that's what this is all about. Okay. So, I can tell you the last 17 months has been a tremendous opportunity, but the key here has been I don't do a certain, I don't do a day without talking to my private sector partners and I meet more of them day by day. And that's really the incentive that drives me through that parking lot at the Pentagon, and has me work long days to try to figure this thing out. So, that concludes my presentation. I'd be delighted to take any questions you have. I still have about 15 minutes, but thank you all very much. Question? I think they want you to use the microphone. Yes, sir. Do you think there's overlap in your mission with the CIC? That's what you're trying to do with the CIC with U.S. SIRT or SIRT-CC or the MSISAC and what they're trying to do in bringing some of this public private partnership together? Right. I mean, the issue of overlap is always a concern because, you know, today there are a lot of investments in the various organizations that gentlemen mentioned. I reach out to all these organizations. All right? I started out Air Force only. I knew that in order for me to be effective, I had to go to DHS. I've been to DC-3. Steve Shirley knows me. Well, same presentation hasn't changed. Okay? What they like is a framework that now works. Why? Because each one of the organizations that you've mentioned is a knowledge note in and of itself. So think of it almost as a charm bracelet. I'm trying to build the mechanism that is the bracelet itself where each one of the charms from DC-3 to the ISACs, etc. hook on to that. Okay? We're not competing. We're enabling. All right? So but we got to have the mechanism that allows, you know, the preponderance of information to flow. So yes, I do collaborate with them all the time. I'm wide open. Again, this is not about me. And the cost is in not talking to those organizations. Thank you. Yes, ma'am? Hi. Hi. I wanted to ask you what you meant by, like, capability. You've spent several sentences on capability in terms of knowing it in advance and treating it as if it was something that could be quantified. Right. Is that correct? Is that your definition? It is. When I use the term capability, I'm talking not just about a tool. I'm talking about a person, a solution, a process, you know, something that will enable the fight. And we're talking about integrated capabilities. So sometimes I use the term to represent all of the above. But when I'm referring to capability, I'm referring to that type of cyber knowledge, etc., that's out there that I can leverage. Yeah, I guess something that I can tell you from my experience doing different open source projects and just working in sort of this community is that capability is extraordinarily dynamic. And I think that you're going to run into some problems with the different definitions of the word. Capability isn't really something you can determine in advance with any, you know, hard, 100% got to know it. And the mindset is that the capability emerges from the situations we're put in. That also has to do with your question about incentivizing. Money's okay. Money's great. It buys stuff and we get paid with it sometimes. But incentivizing, at least for me, I'll work on a problem for free if it's interesting, you're going to have to express them in a way that really engages the imagination. That's going to be up to you to be able to express those things. And these are, I mean, from the capability issue, terminology is huge here. I mean, I, for one, understand that.  People come in and say, you know, don't see tomatoes, potato. Got that. I'm wide open for that. As far as incentivization, I will tell you that the government is trying to understand, especially in OSD. OSD is moving pretty quickly. They're very interested in figuring out how to do it. But we're not quite sure on what will incentivize you. Because I've met with folks saying, I just want to be part of the game. I love this stuff, man. This is great. I got that. I'd love to be able to categorize those folks saying, these are in it for, you know, because they're enjoyed because they're patriots. These over here are a bottom line. They want cash. Okay. What type of cash?  Because we don't know how to do that yet. Okay. And the reason is to turn to, if you've ever worked with the government, we have what's called the FAR, the federal acquisition regulations. It's called nightmare. All right. And that's the incentive of why most people are not incentivized to even work with the government. So if we had this clearinghouse that could very quickly allow us to, you know, purchase that solution, that service, that tool, whatever. But what the government in many ways are considering today is, they mean well, but they'll turn to their typical government group. If you're in the Air Force, a handful of majors and Lieutenant-Colonels and say, I need for you to think differently. Okay. These are folks that never spend a single day in the private sector. I need for you to change a 40-year-old acquisition cycle. Okay. Do it quickly and make sure you incentivize those that are out there. Let me tell you, you don't even want to read that stuff.  You know, yes, you also want to know that when you do participate that you get some type of attention for it, whether it's monetary or just that representation, I get that. Okay. How does that fit into the clearinghouse? Help me out. Okay. Is it a blog? Is it an active thing? I mean, are there new ideas out there? The more clarity I can get, the better off we are all going to be. Great question. Thank you very much. Yes, sir. You talked a lot about the framework and I was just curious is if this were successful and created some useful byproducts, would your plan to be to open that to the public or would it be to keep it within DOD or government or something along those lines? If I didn't make it clear, this is being proposed to be really the entity for public and private sector participation. The last thing I want is for this to be managed by the DOD or DHS. You've already heard from our last panel some of the frustration that takes place. Okay. So you've heard of FFRDCs and 501C3s.  I mean, if there's something new out there, let me know what it is because we ought to make sure that legally it works. But the key is when good ideas start to permeate, we've got to be able to share those. So that is a key byproduct. Okay. But hooked into that clearing house are all the good, are all the various entities. So right now, this framework is a little at 64,000 feet. As you start to break it down and say, okay, you may, we're part of the framework team. We're going to figure out what goes into that framework. Well, we want it to be a common operating picture, want it to be a repository, want it to be this and that. Okay. Those are the types of things that need to go into that. And of course, we've got to be able to share the information. Thank you. Thank you, sir. When the military was looking at how to manage kind of the out-of-war component of its resource pool, they turned to essentially the National Guard component that said, why don't we have a component of soldiers who train, not every day, but train infrequently and are essentially ready to be called up into service whenever this happens. Have they considered a kind of National Guard in a sense component for this, where you in fact incentivize by saying, okay, we can make it an almost kind of paramilitary type arrangement. You can put it in a infrequent training arrangement where you can say, we're going to pay you, like they pay national guard soldiers. You get a training component that's in place and again, a kind of disciplined, flexible component that could be brought in on a regular basis, done in that setting where the incentive is to say, yeah, you're going to go out and work a weekend every six weeks or whatever, or one weekend a month or however the National Guard is structured, but do it in a way that it's additional income, but it's also recognition for what they're doing in terms of ranking. Yes. I mean, what you're identifying is a model that exists today. Now, I'm a retired Air Force Reserve member. Guard and reserve make up what's called the ARC, the reserve component. Now, just assume that these individuals that participate in Guard and Reserve, they have other jobs and many of them do the things that you all do. They're the closest thing within DOD to the private sector. We know that. So in that war game that I just showed you, the Striever war game, one of the nodes when I was presenting it to our leadership said ARC on it, because it's easy for us to go into our files and identify Captain so-and-so, Major Lieutenant Colonel so-and-so that works in this company, that company who brings certain types of expertise there. Okay, so absolutely. These are exactly the types of nodes that we're hooking into. I mean, having spent 22 years as one of them, there's huge value in leveraging the expertise that they bring to the fight. It certainly seems that foreign governments are doing this very type thing. I mean, that's exactly if you look at Brazil, if you look at China, you look at the way that they're essentially bringing on very young individuals, but essentially creating that kind of paramilitary arrangement where they belong to this group and that group has an affiliation and allegiance to the group that's being, that's training. I understand. I mean, as I mentioned in the last discussion, the military thinks very traditionally, all the respect to that. They're regimented, they're here to take care of all of you. But when it comes to the cyber domain in which the intellectual property resides with you, our approach is to basically walk up to an engineer in our military, wave a wand over him and say that you're now a cyber expert. Does that make any sense?  So we need a new mechanism. How difficult is us for it to have a couple of cyber experts, but form a relationship with the private sector where you help us jump start from where we are today to where we need to be tomorrow, which is at your level of competency. That's the partnership that has to take place. Okay. Thank you very much, sir. You've, excuse me, you've touched on a couple of the issues that inherently make this a difficult relationship with the FAR. My initial question was going to be once the capability was described as being available to you in three weeks. How long did it take you to procure that capability? I suspect it took longer than three weeks. With that in mind, this community, the black hat community, inherently we come together for a love of attacking processes and finding the chinks in the armor, the breakdown in the processes. So to approach it with the traditional idea of process driven frameworks is going to be a difficult marriage indeed to achieve with this, with this kind of culture. Have you considered looking further again at the open market model and essentially becoming, if you will, a large investor in that model in such a way that based on performance, we reward the performance rather than rewarding maybe what is going to happen out into the future and trying to fit a niche or a requirement now rather than it's not perfect, but clearly we've run economies, we occasionally have breakdowns, but I would suggest that our country has a long history of based on past performance going with investment into the future. Right, and I clearly understand where you're coming from. So to be able to reward, to be able to identify, I mean for many of you they had within the Department of Commerce years ago they had what was called the Malcolm Baldur's award, reach out to the world and identify all those good things from companies and give you the recognition because these were companies small, big and large whatever that got great attention. When I talk about bringing the right mix of talent in, it's bringing in the economists or the people who have clear, creative ideas and saying, okay if I got a bag of money and one of my rules of engagement is to ensure that we incentivize, we meet the following issues, et cetera, with the right people in there we can put together a pretty good package, okay. I'm not that expert, but when we identify and say in order for us to fix this element we've got to bring in the right type of talent is, in we'll be able to identify and address those issues. I'm all about open source. This has to move quickly, right. Again, this is a collaboration model. Yes, there's a process tied to it, but the key here is I'm trying to enable things. So how best do I do it? If you've got a good idea, I'm in a position to say let's try it, okay, because what I'm bringing to this fight now is the ability to turn to a four-star admiral and say, sir, I've got your solution. It's some company that came out of black hat or from this particular session, because you knew how to provide an answer. He won't care any more about what you're all about, but will care that you brought a solution to the fight. I'm trying to build a mechanism that puts the two together. Okay, sir, I think I have a chance for about one more question based on time. Yes, sir. My question is also around incentivization, but in a bit of a different way. Foreign governments often times celebrate their independent researchers or whatever you want to call them, the hackers. Whereas we get a lot of mixed messages. You have the DMCA and the upcoming ACTA, which try to squash independent research, security, vulnerability research, exploit development, that whole deal. That's going to hamper us in the future, and I think what would help us is some heavy hitters that go in there and say, let's get this legislation right. I'm tired of being told to go talk to my congressman. I think if we had some heavy hitters in there saying, you know, let's get the legislation right to actually encourage people to keep closing the holes in the software and to develop these skill sets, that would make a big difference. I applaud that question.  Look at the size of Black Hat. Look at the size of this organization. You've been around for 18 years for God's sakes. You've got a supporter in me, okay? However, I need to be able to rely on Black Hat. I could see that this organization between Black Hat and DEF CON form in their own legislative team because look at the numbers of people here. The legislations coming out of Congress today, out of the White House, out of Capitol Hill, etc., are looking for answers. However, they're turning to the same traditional mindsets, and the thought process that they're throwing into this legislation just seems so basic, and it's not going to solve the problem that you're all looking for. Part of the opportunity here is to be able to have a voice from not by Congress, but by you all, because I'm talking the numbers that you bring, I guarantee you the members are going to listen. This is my first time here, and I look at the creativity, and I enjoy my Black Shirt, and I'm really impressed by what you're all going to bring to the fight. However, my biggest concern is how are we going to leverage this? And whether it's from Moss and his team or the executive body that makes up these two groups, the cost now is in figuring out how we can make sure that there's an ear in Washington that's listening to you. Thank you all very much. I'd be delighted to answer any questions later as well. Thank you.


All right, so I want to come back to infrastructure. You're talking about moving beyond infrastructure, but there's still a lot of action going on there. Cloud Air, MapR, Hortonworks. You have Adapt now entering the scene. What's going on there? Who's winning? Who's losing? What are the key factors? Well, yeah, to my earlier point, it's not to say that infrastructure is not important anymore. There's still a lot of development to go happening and a lot more improvements need to happen to make Hadoop and other, early Hadoop, quote, unquote, enterprise ready. So what's happening there is, the way I look at that market is, you've got kind of the three big players that have been in the market now for a year or more. Cloud Air, Hortonworks, and MapR. And the way I look at it is, MapR has taken the more proprietary approach they're delivering. Their premise was that the open source world is just moving too slow. We want to add some of our own IP, harden up Hadoop, and get it to market as quickly as we can.  They're doing some great business. They're growing and they've got some great customer-proof points. Then you kind of got Cloud Air, which is taking a little bit of a hybrid approach. They've got some proprietary management software around Hadoop, the open source core. And they've been in the business the longest, so they've got that lead, which has helped them become the biggest of the three. But they're kind of playing the middle game, and then I looked at somebody like Hortonworks is really playing the long game. Completely open source, not charging for their product at all, giving it away, seeding the market at this point essentially. And they're going to build revenue off their services offering, but that takes time. And they've raised a lot of money, and they're not in a rush to start really building revenue. And that's Rob Bearden's specialty, right? Absolutely, he's done this before. So the idea is let's seed the market with our product, let's get people using it, let's get people excited about it, and then as they start to go into production in six months, a year, two years from now, we'll be there to provide the services, and that's how we'll monetize our business. That's the long game. So we've kind of got MapR with kind of the short game, Cloudera in the middle, and then Hortonworks is the long game. And then you've got little startups like Hadaptu and some really interesting things. Hadapt is very exciting. I mean, we saw those guys, they won the Best in Show at Strata in the fall. My sources indicate that they've actually got some serious customer traction. I don't know if you've heard that. Now Impala is Cloudera's announcement. Essentially what these products do, if I understand, is they bring SQL and NoSQL together, bringing real time to big data and Hadoop, essentially, and Hadapt, we had Ming Sheng Hong on theCUBE a while back, and he was really taking us through some of the functions that they've developed at Hadapt, things like the ability to do user-defined queries, which are sort of new in this whole world. So it seemed to me, Jeff, that they had a little bit of a technical lead, even though they maybe didn't have a market presence lead. Now Impala is the sort of, you know, Cloudera's version of that, you know, people are, you know, the knock on his arm, he was rushed to market. Cloudera says, no, what's your take on all this? Help us squint through all the fud. Sure.  It's not ideal for running what you might call fast, near real-time analytic queries at hot queries. It's a good place to run really deep, complex analytics against large volumes of data, but that takes time. MapReduce takes time to run, depending on the size of the query and some other criteria. So the idea has been, thus far, for most practitioners, early adopters of Hadoop is, let's get, almost using Hadoop as an ETL layer, let's get all our data in there, let's put a little bit of structure around it, and we want to do the kind of fast queries that we're comfortable with, doing it in technology like Vertica or Green Plum or Teradata Aster will move the data out of Hadoop via connectors into one of those databases, and then that's where we'll do our analytics. The idea is, the idea is clearly, she was not particularly interested in that talking point, but nevertheless, the idea with what bringing the SQL aspect to Hadoop is, well, there's no reason to move your data out of Hadoop anymore. You're essentially bringing the SQL database into the Hadoop environment and running SQL directly on top of Hadoop. So that's what Hadaap's doing, kind of from a native framework, they've started their whole company, the technology is from the ground up built to do this. Cloudera with Impala is kind of building this as an add-on. This is the latest development on their platform. So it was- Was the timing of Impala designed to try to freeze Hadaap? Was that sort of a ploy, or was it just sort of two companies on a parallel universe? Well, yeah, I don't know if it was a ploy. I think it may serve that purpose to a degree, but this is where- So Impala's real in your view. I mean, it's getting traction in the marketplace, getting customers. Well, I didn't quite go that far. It's still very early for Impala. As far as, the latest I heard is still in private beta. It's very early days. I've done a little bit of research on the number of contributors to the, because it's an open source project, Clutter Open Source, the Impala project, and there aren't too many, just a handful of most Clutter employees. We should try to find the Hadaap guys today. I heard they're gonna be here, although I haven't seen any of them. I heard they're in production, so let's try to confirm that. Yeah, so it's very early days in terms of Impala, but I don't think it was necessarily time to impact Hadaap. I think it's just a natural evolution of Hadoop. I think we've done some research and we've talked about our fundamental thesis that we believe a unified big data platform or data infrastructure is the right way to go. The idea of connectors and moving data back and forth really is just a stop cap measure for now. So I think to a degree that the market's listening to some of our insights, and to me it was just the next kind of evolution, and I'm not surprised to see it, and I would expect similar moves from Hortonworks from others that are playing in the Hadoop space. You think we'll see an acquisition this year of one of those guys that we've just been talking about, one of the infrastructure players, a big move to try to shake things up a little bit by one of the whales who really wants to play in this game? Yeah, I think we will actually. Where do you think it's gonna go? My inclination, going back to my analysis of the three major players, I think the least likely is Hortonworks, they're very early in their cycle. Kladera's the most mature in terms of being around for the longest, but MAPR is really got a lot of momentum. So I would say it's either gonna be Kladera or MAPR, the most likely candidates for an acquisition. Kladera's talking IPO, I mean, I don't know how real that is, but. You know, it's certainly possible. I think in terms of potential companies that might want to do the acquiring, I mean, they're the ones you expect, I think IBM and Oracle are probably top of that list. Microsoft, not so much. Why not Microsoft? I think they seem to be taking a little bit more of an open ecosystem partnering approach and a little bit slower approach. They've done their partnership with Hortonworks for the HD Insights platform. So Hortonworks might be the natural company they might look to if they're looking for an acquisition, but as I said, I think it's a little early in the Hortonworks cycle, so I don't really see that happening. But you know, Kladera, take it for what it is, but you know, Mike Olson's an Oracle guy in his previous job and, you know. How about Splunk? Let's talk about Splunk a little bit. Big IPO last year, really doing well. Rabbid fan base, users love Splunk. What's the update on Splunk? Splunk, well, so as you may have heard, they are the center of rumors around acquisition as well. And again, the same players are IBM and Oracle or the two suitors. That news was kind of hit last week, it's gone a little dark, we're not sure where that stands exactly, but they're happy. Where they smoke this fire, they say. Yeah, well they had a great year last year. They're doing very well this year. I don't believe they reported full earnings for the year, but they're probably gonna top $180 million in revenue. They've got great momentum. They're now evolving their, you know, Splunk really started focusing on making sense of infrastructure, IT infrastructure data, machine generated data. And they did that with a series of applications, essentially for specific functions in the data center. What they're doing now is kind of opening up their platform really to outside developers. So they've announced several SDKs over the last, in the fall. I recently spoke with a company called Pre-Lert, which has developed a non-lead detection software that they have built specifically a version, specifically for the Splunk platform. So it's gonna run on top of Splunk. So Splunk's really getting now into the platform game, not just, so they're offering kind of their own applications on top of their platform, but now they're also opening up to developers, sort of like the Apple iPhone model. You've got Apple applications, you can run your iPhone, but it's also open up to outside developers and you can run those as well. So I think it's a good move on the whole. The risk for Splunk in that case is of course, if someone comes along and starts creating applications that kind of cannibalize their existing business. But on the whole, I think it's probably a smart move and they're under a lot of pressure from the company to deliver, so. Now Splunk's one of those companies, so many companies out there now looking at it and say, wow, why didn't we think of that? So that's great, you know, they gotta love the innovation. All right, Jeff, we gotta roll, get more guests. Thanks for coming on.


Hi, I'm Jean-Faye and this is Jean this way, and in this installment I'm going to be asking, what is homo-nationalism? Homo-nationalism is a term that was coined by academic Jasbir Poir in her landmark text, Terrorist Assemblages, Homo-Nationalism in Queer Times, which marks its 10th anniversary this year. And if you're unsure how to celebrate the 10th birthday of a landmark text in Queer theory and Foucaultian biopolitics, then look no further, because I've already thrown the party. At times, especially in activist circles, the concept has become somewhat simplified, just to mean any time that LGBT liberation politics could be co-opted by the nationalisms of the far right. For example, the National Front's appeal to white gay people in France, based on racist fear mongering about Muslim homophobia, or in the US, the rise of pro-right LGBT slogans and groups, including, among others, twinks for Trump. While this phenomenon of fascism or white supremacy co-opting LGBT politics for its own ends is real and alarming, it's not exactly what homo-nationalism is. Homo-nationalism as Poir intends it, is an analysis or organizing principle of global politics, not a personal ideology that you can choose or disavow as an individual. This isn't about good gays or bad gays. Terrorist assemblages was a rejection of the idea sometimes found in feminist and queer theory of the 90s that the nation-state is always heteronormative, and that the queer citizen is always an alien or outlaw within it. Instead, Poir argues that homo-nationalism is how acceptance and tolerance for gay and lesbian subjects have become a barometer by which the right to and capacity for national sovereignty is evaluated. So what does this mean? Well, in other words, it means that it's not about when LGBT people and racism visibly join forces, but a critique, a critique which basically argues that all liberal lesbian and gay rights movements uphold certain ideas of social progress and modernity. In particular, LGBT equality movements rest on the basic assumption that the western nation-state is capable of expanding to include all marginalized populations and capable of being a benefactor to all its subjects. Poir argues that these assumptions and discourses cannot be true as they always afford some populations full legal and cultural citizenship at the expense of others, specifically racialized others, both at home and internationally. A simple example would be the progress towards marriage equality. Marriage has been considered the pinnacle of gay and lesbian participation in civic life, and marriage does provide equality with straight citizens, but it also upholds an institution which, for example, entrenched Western sexual and family norms as supreme and exceptional in comparison to those of other cultures. Marriage also upholds the nation-state as the supreme arbiter of sexuality in the way that it sometimes allows those inside the institution right to remain or citizenship, while others outside of it can be deported. A more modern example 10 years on could be transgender rights. So, for example, as part of the social and legal process of transitioning within a nation-state, a trans woman like me may apply to obtain a female passport. While this can be considered a success of a modern trans rights movement, and an example of the excellence of a liberal nation-state which allows its trans citizens full participation in national life, the very existence of passports still upholds the sovereignty of the nation-state to police its borders, to arrest and detain and deport those who do not have documentation, and its role in using the passport system for surveillance and monitoring of racialized minorities as part of counter-terrorism efforts, as well as all of the capitalist exclusions that come with non-citizenship. Does that mean that I'm not going to apply for a passport? Of course not, and this is what's meant by homo-nationalism being more of a lens to view queer politics through than a bad political philosophy you can simply disavow. However, 10 years on from the release of terrorist assemblages, which was largely considered the leading queer analysis after 9-11 and the war on terror, we find ourselves in a rather different political landscape. How, for example, do queer communities alive to the critique of homo-nationalism respond to President Trump's attempt to ban transgender people from serving in the US military this year? Trump's intention was not merely to signify that trans people are a burden to the health and vitality of the nation-state, but to use the publicly funded healthcare of trans service personnel as a wider testing ground for policies around publicly funded healthcare and, namely, whose bodies are disposable. However, in order to contradict the symbolism of the move, it's difficult not to recreate homo-nationalist discourses, namely that US trans citizens are fully entitled to participate in their country's military-industrial complex, or, to put it more bluntly, that American trans people are just as good at killing brown people overseas as cis people. When LGBT rights for the white and all-citizen become actively forced into reverse, homo-nationalist discourses entrench themselves in response. So what is the solution? Well, perhaps it's to look at the point of universal healthcare instead of the military, and to fight Trump on the position that some bodies are entitled to public healthcare while others are not. Applying the lens of homo-nationalism should be a call for greater debate and critique of what liberation looks like, its shape, its form, and its limitations. Homo-nationalism is a huge, complex, and expansive subject, which I can only begin to touch on in a social media shareable. But if there's one thing I'd like you to take away from this video, it's that it is not a political position designed to separate good politics from bad politics, but rather better thought of as a structure through which queer politics and liberation are formed, rather than providing us with answers on the right way to struggle if only things were that easy. Homo-nationalism invites us to harshly critique our own role in global politics, even as we gather under the banner of queer liberation or anti-racist politics. I want to know absolutely anything you think about anything I've discussed in the video today, whether that's homo-nationalism, whether it's Jesper Poir's book, whether it's passports, whether it's Trump and trans-military personnel, any of it. Hit me up in the comments below, tweet me at Sean Fay, use the hashtag Sean this way, and don't forget to follow Navarra Media at Navarra Media. Thank you so much for watching, bye!


Hey everybody, welcome back to Average. I'm Steph, I'm the Average Artist and I'm so excited. We've got a new type of jelly paint. I'm just going to open this up. It comes from ink lab gouache paint, no fading cup. And the cool thing about it is it's quite cheap. This box was around £14 which is probably about $12 maybe? Maybe a bit more. Not including postage and packaging. Postage and packaging was just standard. But look guys it comes in this like honeycomb kind of hexagon. Hexagon, is that right? Or is it six? What is it? What's the shape? It is a hexagon. I'm smart. Oh look at that. Okay so it comes from ink lab and yeah we're just going to open these up. Right first of all it comes with its own palette which is super handy. Thanks very much won't be needing this. Look at these shapes. I don't know why this is just this is just very exciting to have a different shape of design. Look at that. How cool. It's two whites. Perfect. Perfect. I just I don't know why I have the urge to just take these all out. Also before I do that I realized that they come with its own brushes which is crazy. Quite affordable. We'll see how good the brushes are probably not very good but you know maybe for a beginner this set is great and if you can't afford very much you can just try it out. I mean that's pretty cool. We're gonna try them out anyway and see like how good it is. I really want to do this. Look at the colors. I don't know why but that just really pleases me. So we've got two whites unless they look a little bit like that one is a bit dimmer. Yeah so this is zinc white this is titanium white so they're two different whites which to the naked eye you probably like what are you talking about those are exactly the same but this one is slightly more yellowy. I don't know. But yeah I'm really excited to try them out. Let's test them out and put them back in its honeycomb shell. I don't know if this is triggering anyone's I don't know what the word is. Tripophobia or whatever it is but let's continue. Okay let's get going to opening these then I guess always my least favorite job. This is how they're looking. They do have like a bit of oil on the top but I'm thinking if I just mix them now that maybe that will just not be too bad. I'm very scared of just completely ruin the aesthetic and putting paint outside these cups. And that's a final painting what do you guys think? I think these paints are really nice you can see that they're kind of drying a little bit here so that's kind of strange because the other jelly gouache paints don't really crack like this instantly so I think that maybe there's like some sort of obviously different ingredients going on in there but they're really good. I thought they were pretty nice they're a little tiny bit tiny bit watery than the other jelly gouache but overall I really like using them and I think they're just fine like worked pretty well opaque when you wanted them to be and watercoloury when you wanted them to be watercoloury so yeah that's the final look and hope that you guys like this video. Please let me know down in the comments what you think of these and thanks to my patrons again for helping me out. Love you guys and see you next time. Bye!


This systematic literature review examined the existing research related to information security in chatbots, identifying potential threats, proposed solutions, and future directions for research. It found that chatbots face various security threats, including malicious input, user profiling, contextual attacks, and data breaches, and that solutions such as blockchain technology, end-to-end encryption, and organizational controls can be used to mitigate these concerns. Additionally, the review emphasized the importance of maintaining user trust and addressing privacy concerns for the successful adoption and continued use of chatbots. Finally, it identified future research directions that included developing more sophisticated authentication and authorization mechanisms, exploring the use of privacy-enhancing technologies, and improving the detection and prevention of security threats. This article was authored by Jingyang, Yanlin Chen, Lippipoor, and others. We are article.tv, links in the description below.