It's made some a slide, and I'm very excited to be part of this fantastic event this year. I would like to thank the Blue Team Village and DevCon team for this great opportunity. And a big thanks to all of you for being here. It's highly appreciated. In this presentation, I'm going to talk about one of the most challenging issue in cybersecurity that may affect almost every type of the engagement. I name it SCOP-X, and I'm going to define it. But before that, let me have a quick introduction about myself. Again, my name is Meysam. I'm a cybersecurity practitioner, consultant, researcher, and blogger. Please take some time and check my blog at MediomSECHOPMedium.com. Every feedback could help me to improve my series. At the moment, I'm working as a senior director at the EGS, EC-Concealed Global Services, which is a consultational division of EC-Concealed. I strongly believe on the collective knowledge. So let's connect. You can connect me at LinkedIn, or Twitter, or whichever works better for you. Together, we share, and together, we learn. Here is the agenda for today's presentation. But before that, let me raise a very typical disclaimer. These are all my words and my opinions, not a view of my employer for sure. So anyway, I will start with the definition of the SCOP-X, what is SCOP-X and why it's a big challenge. I will talk a little bit about subdomains, the issue and challenges, and why I choose a subdomain as a perfect example for this scope. Section 3 covers the process flow for discovering the unknowns and followed by the threat hunting, the structure one. And I will end with the hypothesis, how to convert the finding to the proper hypothesis for the successful threat hunting. So the SCOP-X, what we exactly don't know, almost every cybersecurity engagement begins with the pre-engagement questioner and clarifications. How many web applications are being assessed, how many IPs will be there in the scope. There is nothing wrong with that. Scoping is essential for every service line, but it suffers from a very fundamental issue. It's all about what we know and what we think need to be tested. So there could be a lot of other assets out there in our environment that we simply exclude them from the SCOP for any reason. And we define the SCOP base over point of view, which is not necessarily the same as the hacker things. Hackers are mainly attack every part in our organizations. That's why we can define one of the most scariest part in our environment as a SCOP-X, which is a group of entirely unknown assets and items where we have zero knowledge about them and zero visibility. And they are mostly act as a perfect attack vector for cyber criminals. This is actually the SCOP of work for the cyber criminals. Normally, when we talk about the SCOP of works, we focus on the two below categories when we focus on what we know, whether they are in the SCOP or out of the SCOP, whether we know the risk or we hire some third party or conduct the in-house security test to find the threats. But hackers are much interested in what I call it SCOP-X, where the assets are unknown and the risk accordingly. This is the one big challenge for almost all of the security service line, not only offensive but defensive as well. If you are part of the IR team, I bet you have experienced it several times in your entire career alert generated. We know something happened for an asset, let's say a server, we own that server, but we have no idea where is that located. It's a common issue with the large organizations, with the poor asset management. However, the SCOP-X is not only belong to the large organization, it affects the companies with any size. And to be honest, this is not something completely new, but in my opinion, it has not paid enough attention. Somewhere at 2016, the Gardner predicted that in 2020, most of the attacks will be conducted on the shadow ITs. So that predictions came out through, Buck Road 2021 showed a statistics on that that proved that hypothesis. But I'm glad to see that the Buck Road did not use the term shadow IT because the SCOP-X and the unknowns is something more than just BIOID. The BIOID or the shadow ITs technically, it's every unregistered, unattended, unauthorized and unknown things within our people, process and technology. And subdomains are a very interesting example of the SCOP-X. Why I said so? A very common issue with the subdomains. We want to test something, the external service, we create a subdomain, we run a service and when the test is done, we forget about that. It's remain active there. It's easily used by the attackers as an entry point, walking or breaking our network. There are 100,000 subdomains for the main domain, which not necessarily inherited security controls of the parents. They are not well maintained and obviously they are not under monitoring because we may not even aware of them. As a red team, we normally use them as attack surface. I personally always suggest my client to consider the subdomains as a part of red team when they engage us. However, as a blue team, we should proactively look for them. Look for the subdomains on any other unknown things actively to discover them as a risk and address them. When we talk about the subdomains, I call them the vulnerability-lucky draw because there are a wide range of attacks for subdomains. However, when we search in the Google and other sources, we mainly talk about the subdomain takeover only. No doubt this is a very big issue, even the big company like Microsoft affected. But I did a research on the hacker one recently and only 30% of the findings were on the takeover. More than 60% is another type of the attacks. That's why it's not easy to just jump into the subdomain analysis and the SCOPEX threat hunting without the proper process. So technically, threat hunting is supposed to proactively look for the anything malicious in the past and present. In the ideal world, it watched the areas that not covered by our monitoring system to look for the attacks that not detected. But there are a few success factors for that, total visibility for sure. We should aware of everything that exists in our environment. A situational awareness is all about a good, bad, and any deviation that makes them unknown and we tag them for the further analysis. We should have the advanced knowledge about the attacks and the techniques used by the attackers. And of course, the data should be available and the data should have some certain of the quality to be analyzed. So this is something that we rarely see in the subdomains. So I have a question for all of you. Do you know how many subdomains exist for your organization domain? That's my point. We never checked them, but the hackers may use them, may take over them, may hack you into our registrant system and create a subdomain under our name and use it for the malicious purposes. For that reason, we cannot just blindly jump into SCOPEX and search for something that we have zero knowledge about that. We're still exactly as confused as Alice, but in the cyber-digital Wonderland. So this is the simple process on what we can do to discover the unknowns in the SCOPEX. We start with the data collection. It could be passive and active. For example, for the subdomains, we can do the proactive subdomain enumeration to come up with the long list of the subdomains. All the data must be validated to ensure the accuracy and the quality. For example, I check whether the subdomain is resolvable or not. It's up or it's down. So every collected data should be validated and classified. We can group them, free them, and reduce the unwanted data before we start the analysis. And of course, we ended up with the report and finding. Maybe you think that this is something that you know. Of course, this is not something new. But I merge the second part with the first part, which is a very common in normal penetration testings. So the moment we discover the unknown in the data collection phase, we can feed the other service line with a new SCOPE. Let's say, for example, we have these are the domain and IP in our organization. We do the proactive discovery and we find the new items. Before we do the analysis, before we do any further action, we can update the other teams with the new findings. Okay, the compromise assessment team, the penetration team, the red team could have a new SCOPE of work. But the main point here is when we find a report, means we collect the data, we validate, we classify, we analyze, and we have a set of the finding. This is the time that we can conduct a structured threat hunting by creating the hypothesis. That hypothesis should be testable. We should check all the type of source of data, which can help us on proving the fact or disapproving. All the tools and techniques that we need and we test the hypothesis. The finding could affirm our hypothesis that, yeah, we thought something bad happened and is exactly there. It could come up with the action list, like trigger the IR, or we could simply reject. We have a hypothesis, we think that our server infected, we go check, everything is fine. Peace of mind. There is a third option, which is non-conclusive. Means we make an initial hypothesis, we analyze, but we couldn't find anything. It could be because our hypothesis was not mature enough or we don't use this proper tool set. So let's start with the data collection with the subdomain as an example. So there are a lot of tools and techniques to enumerate the subdomains. I'm not going to talk about them because there are a lot of interesting talks out there. Even in the previous DEF CONS, you can find an amazing techniques discovered by awesome presenters. Just a suggestion that we have, as you may know, we have to use different techniques because the objective is to collect data as much as possible. I'm not going to whitelist any tool here, but the three tools that I introduce here are my top favorite, because simply they work for me, especially the asset finder, which is quite fast. I would like to talk a lot more about potential observation that we could have in this stage as a threat hunter. When we collect the data, when we enumerate the subdomains, we may see a lot of unknowns. The domain is there, but we have no idea what is this. It could be with the unusual name, unusual lens, unusual pattern, or it could be an odd number of the subdomains. A small company with a hundred subdomains, when we did expand the business on that scale, we have no idea. It could be the indicator of the attacks. So once we collect the data, we should make sure to validate them because a good data, a high quality data with the accuracy, reliability, and completeness could provide us with the better vision and higher quality in the analysis. So for example, for the validation in this presentation, I'm going to check which IP can be resolved and which subdomain is done. So I simply filter out whatever is done and I keep all that can be resolved in my data set for the further analysis. So as you can see, for this case study, I had around 6,600 subdomains, which I reduced to the 4,025% data reduction, but the important thing here is that we should not simply filter them out and forget about them because whatever you don't know could be up any time in the future. That's why we should keep the data in the watchlist, keep eyes on that for the further analysis. So there are a lot of prefixes for the subdomains based on the nature, based on the service, based on the stage like development, testing, acceptance, or whatever. But there is a few interesting things that I always go for, which is the admin login API. For example, in this example, I was searching for any domain which has the API and it's in the staging state. So once I do the first classifications, I go for, for example, another round. In this example, I go for the port 80 and port 443. And also I check the HTTP response code of all of the subdomains, which is up, leave, and operating on the HTTP protocol. This is not only for having the better data, this is to having a situational awareness about normals in other organizations. So when we discover the things, we study them, we understand and record all the normals. And if any deviation happened, any sudden change, any sudden difference from what we use to be observed, we look for the justification. Is there any operational justification for that? Or this is something that we are not aware of that could be tagged as a malicious. For example, if we keep tracking up the number of the subdomains that can be resolved and normally up and down one or two per month, but suddenly we have a significant drop of our active subdomain. This could be an example of the, for example, DNS server, attack on our DNS server, the vidas. Even the status we can check, for example, one subdomain changed the status in the code like once in a while, that's pretty normal or could be different stages. But if we observe a subdomain, which is constantly changing the status, it could be some sort of the second communication between our infected server and the clever hacker, a smart hacker. So technically, there are a lot of things to be analyzed from technology to platform to services to applications because it could be anything interesting behind every subdomains. For example, for the HTTP, we could have the source code review, directory, external internal links, login pages that may lead us to the vulnerability, misconvolution, data leaks and whatsoever. The main question is that there are a lot of techniques. What we should do? We should go for manual observation. We should use the automated tools or we should use advanced analytics techniques. So I don't want to debate on the human VS machine here because at the moment a threat hunting still is a human driven thingy because we need the human brain to understand, to sense, to find all the strange stuff. So as well commented by Charles, we should automate what we know and we need to spend quiet time finding what we don't know. So manual observation is the first step in the data analysis, but of course, navigating the subdomains one by one is not an option. As a common practice, we can use the automated tools which make bulk screenshots. There are a lot out there, but in this example, I'm using the eyewitness to make the screenshot and also download the page source code for the review later. So the screen shots are fun to review. We could have a lot of error messages who may could expose the sensitive information, login pages, any other juicy information. For example, here I show you the real example of the Apache Comcat in the real engagement time to say hi to the exploit DB. On the other side, we can check the source codes, page source code to look for any default credentials, login pages or Java scripts. But one that was interesting in one of my engagements that with a simple directory and file traversal, we find the file and we simply search with the zip file which is contained the entire backup of the website. So a simple grip on the configuration file which is contains pass or password lead us to find a username and password. Even if the username, even if the password is no more in use, but still the pattern can be used for our dictionary attack later. So there are a lot of things that this is the challenge. There is a lot to discover and we have to know three main things. Normally what to discover, we find them through the technical checklist and guidelines like everything that covered by the OVAS, by the CIC controls for misconfigurations. We could have our hand dirty with the knowledge bases like Mitre to find both recent techniques and technology used by the attackers. And the most important things in the threat hunting which is at the cyber threat intelligence. When there is a newly attack discovered or new pattern observed by our team, we should adopt them and use them to look for the similar things in our environments. So we have all this huge information there. We can pick up those related to our threat hunting objectives. For example, as for sub-domains, I could choose the application entry points from the OVAS. I choose the application layer protocol for the command and control mechanism from the Mitre. And I look for any sub-domain and port 80 and HTTP and web-based traces in the public available cyber threat intelligence databases. But to be honest, it's too much work. We need a lot of knowledge. We need two big monitor popcorn coffee and the nice eyeglasses like me. But still the manual observation is an integral part of that. Personally, I enjoy doing the things manual, but don't take me wrong. We should do it mostly during the training and capability developments and we are not supposed to avoid tools and waste our time and energy when we are in the real battlefield. We are all human. We need the tools. There are a lot of tools out there that help us to automate the process, make it faster in the larger scale. So again, I'm not quite listing any of these tools, but I personally have a lot of fun with the spider food, which is the O-Synth automation tool. Get fits from several data sources, classify it for them and make it easy for us to navigate. But what I really love the most about the spider food is the ease of installation. If you work enough with the open source tools, you know the pain. So the famous amas from the O-Bas, which is the tool for the network mapping and the attack surface external asset discovery. The project discovery is interesting because this is not one tool, it's a set of toolkit that we may use. The sniper to me seems good, but I only check one or a little bit of it yet to explore. These are all good tools, no doubt. They can automate the things and make things faster for us. But when it comes to the complex sophisticated attacks, multi-stage techniques where the huge volume of the data is involved, the data keep changing, the data comes in the very different and variety of the format. We need something more than the human, the ordinary hunter and tools. This is where we should use the advanced analytical techniques like data mining, machine learning, pattern recognition, neural networks to look for the sophisticated things. I always use the botnets as an example because they are really sophisticated and the botmaster use a lot of techniques to remain persistent as much as possible in our environment. For instance, I picked up a few subdomains from the IBM X-Force. These are the subdomains belong to the certain botnet activities. So for all of them, the main domain are still up. For the first one, the domain are up, but it's operating in the different IP which is not belong to the main entity. The domain is in this company and the subdomain which is under the main domain is operating from the another entity. It could be the obvious indicator of either the takeover or the shadow domain attack. On the other hand, I found another one which is interesting. So the domain was up and the subdomain is up, both operating from the same IP. It could be an indicator of the server compromise. The last one was interesting. The domain was up, but the subdomains are no more there. So when we look at the threat hunting and threat feeds, we should not only focus on what is active at the moment. We should consider those who are not active as well as well for many reasons. They could be simply just disabled temporary by the attacker as a strategy to evade the defensive solutions. Or it could be disabled, retired and replaced by another new subdomain. In that case, we can use the pattern, identify the pattern and look for something more or less similar to that. This is what we call it a similarity-based detection. So the main point of this slide is, as you can see, there are a lot of patterns, a lot of behavior, a lot of scenarios, a lot of detection evasion systems. For example, one of the web-based generation of the botnet used the HTTP and port 80 to communicate with the command and control server. They can impersonate themselves as a normal web traffic and it would be difficult because port 80, we cannot easily block it and we cannot just monitor it because there are a huge amount of the web-related activities in our company. That's why we need some advanced technique to look for the pattern of these activities combined with the manual observations and, of course, the automated tools to detect these kind of complex attacks. For example, in two of my last researches, I proposed some classification based on the neural network and pattern recognition to detect the HTTP botnet traffic based on the periodic nature. If you are interested to know more about them, I would be more than happy to connect and explore together. So let's say we have a set of the findings. So this finding, actually, this is not the final finding and the final report should be more advanced. This is just initial observations. I list down three examples for you, one with the manual observations. For example, the unknown subdomains during the red team exercise or pentest exercise or we have just found fun with the discovery. We observed two newly created subdomains under our company name. So we have to go and validate them. Is it something that one of the development team did for some purposes? Any logic behind that? If no, we should definitely tag them as a suspicious for further analysis. It could be as simple as the result from our vulnerability scanner for a known vulnerability. For example, in this example, there is a CV with that code with the unverified exploit available in the exploit DB. As a turret hunter, we should consider everything as a top severity because if the exploit is not available public, it doesn't mean that no one out there could exploit it. It simply means it's not available. It's not publicly available. We should still consider it as a top priority. But I'm more interested to discuss about a new botnet. Let's say our cyber threat intelligence monitoring discover a newly web based botnet, which is targeting a Microsoft Windows operating system and exploit three different vulnerabilities. Not to mention if we are heavily operating with that kind of industry or that type of industry or that kind of infrastructure. This is something that we really need to go depth into. So now we know the different issues. What would be the next step? Let's say for that button infection, we have two things and we can go minimum three directions. So we have the indicator of the compromise. So that botnet have a certain URL and a certain house name in the turret intel feeds. The first thing that we could check is any of our system communicating with those known bad URLs or IPs. This is something to easily search from the different source of the data logs and firewall in our environment. So but the second interesting things is a still IOC base, but instead of searching about communication from other organizations with those URLs, we do the other side. There is it because that URL that botnet working based on the three non vulnerability. And if by chance we have that vulnerability in our servers, there is a chance that we get compromised. The botmaster take control of our server and use other sub domains as a part of its domain is a C to C mechanism. So in that case, I highly suggested to search to discover our sub domains, list them all the IPs and subdomains and start searching for any traces of the existence in the public available. Threat intelligence if we found means be compromised. So IOC is a very good things for the rapid testings, but they are most basically on the rule base because it's pretty obvious so I have this pattern I have this signature I have this hash I have this IP if I find it I find it. So as well mentioned by the Anton if we can simply write a rule, let's write a rule we don't need to really go for the hunt. The challenging hunt comes when we talk about the TTP, the advanced technique, the procedures used by the attackers. So for example, as we just mentioned, but master could use the web based protocols to communicate with the command and and and controls and they can hide amongst the normal traffic. However, based on the research out there, the bot nets mainly chose some sort of periodic pattern because they need to periodically communicate with the bot master to receive the new commands and updates. So any periodic web based net for traffic that do not follow the expected standard protocols. For example, they use a different ports on the on the HTTP rather than the 80 and 443 could be considered as a sign off infection. But this is something that we should collect a reliable data. For example, we collect the network traffic based on that hypothesis. We identify the parameter that use for the detection we go for the advanced analytics and we either affirm the hypothesis or we rejected. So actually, the scope x is more wider than just a 30 minutes talk so the talks is ends here, but the threat hunting is a never ending battle. I would like to say the very special thanks to Joko, a very good friend and the teammate for all the brainstorming and the support not only for this talk but for all the other things. Feel free to connect to me if you want to know more about these concepts, discover your unknowns, be proactive, dive into the ocean, happy hunting.


My happiness and my function are one when you're in your function when you're in your Holy Spirit given function You are guaranteed to be happy And when you seek for happiness through external means I'll be happy when My life looks this way when I have the right partner the right this the right that and then it's almost like a dream You chase that never works out We have to let that Holy Spirit Inspired communication come in us and through us How does it feel when you write a love letter from the heart? Whoa, your heart is exploding with love and joy. You feel completely filled up and just engulfed in love and intimacy We have the light inside of us, but we have to let it Shine in us and through us really to feel that practical day-to-day experience


All right, then I'm going to get started now and I really am pleased to have today for today's OpenShift Commons meeting briefing. Delano Seymour from Sixth Fusion, who's going to talk about metering microservices on OpenShift, and I saw this demo probably three weeks ago and got really excited. The idea of being able to meter and do the billing and charge back and figure out everything that's going on on OpenShift 3 and deal with all the containers that are being scaled up and scaled down has been an interesting thought process for me over the past few months while we've been moving towards OpenShift 3's release in the upcoming months and how we're going to do this. And then when Delano showed me this wonderful demo, I was like, okay, we have to get this on and up in front of the Commons folks and get everybody's feedback on it. They're doing this in part and parcel to help one of their product offerings, which Delano I'm sure will explain what all that is and what Sixth Fusion does. So without any further ado, I'm going to let Delano explain who Sixth Fusion is, who he is, and why metering microservices on OpenShift is actually doable and pretty cool to see. So I'll let you take it away, Delano. Great. Thank you, Diane. How's everyone today? I'm going to go through some brief information and then a quick demo afterwards, and then we'll have some questions and answers so that you can ask questions. My topic of discussion today is metering microservices on OpenShift. My name is Delano Seymour. I'm a CTO of Sixth Fusion. I have some Twitter and email there if you want that. But Sixth Fusion's goal is to create a method in which to meter infrastructure, like we meter electricity, being able to create a true utility in the cloud space. And so what I'm going to do now is just talk about some of the advancements that have happened in the cloud space, specifically around containers and how we've done some adjusting to take advantage of containers. So the first thing I want to show is that with containers, it creates some great benefits for our customers. For software earners, containers provide a consistent way to package and deploy code. Containers provide an efficient way to distribute software. Containers improve the portability of software between platforms. For infrastructure earners, containers provide a standard way to deploy software. Containers provide a standard interface to orchestration and running software. Containers also remove the need to have deep knowledge of the software to deploy and manage it. So it creates a way for infrastructure earners to just basically take that container and just run it. So on the technology side, OpenShift has got that part covered. They allow us to provide a platform to deploy containers that you create or containers that are created by others. They also provide the ability to deploy directly from code using their SDI capabilities and provide a simple way to run containers at scale. So OpenShift becomes a great home for microservices. Well, what are microservices? Microservices are small, lightweight services that can be combined to deliver a single application. In the old days, we had a monolithic application or we would break down application into multiple libraries. But with microservices, we can break down that application to multiple services. Each service independently operating from the other and communicating over a lightweight interface like HTTP. This microservices style architecture fits very well with OpenShift and the OpenShift platform and allows you to then put individual services that can be scaled automatically and independently of each other. Then another question comes up. We've got the technology side sorted out. We had the ability to create many microservices and split our application up into groups. But what about the financial side? What about how we want to share our software with others if it's a commercial package? Well, how do you track that usage? How do you charge for that usage? And then the big question is, can you use that same method regardless of where that microservice is running? Because as we all know, microservices can run anywhere. They could be internal to your enterprise or they could be on a public cloud. And OpenShift provides both an enterprise product and a cloud product. So you want to be able to get that holistic view. So what the solution is, 6Fusion believes the solution is that you can meter it like a true utility where you can track resources used by all microservices that make up your application. You can provide a single metric to make that financial conversation easy, remove the friction and allowing users to understand their consumption intuitively. It also provides a method to baseline and train your consumption over time. Now an analogy so that we can all wrap our minds around what we're thinking is electricity. With electricity, you will purchase a refrigerator, a air conditioner, a laptop or cell phone. And all of those devices, all you have to do is plug it into the utility grid. Once you plug it in, it gets metered. And then they use that consumption data to either bill you or to recommend any other changes that you might need. We want to do the same thing that the utility grid has done for electricity. We want to do the same thing for compute. So real quick, how does it work? Step one, we collect resource usage for compute, network and storage, but each container that we find on a particular platform. Then we take six metrics and we'll go through this one at a time. One is CPU, which we measure in megahertz. The second is memory, which we measure in megabytes. The third is storage, which we measure in gigabytes. And then there are three IOs. There's disk IO, which is the amount of bytes transferred from disk to memory. And that is measured in kilobytes per second. And then we have two network IO metrics. One LAN IO, which is the network traffic or the bytes transferred between machines within the same network versus WAN IO, which is bytes transferred or received across networks. We then take those six metrics, hence the name, I guess, six fusion, and fuse those metrics into one universal metric. We call it the workload allocations view. So when a particular system is measured, what you see is how much workload allocation cubes are being consumed at any given time interval. And we can measure that just like the kilowatt, it can have different levels. CPU, it could be kilowatt, kilowatts. It could be megawatts, et cetera, et cetera. So let's go to step three. In step three, we would then collect the state of the current inventory. In the case of OpenShift, we will collect the services that are running, the pods that are running, and also the list of containers that are running. And we will take that data in step four along with the meter data and submit that to our UC6 platform. That allows us to store those metrics. That allows us to do analysis on those metrics. And then in step five, we can provide consumption dashboards and reports. We can also use that for benchmarking, baselining, and other analytical processes. So what I'm going to do real quick is just show you how we're doing this with OpenShift and containers. I'm just going to switch over my screen. So what you're looking at, I've created a version where we can see the usage in real time. And what you're looking at is a list of pods that the system has interrogated. We have an Apache pod and we have a Docker registry pod. It's a total of four containers across the two pods. And in this scenario, we're looking at the aggregate usage of a particular pod over time. And you can see that this pod is consuming about one milliwatt of usage per hour versus another pod, which is using three milliwatt of usage per hour. It makes it very simple to look at your consumption and communicate with the financial managers of your infrastructure how much is being consumed. And then when you combine the usage of all of your pods in wax with a price per wax then you can calculate very simply how much it will cost to run that particular application over time. You can also use that for planning in the future. You can use that for optimization as well. I mean, one thing that technologists want to do is make efficient applications. So being able to look at the usage of a particular application and how it's using the resources of that infrastructure and being able to tweak your code or your settings or any type of configurations will allow you to then optimize and make more efficient that application or another particular infrastructure, thereby reducing the cost of that application. Some other use cases could be for ISVs. ISVs have softwares. A lot of software is open source, but there's also software that's not open source. And for the software that's not open source, this provides an excellent way where you can practice consumption and usage of your applications and your containers as a whole for chargeback purposes, for billing purposes or just to report the consumption over time. So with our technology today, we actually meter virtual machines from the hypervisor level. We can meter individual operating systems using agents. And now with containers, we want to take that to the next level to sort of move up the stack, being able to get closer and closer to the process in which that application is running. And if we can get all three levels combined in one view, then you can get a holistic view of what your applications are consuming and how much of those applications are actually costing you to run. Some other advantages are things like being able to benchmark how much would this application cost me to run on OpenShift in my private infrastructure versus if I had to take this and put it out in the public infrastructure. And with the use of the consumption metrics with the workload allocation view, we can do some measurements on both infrastructures and give you an idea of how much it will cost in either location. So that's as far as I really have for today. I wanted to sort of open the discussion up for Q&A. I'm sure we're going to have a lot of questions. And Diane, if you can do that for me, I appreciate it. We have one already. And I think this is probably where we start to get into the deeper dive on this. And John Osborn has asked, how do you collect this data? Is it an agent-based solution where you install the agents in each container that reports back to a service? So maybe if you could tell us a little bit about how you architected this solution and where it's working and at what level it's working inside of OpenShift. Okay, so what we've done here is created Docker containers that will basically be run inside of OpenShift. They will then have to be given access through the privileged parameter. And you have to mount the Docker socket, and it will then be able to pull the metrics directly from OpenShift. This will have to be done as an administrator, for instance, of your OpenShift installation in the enterprise. So this is really something that the OpenShift administrator would have to install and manage for the deployed applications on that installation of OpenShift. That's correct. And it will pull all pods, all services, and it will map that back to your namespaces so that you can then reconcile or report on usage. We also have APIs that allow systems to be able to pull that information directly. So once it gets collected, aggregated, and all the analytics are completed, you can actually get out APIs and create queries that say, give me all my usage for the month, for the year, for the day, for the court. So John just asked, so does it run in each container or is it just monitoring each container? It's just monitoring each container. It runs outside of the container. In our world, we try to meter from the outside so that we don't affect the usage of the container itself. So this is definitely outside of the container. So John, I think I have the mic off on you. Did that answer your question? If you have your microphone turned on. Yes, thank you. Okay, so Michael Virgil is also asking a question. He says, the WAC aggregates six underlying utilization metrics. What are the numbers of each of these that equal a single WAC? So the way we do it, we create a contribution module. So we have what we call compute types which define a sort of ratio of those numbers between CPU to memory to storage disk. And so at this point, we haven't actually published those numbers, but we can definitely have a conversation and talk about that in more detail. So that I think that that answers Michael's question. So if there's any other questions at all, those are the two key ones that came through today. That's one of the things that I'm interested in getting people's feedback. And I wanted to get you in front of this audience and as well as everybody who's watching it as a recorded one, because I think this is where we can get feedback from the community, the commons group on whether or not this, I personally think this is hugely useful to anyone who's hosting or operating, whether on-premise, behind an enterprise or doing the hybrid model you described. And so I think this is kind of one of those things that everybody's going to end up either having to build themselves or we're going to have to open ship engineering is going to have to build it or we can get you to do this for us, which I think is what's awesome is that you've stepped up and done it. So I'm really pleased with what you've done and I'm wondering about other folks, if they could email the mailing list or Six Fusion or me and regular folks on the IRC channel and just talk about what else they need, what it's missing. So I've got another question that's just come in from Nicholas Schutz. What sort of performance impact and resource consumption does this have on OSCE as a host? Yes, so the performance that it pushes on the host is relatively low. It does introduce some performance because it has to collect those metrics. That could be a fine-tuned though because what I'm showing in this demonstration is that I'm collecting the metrics every single second, but that could be fine-tuned where we don't have to collect it every second. We could probably collect it once every five minutes or different into fine-tuned it. One other thing I want to point out is that we're trying to get feedback that we can sort of fine-tune this product so that it can be something that customers can find useful and performance is one of those things that we're looking to fine-tune. So I know that you have a visualization tool for WAC or other offerings because I've seen your demo before but Michael Virgil is asking what reporting visualization is available. So maybe if you can explain a little bit about Six Fusion itself and what you do, that might be helpful and how this terminal view and the API that you can call to do this is a wonderful thing for us but maybe a little bit more about your user experience and visualization stuff. Yes, so we do have a console that allows our customers to log into our platform and be able to bring up reporting. So I'm going to just switch here. And I did not stage these questions there. They're all coming in live, so perfect. So I'm going to just go ahead and show a particular account that I have here. And in this account, you'll be able to see a list of machines. So we have this concept of machines and infrastructure. So an infrastructure is a collection of resources that can run particular work needs. In this case, I have a bunch of different resources. I'm just going to go ahead and pick any pick of a particular one. I'll try to go back in time because this has some old data in it. But I'll be able to select a different account if that one doesn't have what I want in it. This is why live data is a wonderful thing, live demos. So we do have this concept of organizations, and I'm just going to switch to demo organization. So in this demo organization, I have a few different infrastructures. So we have a meter for every infrastructure, and we have different types of meters. So if you look in this list, I have a meter for AWS, VMware, Zen, and then the OpenShift meter is the one that we're working on right now to integrate into our console. And for this particular infrastructure, this VMware infrastructure, so it's a VMware-based infrastructure behind this, I can see that I have some consumption, some minimum, maximum, average consumption, and my total consumption. I also give you the average usage of the individual six resources, CPU, average memory, average storage, et cetera. And you can also see a chart that shows the six metrics alongside each other. I can go ahead and turn off any one of these six metrics to just show one. So in this case, I'm showing my consumption over time, and I can move, of course, to see that consumption. And I can then compare that consumption to maybe a CPU. There's a correlation of my consumption going up and down based on my CPU, the variability in my CPU. We can also do things like get a list of machines.  There are two that are started. There are 15 that actually are stopped, and there's one that's deleted the total of 18 in this particular environment. And it gives you how much wax across those categories. We also have a more granular report where you can pick a particular machine out of that infrastructure. I'm going to go ahead and click this filter here, where I'm going to select that infrastructure we were looking at by filtering that infrastructure. And then I'm going to pick one of these machines. So I'm going to pick this prod after one. Actually, I'm going to pick the firewall since it's actually started. And I could do one or more machines. So in this case, I'm going to pick just one machine, and I will go ahead and show the report by default. It picks the last month of usage. And it shows a similar report. So as this rolls on, I can then get again that chart where we can see all of the different metrics at any given time. You can see that we had some drop-offs in the middle here. It shows me my total usage in kilo-wax. It shows my average maximum with all the six metrics. And I can go ahead and drill down into the details and look at this at an hourly point of view. So every single hour, what was the consumption? And I could scale this back in time to any hour at any time. Can I stop you? Can I get you to click on the Reports tab? Somebody's asking to take a peek at that. So let's click on the Reports tab. On the Reports tab, we have a few reports that you can access. One is the machine report, which you actually saw already. So I'm not going to make sense bringing that one up. But we also have a tagging report, a utilization report, and a chargeback report. And then we have an Amazon price comparison report. That one, I should probably go into another conversation for that one, but we do have that report. In the utilization report, for instance, I could pick that particular infrastructure. And I can compare, I can look at how that infrastructure is being utilized from a financial point of view. So my target cost in this particular infrastructure is $41.20. But my actual cost is $148.00. Why? Because I'm not really using all of the resources that I actually have. We can also look down and see what the consumption looks like over that same period of time for that infrastructure. The actual versus target consumption. So I got another quick question here. Is this reporting interface available for on-premise installations, or is it just a SaaS offering from Susan at the moment? So at the moment, it's a SaaS offering. We've been entertaining the concept of making this an on-premise offering. We have a couple of customers that have access for that, and we've been discussing it in our dev meetings about making it on-premise. It shouldn't be that difficult, actually, to make this on-premise, but today it is a SaaS only offer. Well, that looks like all of the questions that we had. If you want to throw up your last slide with your notes in your email address again, that would probably be great. And I think that you've really inspired a lot of conversations here. This is going to be a wonderful resource for people who are deploying OpenShift on lots of different infrastructures, as you're showcasing that you can show off Amazon and VMware. Once you've got the containerization added in. A quick question, though, you're doing machine... Nicholas, the quick question is, is there a solution for V2? At the moment, no. Everything right now is really focused on V3 and the next release. So this is really a forward-viewing opportunity for metering. So I think partially why I'm so excited about it. The other thing is on your utilization of reports, this is by machine, is the thought that when we go to dockers and containers, that you'll be able to meter and show which applications are consuming or which can break it down even further than just the machine level? Yes. That's exactly why we love OpenShift and the docker and container world. Because with machines, the best we could do is hit it at the machine level. So all of the applications and processes running inside that machine are all aggregated, and therefore you just see one aggregated view of that machine. With containers, we can move up the stack. We can get inside the machine. At this point, now, we can look at all the containers that make up a particular application, and we can then report on the application itself. What we want to add to this is another tab or another area where you can look at your namespace or your project, and you can look at your project as a whole, and then you can drill down into your project and look at your services. And then for every service, there's a pod or a group of pods behind that service. So then we want to be able to drill down into that group of pods, and then eventually drill right down into which containers inside that pod are actually driving consumption. So we want to better give customers the full view of their users. So this to me is a necessity and totally exciting that you guys are working on this, and this is what Commons really is about, is we don't have to invent everything on the OpenShift community side or by Red Hat engineers, and this is a wonderful example, and I'm hoping that everyone who's listening to this call now and others folks will help you with some feedback on what they're looking for in terms of reporting and charge back and the detail level. So if you have questions or further feedback, please post it to the Commons mailing list or send an email directly to Delano. If you want to throw up your slide now with your email address on it again, that would be a good way to end this conversation. And we'll get you the feedback you're looking for and move this forward as quickly as possible because the general availability of OpenShift 3 is coming very soon to the world and we're hoping that we can do a lot of work with you guys. So thank you very much Delano for taking the time today. This is really cool stuff, and I think everybody who's in the chat now realizes that too. So thanks again and thanks again to Six Fusion for making this possible today. Yes, thank you Diane. Alright, take care everyone.