Yes, we have the starting soon. I don't know if they, in the registration, there are also our voices. I don't think so because the Zoom window is not there. Okay. Okay, I have the chat on Twitch here. I have the YouTube also. YouTube is not started yet. Okay. They say online, online, online. Let me check our restream chat. There is Didier in Twitch. Okay. So it's, I see it's 10 o'clock here. What do you see in your clock? Yes. 10 o'clock, yeah. So starting soon. I'm going to show the bumper and then we go into the main Zoom session. So that means that we will start and we will check that everything is fine. Let me do that. Usually the mic is always open. If you hear any noise in the nearby, please switch off your mic and then you can open. Okay. I'm going to switch to the bumper. Welcome. Welcome everyone to our first Openshift coffee break show here at Openshift TV. My name is Natalia Vinto, product marketing manager for Openshift. And I'm pleased today to present everyone in this session of today. So let me introduce the other one. Jeff Farr, you want to start? So hi everyone. Welcome to our first edition in the middle of the Openshift coffee break. So here's to everyone connected. Thank you very much Natalia for hosting this. So this will be a bi-weekly event. So meaning every two weeks. And I will be alternating with Natalia to host this show and have esteemed guests talk about experiences with Openshift. So again, thanks a lot. I also work as a product marketing manager for Openshift. And I hope you guys enjoy what you see in our show. Thanks. So thanks Jeff Farr. So before introducing our fantastic guest today on this very interesting topic, which is disconnected on public cloud with Openshift. So let me introduce a little bit this show. So the idea around the show was to having like a kind of a coffee break. Like we had in the office where we were talking about many things that we were relaxing every some cup of coffee. So I would like to take some coffee together with you. Everyone in the world together talking about cool topics and technology. Feel free to ping us in the chat if you have any question for today at the other show. As Jafar was saying, this is a bi-weekly show every Wednesday morning at 10 a.m. set. If you are in the UK, it's 9. If you are in Eastern, it's 11. One hour show. And I'm pleased today to introduce our guest. So if you want to start, Raf. Good morning everyone. Well, my name is Rafael Cardona. I'm one of the solution architects. Specializing Openshift for the EMEA region in Red Hat. I have a reasonable good experience with Kubernetes. About three years deploying and playing with Kubernetes all around Europe. Today I came to talk a little bit about my experience deploying the Openshift in a highly secure environment. There are many, many features interesting to allow us to unleash the enterprise capabilities of Kubernetes in this fashion. And well, I'm very happy to jump into those contents in a few minutes. Nice to meet you. Thank you, Raf. So we have other two guests today because we want to talk about those US cases. And I'm pleased to start introducing you Hamid and Ryan. You want to start, folks? Yeah, sure. My name is Hamid Hussain. I work for a company called FightServe. I'm an architect. So my main job with FightServe is evaluating new tools and frameworks and then onboarding them onto our development teams. My experience with Openshift has been very new. This is the first time I've worked on Openshift with Red Hat and IBM. My background has been enterprise application development, mainly on the Java applications. I come from an investment banking IT background, been working there for most 20 years. And I moved to FightServe a couple of years back and into an architect role. Yeah, that's about me. Thank you. Natalia, sorry to interrupt. So it seems the event is not streaming yet on YouTube. Can you please check? Yes, yes. It's fine. So yeah, and while I check this, I think we can, because I see it started, but we can continue with the presentation. So if you want to start presenting yourselves. Okay, so it's on now. Okay, cool. So thanks, Hamid, for your presentation. Brian, you want to go ahead? Yeah. Hello, good morning. My name is Brian Olmeda. I am a customer success manager architect in IBM. I usually help clients with the adoption of open ships and cloud packs. The pieces in this case, we were working with Hamid and Raph quite close to install open ship disconnected and on top of that, having a disconnected installation around the cloud pack for integration, especially one of the product data power. It was a pleasure. Thank you for inviting me and I'm looking forward. Thank you. Thank you. Thank you, Brian. Thank you, Hamid. Thank you, Raph. So today's topic, we invited you because I think you can share with us some very cool example on how to do production grade open shift cluster on public cloud. What does it mean production grade? So we understand and we know that open ships is very easy to install, very easy to use, but how to make it production ready? So the title word about disconnected, what does it mean disconnected Raph in a public cloud? What does it mean disconnected? Okay. When we talk about security, we already understand that this is probably the biggest issue and the paramount of all enterprise environments. Customer got different needs and with their needs, we need to try to adapt our technology to address those concerns. So customers want some to be able to store their application infrastructure into environment that is completely isolated from the outside world. But at the same time, they want to enjoy the enterprise capabilities that bring us to a dual world. If we want to automation, we need to be able or the customer need to be able to give up some security constraints. So what we've been working the last month heavily in this integration is try to keep using those amazing level of automation that we are now enjoying with enterprise Kubernetes with open shift, but at the same time addressing those concerns. The two approaches for that is disconnected installation and private installation in a private VPC. They might sound the same, but actually they are not. When we talk about installation in a private VPC, it really means installing open shift with all the capabilities and automation of the installer, but doing that through a proxy. The cluster is still isolated from the outside world, but the access to the internet is needed and that's where the challenge came. We want to store this platform, but we cannot use the internet. Okay, not literally installations. The other approach could be disconnected, but disconnected reduce a bit the facility of the installer to do all the magic. So I created infrastructure to configure all the environment. So what we did was to use the bit and byte of the installer in a completely isolated world and that's what we did. We did a mirror installation in a completely private VPC. I want to share something with you. I prepared especially for this call. One second. I want to share my screen now. The right window? Yeah, this is the right window. Okay guys, this is what I prepared actually last night. I hope you guys like it. This is a small procedure. This is small because it reduces a lot of the tasks we suppose to roll out in order to have this type of deployment. Here I explained a little bit what we want to do. I just mentioned that we want to do a private installation in a mirror environment. This is very important to mention that because of doing that we are actually telling the installer to behave in a way that is not meant to be. But at the same time we are using all the capability and all the intelligence behind the installer. The tonnage is a very powerful binary because it reduces the complexity of preparing all the infrastructure around Kubernetes. At the same time, it has trapped that complexity providing you with a ready installed cluster. How the architecture of the deployment of today? We can start by defining what are the components or the minimum necessary component to have an open shift installation. We need of course the control plane that is composed at the moment by a default of three master nodes and two compute nodes for the compute area. Those are the core of the cluster but around the cluster we need other components. My variant is with a bare metal or with a private cloud. But in general, it is more or less equipped in all types of environments. We need always a low balancer. The newest version 4.6 and 4.7 are a little more flexible about the requirements. But still we need a low balancer one who provides the traffic management by traffic distribution to the internal API of the API server. The other is not public but an user API called the API server. And the other is the endpoint that is both the router who is the one who managed the traffic directed to the ingress router of their applications. The other component is very important is the S3. The one who will be used not only to store the applications or loads in case it is needed but in this case the installation will save the images of the bootstrapping and will support all the back and forth of data during the installation. And route 53. That was also a very interesting issue because before we needed to deploy our own DNS, we weren't able to use route 53 because there were some constraints but in the latest installation we can do that. We can do route 53 in a disconnected environment in the way we desire. Moreover, how our VPC will look like. This is the idea of work. We got three private subnet, three different availability zones. You might ask yourself what do you see now Internet Gateway here? Let's do this question. Why do we have an Internet Gateway there if it is disconnected? That's a very good question. Yeah, that's a question that all our customers ask. Okay, we want a completely disconnected. Why do we have that? Well, at the end we need to download the images. We need to mirror the installation images because the installation takes place by deploying those images and spinning up the container. One way we avoid this in several use cases was to create the mirror registry in a less protected environment. Then we mirror all the images. We did all the testing and then we create an image out of that server. Then we move that to another easy to install in a privileged environment. We did an error gap. That's one of the modes. The other mode would be to use a VPC peering that one VPC is completely isolated from the outside world. However, it's connected to another VPC who serves like a gateway to download those images. That VPC is less protected. The connection to Internet is not completely enabled by the VPC itself, but it contains the bastion and all the mirror images. The use case I want to bring up today is none of them. The one I want to bring up today is the one who uses Direct Connect. Basically, the VPC exists in the customer premise. It's completely isolated, but instead of being connected to the outside world directly, it's connected directly to the AWS premises. They got a certain level of access to the outside world, but that's something that the customer can self-determine. Of course, I don't have Direct Connect. If I do that, I will have to pay a very big bill. I'm sure my money will not be happy about that. What I did was to mimic the case to have a Direct Connect here with this Internet gateway. I would like to show you something else. Here goes the template. I told you that we're going to be using an existing VPC. With that VPC, we have to have certain requirements. Those are not very special requirements, but anyway, you should be aware, especially about the security group that will govern the internal mirror registry. Here, if anyone is interested to do this deployment in a Direct Connect environment, the only thing they need to do is to come here, change the Internet gateway for the Direct Connect gateway, and change it here.  A rough question. This is a cloud formation template. This is the cloud formation template for AWS. How do you link this to the OpenShift installation, or how do you link with the OpenShift cluster use it up? Very good. As I mentioned before, what we wanted to look at with this type of deployment is to use the best of both worlds, the highly level of automation of the installer, but also use our own premise, ready to rock, let's say. But the VPC needs to have specific settings to be able to hold the installation. Because the installer will take over the installation. I will say, okay, I'll do the installation. But when it reaches the VPC, it expects that certain settings are done. Those settings are well explained in the documentation. But if you go to the template itself, you can relate what exactly it needed. In a completely disconnected environment, you might replace the installer in the area of provisioning. You can do the provision only through a cloud formation template. But this is not the case. We got our VPC already up and running. I can show you. This is our VPC. It's a brand new VPC. I haven't done much on that. The settings, as I mentioned, is to compare with this. But my experience with this is what customers have on place. It never differs so much what we have here. The cloud formation template for this particular case will help to have a role model to follow. But it's not part of the procedure. It's just to see, okay, my VPC is able to handle the installation. I got another cloud formation template. How it relates to the installation of security groups. This cloud formation template that I put here is quite complete. Because it doesn't only cover the security groups of the installation. It doesn't only cover the security group of the vacuum hose, but also the security group of the home installation. For this installation, we don't need that. Because we're going to leave the installer to the rest of the magic. But it's important that you have access to see how it's really deployed in a similar way than the installer. The installer doesn't use the cloud formation template, but the interaction and the flow of how those components are being deployed is exactly the same. Does that answer your question, Natalia? Yeah. Thank you. Let me ask you another question. Just to summarize what you mentioned about the Internet Gateway and the need to have all of the required images and components readily available somewhere. What you mentioned is there are different options. I'm just doing a quick recap for the people who are not so familiar with OpenShift. You said there are different ways of getting the Red Hat content to make it available offline. In this case, we are using a direct connection and we are using the OC command lines that allow you to mirror the content for a specific release. Basically, you say, I want to install OCP-46. Please get all the images, all the content. Download that for me and make it available as container images somewhere. If we wanted, we could have that downloaded separately on a different host and then copy that on the version, for instance, and make it available on an internal registry for the private VPCs to be completely isolated. Is that roughly...? Yeah, definitely. Thank you for completing that. I appreciate it very much. Yes, indeed, we got the OC client and the OCP install who are going to help us not only to mirror the registry but also to the deployment. I see the time is running. Guys, if I should start with the deployment straight away. If you can increase a little bit the font, that would be better. Oh, excuses. We can see that topology pretty good. Thanks, Didier, in the chat for having shared the link of this repo. You can access to this repo. It's publicly available. So you can use it for your POC, for your use cases. And then I would like also to know from Hamid and Brian what they think about this kind of topology. And if this kind of use case, Solved addresses the requirement that they had for a production grid. So Hamid, you say this was an example for Pfizer. So how did Solved your use cases? Yeah, so we had a couple of requirements. I would say challenges. When we started this setup. So the first one was due to Pfizer policies. We were not supposed to use all of the AWS services. Services like the Route 53, the load balancer and the S3. So we had restrictions on those. We were not supposed to use it in our installation and the setup. And the other challenge that we had was we were forced to use AWS CLI rather than the AWS console, which made things a little bit difficult because CLI is more manual and is a lot of syntax checking and all that stuff. But the scripts that Rafa built for this whole thing in GitHub, that helped a lot because it had all the command that we needed to run quickly. So that helped us making this whole thing quicker. Yeah, so overall, I mean, if the whole disconnected mode that we wanted was already there, designed by Rafa. So we just had to use it, make small changes and then use it for a specific installation. Cool. Thanks. Thanks for your explanation. So as I understand, this is rough. This is a prod-like topology. This is something that people that look at your repo can say, hey, I want to try this on my prod and I can set up my OpenShift Disconnected on AWS with this strategy. Yeah, definitely. That's a simplified version, as I mentioned before, because I didn't include the Red Connect. But if someone is familiar with CloudFormation on AWS, that's exactly what they need to change and they can mimic completely this into the Red Connect environment. Actually, Hamid mentioned something that came to my mind during the installation there. Yeah, indeed, we had to use CLI. When a product is mature enough and is enterprise enough, it has to be resilient to the type of constraints. We were able to, it was a teamwork. It was a few people working together. We managed to translate the Terraform and the CloudFormation template into the CLI command. It's a comprehensive way to do that.  Every single component has been deployed in the environment. Yeah, that was probably the biggest constraint. But as Hamid mentioned, we are able to do this and we're able to provide, to provision the cluster in those three fashion ways. That would be Terraform, our installer, CloudFormation template and also to the CLI. And the result is the same. Yeah, thanks. There was a question in the chat on Twitch asking, what does it mean updating a disconnected cluster? If you have this topology, that looks great for production grade using disconnected on AWS. If I need to update to OpenSheet for that seven, what I have to do? Well, actually, it's a very nice question. At the time it's running. I think I will do the mirroring because you can see how easy that is and it's connected to that question. We do the mirror. We mirror to any version, any level. For those six, four, 15. And our set of operators, that's a specific level of release. But when you release, you are not able, as a private-holder cluster, you are not able to acquire those updates on the outside world. So you need to mirror those content and you execute the upgrade in the same fashion that you need to just mirror the exact version of the operators that you want to upgrade. It is quite handy because I saw the case that the customer wanted to upgrade only a section of their clusters. There were many clusters installed and they were able to have different mirrors, different versions, and then they manipulate the updates in the way that only a specific cluster had a specific version of the OpenShift in that moment. You want to say something? Yeah, Rafa, I was going to offer to explain that if you wanted to start doing things, otherwise you can't carry on. So basically what we all need to understand is that OpenShift 4 installation and the whole operation of the platform heavily relies on operators. And those operators use container images that are versions that are hosted on the Red Hat release. And basically upgrading means deploying the new set of operators, running the containers from the new versions and to do that, as Rafa explained, you have to get those images from the Red Hat registry, extract them, so we provide the tooling for that, host them on a private registry and then trigger the upgrade, meaning that you will replace the operators that are in place with the new images, exactly as when you publish a new image for your application and your new containers get started from that new image, that's exactly what happens with the operators. So there is a global operator that is responsible for the overall release. It gets its image and it knows what other images it needs to deploy as part of the release upgrade. So basically that's in a very high level summary how it works. And that's why you need to have those versioned images on your internal registry. That's pretty cool. So Rafa, if you are showing something in the terminal, can you please increase the focus so everyone can... I would like also to hear from Brian what he think about this disconnected public cloud experience, because you know, when you think about public cloud, you think, oh, it's all on internet, no? It's all connect. So Brian, what do you think about this specific use case? Do you see... Is the normality in the production grade cluster on public cloud? Yeah, I think... Thank you. I think nowadays customers they are turning to this disconnected mode and I can see... Yeah, Opensiv has improved a lot and we are pointing to this direction. And I will say the most common, especially in private industries, in banking, I will say governments, they are all going for this disconnected installation. And as Rafa and Hamidis is highlighting here, of course you can have... You can see a lot of challenges, you can find a lot of challenges, but the good thing or the maturity of Opensiv that you can find workarounds for everything. And that is probably the key here. And yeah, as I said, on top of that, no difference once that is installed, no difference on Opensiv, you can run the sense of work in disconnected mode or in public mode, that is helping a lot in our case for running the software that we install on top of Opensiv. Yeah, I agree on that. And to link to this discussion, there was a question in the chat, say, hey, do we have to enable telemetry in the disconnected to have the support? So the telemetry service is only needed for having insight from Red Hat. Like, hey, your cluster is running under pressure, hey, you should fix this, but it's not really needed. And I imagine for a disconnected cluster, the telemetry is not active unless only that specific host is enabled. So this is another topic, interesting topic. So in the disconnected cluster, telemetry active or disabled, so I don't know, Raf, what do you did in this case? One second. Yeah. That's an excellent, excellent question that out of the box, you got a monitoring setup. You can avoid that, you want to be there all the time, but in case you want to extend the telemetry, you can always do, but in the case of the disconnected doesn't make much sense to go as far as monitoring that we do in exposed systems. The basic out of the box setup allows you to have the minimal required to have a good overview of how your cluster, how your resources are being consumed by the cluster. But you are able to even reduce that in case you don't want to see anything, but the out of the box stack, how this setup is reasonable, not to abuse the amount of resources it consumes, but also to keep a good overview of how your resources are being utilized by the cluster. Thanks, Raf. Hamid, I was wondering also from your side, if you know what is the telemetry services, if you think it was good for your use case, what do you think about? No, we didn't explore much on that aspect. Okay, so the monitoring, internal monitoring was enough for you, or do you have also, you use also other monitoring tool to track your cluster? No, we didn't use any other tools. So for now, it's just what we have. Okay. Yeah, but we might be adding something in future. Okay. Once we have this whole set of stable and running. Cool, thanks. Yes, and Natalie, so maybe a quick also reminder of why telemetry service has been at OpenShift. So it's basically a way for Red Hat to improve our support relationship with our customers, but in a proactive manner. So what we ask for the customers is to agree to provide some non-sensitive data about their clusters, about its health. And because we collect information from those clusters, and of course it's, there's no, like it's mostly anonymous information about the components of the cluster itself and not anything related to the workloads that run on those clusters. It allows us to see the overall health of the clusters. If there are some upgrades that are failing for some reason, if there are some components that have problems, we are able to see it ahead of time. And when a customer will open a ticket to say, hey, hi, I have a problem on this, we know if it's something that is only with this customer, or if it's something that we have identified with many other customer installations. And that's something we can proactively start working on and dispatch the information to our customers that something like a new fix has been released ahead of time. So it's really to help being proactive in the way that we handle the support relationship. Yeah. Cool. Thanks, Jafar, for this explanation, for having some context around the telemetry. That was a good question. So thanks. I think it was the UNIX 365, I made this question. So if you have any questions, please use the chat. We want to use this coffee break to have a coffee break with all you sharing all the experience and the use case. And as it is a bi-weekly show, if you have any idea, please reach out to us at OpenShiftTV or we will share also our Twitter. So please reach out to us to propose any topic you would like to discuss around OpenShift use cases. In this coffee break, let's say virtual extended coffee break in me a morning, but we may have also people from Epoch. I don't think US because it's very early there, but yeah, it's an extended virtual coffee with all of you. It's a pleasure. So Raph, what are you going to show us? Yeah, I will show you guys how easy and straightforward is to mirror a register for the installation. Here we have brand new AWS, AWS C2 Instant with a real image. Can you please share this link in the chat so I can reshare in all our screens? Sure. Thank you. I'm trying to look for the... Yeah, you can use the Zoom chat and I will share to all our channel. Okay, I cannot find the Zoom chat here. I got more than 30 windows open. I'm going to share it with you. Okay, no problem. Go ahead. You got that. Okay, first I will increase the size. I hope you can see this. Yeah, okay. Here we got our client already ready to rock. Describe stacks. Okay, everything looks good. We got our bastion we created, the security group and the VPC. Okay, that's fine. And now we're going to do the most important step of the whole installation. That is to mirror properly our internal registry. I created... This is the script that Hamid was mentioned before. Everything is new in this server. So I need to store almost everything. Yeah, Veeam is a good package to install. I think you heard it already. Yeah, perfect. We need to find out first what our host name is. I think the host name is, of course, but it's not what I want. Why? The following. I configure a most reliable host name, let's say for a mimic of a real environment. That was then the... And folks, this is real live coding, live hacking. No jokes. No preparation, no recording. Let's hope that everything is going to be fine. Yeah, we got this created. Very good. But I was expecting another host name for this. It's playing me with... Yeah, we're going to use that one as well because I wanted to use the test lab that I wanted to use. So can you give another little bit of context of what you're doing now? I'm going to create a mirror registry and for that, I'm going to use this magic script here. It looks a little bit long, but actually it's not that interesting. Sorry, sorry. What it does is first it install the package. I need a package to do the whole magic of the operating system, but it also installs the installed packages, the one what Jafar mentioned before, the OC command, the OC pistol, later for the installation. Fair tools like POTMUN, SCOPIO, HAProxy, in case we will use the proxy. This is a very complete script because it brings you from here to zero in respect of mirror registry. You mean the opposite from zero to hero? Zero to hero, sorry. That's very important point. Thank you, man. However, when you mirror the registry, you need a fully qualified domain name. Yeah, to specify very clearly that guys, don't put IPs because then it will not create the proper certificate. Use the fully qualified domain name for your environment. That is an internal hostname in the AWS machine or is the reverse? It's reversing to this, which I actually don't know why because it should be reversing. This is the internal hostname on an AWS virtual machine. Yeah, but I created a record in a private zone and that's what I was expecting. But anyway, we can also use this one. It's no problem. Okay, as best practices, it's better to create a record on Route 53 and in the private zone so your machine has this hostname, which is better than IP, whatever, and more mnemonic names. So you did it, actually. Yeah, I did it. However, I believe I'm missing the endpoint that is necessary for being able to query, to query back DNS entries. But I'm going to check that later. But in any case, we can use this. It's no problem because this is what, anyway, the bootstrap hosting when are we looking at, okay? Come here. That's what I'm talking about. You test everything and then at the last moment, it's something missing. It's never changed for demos, for live demos. It's really crazy. I mean, the important that worked on your cluster, right? If that doesn't run in the live demo, we don't care. It worked for us, fortunately. Okay. I don't know if it's very relevant in this short time to go through every single part of the script, but guys, you can go through it and then you have any questions how to use it. I'm more than happy to go deep a little bit on that. Okay. Rafa, and keep in mind that we are here in case you need us to do whatever explanations while you are doing things. Oh, that's very good. Don't hesitate to use us for that. Thank you very much. First step, we're going to install the appendix in the matchup before. Wow, that looks good. It's quite automated. I think Hamid can remember that I saved a lot of headaches in a given moment. You just launched the script and you can see what you need. Actually, it's very good that it happened because it means this part. It requires that on the root directory you got this file. I'm going to place that off camera because it's a secret file. Yeah, so this gives us the opportunity to discuss again a little bit about it. There are kind of 12 minutes left. I would like to know from Hamid and Brian what do you think about the experience you had with this disconnected on AWS and what do you think could be the next step? If you need to scale up your cluster and if you need to create multiple cluster, if you are planning any hybrid strategy or you want to use full public cloud with multiple cluster, what are the next steps? For us, this was the first OpenShift install within Fitesurf and it was very important for us to get this right because there are many other projects which would be needing a similar kind of setup. So yeah, so now we've got the AWS setup done. We might be looking at hybrid cloud with Azure as well. So yeah, we would need to work on that in the future. But yeah, so getting this disconnected was important for us. Okay, thanks. So a multi-cloud strategy always disconnected as far as I understand? Yes, yeah. Disconnected is very crucial for us because of the concerns on security and all that stuff. Cool, cool. For me if I'm wrong, but to complete this hybrid cloud a non-prem installation is going to happen too. You will have the complete hybrid picture. Thank you. And on top of that, again this I will say disconnected installation is quite standard and we are taking the advantage to get in the same procedure to install the software on top of OpenShift. In this case was the cloud packs on IBM and basically they are following the same steps duplicating this registry and continue, I mean that is an industry standard, I would say. Cool. Yeah, it's going to be an industry standard. That's why I wanted to understand a little bit more about this use case. So this disconnected on address is kind of a template let's say hey, this works, let's continue to do this and let's have also an hybrid cloud strategy after so we can have the resiliency, you know. We can have multiple points where running our workloads. That is cool. And how about this disconnected cluster? Can you say it runs many apps or it's running and only some pop up or how is this production? Yeah, I mean it's not been productionised because we are still in the process of evaluating some of the applications that are being onboarded. IBM and Brian has been involved so we are trying to onboard some of the IBM products onto this cluster. So we are still in process of testing our applications. So once this is done then we would be looking at moving on to the next stages. Cool. Thanks. Thanks for all. Hey guys, I would like to show you before the time. Yeah, actually we are looking at you Raph. Oh, sorry. You are seeing the screen. So we are talking and waiting for your message. Okay. Actually I found out that it was retrieving through, it was resolving two different host names but actually this is what I was looking for. So this is that, can you increase a little bit the phone because I think it's better for visibility. So you are using DIC to resolve this host name which is the bastionos. The bastionos is the host used usually for the installation. It contains a mirror registry we were talking about before. So it's kind of helper node. We call it also helper or jump node. Many names for this node. I like helper node because it helps. And you are now trying to resolve the host name for this node, right? I actually execute this command before and actually it didn't happen to me before that it was doing a reverse to this internal because actually I deactivated it. So I don't know why it didn't want to help me this time but this is the most important. This is my private song and you mentioned before, Natale, I would like to do this because you have control exactly of the DNS resolving and anyway, it's resolving properly from this server. So now what we want to do is keep checking what our steps are for the mirroring. We did the dependency, we did artifacts. I don't remember. You can do it again, no problem. Yeah, we only did the dependencies. Thanks, manager. The next step for this is an Ansible playbook, right? Some Ansible roles. Actually it is a quiz already but this mirror script is hot, as it was up there last night. So I need to really translate into a proper playbook. Okay, we carry on with the artifact. Then, interested enough, we're going to go to create our, all the preparation for our internal registry. I'm going to use admin, admin. We left the OC tools that Jafar mentioned to the magic. Then we come to the registry. It will last, normally last seven minutes. I see success. Hey, Hamid, working with Raph is great. It looks like everything is easy. I feel I can install disconnected blind now. Yeah, that's true. The script that Raph is running now, I've used on my own multiple times to create the registry. It works wonderfully every time. Cool. So you can try your script for people watching in the streams. I've also shared in the chat the link of the OpenShift, the Try link. If you want to try OpenShift on AWS or public cloud or any other installation, you want to just try it out. Please go to that link and start trying OpenShift. And you can use this script, this repo, if you plan, if you want to try disconnected installation. It's an helper that can make your life easier in this disconnected scenario. One more thing I would like to add to that. Guys, we need the mirror now, but that's the minimal necessary to do the vanilla installation. There is another flag here. It's a prepare operators catalogs. We actually do the mirroring for all the red hat and community operators. We don't want to do that now because it's not necessary for the installation. And also because it's quite big, we're talking about more than 120 gigabytes of beautiful operators, not only produced by Red Hat, but also by the community. But I just want to let you know that in case you want to mirror also the operators of the community, you can do that here. It's a bit of a script. Yeah. A bit of caution on the operators. I think it's more than 120 gigabytes. Really? Yeah. I did remember Hamid. Yeah. So we started it and left it, but the EC2 instance didn't have enough space. So I think it was up to 200. I don't even remember. I remember now. I was really sorry. Sorry. You know, the speed of the Pfizer was really amazing. Those guys are very professional and IBM are also doing their stuff with the integration of the Cloud Pak. It really was a super interesting project because we could see the power of a Cloud Pak serving the customer running on OpenShift doing what they needed. It was really, really good. It's a great team. Thanks. And the collaboration is still growing. It just starts. Okay. Yeah. That's wonderful. That's wonderful. The collaboration is key. So thank you for sharing this use case with us today on OpenShift TV and having this collaboration is key as a final dose because we are going to the end of the show. So I would like to do the closing. Raf, if you want to show something, I don't know if we have the time. I had hopes that it will finish and then we could see, we can query the registry. But if we don't have time, you know what I'm going to do? I'm going to run this. I'm going to load the video so you can see the whole thing in this repo later. Actually, that's a good idea. I see that I'm going to do. I'm going to do the whole procedure with a small shot of video and I'm going to play it in here. So if you want to try it at home, you are safe to do it. Thank you. Thank you very much. So the topology you shared that is in the repo is very interesting. It's very cool. And also this collaboration you were talking about with Amit, Brian, that is fantastic. This is really the spirit of how to build great things. Thanks for sharing this use case today. It was very interesting. If you want to reach out to Raf, to Amit, to Brian, you can ping us and we will also folks, if you want to share your Twitter handle in the chat, but we are always available on Twitter at OpenShiftTV. If you want to get more about this, please reach out to us. We will keep you in link with Amit, Brian and Raf. In the while, thank you very much for this session, Raf. You can stop sharing the screen now. So we end this session. I only would like to thank you all. And I would like to thank Chris short specifically for this big, big help on setting up this first EMEA Times on OpenShift TV show. Thank you, Chris, for having set it up all for us. So thank you very much. Appreciate it. Appreciate it. Talking about the collaboration. Natalie, do we have one more minute to show how the mirror finished? Or not? Just, we have a couple of seconds. While you show it, I have to close, but if it's finished, you want to show just that it's finished, it's fine. Okay. I'm going to show that right now. Yeah, so just before we close up, this is a show that we are hosting for the EMEA Time Zone. But I linked a calendar for the main OpenShift TV which shows, which have also a lot of great content. It's just maybe a bit later in the day for us. But yeah, I'm free to check it. And the good thing is that it's also recorded. So you can also take the... Guys, look at this. Here you can see the little pattern to be added to the installation definition file. And here you see the query. Query our internal mirror registry. So happy work. We got a very nice internet rate to do our magic. That's it. Thank you. We want to show, okay? That worked good. Thank you. You have it live. This was a totally live code. Live hacking. Thank you. Thank you very much. So I want to thank Hamid, Brian, Raph for the session today. Thanks for having joined this first episode of our OpenShift Coffee Break show. Just a quick reminder, as Jafar was saying, if we on OpenShift TV, the next show would be at 9.80 time, the level up hour today. And we will come back on two weeks, on the 17th. And the topic will be an inner and outer loop for Java developers on OpenShift. So if you like to watch all the shows, go into the calendar. I think Jafar shared it in the chat. In the while, folks, thank you for attending it. Have another coffee. It was a great coffee break. And talk to you soon. Natali, we see the Italian coffee is not the same as the rest of the world. Of course. You have to go to Espresso. You have to go to Espresso. We are a margin of improvement. This inside, my friend. Guys, thank you. Bye-bye. Bye-bye. Bye. Master, we are offline already, right? Not yet.


Alright, I'm Raphael, I work for a company called ArcSight and for a living I do enterprise security management or log analysis and correlation. And somehow over time what happened to me was that I got kind of bored of the textual log files and parsing and everything so I somehow developed a passion for visualization and that's what I want to talk about today. So the rough outline is going to be that I want to introduce the problem very quickly. I'm going to show you what graphing really is and what I mean by that. And then I'm going to show you a tool that I developed over time and some of you might have seen my talk last year where I introduced Aftergo a little bit. Over the last year I have made quite some improvements on the tool and there are new features in there which are quite interesting and I've gotten some really good feedback from you guys on what I should change and how you're using this tool to actually visualize your own data. And what I want to do in the end which is kind of the main part is I want to go through a firewall log file and show you some of the things you can do with it and how you can visualize it. Just to send that up front and you might have heard that from me already but I need to do it because of legal issues, IP addresses and host names showing up in event graphs and descriptions were obfuscated or changed. The addresses are completely random and any resemblance with well-known addresses or host names are purely coincidental. But don't try to play with these IP addresses and think it's me.  Alright well the motivation for doing visualization or looking at graphs is really the question here. What would you rather look at? Do you want to look at the left-hand side or the right-hand side? I prefer the right-hand side because I can't even read the font on the left-hand side. And I guess this is something probably everyone is hurt by now but a picture is really worth a thousand log entries. I hope to be able to prove that in the next 50 minutes if I'm not going over time. Alright so how does graphing work or what do I mean by that? There are basically two steps if you simplify it a little bit. You have a device that generates a log file or maybe events. You record them and you want to make a visual on the right side some kind of representation bar chart or a link graph or a tree map or something. What you have to do is you take that log file on the left side and you need to parse that thing. You need to figure out where are my source addresses in there, where are the destination addresses, where are the ports. So you need to parse it out and figure out all those things. And then you throw it into an event visualizer, whatever that is. You can use afterglow which I will show in a second and you generate a graph. What I'm going to look at for graphs today is I don't want to cover all the different types of graphs that are out there or visuals. There's bar charts and pie charts and line charts and so on. What I want to cover is just these two types. There are link graphs and tree maps. Link graphs are basically you have a couple of nodes or three nodes and they're connected to each other and they show you a certain relationship. Tree maps, they look a little complicated when you look at them the first time. You might not quite understand what they are but I want to introduce them to you and show exactly what they are in a second. Now if you take link graphs, a lot of times you will see graphs that show you communications between different IP addresses. You see that this machine talked to all these other machines but the interesting thing is that you can actually reconfigure these graphs to also show you other things in the log file. So what I have here is a snort event. You don't have to be able to read this but I basically take five fields out of this snort event. It's the event name, the IP addresses and the ports. And there are different ways to configure a graph now. The regular way would be that you take the source address, then the event name and then the destination address. So you will see things that what has this IP address done to other machines. However, if you configure this graph a little differently, if you say, well, let's visualize a way that I have a source address, a destination address and then a destination ports. You will suddenly see port scans because you see source machine going to some other machine on all kinds of ports. So you wouldn't see that in the first configuration where you see a source IP, a name and then a destination IP because you're not showing a destination ports. So the point of this slide is really you can configure these graphs all kinds of ways and depending on the configuration, you will see different things. Now let's have a look at three maps. And I'm not sure how many people are familiar with those but they're really useful for a lot of cases. So let's assume this rectangle that I have here is all your network traffic. Everything you see on the wire. Now what I want to do, I want to see how many percent are UDP or TCP. Basically I want to see the transport protocol here. And what I do is I just configure the tree map to use the transport protocol as a parameter. So what happens, I have 20% UDP, 80% TCP and it just partitions it that way. So 20% in the rectangle is UDP here. Now if I change the configuration, do not just be the protocol, the transport protocol, but I also want to see the service. So what happens then is basically this space is subdivided now and the protocols are shown inside. So for UDP you might see things like DNS, the rest of them P and this is probably like a ratio of 90 to 10. And then on the other side, the TCP, you see the partitioning to all the TCP protocols you have. So you can set up a hierarchy and the hierarchy is going to be displayed inside of this tree map and you could go deeper. I could say now that I want to see the target ports or the source IP address and it will just be subdivided inside of these little rectangles and it will show the percentage again. And you will see a real example later on what you can really do with this. Now let's switch over to Afterglow which you will find either on the CD or on SourceForge. There are two versions right now, it's a little confusing, I apologize for that. There's basically a version 1.x, currently the one on CD is 156 and it's basically a Perl script that generates your link graphs. And then the version 2.0, I started coding in Java and I used the library and I thought I would be able to get all the capability or functionality from 1.x into that new Java framework. But it turns out the library I'm using is a little buggy and the author was not really responsive. I traded a few emails and so far I haven't gotten any patches for the bugs that I found. So for now there are two versions, I'm sorry. Hopefully at some point there's a 3.0 which combines everything. So what you will also find in Afterglow is when I started playing with the log files I needed to write parsers because otherwise I couldn't really visualize my log files. So what I decided to do is the parsers I'm writing I just put under the Afterglow distro as well. There are three parsers, one for the BSD packet filter, one for TCP dump and I updated that to 3.9 recently. So if you're using an earlier version of TCP dump it might not quite parse it out anymore. And then one for send mail. If you look at the parsers quickly they might actually be quite useful even if you don't want to do visualization. Here's an example of how to use TCP dump to CSV. So what it basically does is you pipe TCP dump into it. That's just a long common line on top. And then you tell it what fields you want to extract from the original TCP dump output. And here what I said is basically I want to see the source IP, the destination IP and the source port. And if you want to know the names or the fields that are available look at the source code there's a list of all of them. So what this thing also does for you it takes care of connections. If you look at a connection or if you look at TCP dump and you see a client connecting to a server you will see a request going over the wire and the source address is the client, destination address is the server. But then immediately followed by that you will see a response. And now the source address seems to be coming from the server going to the client. If you visualize that you will suddenly see that there was a bidirectional communication. So you're thinking that the server actually connected back to the client. Which a lot of times you don't really want to do you want to see the real connections who was connecting to whom and not the bidirectional thing. So the parser takes care of that for you. It will keep state and remember what initial setups it saw and using some heuristics if it sees a syn by itself. Then it's probably the session setup if there's a syn act that's the response. So it keeps a hash of all those connections and inverts the output if necessary. Then the send mail parser. Send mail is they have certainly not designed this thing with logging in mind. What they do is for every email they generate two log entries. One entry tells you this email came from someone and then the other entry tells you it went to these people. And the way you connect them is the message ID which is in bold here. So what I did is I just wrote a Perl script that basically remembers all those message IDs and whenever it finds the combo of two went froms it will output one entry showing you the two went froms in one line so you can actually visualize connections now. And then the third one is the packet filter to CSV and I'm gonna use that in a little bit so I won't waste any time on it right now. Now afterglow and we saw a similar diagram already on how do you graph something. So basically you have a device outputting some log file. You have a parser. Either you use one of the threes I provide or you use a awk or something or Perl to parse the logs into a CSV file and that file you pump into afterglow and afterglow outputs not a graph directly but a graph description. So it basically uses the dot description from the graph is library and it just says okay there are these nodes A, B, C, D, E and they're connected that way so you have a node from A to B, one from B to C and so on. So it just gives the description and then you use some kind of a grapher and I usually use graphers to generate the graphs because what that does, it does all the layouting. So it figures out where to place all the nodes so the overlaps of the edges are not too high and you can kind of identify groups and so on and I didn't want to tackle that problem because that's other people have done that. So what are some of the features of afterglow? Well, it generates link graphs. Then what you can do is you can filter nodes but you can say well I can do that with grep minus V. Yes, you can but it's a little nicer to actually configure it in afterglow and what you can do is you can either do it on the name so I can say I don't want to see any nodes that contains windows, let's say or you can say based on the number of times that node was seen I only want to see nodes that have been seen more than 10 times or more than 100 times or something so you can kind of filter noise out. Then you can do fan out filtering. A fan out of a node on the right side there that blinking node has a fan out of three. So there are three connections going to other nodes so you can specify when you draw a graph that you only want to see things that have a fan out of X so let's say only everything that has more than five connections going out you want to visualize and you might already have in your mind what you want to do with this and it's basically detecting port scans very, very quickly and filtering the noise. And then you can color everything you can color the edges and the nodes and finally you can do clustering which is very useful as well because especially if you start visualizing firewall log files you have this hundreds of machines or thousands of machines connected to your network you're not interested in every single one because that graph is just going to be cluttered so you want to cluster all those together and that will show you what I did with that. If you start after glow you can use minus H and I think the usage is actually up to date I hope. These are not all the parameters that are there but the most important ones and the most important one in here is minus C where you provide a configuration file and it will show you a few of those files in a second. Then you can use minus D and that basically adds the count of how many times a node showed up into the node itself. So you can kind of figure out what was more severe than other things because you might have seen them a lot of times. Then there's an easy way to change the edge length you just say minus E and then give a number that's basically the default edge length that it uses to have the nodes separated. Then you can also use minus N if you're not interested in node labels at all you just get rid of them and sometimes that's really useful to just see kind of patterns in there if you're not interested in seeing all the labels and knowing exactly what it is but just kind of figure out what does this look like. And then you have minus O it's a horrible way I named it here omit threshold it's basically how many times does a node have to show up to actually be in the graph that's what I kind of described before and number of occurrences. And then you have a minus F and a minus G which is used for the fan out filtering and you can specify two different fan out filters one for the source nodes how many time or how many connections does that guy need to have and one for the event nodes or the middle node how many connections that guy have to have to show up and minus F is for the source and minus G is for the event node. Now we're getting into the property file and this is really where you configure everything for afterglow. Let's start out with coloring.  The easiest pearl expression is just a string you just say red and that the source for example is gonna be red all the time. Then if you wanna get a little more sophisticated use the next line color event equals red if and then I provide some kind of an expression there and afterglow keeps track of the current log entry in the at fields array. So whenever if I do this here dollar fields one that means that the second column starts at zero the second column has to be matching these reg X basically starting with 192 dot something. In that case I would make the node red and you can do all kinds of things here it's really just doing an evil on this pearl expression so whatever you wanna do you can read from files and all kinds of things. Then if you wanna filter the nodes you just use the color invisible. Same idea you say invisible then if the first column equals IS action it won't be shown.  What this does now is it evaluates that for every node and basically returns or changes the node name. So if I have a whole bunch of 192.168 addresses and I wanna cluster them together into one node I can basically say that the cluster source equals and then I just give some kind of a string I can say 192.168.x.x if the field matches my regular expression 192.168.star. And that would be the node that is assigned now instead of all the other so that's evaluated every time a node is drawn. We'll see an example of that in a second. So this is kind of my hello world example. My input data is on the upper left there are four entries in my log file A, B, A, C, B, C and D, E. If you would just visualize that you would get something on the lower left the nodes are connected. The command to run after glow is on top and I think there's a pipe missing in your slides on the top right but you will realize that yourself. You cat the file, you pipe it into afterglow you give it a property file and you use minus T. By default afterglow expects a three column input basically the first column being the source the second column being the event node and the third column being the destination node. So it's always a three tier thing here but if you use minus T you just give it a two column input and that's what I have here. So now to color these nodes as you can guess how that works you just say color.target is blue and by the way that's all my PowerPoint skills I can't go any further. So target is blue if the second column is not an E. So you look all the target nodes the ones that get an arrow pointing to them they're blue, the E there is not blue because it's not E or it is E. Sorry it's early. Then the source I wanna make green if it's not D. So that's that one guy there and by the way it's first match and then it's done so it doesn't go and evaluate the rest. And then everything and all the other sources are gonna be red and then whatever is left I just say color is green and so I don't have to say color.target now or anything I just say color and that applies to all the nodes. All right so that's the easy example and it's getting harder from here I hope you're all still following, okay? Good, all right but let's quickly talk about Africa 2.0 so in case you wanna play with it it's a little simpler. You have two comment line parameters. One is for the property file and the other one optional is for the data file. You can also provide the data file inside of the property file if you choose to do so. It's taking CSV input again, it's kind of my mantra I wanna use CSV files because then you can use your own parsers I don't have to care about your data set at all.  And the interesting thing about Africa 2.0 is that you have the tool generate the graph itself so it doesn't just generate a description and then you have to use some kind of graph to actually visualize it but it's a Java itself that generates the tree map right away and what I can do with that is you have a little bit of interactivity. You can basically zoom in into the different parts of this tree map if you wanna look at close ups and you can change on what level of the hierarchy you wanna use the color. So before, if you remember the example my top level hierarchy was the transport protocol so you can say I wanna color this graph by the transport protocol so you will just have basically two different colors in your graph or you can choose the color on the service and then you will have all kinds of different colors just as in my example. So you can kind of play with where you wanna use the color from what depth in the hierarchy. And a simple output looks like this. All right, so let's get a little more practical and let's see how we can visualize a firewall log file and you gotta stop me if I'm going to fast here. So if I'm analyzing a firewall log file there are a few steps I wanna take. The first step is certainly I need to parse it. The second step, I probably wanna look at what is coming into my network. What is allowed through the firewall so I kind of know what is hitting me so I can probably verify the ACL and figure out whether there are any gap or any holes of things that are coming into my network which shouldn't. Second, I probably wanna look what is leaving my network.  Then I usually look at outgoing blocked traffic and that tells me that either I have misconfigured machines which are not configured according to my policy because they should not be trying to connect outbound if they're not allowed to or maybe my firewall rule set is misconfigured and there is really a need for this service to communicate out. So if you have ever set up a firewall yourself and nothing worked anymore, you might have realized that you might have blocked ICMP and it doesn't do path empty discovery anymore or something like that. So communication is gone, especially in the older DSL worlds. And then if you're brave, you can also try to visualize or analyze the traffic that is blocked from coming in and you will see all kinds of things in there. So the parsing part, I have a PF output here, a log file, just two lines. Then what you do is you cat that file and you use PF to CSV to parse it. And in this case, again, I wanna use source IP destination, IP destination port and the output is just gonna be those three fields delimited by a comma. Now this CSV output is the input for afterglow. Well, putting it all together, you basically cat the log file, PF to CSV, give the fields you want, pipe that into afterglow with a property file and then pipe it into a graph viz. And there are basically four tools that graph viz provides that use different layout algorithms. One is dot, which uses a hierarchical layout and you will see that in a second in the first example I have. Then there is a needle, which does a organic layouting. So it tries to place the nodes optimal so there are not too many overlaps and similar things fall together. Sometimes in a few cases that doesn't work out too well and there are two others, there's circle which uses a circular approach. It basically starts by putting all the nodes into the middle and then there is apparently too much overlap there. So it starts to just move some of the nodes out and out and out. So it does like concentric circles until it finds a layout where the nodes are placed in a decent distance to each other. And then there is a 2PI which uses some other weird algorithm to place things. So if you get stuck with the others you can try 2PI and see what comes out of there. All right, so let's take the first use case we wanna go after here. I wanna look at all the traffic that got passed coming into my network. So what I do is I somehow grab all the passes in my log file and this is, don't do this literally, I just use this place all the pass in. It really depends on how you filter for that in your log file. What I did in my log file is I grab everything that is coming in on my outer interface which I think was Excel one or something. Then you pipe it into PFCSV. I get my source IP destination IP destination port and then I go into afterglow, give it a properties file. I use minus D because I wanna see the counts on the nodes. How many times did these things show up? Something showed up just once. I might just close my eyes and not think about it. And then I pipe it into dot. That's the hierarchical layouting. And that basically looks like this. Do you have a node on top called external? Which is already kind of strange because there is no external IP address or an IP address that's called externally my log file. So we'll figure out how we do that. And then you see all the machines in my network tab where it exists. And then the bottom circles are the target ports or destination ports that you connected to. So how do we get the colors and the clustering? Well, use a few features to make this not too complicated. So the first thing we wanna do is the clustering. So what I do is say the cluster.source. So all my source addresses. Because I'm assuming the sources are in the internet. That's what I'm looking for. Everything that comes into my network. So the source is external. That's the cluster name that I wanna give it. If it doesn't match my internal IP range. And match is a function that Afterglow provides you. And it basically, I told you before that there's this add fields array that it basically keeps track of the entries in the log file for the current entry that it looks at. The match just looks at all three of them and makes sure that none of them matches. And if I mean source, it only looks at the first one. If I just set cluster, it would look at all three of them. So it always uses the one where it's evaluating currently. So this is a very simple way of clustering all my external nodes. Then I say, well, I wanna have this thing red. So basically if the field is external. I could also use matches here if I use external as a reg X basically. But I chose to use field, which is another function that I have that just returns to current field it looks at. And then you can compare that to something. Then the event. So that's a dark blue nodes there. I wanna have them blue if they match my internal address range. I hope that all my nodes are gonna be blue now. However, you see there are a couple of light blue ones in the bottom left. So what I did there is I say everything else that's not blue is light blue. And that's actually kind of interesting. Why do I have the light blue ones on the bottom left? Well, if you look a little closer, you see that my machines were connecting out to those guys in the internet and they're using port 20. And that's an artifact of FTP because the way PF to CSV parses my file, it doesn't know that there is a back connection out from FTP, so that still stays in there. But it's kind of nice to see that. So I know that two machines in my network were actually FTPing back out or sending traffic ban data out to the internet. And everything else is just red. And what I see in this graph now, if I look a little closer, I see all kinds of different ports. I see port 25. Yeah, I have a mail server. I have port 22. Yes, I do SSH administration on there. I have some echo requests. Well, hopefully that was me pinging those machines and no one else. FTP, I have web, I have HTTPS, I have some DNS. And then I have port 100.  And it showed up only 21 times. And I remembered, yes, that was my backdoor SSH access whenever I updated SSH on port 22 and I didn't quite know how to do that back then that I don't, if I bounced it, I always locked myself out. So I just had another SSH on port 100. So I could connect in there, bounce the one on 22, update it and whatever. So that's, I remembered it was all okay. So I was good. All right, so the next use case is a very simple one. I wanna see everything that goes out of my network. And here again, I'm using very loose syntax. I don't, you don't see the PF to CSV up there in the command.  So basically, I look at all the things that are going out and this is the graph that shows up. You can't read the node labels, I know. In the middle, it's an external node again and everything connects to it. Those four machines or three machines, no four of mine, the light blue ones, the circles that you see, they all connect out and you see the red ones are all the target ports. I don't see much of concern in here. There's a lot of web traffic. There's some echo replies. So maybe I could actually filter that because I have people pinging me. So I might wanna restrict them from knowing that my machines are up. I have some DNS traffic going out and one of the traffic, a little bit of the traffic is 123 NTP. That's probably okay if I'm sinking out with some external source. What I might wanna do is verify what the target addresses of those NTP connections were but they were really NTP servers I configured and not some other machines and someone is trying to tunnel something or so. Well, the next thing I wanna try here is I wanna look at all the traffic that gets blocked going outbound. And remember, these are a misconfiguration of my internal machine or the ACL is misconfigured because there is a service that is needed by the internal machines and it's not allowed. What comes up is this really nice cluster of nodes. What happened? There's a lot of target ports that are being blocked. So does that mean all my machines are misconfigured or the ACL is wrong? Probably not. Let's think about this. We have a client and a server simplified. I have some machines in my network which are the servers, the clients probably on the internet. I have my firewall with two interfaces.  So always think about this when you have, if you get a graph like this, whether there's something wrong. So what happened here is when I built my firewall rule set, I was not very consistent on what interface I'm logging and I logged on the responses going outside. So not just did I not only log on the requests, I also started logging on different interfaces. So always think about it. Already when you set up your firewall, what do you log? Do you just log requests and responses as well? I don't know what interface you log them. And it also depends on where you wanna have your ACLs on the internal interface or the external interface. A lot of firewalls don't give you that option, but especially if you start playing with IP tables or PF or anything else in the more hand-woven firewalls, you will run into this problem. So think about it before you do it. So this is the reason why we have this huge cluster of nodes here. So, well what I did is I basically extended my PF to CSV and unlike the TCP dump parser, I didn't wanna take care of the swapping right away and kind of transparently to me, but what I wanted is I just wanted to see it should just tell me if it thinks that it should be switched. So if I say reversed as a parameter to PF to CSV, it will write an R in the last column. And it was really late in the night when I coded this and I haven't changed it yet, so bear with me. If it's reversed, if it thinks it should be reversed, it just writes an R in the last column. So if I do a grep minus V of an R in the end, I won't see all those responses. So if I filter out all the responses and you could use any other method of getting rid of your responses in the log files, it's just one way to do it. Then I end up with a more sane graph here. It's still a lot of target ports. If you think about it, everything that's blocked going out, there really shouldn't be anything. But there's some artifacts, like very high port numbers where the heuristic to actually determine whether it was reversed probably didn't kick in. It didn't have enough state to figure that out. You could probably go ahead and improve that heuristic, but I was kind of satisfied with the graph I got here. I can read this and kind of filter them manually. But there are some other interesting ports. One is 427, a service location. And what this told me is that I had four Macs standing in my network, and they're using, I'm not a Mac person, so I'm not quite sure what this service actually does, but it's some kind of an automatic setup, service broadcasting and it advertises itself to the network saying, hey, I have these services, I'm here. So this concerned me a little bit and they actually showed up like 16,000 times in my log file, it's probably more than, that's probably almost all the block traffic that I had. If you had to go through a log file, this log file manually or textually, you would have seen about 16,000 of these guys and somewhere intermingled, you probably would have gotten tired finding those other ones, the other ports that were blocked. So this helped a whole lot to actually find this and hopefully none of this traffic was actually leaving the network and it was all blocked. All right, now we're looking at the blocked incoming traffic, yes, it's gonna be way messy. So I'm not even gonna show you the graph that was computed by this. I decided to still tackle the problem and kind of figure out what is in the data set or what is in the data that actually tries to hit me. And one of the things is, it's always the low hanging fruit, find your port scans. You can do it a thousand ways, I'm just gonna show you how to do it here. So basically I'm using the minus G capability. So remember that was the fan out filtering and G looks at the event node, so the middle node. And if I say minus G2, that means everything that has a fan out of three or higher or not two or one, I don't wanna see. All the demands of three or higher I wanna see. So everything that has more than two ports that were connected to, I wanna show them. So if someone with port scan me, I would see that because they hope the port scan is more than two ports or whatever threshold is set. It's actually very interesting what shows up on the right side in the graph. I didn't have a port scan. So I don't see any fan where things are hitting me. I have three nodes at most there and there were people probably just trying to get to some well-known ports of mine, very specific targeting, port 80, 22, 443. And it's only about three or four attackers that really did this. So the kind of yellowish nodes that you see. So that's actually kind of interesting. And the data set was probably a couple of days in a very, very busy web server form. So this is pretty good. How do you configure this? And actually I cheated a little bit. If I look at port scans, I'm not necessarily interested in the things that hit me on ports above 1024 because I usually don't have anything running there anyway. No services are up there. Maybe there's some Oracle running on 1521 or some other SQL server. And you might wanna include those as an exception here. I didn't do so because I didn't have any. But what I wanted to do is I clustered my target node, the ports, and I wanted to make a cluster for all the ports bigger than 30,000. So it's basically say the node should be bigger 30,000, just that string, if the third column is more than 30,000. And then, so that's taking care of all the ports higher than 30,000. And then I say, well, everything else, if it's bigger than 1024, I wanna cluster it again. And then the rest I'll leave alone. I wanna see those individual ones. And again, I used the minus G and this basically gave me the graph on the right. This is kind of how you can find port scans. The same you can do with tree maps. By basically what I did here is I set up the hierarchy as source IP, destination IP, destination port. So you will see a big rectangle for the source IP. Inside you will see all different destinations that that source went to. And inside of that, you'll see the different destination ports that were accessed. Who can see the port scan? Exactly, it's very easy to find. There's just a lot of nodes there. And what you can do is you can actually zoom in. And now I just see this one source IP you go into destination IP and I see all the destination ports. What is kind of interesting is there are all these random port numbers on the right side and then on the left, about 50% of the traffic hit a port 6346. I think that's some kind of file sharing. So someone really hit me hard on that port and didn't get in. Something else I usually do is I wanna see whether there are any IP addresses that are spoofed or abused and there is a Bogan address space. Basically all the IP addresses that are not assigned to anyone by IANA. So no one should be using them. No one is really allocated that space that's published on the IANA webpage. And it changes frequently. So make sure if you do this, you update your configuration. But I wanna see all the connections coming from those IP addresses to my network because no one should be using those.  If you thought that it was complicated before, it's gonna hit you. So what do you do in the configuration file? First I need to let Afterglow know what is the Bogan address space. So there is a command called variable and then you just provide any Perl expression you want, any Perl command and we'll just evaluate it. And if you assign what I did here is I build my array ranges with all the subnets that are in the Bogan address space. You could put anything in there, any Perl code, it would evaluate it, it would assign the variables and keep them and you can use them in further assignments and do all kinds of things. So this is just, I'm just defining a variable here or an array. Then for my color assignment, it gets a little complicated. So basically what I wanna do is every value I get, I wanna check every IP address, I wanna check against the array and see whether it's in any of those subnets specified. And you basically start with the subnet assignment here. So you check if the field is in the subnet that we're looking at right now. So it basically runs through all the entries in the array and the array ranges up here. So it goes through all these entries and checks whether the field that I'm looking at right now is in the subnet. If yes, it just returns one and I'm basically counting in dollar value how many times or in how many of those it was, it's always gonna be one because my definition there is mutually exclusive of the individual entries. Anyways, so what it does then, it just says it's gonna be red, the node, if it actually founded. So if value is set. So if you don't quite follow how this map thing works, just copy it and use it for your own things. And then what I wanna do is I wanna have everything else green if it's not in my, or everything else that is in my internal network is green. So I provide the match of my internal address space and everything else is blue. We'll see in a second how that looks. But we're not quite done yet because what is gonna happen if this graph, you'll see a lot, a lot, a lot of source IP addresses that connected. There's gonna be a million individual nodes so you won't be able to find those red ones that we just defined. So what we need to do is some clustering. This looks horribly complicated but that's just because I was a little anal on what I wanna see here. Basically what I wanna do is I wanna cluster by class A network. I don't wanna see each individual address, I just wanna see what class A's connected to my network. That's basically what I do here with this regx replace. What you can do is it's a function I provide for afterglow and what it does, it just matches the regular expression I give there, the backslash backslash d plus. If that matches, so if it has a number, if it starts with a number, then it returns that number because I do a match there. So it will return the first octet and add the string slash eight. So I see that it is an eight class that I clustered. If it does not match my internal addresses, because I wanna see if there's actually a source that it's my internal address. And if it's not one of the bogon address spaces, because I don't wanna cluster those together, I wanna see exactly what IP address hit me. And not just, oh, it was in this class A. You follow me? It's early, I know. All right, and I do the same thing for my targets. Just for your own reference, these are the new features we just introduced. And this is how it looks. It's actually quite sane. So you will see the bogon address space is the red ones. Then you have the regular external addresses clustered by A class. You have my internal machines, the blue ones. Suddenly, it's actually interesting. Suddenly there are more than my four machines from before. It's actually kind of funny. I haven't realized that earlier. But the interesting part is the red nodes. I had a lot of bogon address space. And this is real, I didn't make this up. But if you look, there are 10 addresses. There are 192.168 addresses. There's some multicast addresses. But still, this was in a hosting environment, in a data center. And why would there be other things leaking over to my network from 192 spaces and 10 spaces? So maybe some other people that were hosting there were trying to play tricks and try to see what they can do. Spoofing IP addresses are just like leaking over to my connections. All right, I'm actually right on time. Before I give you the summary, I have one thing that I keep doing with Afterglow, which is kind of interesting. I have a, basically run a Unix command on my box. It's just a while loop. And you'll find the command on, if you go to Afterglow.sourceforge.net in the FAQ, I have a little command line. So basically what I wanna do is I wanna show the network traffic in my screensaver. It's kind of cool if people walk by your desk and they see the traffic that's on your box. Some people might start playing with you and sending you traffic so they show up in the notes, but so don't encourage them. But so basically what you do is you run a while loop, continuous, just never stop, run TCP dump, pipe it, the output into TCP to CSV, get the fields you want, and then just do like head minus 1,000 or something. So every 1,000 packets will actually continue. Then pipe that into Afterglow, into NEDO, and the output file that I usually write in NEDO is just a file with the date in there. Just add the date as a parameter to the file name. And then it will just write every 1,000 packets it will write a file. And then you can have a cleanup script that kills the ones that are older or just keep two around or something. Configure your screensaver to pick up the pictures from that directory and just slide show them. And it's really cool, I like it. All right, so let's go to summary. We looked at Afterglow. You can do filtering, you can do coloring, you can do clustering if you get brave. And why would you, what we do all that? Well, I hope I was able to show you a few use cases where I was able to find my port 100 that I connected to or if we find the Bogon address space, you find outliers, you find relationships among things. Who is talking on the network? And I know people in here that use Afterglow to find people that are connecting from countries of concern, things of that nature. Suspicious activity basically. All right, so don't read log files, visualize them. Thank you. I think I have time for a couple questions if anyone, yes. Did I try cluster by country? Yes, basically what you can do is there is a mapping that's published that maps IP addresses to countries. So just block that in somewhere, exchange your IP addresses by country, and automatically it will have them clustered by country. Yes. No, not at all. So I just gave you one example, firewall logs. I actually yesterday sitting in one of the sessions and sorry, I was a little bored. I was basically running Kismet and I quickly used an awkscript to pick out the access points that are there, the machines connecting, and I visualized that quickly. So I see that all these machines go to this access point and suddenly you see this machine goes to the other access point. There's really cool things you can do. Do I have any competition? Lots. There are a few people. Greg Conti does some visualization with Rament. There's a lot of university work that's done visualization of network traffic. I'm not sure if anyone has a very simple method of visualizing just open source. If you have any, let me know. Have I extended this to look at the amount of traffic, things like amount of bandwidth and so on? So basically what you have to do, you have to do that yourself. So I kind of offload that work to you because I just visualize your CSVs, right?  You just awk and do sums in there. It's not too complicated. Oh. All right, so questions please go to the mic so they're recorded. Thank you. Can you weight the size of the nodes by the number of packets from them? Can I weigh them? Can you make them larger if there's a lot more traffic? Oh, can I make it larger? I didn't put that in. It shouldn't be too complicated to extend after go if you want to do that. Would take me a little bit of coding. Download it and extend it if you want. I might do it. That's actually a good idea. Thanks. How about parsing server logs? Parsing server, any kind of server logs? Yeah, any kind. Sure. You can, for example, Linux, you use Syslog. It's a pain to write your parser because every line is different. Every entry has a different format. But you can visualize logins, for example. You just pick all your SSH messages or all your login messages, pick the source address, where did it come from? If it's locked, it's not always the case in server logs. Lock the username that's come in. You can do all kinds of things, really. This is giving you the toolbox to go off and do your own visualization of your logs. And please send me use cases if you have them. I'm collecting them. Server logs are very interesting in cases where you're not trying to go after the perimeter threat where things are coming out from the outside. But if you're looking more from inside or problems, so people that are logging into machines that might not be allowed to access machines for if you do any compliance reporting or something, who got into my financial servers, what networks they come from, things of that nature. Anyone else? Are you trying to make it real-time or? Great question. I was trying, yes. The real-time problem is, that's kind of why I started to switch over to that library I'm using for Java. Because I wanted to have more control. The way I set up Afterglow right now is that it basically takes the log file, the CSV file, and generates one graph. So I kind of lose state. The problem is, I mean, you can do what I told you about the TCP dump, the while loop, and then just get one picture after the other. The problem with that is, every picture you calculate is gonna rearrange all the nodes because of the layouting. So what you have to do is you have to remember, you kind of do a sliding window, you remember the current layout, you feed that into the next layout, and you just add the new ones and then drop off things over time to get rid of them. It's not as easy as it sounds, or maybe it already sounds complicated, but I was trying for a little bit, I just don't have the time to do it right now. But there are, I think Dan Kaminsky with Xovi does something of that nature. He tries to animate it. You can try to download that. It's slightly different what he does there. Thank you. All right, I think. What kind of limitations do you have on log file sizes? Your box. Okay. And afterglows memory.


So, we're here at the SID display week, and who are you? My name is Ross Young. I'm the CEO and founder of a company called Display Supply Chain Consultants. So, do you work in the display supply chain and you consult them? Yes, we track and forecast what's happening in the entire display supply chain, ranging from the equipment that goes into the FAB and the materials that are used on the displays to the panels and to the OEMs and brands and retailers and the financial analysts that follow the display industry as well. So, what's going on with China? China has announced a number of government subsidies, both at the local level and national level, to subsidize new FABs in the display industry to build capacity. And essentially, they are taking a lot of market share and forcing companies in Korea and Taiwan to have to innovate in order to survive. And we expect the Chinese to take a lot of market share, the Chinese companies to take a lot of market share in order for the Korean and Taiwanese companies to survive. They're going to have to innovate, come out with new products like foldable displays and micro LEDs and products like that. Sunlight readable displays? Sunlight readable displays is another one, something that the Chinese companies, the commodity displays that aren't available, something you could use outside that has a very long lifetime. It's still very readable, reflective, things like that. Because we want to have phones and laptops that last for 50 hours, 100 hours on the battery, not just like five to 10 hours, right? Yeah, one of the problems with the Apple Watch, for example, is you have to charge it nightly. So if you could have a display that could last a week without charging, that would be a much better solution. Like my memory LCD, but it's a little bit hard to read. So how about some, are they going to, because in the last decade, maybe Taiwan and Korea just didn't want to take too many risks. So they were just doing basic kind of displays, and now they have to take risks. I would say that they didn't want to compete in the commodity markets. And so they've gone after a lot more niches. But now you have a number of companies all going after the same niche. So that means it becomes a commodity. So they have to work harder at finding different niches. And we see a lot of interest in all of these new technologies going after more niche applications so that these companies can survive in the market. And just as a smaller side, Europe and USA, did this just nothing? There is the possibility of Foxconn building a factory in the US. The state of Wisconsin has agreed to subsidize that factory up to $3 billion. If they build less capacity, the subsidy will be smaller. So it will scale with the amount of jobs that are created. And in Europe, there's certainly nothing. And that will be displayed? That will be displayed. The intent was to make very large TVs in Wisconsin. The shipping costs are actually a big chunk of the cost for a large display. So if you could build it locally and ship it to Best Buy in Minnesota by truck instead of by air or by boat, you could save a lot of money. And it would create a lot of jobs. And that's something that President Trump has been very focused on, is bringing more manufacturing jobs so that it actually could include TVs. And there was tariffs that are rumored to be created to make it even easier to justify building display fabs in the US. So there's definitely in the SLE display, there's lots of examples of awesome new things happening. So there's plenty for them to choose from, the Korean and the Taiwanese. Like, micro LEDs are pretty awesome, right? Yeah, micro LEDs are great. But right now they're very expensive to manufacture. So you could build a prototype, but if you want to compete in a very price-sensitive market, it's not going to be acceptable. So they're still working on how to bring the cost down. And but like the Apple Watch would be a logical implementation, initial implementation because the brightness is very good. The battery life is really is much better than where OLEDs are or where LCDs are. So it's a natural application. And it doesn't have too high a resolution. So the cost is more manageable because LEDs scale with the cost scales with the number of pixels. So are we definitely going to see this dream-like flexible phone that just folds and becomes huge? Or that you flex and put in the pocket and doesn't scratch? Very interesting. So right now the flexible display costs three or four times more than a rigid display. Yet you cannot flex it in the final product. So there is no incentive for manufacturers to buy flexible displays. However, the industry is building all of this flexible display capacity. So these flexible display prices are going to come down and companies are going to work harder and harder to differentiate their flexible displays by creating these innovative foldable products that are going to consume all of that capacity. So we see a lot of interest in foldable displays. The problem is there are a lot of challenges to be able to make a display that can last three years with all of these folds, given the materials have never been folded before. So you have adhesive materials that cannot delaminate over 100,000 folds and must also now create some sort of slippage between the layers so that it reduces the stress.  And it has to be very, very thin. And yet when it's thin, it's soft, but it also has to be hard so that you can't scratch it. So you have a lot of challenges to build the materials that go on top of the OLED or the TFT to withstand all that folding that have never been in that position or asked to do that before. Would it make sense because people pay $700-800 for a phone, would it make sense to kind of have the display be kind of modular? So like people before would swap a battery after a year or two, maybe you'd swap the screen after a year because it gets scratched up. But you'd just pay $100 or something just for the display. Is that enough to cover display, flexible and everything? Probably not. I mean, you could buy a flexible display right now for $70 or $80, but then the cost of shipping it to the regional Apple store and making that replacement for a flexible display would certainly add a lot of cost and the labor to do the replacement would add a lot of cost. Maybe it should be user-replaceable. Yeah, I think you're going to see the foldable phones come out for like $1,500 or $2,000 because the volume is going to be pretty limited and they're going to charge a lot of money for it. But it's going to come down in price as more brands start entering that market and offering lower margins. But it will start high because you'll have companies like Apple and Samsung who have high margin requirements introducing those products. So I love those reflective displays and I've been doing videos about like a decade ago about PixelG. And then there was Liquid Vista who was trying to do stuff and it was acquired by all kinds of companies, Samsung, Amazon and all that. I don't know what happened with that. And then there's Clear Inc right here. And E-Ink is awesome. So what has happened? Is it really happening? Are we going to get those amazing displays that just works in the sunlight? Yeah, I don't think the... I heard that the Amazon investment in Liquid Vista is not panning out so we don't expect to see that product come to market but I think there's renewed interest and new investment has gone into this company Clear Inc that can do both color and video in a reflective display and they're ramping up now. They're working with Merck who is also an investor and they have IP on the material so Merck is heavily motivated to do this as well. And the prototypes keep getting better and better and so they have a real opportunity.


Now in business, the federal government says it will not rest on its oars to ensure all its employees are migrated into the integrated payroll and personal information system IPPIS platform. The accountant general of the federation Ahmed Idris said this at a nationwide IPPIS training for role players in Lagos. Justin Akadonye tells us more. The integrated payroll and personal information system is one of the key reform initiatives of the federal government, which maintains a database for information and actions on government workforce. This position honors course to resolve by the federal government to evolve permanent solutions to heaps that may be observed in the operations of the policy, hence this training. The accountant general of the federation who is represented by the director of consolidated accounts Salaw Zubairi hints that 711 MBAs have been enrolled into IPPIS. So the essence of the training is again to deepen the knowledge of the role players to know their specific roles and the issues and challenges they normally face, how to address them, so that they don't have to be coming all the way to Abuja every now and then, the high visual of variation, the high visual of of minor issues that could be handled within those ministries and department What is expected of the participants is highlighted just as the AGF says the policy has brought greater efficiency to the planning of personal emoluments and budget. It is also to bring home to the role players that the accountant general of the federation, the office, is desirable to decentralize the management of payroll, so that all the issues today, because what we have now is like you have a centralized system that at the end of the day every year that comes up you have to go to Abuja to resolve it, but the message that has been brought to this training is that the IPPI has been decentralized, each ministry department and agencies can address quite a number of the issues that they currently bring to Abuja. You will not be able to register a terminated employee, you will not be able to pay part A area of an employee, you will not be able to create elements on the platform, you will like bank property or change your bank details, you will not be able to pay sabbatical employees or contract staff. This special training commenced three weeks ago and has been successfully held at the four geopolitical zones in the country. Justin Akadone, for Plosive Africa.