All right. Hi, everyone. Really happy to be here in front of you today. I'm Adrian. I'm a senior ops engineer at Decathlon in the Cloud Platform Engineering Business Unit. So at the name suggests, we provide the Cloud Platform product to our internal users. So the idea is to standardize and ease the adoption of Cloud solutions for all our IT users. I'm more specifically in the Cloud Native Partner team, which handles everything related to Kubernetes, content orchestration, and serverless components to integrate them into this platform. And today, I will talk to you about Service Mesh. And first, I wanted to talk to you about the study we did about Service Mesh last year. But then I changed my mind, and I'm going to talk to you about how we can improve Service Mesh with AI at the edge. Basically, the idea is to see if you can connect your, let's say, your connected bike into your Service Mesh and talk to it when you go to work in order to configure your various Service Mesh. I don't seem so excited. But I actually talked to you about the study we did about Service Mesh. So basically, this is the conclusion. Service Mesh is not a solution. Thank you. Feedbacks appreciated. We are hiring, by the way. Thank you. I saw some up in the eyes of my colleague in the front. We're thinking, yeah, we can go to lunch early. But I owe it to you to dive in more on that. So why is Service Mesh not the best solution? We'll see a bit about that later. First, a few words about the Catholic. So the Catholic is the world's largest support-good retailer in the world. We had $15 billion in revenue in 2022. And we have around 5,000 teammates, as we call them, working in the IT overall in digital. A few words about our container journey. So we started with containers quite early. We did our first test with Scabrantees in 2017. We put our first Scabrantees in production in 2019. In 2021, we went full public cloud. So we basically shut down our latest data centers in 2021. That was the end of a long journey that started in 2016. And now we're in 2010-24. And in 2023, we did this study about Service Mesh. So we look at why we did this study. We look at what our user needs and who are these users. We'll focus a bit more on Service Mesh to see how it works, what it does. And then we'll focus again on the study itself and these conclusions. So first of all, the context. Before diving into the container context, I want to say a few words about our API management system, because it's quite relevant for the next part of the talk. So basically, our API management system, it's an API gateway like all of us use. The idea is to centralize the exposition and the API calls so you can enforce quality and security. The thing to note here is that the architecture of our current API gateway is that we use big centralized data main, big centralized gateways that are, let's say, continental. So we have one big gateway in Europe, one big gateway in Asia, one big gateway in the Americas. And basically, all the API calls, all our APIs are exposed through this API gateway because it's like the golden rule of the Catherine. If you have an API, it must be exposed through the API management system. And so basically, all the API calls go through this gateway, whether they are external or from service to service. So what are the current issues with our communities and our users? Well, as I said, we are using Kubernetes for quite some years now. So we are past the day one and day two issues. And our users are starting to have more issues related to the business. So basically, this issue is all into costs. In the Catherine, we have, let's say, it's directly a lot of small clusters. So this includes some costs because there are a lot of clusters to manage. And well, you have to pay for them. This is changing, by the way, but still, we have a lot of clusters with couple of services. The other thing is related to the way we use the API management system. So you can imagine that having this big centralized gateway, this creates some latencies. Because if you have a service that you want to talk to another service, that is in the same cluster or in a cluster nearby, it has to go out to the gateway. And then the gateway will reach the other service. So this includes latencies, of course, but also costs. And for the tricky ones that don't use the API management system, well, they are in the pickle because, well, they don't have a lot of observability because observing network inside communities, inside vanilla communities cluster, well, it's quite opaque. So they don't benefit from the metrics and such from the API management system. So if we take a step back, we are actually looking at east-west traffic because we are looking at how we can optimize service-to-service communication. So, well, east-west traffic. So let's see now where the users that express these kind of issues. These are the domains that reached out to us and said, yeah, we are having these issues about latencies, about costs. And if you look at them, you can imagine that for the Catholic, there are not some small group of users in the corner of the Catholic, coding in the dark. These are basically some of our biggest internal users. And they represent a large portion of our IT users and also represent a large portion of the Catholic revenue. So we really had to address their issues and their needs. So we have these users. They have, well, business goals, you know, like every one of us, the goals link to improve performances or to solve issue rasters, et cetera, or to reduce and improve costs. And we have this context how to improve service-to-service communication and east-west traffic management. And if you combine this, you can deduce some technical needs on which you can act on to try to solve the business issues for the users. And, well, these technical needs, they are these ones. And multi-cluster, dynamic routing, east-west filtering, and more observability. And if you know a bit the Kubernetes and cognitive ecosystem, you'll think service mesh, right? This must be the answer. Well, service mesh is not the best solution. But what is service mesh anyway? So service mesh is like an additional network layer, an additional communication layer inside your community clusters that sit on top of the other network layer that already exists. So you have your group provider network layer, you have your community sovereign network, and often some networks in between, and then you add yet another network layer. And all this network layer works is that without service mesh, your pod communicate over a network of the community clusters. And on the most common implementation of service mesh, all the traffic goes to a cycle proxy that are deployed inside each pod. So now all the traffic going in and out of the pods go through the proxy, which is a layer seven proxy. And then you can do a lot of things with this layer seven proxy. And you also have the control plane, the control plane pushes configuration to these proxies to basically tell them what they need to do with the traffic they receive. So a layer seven proxy that manage all your traffic, this means a lot of functionality. And this is really one of the main selling points of service mesh is that you can do a lot and you can offload capabilities or functionalities from the code. The most simple and common example is TLS. If you ask your developers to implement TLS encryption between services, it's not an easy feat because they have to connect to PKI, they have to manage certificate rotations, certificate revocation and so on. In service mesh it's quite easy. It's one of the basic feature. It relies on the proxies.  All the traffic inside your mesh is TLS encrypted because all the proxies have a TLS certificate and they use that to encrypt the communication between the proxies. So you can imagine there are, as I said, a lot of functionalities with service mesh. I divided these functionalities into four big categories or domains and what do you think these categories or domains are? You can shout out, read it out. Security, yeah. Sorry, observability, yeah. Matrix, yeah, observability. Discovery, yeah, it's, yeah, yeah. Yeah, good. So, you basically have most of all. First is, well, traffic management. Layer seven proxy watches all the traffic so you can act on the e-eaters, act on the requests, et cetera. Security, of course. I was touching on that with a mutual TLS so each pod knows as an identity so you can do authentication, authorization, et cetera, filtering. Next is observability. Once again, a layer seven proxy so you can watch all the traffic that goes by, that goes through, sorry, and extract number requests, extract latencies, and so on. And the last one I call it topology. It is basically the ability to expand a mesh across communities clusters or across virtual machines or other workloads and this way a service in a cluster can communicate with a service in another cluster, completely transparently, as if they were in the same cluster and the mesh handles all the underlying things. So, next question for you. What do you think the benefits of all these functionalities are when using service mesh? Well, this is the benefits we found during our first assessment so it's basically based on the various topics I was talking about earlier. First is simplify the developer experience by offering some capabilities from the code. Next is better visibility, so observability in scenario traffic. As I said, in a basic communities cluster, networking is quite opaque without any additional tool. Secure communication between services. Of course, the security and TLS, this is service mesh is the way if you want to implement zero trust, for instance. And functionally, it can ease multi-tenancies with all that, so it gives you more tool and more solution to easily locate services in bigger communities clusters. And, well, pragmatically, it's service mesh is one tool that packages a lot of functionalities. So, it's one tool you have to maintain and it offers you a lot of possibilities. Of course, there are drawbacks with service mesh. The main one is its operational complexity. As I said, another network layer, a lot of proxies with each pod and you have to manage that, operate that, maintain, upgrade, et cetera, et cetera. And it can be quite tricky. When it works, it works. And when it doesn't, well, you can't lost. You maybe can't lost. Satcar, a cool feature to have a proxy, but, well, it has its own shortcomings. It consumes resources. It has tenancies and ops in your traffic. The last two ones are more linked to the DeKathlin context. Basically, service meshes advertise like one click observability and dashboards. Well, this often implies having other monitoring tools like Prometheus, Jagger, Zipkey, et cetera. And if you don't want to install them and manage them, then, well, it takes more work to integrate maybe in your existing tooling or existing solutions. And the last one is functional because there's a new redundancy that appears between the API management system, the API gateways, and the service mesh for Narsal Traffic because where do you implement your routing? Where do you implement your filtering? And you can even take into consideration the ingress if you have this additional ingress layer. So, Satcar is a big pain point of service mesh, right? Consumers a lot of resources. It has a lot of drawbacks and shortcomings. And service mesh developers and service mesh solution, they thought about this and they came up with some solution in order to remove this sidecar and ease the operation on the service mesh. And the idea is to replace one proxy with two proxies. So, yeah, simplicity, right? Well, it's simpler anyway because they are not at the same level. And the idea is to have a layer four proxy that will handle all the layer four traffic, basically all the traffic of the mesh and apply layer four policies and an additional optional layer seven proxy that would handle all the layer seven policies inside the mesh. And by using, by being optional, by making the layer seven proxy optional, you can basically have better performances because you don't go through the additional layer seven proxy if your traffic don't need it. There are currently two implementation of this model. The first one is ECOS ambient mesh. So, in the ambient mesh, you have layer four proxies on each host that are called zero trust tunnel or Z tunnel. They're basically established tunnel between the host. So, all the traffic of the service page go through this Z tunnel proxies. And you have a layer seven proxies that are called waypoint proxies that are basically Android proxies. And the thing is that there is one proxy, one waypoint proxies per identity. So, basically per service account. And the idea is that if you have a pod with a service account that wants to talk to another pod with another service account, well, your traffic will go through this waypoint proxy. And if you have additional layer seven policies, it will also go through this proxy. The other implementation is Cilium service mesh. So, once again, you have tunnel between the hosts. These are IPsec or wire guard. And the layer four policies are handled by the Cilium CNI and EBPF. And for the layer seven policies, you have an Android proxy on each host. The main difference between Cilium and ambient mesh is that the layer seven proxy in Cilium is multi-tunned because there is only one per host. So, it handles all the identities of the tenants. I also have to mention that this implementation, I quite knew, ambient mesh is still experimental. It's going to beta soon. And I think Cilium service mesh is stable, but well, it's still quite new. But if you're looking at service mesh in a couple of months or maybe a year from now, you should have a look at that. It will not be your choice, but well, more choice is good, right? All right, let's go back to our study at the Kaplan and the conclusions. So, the goal of the study was really to assess if service mesh was the solution to our user needs. It was not to choose the best service mesh solution or choose the best implementation of service mesh. It was really to see if service mesh was good enough for us to use it every day and deploy it at scale. And we conducted the studies quite simply, actually. So, we took the needs from our users. We translated them into scenarios or user stories, whatever you call them. Some were required because they answered the needs of our customers and some were optional because a lot of functionalities were service mesh. So, opportunistically, take some advantage of these additional capabilities. This is an example of one of these scenarios. This is basically the scenario that says that we want service mesh to be able to be expanded between Kubernetes clusters to handle service-to-service communication between clusters for fellow over, et cetera, et cetera. We also took a hard look at the service mesh ecosystem. So, the idea was to have the broadest overview of the service mesh market. So, we looked at open source solutions. We looked at commercial solutions. We looked at various implementations of service mesh, the Serium one. And the idea was to see oh, each implementation or each kind of solution would react, would be used with these various scenarios. So, we took that. We went back to our users that expressed their needs and their issues. And we told them, okay, we may have a solution to what you're facing, but we need you to test it. And so, the idea was to test it inside some of our users' environment, development environment, or environment built on purpose, but with our users' applications to see all the solutions would actually answer and operate them. So, we took all the tests. We aggregated the data. We reviewed it. And we asked ourselves, well, is service mesh the solution? Do we need to put it everywhere? And before answering that question, I want to go back to our old little friend, which is the API Gateway and API Management System, because when our users expressed these needs, our API Management Team, they didn't sit back and relax. They also heard them and they got to work. And they came up with this. And this is, let's say, a new design pattern of our API Management System or for API Gateways. And the idea is to deploy an API Gateway as close as possible to the user's workloads, directly inside the Kubernetes clusters. And what's cool is that the users would use the same control plane, so same interface and APIs that they were using to configure the centralized data and centralized gateways. And this way, this would alleviate the cost and the latencies that were occurred by going out to the centralized big gateways. So basically now we had two solutions, which is pretty cool because sometimes you have none. And well, it's not as easy. So if we take back the needs I was talking about earlier, can service mesh do multi-cluster? Yes, of course. It's one of the nice feature of service mesh. Can API Gateway, can the micro-gateway, and the nickname for this design pattern is still micro-gateway because, well, you take a gateway with a smaller configuration and only the configuration that is relevant to the context where it is deployed. So can the micro-gateway do multi-cluster? Yes, ish, because the idea here is that when a service requests another service, it just requests its internal, let's say, or neighbor micro-gateway. And the micro-gateway and those contacting the service may be external or inside the same clusters. So all the management of reaching and authenticating to the services is handled by the micro-gateway. So yes, ish. Can service mesh do dynamic routing? Of course, layer seven proxies, et cetera. Can the API gateway do dynamic routing? Of course, observability. Once again, both do it well. They use proxies, layer seven proxies, after all. And last one, east-west filtering. Of course, once again, both do it. So we basically have two solutions that Ansari needs. One is one new cool solution, one new cool tool, which is service mesh that will have to deploy everywhere on the cathode, and one is the micro-gateway, which is a nice new design pattern of a tool that we already know and use. So that's basically why service mesh is not the best solution. Actually, the real answer is service mesh was not the best solution for the cathode, because we had another solution that was just taking something that already existed and adapting it to new needs and new usage. And even if service mesh was not the best solution for the cathode, it may be a good solution for you, because basically you have other needs, you have other architectures in place, you have another kind of infrastructure. So you have to look into service mesh if you have needs around dynamic routing, zero thrust, et cetera, et cetera, because it may be a good solution. But always go back to your user needs, that's the main point of the talk, is to always go back to their user needs and not implement a complex solution just for the sake of it, and because it looks cool, because in the end, you will have to maintain it, operate it, et cetera, et cetera. And that concludes my talk. Thank you. I guess we have time for questions. Oh, hi. Well, first, congratulations for the talk, it's bold. I mean, I was not expecting the final result, but usually when you think about putting something like service mesh, it's because you are also sometimes foreseen something that you may use later. And in our case, for example, we use because we wanted to build almost like a platform on top of service mesh and so on. So what kind of consideration in the future of the Catalan needs and so on took place in your decision in not to use?  We took a look at what we could do with service mesh in the future. And that's what we had some optional scenarios that we want to test service mesh against. And in the end, we, well, we didn't dim the service mesh necessary even for those because they were not, you know, we wouldn't have to take this into account right now. So it would be maybe a few months from now, maybe a year from now. And well, in a year from now, lots can happen and a lot will evolve in our ecosystem. So as there were no big, big priority subject to end up or tackle with service mesh, we said, well, let's for now just say use the micro gateway and use service mesh for other needs. Well, marginal needs. So we didn't forbid the use of service mesh. We said, yeah, first look at the micro gateway pattern because it would answer 80, 90% of your needs. And then if you want to implement zero trust because you want to be ahead, well, use a service mesh solution and we will use only a, we'll select a service mesh solution for on the cathlon and we'll go from there. And the idea is to revisit the needs in a one year, two year from now and see if this decision still holds. Hi, thanks for the talk. So you decided to choose basically to expand on your existing solution which was the API gateway, but make it in the smaller units and have it aggregate them at high level. How, so how far were you from still choosing service mesh and like in what world, what in what event would you not have chosen to use your existing solution? What drove you to actually do that and was this choice difficult? The way it went is that we basically did the study of about service mesh and our API management team and they came in at some point and said, hey, look, we have this new cool pattern and we should test it. And basically our users went for it because well, it was easier for them to implement this new pattern because it was a tool they use every day and they have a lot of experience with it and it was in the end an easy decision to make because for service mesh, they would have to learn a new tool which is a complex one. We would have to then implement it at scale because well, we use one solution and we would deploy it, we would standardize it, automate it and it would take a lot of work for their teams and for our teams to implement it. And that's why, as I said, we went for the micro gateway pattern and said, go for that for one year, two years and maybe revisit the decisions, see if users have new needs or if the pattern doesn't answer all their needs and then we'll maybe look at service mesh again and maybe implement it. Yeah, so basically it's because user adoption was natural that went for it. Yeah, all right, exactly. Thank you. How did management react when you told them you spent all this time and money on service mesh research to go back to the... I have my manager here, so I cannot answer that. No, I have to answer those kind of questions. No, no. Well, basically the idea was it's two-fold. First is we did this study, we did it well, so the decision, we wouldn't go back to the decision in two months from now because one user would say, hey, look, there's new service mesh school and no, it's a decision for one year or two years maybe more. So it was good for them because there is a decision and we wouldn't come back to it and the other is, well, it's just the user needs are what's most important. So we took a couple of months to do a certain service mesh, it doesn't fit the needs. Well, now we know more about service mesh and well, maybe we'll go back to it but the most important things is looking to your user needs. Hi, I had a feeling that cross zone traffic has never been a challenge for you because I thought you are not paying for that but saying goodbye to protocol, layer seven protocol and using layer four, did it help you to reduce the traffic overhead and do you have a figure in how much did you save the traffic but not using layer seven protocol? Well, we are using layer seven protocol actually because we are using the micro gateway, API gateway, so it's a layer seven. Okay. So there are actually, the savings are more on, you know, aggressive traffic and inter-regional traffic and it covers all the traffic and the idea was to, a lot of services are collocated on clusters and even more will be. So we'll keep the traffic local to the cluster and so we'll have on the bandwidth on the code cost of going back and forth between the centralized API gateway that are in other GCP or AWS projects or other regions and so on. Thank you everyone. I got one question, sorry. In your test, what's the point you're looking to site KLS model? I'm in a model where we're using service mesh and we need layer seven for high use case. Do you see any benefits in going to site KLS model while still using all the waypoints in issue ambiance or was it pointless in this case? Well, I talked about the site KLS approach because I wanted to look a bit into the future. We didn't really take into consideration site KLS solution into the study because we thought there were not mature enough. The advantage I see anyway is that you to remove the site car container and the site car container, which is a proxy for service mesh, it's quite hassle because you may have some ordering problems and always in it containers and so on consumes a lot of resources. So even if in a site KLS approach, if you don't, if you always use layer seven policies and they always go through the layer seven proxies, you can save on the operational costs anyway and on the cost of, let's say, the communities costors because there's less resource consumptions and it's simpler to maybe debug because you have one proxy per host or one proxy per namespace or service accounts. So it's a simpler approach even if you don't maybe gain much on the latencies and so on. Thanks everyone.


North Korea not only famous for chocolate, but for being a surveillance state and As a good surveillance state as has to his own operation system Because how will you do proper surveillance without your own operation system? Today we get a brief introduction How Red Star OS is working? The introduction will have a specific focus on the custom code which was inserted for Surveillance and especially how to get around it So please welcome Florian and Nick Nicklaus with a big round of applause Hey everybody. Thanks for having us We are going to give you a deep dive into Red Star OS It actually we were kind of surprised that there is not so much information on the net about Really the core of Red Star and what is it is it doing? so we thought we would change this and Give you an insight in how this operating system works and by looking into the technical aspects of Red Star You can also draw conclusions about how development in North Korea is evolving and Is maybe catching up so what we're going to talk about is First of all a short introduction into the motivation. Why are we doing this? We are going through the architecture of Red Star We are going to show you the components in the core in the operating system itself And then we will take a deep dive into the additional components all of the programs that are coming from North Korea And are supplied with the ISO of Red Star OS After that we are going to give you a deep dive into the most interesting features of Red Star OS and then we will be able to draw our own conclusions and Afterwards, we will have time for questions. We hope by the way this picture on the left You can see here is actually one of the I think it's a screensaver right from Red Star OS so Yeah Okay, so before we begin we need to do this disclaimer For your information, we have never visited DPRK. We have never been to North Korea All we know about North Korea is from public sources from the Internet's from media Whatever so what we are going to say about North Korea is Has to be speculation because we don't know exactly what happens in North Korea also the Isos that we have been analyzing are found publicly available on the Internet Maybe fake we don't think that they are fake because will Scott has shown last year on the 31c3 How Red Star looks and everything that he has been showing is basically in the ISO so we think it is legit Remember that we are not going to make fun of anybody in this talk We're not going to make fun of the developers and we are certainly not going to make fun of the people in DPRK because we think that Our presentation might have some funny aspects or something that makes you laugh, which is perfectly fine but Looking at Red Star in detail is kind of like it's it's a surveillance mess I would say and it's a security or a privacy nightmare and So the keep these aspects in mind Also, this talk is not going to focus about security We're not going to talk about security many of the publications that are available on the Internet Are about security and we are not going to focus on this in this presentation. So why are we doing this? Red Star ISOs have been leaked some time ago There is a version 2 hanging around the Internet's and there is obviously a version 3.0 Which has been leaked at the end of 2014 And we were quite surprised at the mid of the year that there is no in-depth analysis of this operating system So most of the blogs and news articles are quite superficial that you can find out there and this is kind of surprising because if there is some kind of State that doesn't put focus on transparency and free speech and they are putting out an operating system Actually, you kind of want to know how do they build their operating system? So that was one one of the major aspects for us to look into it The other aspect was on to find out how is the state of software development in DPRK? How are they developing software? Is it like? Do they have like a well-thought? Architecture, are they thinking about what they are doing? So how is the level of or the skill level of a software development in North Korea? So do these were the two Aspects that we wanted to find out so if you look at previous work as I said There is mostly superficial stuff. There is some information that Red Star S actually looks like Mac OS X We will go into this a little bit further Then we have this talk from Will Scott last year at 31c3 who was talking about computer science in DPRK Which was very very interesting and gave a pretty good insight into what's happening in DPRK And then we have a few guys a bunch of guys that looked into the browser of Red Star, which is also quite interesting. So What we are going to do now is I'm going to show you the custom basic components I'm going to talk a little bit about integrity on the system Then I will hand over to Niklaus who will be like Looking into the the core and the surveillance features and then as I said, we will have time for questions afterwards So there are different leaked versions out there as I said, we have a desktop and a server version of Red Star So you can also use Red Star as a server and it turns out that Server version 3 is actually used on the internet right now as you can see there is a server header returned Red Star 3.0 This is an IP address of the server and it is pointing into North Korea So this is one of the few websites that is publicly facing the internet from North Korea And they're obviously using the server version in version 3.0 So 3.0 might even be the latest version. There is another version 2.0 which is also has also been leaked to the internet and then there is supposedly something that looks like 2.5 we have found some South Korean documents that seem to be analyzing the system quite superficially and It looks like 2.5 actually resembles the look and feel of Windows XP So you kind of see this evolution right now from 2.5 XP going to 3.0 mimicking Mac OS X Our talk will focus on the desktop version, which is desktop 3.0 If you look at the timeline, which is a guess There's no documentation available on how they did it obviously But if you look at the the 3.0 version you see that it is Based on a Fedora 11 which came out in 2009 So our guess is that they started developing 3.0 in 2009 with this Fedora 11 release The kernel that they're using is 2638 which came out with Fedora 15 in 2011 so could be that the operating itself is a little bit older the kernel is a little bit newer and The latest package build dates that you can see in Red Star as date to June 2013 so our guess educated guess is that Red Star came out in June 2013 or a little bit later a few weeks later or months later in December 2014 We had the public leak so the ISOs have been leaked to the internet and are publicly Available right now If you look into the operating system, it's basically a fully featured general desktop system You might imagine it's based on KDE and Fedora as I already said and it tries to mimic the look and feel of Mac OS X You have an email client a calendar a word processor You've got quick time And all of that stuff you even have a disk encryption utility that the way Scott has shown last year They implemented additional kernel modules and they touched a lot of kernel modules, so They have this kernel module RT scan which Nicklaus is going to say a little bit more They have an kernel module which is called Pilseung I was told told that this means victory in Korean and that kind of is a kernel module that supplies AES encryption so they implemented an own kernel module to supply something like AES Then there is the kernel module called KDM, which is the Korean display module and KIMM Which is not what it's like it's not looking well, I'll just go on which which basically just does something with Korean letters and Displaying Korean letters on the screen Developed is a word so has been delivered by the KCC the Korean Computer Center It's quite interesting that they like since I think that a few years ago. They had an office in Berlin Like I don't know what they did there, but they obviously had a had an office in Berlin maybe for knowledge sharing whatever and If you look at the system hardening, it's quite interesting that they took care of system hardening So they implemented say linux rules with custom modules They have IP tables rolled out immediately so you don't have to activate it or put your rules into it firewall is working They even have snort installed on the system It's not running by default, but they are kind of delivering it within default And they have a lot of custom services that we're going to look into right now Quite interesting is okay, so why should North Korea mimic Mac OS X? That might be one reason right there because this young fellow sitting on the left is actually using an iMac right here So this is one reason So why should they implement their own operating system? They actually are so-called anthologies that are put out by the leader and one of anthology by Kim Yong Il says that If you translate it correctly and we try to in the process of programming it is important to develop one in our own style and with one he means basically programs and operating systems so there is this clear guidance that North Korea obviously should not rely on third-party Western operating system and programs they should develop this stuff on their own and by looking at the code and everything that we have Star by Red Star is this is exactly what they did they touched everything nearly everything on the operating system changed it a little bit added custom code and so this is actually what they are doing right there The custom applications that you have is a browser which is translates to my country you also have a Crypto tool that will Scott has shown last year, which is called Bokem Which is if you translate it kind of translates to sword You have so one office, which is an open office customized for Korean North Korean use a software manager You have music score, which is an application that you can compose music with Then you have a program which is called root setting And it basically gives you root. So if you look into the documentation It says you're not supposed to have root on the system for integrity reasons But if you want to get root you can use this tool. So this is like they're not hiding anything So there are rumors on the net that say that you're not supposed to get root on the system because it's so locked down This is not true obviously because there is intended software intended to give you administrative privileges They even kept touched KDM And so the the code base that they touched is is really really big nearly the whole operating system We are now going to give you a demo actually the first demo that we're doing is we're doing it right now because We are actually doing this presentation in Red Star OS So what you see right here is basically Red Star OS We're going to show some of the aspects to you there are many many screenshots on the internet Some of you might already know how Red Star works or might have experienced yourself. We're just going over a few interesting issues so as you have seen there is like a full-blown set of Word processing PowerPoint presentations and stuff. I'm going to open up the browser whatever And going into the preferences just to give you a quick Yeah, a quick no Yeah, so give you an insight on the Certificate authorities that are implemented in this Firefox version. It's a Firefox 3 So you see there is not so many certificate authorities right here and they all Are basically well, I guess from North Korea So the browser is a totally created to not be used outside of North Korea, which you can see you can see like in the In the in the URL bar that there is an internal IP address Which they which goes which points into the network of North Korea and all of the settings proxy settings hard-coded IP addresses or whatever all point into this Internal infrastructure of North Korea. So this browser and the email program was never intended to be used outside of North Korea Okay, what else do we have Okay, we have a quick time player So actually speaking about mimicking Mac OS X you all have seen like this whoo swoosh, right? Okay, so that perfectly mimics Mac OS X How many sound on is on yeah, okay Okay, so let me try to find I'll try it with our play right here. So this is a shell quite interesting is that When we were looking through all of this stuff there are a bunch of files that Have a certain protection and they seem to be pretty important for the system and then there is a wave file An audio wave file that actually is protected Lip warning dot waf I don't know if we can hear this I hope that your ears are not going to explode right now.  Does anybody know? That's stolen from Kaspersky antivirus because in the older version of Kaspersky antivirus if you find a virus It actually will play this sound and it's exactly the way file from Kaspersky We verified this by doing checksums. Okay, so We have a copyright violation right here So what else do we have? I've been talking about this you can create your own music But I'm not going to do this now because I'm not good at making music What else do we have we have the browser did we want to show? Ah, yeah, I'm going to show you one more thing I'm not going to show you the encryption tool because will Scott has done this last year, but to give you an insight Into the crypto tool is pretty interesting if you look at the description of the Bocam 3 is Bocam is the tool that is used for disk encryption So it provides the user a tool to encrypt files or even the complete hard drive and if you look into the description It says this allows the user to store his her privacy data with encrypted which is quite nice Right. I mean like we didn't expect like to have something like this in In red star so the user can actually try at least try it to encrypt files Bocam is using out of the box crypto that comes with the camera. It also uses Pilsang Which we don't know if there are any backdoors in it or not So we have no idea if this is possible to kind of Decrypt with a master key or something if you want to look into this We would be happy if somebody with with big crypto experience would look into it Okay, so Let me get back to the presentation Ah, one thing I need to show you is this red flag on the right corner right here if you look into this and If you are going to translate I didn't click the right one So if you are going to translate all of this You will find out that all of the strings and all of the text that you see right here They kind of come or seem to be an antivirus scanner Okay, so they even implemented from scratch an antivirus scanner in red star os You can now select a folder or a file and say run a check on the file and if The file actually is a malicious file. We will come to that part later. What malicious is It will instantly be deleted from the hard drive. Okay, so this is an interesting feature having an And virus scanner in a Linux OS Okay, so let's look at the custom components We have been looking into like the user space a little bit and all of the programs that come with it There is far more stuff Download the ISO play around with it a little bit first change the language to English you will Obviously not get far if you're Korean is bad So Change the language and then just play around with it a little bit. So red star comes with interesting packages They touched KDE as I said they are getting out an integrity checker and a security demon There are signature packages right here, which Niklaus is going to talk about a little bit There are policies for say linux and I'm going to talk about Two of the integrity checking mechanisms that red star has so by looking at red star We saw that one thing was pretty important to them they wanted to preserve the integrity of the system and One way to do this is using this Process right here. It's called in check It comes with an SQLite database with hashes of files on the system like signatures for the system And you can configure it from user space. So it's not pretty hidden. It's not it's pretty transparent to the user I think there even comes a UI with it where you can like configure everything and it's run at boot And it checks the files and if it sees that the files I've been manipulated or tempered with if the checksum changes then it will issue a warning to the user So you get a small pop-up that says this file has been tempered with the security or the integrity of the system Is not where it should be So that's pretty much it what this thing does Security D is kind of interesting because security D is also process that is known to run on the Mac OS X And I think that I'm not a Mac user And I think that Mac OS X with security D is kind of keeping track of certificates and stuff like that So what they did is they re-implemented security D for Linux and they included various plug-ins and one Interesting issue with security D is that it comes with a library that provides a function called Validate OS and what this function does is it has a hard-coded list of files You can see like our way file right here. You can see configuration files and Autostart files for SCN PRC is the antivirus scanner So it checks of the if these files are untouched and if these files have been tempered with it Initiates a reboot instantly. So if you touch one of these files, your machine will reboot instantly The same library is also used From KDM so during the startup process when KDM is starting it is also doing an integrity check and if it finds that one of these file has been tempered with it actually Immediately issues a reboot and the problem is that if you start tempering with the system You will end up in reboot loops all of the time if you're doing your research because Once KDM is saying reboot the system It's going to check it again if it's rebooted and sees like it's still tempered with and it's rebooting again and again again And then your system is basically dead So what they try to do with in check and security D is kind of protect certain files Conserve the integrity of these files and if these files get tempered with they Assume that it is better to have an operating system that you cannot work with anymore Than to let still let it run or issue a warning. So Integrity is one of the main aspects They were looking for by implementing Red Star Okay, I will hand over to Niklaus and he will go into the guts and the surveillance features a little bit more Package we found was this easy package easy B package Which actually says in the description that it's an electronic signature systems But but we found that is doing a lot of weird stuff And this is actually one of the pictures which is included in the package, which is also protected We don't know really why but it says something like This is our our copyright and don't break it and don't copy it and stuff like that But it's actually doing something really different It includes several pretty interesting files. We have some configuration files. We have a kernel module We have also this red flag BMP which is the picture you just saw and we have the warning file and some Shared libraries and we are going now into detail. What's these are actually doing? So the first thing we looked at was because there is a kernel module which is loaded by default and we thought If you want to put some backdoors in it where we want to put it right in there in the kernel module probably And what it does it's actually just hooking several system calls Which provides it provides a Device which is actually an interface to the kernel. So you have different services running on the system Who are actually talking to this kernel module via this device and it has some functionality like it can protect PIDs So when you're protecting a specific process, then even root cannot kill this process Which will be quite interesting in the next slides and It also provides functionality to on one side protect files and on the other side to hide files So protect mean you cannot edit the file and hide mean you can even can't even read the file So even if you are the root user, then you can't even read those files and on the right side you see actually how the Services are interacting with this kernel module and this is one function where it's mostly mostly Protects and hides the files which we just saw which I included in this e-signature package Then like Rowan said we have this virus scanner, which at first glance at least looks like a virus scanner And this is this SCN PRC process It provides a GUI to the user so it's quite transparent So the user can see okay have something that looks like a virus scanner And I can also trigger some scans of different directories and it started by KDE in it So there's this SCN PRC desktop file, which is quite interesting because what you want to do is like disable it But you actually cannot edit this file So as soon as you edit this file and save it that the system will immediately reboot So disabling it is not so easy Like I always say already said you have different rays of scanning like you can just click on a folder and say scan this But also if you for example plug in a USB stick to the system Then it automatically will scan the files on the USB stick and this SCN PRC Service is actually loading the colon module and it starts another service which is called op PRC Which we are going to look in detail in a minute and This is quite interesting also the next service, but the petamatching We looked into this a little bit. There's another package So we have this E6CB package and you have E6CBDB package, which actually just provides this one single Angi file and this means as far as we know it means fuck in Korean And this is basically a signature file or how the code references it a pattern file It's a file with a well-defined file format and it includes several patterns Which are loaded into memory and as soon as you're scanning a file It just checks if these patterns are matching and as soon as the patterns are matched then it just immediately deletes the file and it plays the warning and This is one of the hidden files So even if you are getting root privilege on the system, you are not able to actually read this file So a user of the operating system won't be able to check Okay, what does it check and can I produce like documents? Which are which won't be detected by this because you are actually just cannot read this file and we took a look into this our Most likely our best guess is that these contains a lot of their files are a little Indian so you always have to switch the bytes and We see we saw that it looks like at least that they are utf-16 strings with Korean Chinese and some other weird characters and if we put this in Google translate then there are actually some pretty weird and disturbing Terms in those files, but we actually cannot confirm this it looks like they are actually not scanning for malware in the system So they are most likely they are checking documents and if those documents match those patterns Then they are most likely for example governments I want these files to be distributed within the internet of North Korea Then it just deletes those files, but actually we cannot we cannot confirm this because we are not quite sure if you put those strings in Google Translate that they are actually real translations But you can always update these pattern files So on the one side is SCMP or she has a built-in update process where it just updates the file itself Or you can just when you're doing operating system update by your pocket package manager and you're updating this E6 ebdb package and you also get a brand new file and The interesting part of this is that actually the developers decide what is malicious. So it's not necessarily like malicious Means that it's malware that it's bad for you, but Somewhere the developers and Officials will actually say okay. We don't want those files distributed just delete them because we think they are malicious There's this other service which I was always also talking about this op PRC This is like even more interesting than the virus scanning itself. It's running in the background So actually a user will not see that there's actually another service running You don't have any GUI or something like that You cannot trigger something with this and this is one of the protected PIDs So as the NPRC for example, you can just kill with root privileges but this is a process no one can kill on the system and This is quite interesting because you cannot unload the kernel module unless this this service will be killed So they're actually protecting each other so that no one can stop the services at all and this service Shares a lot of code with the SCN PRC. We just did some entropy checking and so okay I will talk in a minute when we are Comparing more of these files why we think that this looks like pretty much the same why they are sharing so much code Because we found something interesting with older versions of those services But the most interesting thing this service is doing is it watermarks files And now we are going to look deeper into what this watermarking means So actually as soon as this system will be started It reads your hard disk serial and then scrambles it a little bit and as soon as you are Plugging in for example usb stick in your system. Then it will trigger A watermarking process where it takes the serial Takes an hard code a desk key from the binary itself and then encrypts it and puts it into your file When you're decrypting when you're converting this hex key into a decimal representation Then you see that it's actually two dates We actually cannot confirm what those dates mean, but one of those matches Madonna's day birthday date And there are rumors that some people in north korea might really like Madonna, but this is like Just speculations, but if you have a better conspiracy if you read and just let us know Because we found some some pretty interesting stuff, but we actually cannot confirm this So technically the watermarks they have an s key e o f appended Which is most likely used by the code itself to pass the files and see if there's already a watermark in there And for jpeg and rv files, for example It just depends this watermark to the end of the file And when you have a dog x for example, then it depends it like near the header Where a bunch of null bytes and then it just puts it in there So the watermarking itself is like as soon as you open a document file with the office, then it will be watermarked And actually they have code which which watermarks files, even if you don't open those files, but it's As soon as we saw this it's like pretty buggy. It doesn't work every time But they have code for this implemented and mostly it works, but sometimes it just it just Fails and the supported types that we can confirm are dog x files Image files like jpeg and png and rv video files But the code indicates that there are several more file types available for watermarking, but we most likely didn't look into this, but the the most interesting Thing here is that all only media files are affected. So they don't watermark any binaries or something like that. That's just They're reducing their their surface to files which could be used to carry information, which is actually which could be used to to Put information which for your free speech purposes And actually what we think is that this is not a security feature So they're actually trying to watermark free speech in general So that every time you might have a document file an image or a video file Then they want to know who had this file and they watermark it so they can track the origin of the file We have a short demo Where you can see like for example, I have a usb stick I'll put it in my system And there is a file on the usb stick Which is a love letter from kim and it has a checksum which starts with five two nine And as soon as I plug this into the system, I hope that it will notice this Stay you can see okay It it recognized something like a usb stick on a system, but I won't open it and I won't open any file on the on the usb stick I just will eject it back there I hope that it worked Actually opening this is what I meant is that it's kind of buggy So it doesn't always work with the with the watermarking But most likely if you open the file itself, then it will work It will I guess we didn't have the case that it doesn't work when you open it Um, which an open stuff is and we close it again Just close this Back and then hopefully if we mount this again Then you can see it has been changed So, um, we didn't change anything in the file. Um, it was just the operating system who's changing files And this was initially the part where we, um Started to look into this more deeply because we thought the operating system who's just changing files on the which Which you're plugging into the system, um, is is kind of annoying. So Um Just to make this easier for you. Um, so what it actually does in the file We have here now the header of the file, which is a document the doc x file And it just uh added this string which is which is marked right here And this is actually the watermark. It's putting in there Up here there you can see the the plain text which is actually encrypted and then put into the file And the serial starts with b 4 8 So every time it puts the serial into the file it, uh, it prefixes it with w m Which we think is stands for watermark probably and you can see the e o f at the at the end of the file And this allows basically everyone who can access this file who can decrypt this This watermark, which is actually encoded with the hard coded key So pretty much everything everyone who has access to this iso get this can get this key and can decrypt this And this this allows it to really, um, track back to origin of the file where it came from Um, but there's a pretty pretty pretty funny example.  So you think, okay They might intercepting all of my email and my browser communication So you put it on a usb stick and give it to your friends so that you think, okay No one actually on the internet or on the internet, uh, Can can access this file and you give it to someone else Then at at the beginning you have this the situation where you have the original file This is the end of the jpeg file, uh, which by the definition always ends with an ffd9 hexadecimal And as soon as you give this to your friend and he he plugs the usb stick into his computer Which is running a red star os Then the file will actually change and it would look like this So uh for jpeg files, as I said, it just depends the watermark to the end of the file So you can see the ffd9 This is the actual end of the of the image file and they're appending the watermark there like you can see with the eof But where it gets interesting is when your friend is actually distributing the file to another friend Um, so what red star os is actually doing, uh, it depends also the watermark of your third friend So what you then can basically do if you, uh, technically combine this together Then you can not only see okay where the file has its origins But you can also track each and everyone who had this file and who distributed this file And with this knowledge you might be able, uh, to construct something like this Where you can actually track the distribution of all of the media files Which are distributed over the internet in north korea and you can see then in the center Okay, we have this one really weird guy who is always distributing images, which we don't like And you can see also who gets this files and trace it back to all of the persons who ever had this file And then you can just go home to him and then shut him down and take his computer and we have actually not seen any functionality but um Probably there is a functionality in the system implemented where it always sends your heart disk serial to their service So they always can probably always Be able to match your ip addresses to your heart disk serial And then they don't even have to go to your home and get your computer and check the heart disk serial Um, they just can it can do this remotely and can track all of the distribution of all malicious media files within the Within the internet of north korea Um What we what we thought is pretty pretty hard for someone who don't has access to a system other than red star s Who just has this one system and tries to disable all of this malicious functionality like the virus scanning who can't delete all of your files Someone else doesn't like or the the water marking the tracking of those files And this is actually quite quite hard because some of those services are depending on each other and can always be killed Only be killed when the other service is not running So what you actually have to do is you get you have to get root privileges Then you have to kill those two integrity checking demons Which florian was talking about so that it doesn't always reboot the system when you're changing anything Then you can talk via ioc tr chords to the kernel module and say just disable because it has this nice feature It can enable and disable it and then you can kill sc nprc op prc and the best The best thing you can do is just uh The weirdly the the lib os file is not protected by anyone So you can just exchange this with a validator s function, which always returns one which says okay everything is fine And then at the end you can delete the desktop file Which is used by kde in it to start all of these processes And then you are fine and we don't think that actually anyone in north korea who has only access to this one system Uh, it will be extremely hard to figure all of this out and completely disable it so they did a pretty good job in In building an architecture, which is quite self-protecting and They they they put a lot of effort in it to just prevent you from Disable all of the all of the malicious functionality um, we always we also took um a quick quick look on the Second version of red star s just to compare some of those services And there we can see there is actually quite an evolution from the older version to the actual the the current version and The the thing which i was talking about that the binaries are looking quite similar is that in the older version They used a lot of shared libraries And in the in the current version they they're statically linked a lot of code into the library into the binaries itself Even if they don't use it so um the code base looks looks quite the same And the the the chain of starting the processes is a little bit different So they put a lot of uh in the in its um Process which will be started at first and not like this this depending on each other infrastructure Which they have in the current version and they on the current version also they have a lot of problems with file privileges So a privilege escalations would be pretty easy even if you don't have this root setting file um, but also they have a lot of binaries who are just um Just setting like everyone can read and write this interface to the kernel module Which basically allows you even as a non-root user to disable the kernel module And then you can kill all of the binaries, but you actually cannot delete something because then it will end up in the real In the reboot loop and they actually when you are doing something malicious Then it always reboots in the old in the older version.  So um, we thought this is a pretty pretty interesting thing and there's a We we think and we saw that there's a more advanced Watermarking technique in there, which is not just appending watermarks into the files Um, but it looks like they are doing actually for video and audio files At least something like they're putting the watermarks as filters on the files So this will be a little bit harder. Um, I guess to to actually see those watermarks and read those watermarks Because it's not so obvious like when you have this uf string at the end, which is always quite weird But it uses a file this user lip user lip organ file, which is actually not present on the ISO we had And we have we're going to talk about this in the conclusion why we think this might not be there But it's actually not available. It's it's referenced a lot in the code, but we actually hadn't this hadn't had this file and we Unfortunately, we couldn't look into this more deeply Um, so what we didn't found actually were quite obvious backdoors, which we thought Would be in place and there would be pretty easy to spot Um, but we didn't see any of those. Um It doesn't mean that there are no backdoors, but we have some some speculations for this And one of the diseases that like we saw at the beginning of the talk that there are actually systems on the internet running This version of red star s so it would would be pretty weird if they would backdoor system And then put it on the internet and as far as someone gets gets actually gets the iso filing can Can look for backdoors and can find some of them and they would be actually able to exploit the system from the internet Actually the system um has a package manager and as we saw with the pattern file It has built-in update functionalities in different services. So backdoors could be just loaded via updates because um, probably they thought okay, these easels might be leaked into the outside world and You just get the iso install it update your system, which the updates are only a Possible from within the internet of north korea. So this is all with hard coded internal ip addresses So probably they thought okay only we want only our backdoors on the system, which are actually located within north korea and yeah This is like what we thought is the best guess that they were so um were Really thought that the iso might be leaked which actually happened and Another problem is that like floriano already said is that they will touched a lot of code in the operating system And we we did manage to check all of the code We mostly focused on the water marking and the bioscading stuff And there might be a lot of code that should be checked further Um, and the conclusion also is that the system is quite self-protecting. It's um, they not only implemented Several services for integrity checking themselves But also they they configured and implemented like se linux and something like that To just protect the system to make it more secure Um What is really bad what we think is this virus scanning and water marking because it actually allows you to um track not only the origin but a complete distribution within the network of those files And combined with the virus scanner where the developers are able to actually um say okay, what files are really malicious and What shouldn't be distributed within our network then it just deletes those files So these two combined are really a really really strong Framework which can help you to track malicious people within your network But two some words about security. Um, like I said, they have a lot of problems with file permissions They're actually having some documents on the server version of the of the iso In the eyes of the server version of redstone s 3.0 And there are some user guide and administration guides which are quite interesting and they are talking a lot about How to make the system secure how to run it secure and Why why they're doing different kinds of stuff just to save the integrity of the system and That are always that they have a huge chapter talking about file permissions But they actually didn't manage to fix them themselves. Which is quite weird and even the custom code uses a basic memory corruption protections like stack cookies and non executable stacks, which We saw that a lot of security vendors doesn't bother to use. So we thought this is quite funny. So Some of their code is more secure than a lot of security appliances So to wrap this up am I going can you hear me? Yes. Okay. So to wrap this up Again, we think this is a guess that primarily they try to protect and to Save the integrity of the system Which totally makes sense if you're putting out an operating system from north korea The system was in our opinion definitely built for home computers. So it's not like industrial control or Something else. It's definitely built for a home user because it mimics mac osx and gives you all of the tools Maybe the reason why we didn't find backdoors is they actually know that backdoors are bullshit Could be a reason we don't know If you want to look into red star s and help us out, especially with the crypto the pilsang kernel module which provides custom crypto With a look into the kernel to see if there is something hidden there if maybe there are backdoors there Take a look at our github and please contribute if you find something because we think that this message and all of this stuff Has to be put out to the public. So it is a well known fact that this operating system is Actually abusing free software to Actually make free speech harder in a country that is quite oppressed So with this we are at our end and we are going to take your questions now Thank you very much. We have about 15 minutes time for questions If you want to ask a questions, please come to the microphones. There's some on the right and some in the left isle If you for any reason can't come to the microphones, please raise your hand and I'll come to you with my microphone Okay, please line up there if if you want to leave Now, please do this quietly through the front door Could you raise your hand if you have questions on standing at the microphone? There are like three questions over there on the left one, please Hello, hello. Yeah So thank you very much. It was very interesting. Um, I have two questions Have you tried isolated the isolating the system in a ch fruit jail? and the second one is Were there any outbound connections like automatically outbound connections it made Okay, so for the first batch, we did not try to run it in an isolated environment We actually didn't did we install it on a live system? I don't think so, right? Did we yeah, okay, but we didn't do any observations that this changed the behavior of the system Concerning the second question. There actually is not really outbound traffic What it is doing is on the local network. It is talking a lot of net buyer stuff so there is an snmb and nmb demon running on the system and it's talking a lot of net buyers But this is basically everything we could find we have even we left it running for like two days to see if there is Like an outbound connection for one day or something like that. We couldn't see anything like that So the only stuff That that red star is talking to the to the to a network is like this windows net buyer stuff And it is if you push the button on the update feature of the virus scanner It's actually trying to initiate an update process that goes to five hard coded ip addresses that are all local So like 192 1689 Something and 172 whatever So these are the only network connections that we could trigger or that we have observed so far Thank you Okay, the next questions also from this phone from this microphone Uh two questions, uh might it be possible that it like when you install the system It gets caught from north korea. So you cannot find out if it's calling home because you don't get the Could be could be our guess is actually that um, there is far more stuff that you get when you Pull up the the operating system in north korea. So One reason is the organ file that Niklaus mentioned that is missing on the system with the additional crypto information that is used for the extended watermarking that they are applying We don't know where this file is coming from and from our perspective It totally makes sense to not distribute this file on the iso But to kind of give it as an I don't know somebody has to come to your house to install the software And then he puts like this dedicated organ file on your desktop that is specific to you For example that would totally make sense because as you know, like stuff works a little bit different It's not like downloading an iso and installing it It's probably more complex to get this onto your system if you want to use this Right, so there might be stuff that is either pushed via updates Only internal and like this organ file and other stuff that can get to your Computer, we don't know if this is possible of something is happening like with this feature And the second question is like if you look at it from the north korean view, that's like they have the problem They are quite happy. You have a nice state. Everything's working fine from what they say And now people come from south korea from western countries bring their usb sticks with like western propaganda To like have stuff like this watermarking even if it's like evil like a natural reaction from like a closed system So actually it totally makes sense to develop the system in the way they developed it It totally makes sense because it kind of reflects a little bit How the government is working because integrity is not only like a critical part for the operating system It's also a part for like the state itself Like shutting down everything and closing off everything. That's by the way the screensaver And closing down everything Also totally makes sense and tracking stuff that is distributed in the country or deleting unwanted stuff also makes sense So what we think that red star kind of resembles all of this and is like matches How culture is in in north korea actually Okay, we also have two questions of the isc which I would like to shift in Yeah, thank you. Okay. The first question is if you have any theory on how and why the iso got leaked Can you ask the question again? How and why the iso image got leaked We don't know this actually why is like we don't think that it was somebody from north korea We think that it might be a foreigner that got out like wil scott Told us last year that he was able to get a copy of it and get it out of the country There might be others that are able. I mean there is actually tourism in north korea, right? You can go there for your holidays So I guess that if you put a little bit of effort into it, it's possible to get Like nearly anything out of the country if you want to try to take the risk But we don't know who leaked the version and we don't know why it actually was leaked So there are actually rumors that it was a russian student with which was studying in north korea And he bought this on the street and it just brought it out of the country and put it on his block But we cannot confirm that this is actually true Okay, thanks. And the second question is if there has been any attempt at the custom kernel modules yet like reverse engineering or something Well, we reverse engineered rt scan, which is pretty simple because it just hooks a few function calls. That's it We have taken a look At the korean display module on a first glance it seems to do what it is supposed to do having something to do with with display management But we didn't take a look at all of the kernel modules or the rest of the remaining kernel modules because the code base is so massive That we need actually we need you guys to kind of help us out a little bit Next question from the mic, please. Yes. I have another question You said that most of the software is based off other open source software Which is uh, and you're not you don't have the source code and it didn't come with the iso So it's pretty much a massive violation of all sorts of licenses. Yeah, absolutely my question would be Could you get get an insight on what other packages are available? or From the package manager or in what other actually there is a dvd that you can Which also was leaked. I think that it was for red star 2 I'm not sure if it is also for the for the uh for the latest version But um, there is a cd with additional software and you have stuff like apache my school I don't know like all of the stuff that you basically need to run a full-blown operating system on the linux So there is additional software out out there. You can download the the The dvd and install the software on on the machine If you if you go through the rpm descriptions, you will see that They even for some of the software they wrote they kind of used a description for the For the license that says kcc, which is the korean computer center and sometimes they use gpl And sometimes they use gnu and yeah, so massive violations. Did you ask them for the source code? Actually, we think that there is an internal git in north korea where you can just check out everything So we suppose it is this way because it's like open source, right? Okay. Yeah by the way open source Very nice One more question from here Are you having a question? No. Okay, then we have one more question from the internet Yes, the question is if there is a possibility to um, fake the watermarks to get like some innocent north korean into trouble You you're probably yeah, no problem because the key is hard coded you could like, uh, you know how to scramble the the Hardware idea of the disk serial and you could perfectly forge documents. That's that would be not a problem Not not a problem at all. You just need the serial number basically Okay, and I just got another question. Uh, that is uh, does the warning dot web have a watermark? Um, no, no, it's actually it has the exact same checksum as the original file So um, actually like we didn't check if it had no it doesn't yeah, so it does not have a watermark because as niklas said Yeah, it's the same checksum as the kaspersky one Okay, thanks Okay, thank you very much Please give florian and niklas another big round of applause for the major talk


Born or not. I never really born or not. Just say German. Say German, Schmidt. Born or not. Born or not. Yes. But super easy. Born or not. Okay. Easy. Okay. Welcome everyone. We are here with our next episode for our Azure Stack Partner Solution videos. And again, I'm here with Tiberio Radu who is a Principal Program Manager at the Edge Solution Team to talk about different partner solutions. Hi, Tibi. Welcome everyone. And so what are we going to talk today? Hello, Thomas. Hello, everyone. Today we're going to focus on a couple of partners that are actually providing a SaaS solution to their customers. So in the previous episodes, we've had a focus on something more like an enterprise customer, which created their own solution and they served their own internal customers. Now we're going to look at a set of customers that are actually creating this SaaS-like offering for their customers. But to make that happen, they have, they need to have the platform in place. They need to have the solution on top. And they need to make all of these things work as well as create a development environment where they can build things. But I don't want to get too much into details on this. I'll let Julian and Dino go into these details and take it from here. Okay. I'm here with Julian from Knowledge Park. Hi, Julian. Can you introduce yourself a little bit and your company you're working for? Hi, Thomas. Yes. So I'm working for Knowledge Park since 2011. We have a patient management system for dialysis in Germany. My field of action is the medical application management, infrastructure and operations and Azure and Azure Stack. Okay. That sounds pretty cool. It sounds like a very good job there. Can you tell me a little bit? I mean, we have you on the show for Azure Stack partner solutions, right? Can you tell me a little bit about your Azure Stack Hub integration, your Azure Stack Hub solution you're offering? Yes, of course. So we have two Azure Stack Hub all-flash systems from the hardware vendor, Dell. We run them in two data centers of Acquinet and together with the Azure Stack Hubs we also have an object store from Cloudion in three data centers to store large data for long-term purposes. And our business need for the Azure Stack solution is that we need to run cloud services on-premise because we need the data sovereignty at all times in our own data centers to meet our compliance requirements. Today we are running very business critical systems on the two Azure Stack Hubs. We are running over 200 dialysis centers all over Germany on our two stems. And today we treat patients like 18,000 chronicle patients in a quarter and almost 80,000 in-and-out patients. Okay, so you're basically offering really the cloud services we get with Azure Stack Hub, this consistent experience we offer with that, but that really helps you to bring that into your own data centers, right? Is that what you're saying? Yeah, that's the point, exactly. Okay, so I know that you're working in the healthcare space in Germany. Can you explain me a little bit more about what you're doing and actually how Azure Stack Hub helps you in that specific case? Yeah, of course. So we, as I mentioned before, we have a patient management system for dialysis in Germany. So it's a 10-year-old grown monolith, and now we have the need to transform it to new microservices. So we had a look at the market, and we were trying to figure out what is the best platform for our needs. So what we're trying to do is we take the monolith apart, build new microservices, and we want to offer the services as software, as a service in the healthcare market in Germany. Okay, so you're building these, you're modernizing these traditional applications to more cloud-native and modern SaaS applications, if you will, on top of Azure Stack, right? Right. So what we try to achieve or what we achieved is that we now use infrastructure as a service and infrastructure as a code, as well as ARM templates. So we can standardize our infrastructure in terms of code. So this is interesting, because I like the part where you said infrastructure is code and ARM templates, and I know that many of our customers obviously come from the Azure space, and they know that they can use the Azure Resource Manager templates to deploy infrastructure and services on Azure. But you can also use them on Azure Stack Hub, right? So you can exactly use the same deployment mechanisms. Interesting. So can you tell me a little bit more how you're using these ARM templates and how you're actually deploying to Azure Stack Hub? Yeah, of course. So what we are doing is we use the ARM templates together with Azure DevOps. We write our infrastructure into templates and code, and we deploy them either per script or we deploy them with build pipelines. Okay, that is pretty cool. So that would give you the flexibility to basically, as other customers do, or as you probably also do, to use Azure DevOps, to use these Azure DevOps processes to deploy the whole infrastructure to Azure. You're using the same process and the same tools to also do that with your Azure Stack Hub environment. Right. That's pretty cool. So as far as I understand, you're building the solution on top of your Azure Stack environment. You're really doing the modernization of the application and building these modern SaaS solutions. But there is also a site where it's about operations and about operating the Azure Stack. Do you have any help with that? Yes, we do. So to be successful in our project and that we can focus on software as a service, we have a partner, and our partner is Bordonaro IT in Germany, and they are doing the operations as a service for us. Okay, so they are taking care of your environment. So I think that is great. Thank you very much, Julian, for sharing this. We are switching over and let's talk to Dino about how they are actually helping to architect and operate an Azure Stack Hub environment. Sorry to interrupt, but before we switch over to Dino, I just wanted to add one thing. As Julian's team sort of started to work with these stamps and to build a solution on top, he also needed a way to have visibility across all these stamps and understand their usage and that sort of thing. And he came together with, he put together this dashboard. So it's actually using a combination of things from PowerShell to Prometheus to Grafana to actually build this view across all the stamps. If you want to learn more about this, Julian actually has a very nice set of articles that describes how these are put together and describes how you can go through something like this if you're interested and actually set one up for yourselves. Okay, so I'm here with Dino Bordonado from Bordonado IT and you helped Julian with their Azure Stack Hub platform for actually Knowledge Park and the Acquainted Health Service. Can you explain me a little bit more who you are, what your company does and how you actually helped Julian operating and architecting the Azure Stack solutions? Sure. Hi, Thomas. So my name is Dino Bordonado. I'm in IT for more than 25 years now, coming from Classic Hosting, Data Center and Outsourcing and my company transformed a couple of years ago to Cloud Platform and this is why we changed our company name to Multicloud Experts which means we are on Azure Stack Hub as well as Azure for a lot of years. Azure Stack Hub started on technical preview free and since this we are on the boat and love the product and we specialized on two different business areas. One of it is the operator as a service role to keep the trial and pain away from the customer to learn how to operate infrastructure in the right way and the second part, now that we know what infrastructure needs the customer has, it makes a lot of sense that we also offer the support on the platform on top of tenants as well as the consulting service and help the customer build awesome edge solution and most of the time even hybrid which are bridged to public Azure and this is what my company, we are 25 people located in Spire in Germany and we have customers all over Europe and even some outside Europe and we love what we do. That's awesome, now I remember I think that is also why I know you for such a long time is because you were very early involved with Azure Stack well, was it called back then and now Azure Stack Hub and I think you're also a Microsoft MVP awarded for your work you do with Azure and Azure Stack Hub so that's absolutely great. You mentioned that you're basically operating that one of your offerings is operating Azure Stack Hub and but also architecting integration and solutions can you explain me a little bit what that means and what the differences are? Yes, sure, this means that at the end of the day the customer have consistency across his platforms so when we build the platform for the customer we have a turnkey solution, we work close with OEMs like in this case Dell Technologies so we hand over the platform to the customer and in Julian's case with infrastructure it was just brainless for him, he could just use the same code he started to work in public Azure on their proof of concept and run it against the Azure Stack Hub platform and then transform from there his classic infrastructure from VMware with a lift and shift to the in first place Azure Stack Hub standard platform and then we started to leverage all these new possibilities like AKS Engine, Kubernetes and everything around it to start to cut the old monolith solution in slices and we helped them to find the best way and path step by step in different milestones to transform their future version of the software on the same platform as they were used to use already in a public cloud and that's the awesome thing about Azure Stack it's just out of one hand the first party solution that comes from Microsoft to your data center and it's not I run it on premises or I run it in Azure's public data center it's beside you can build very amazing cool stories around these platforms and this is what we try to use there for example we also had a couple of cool additional projects where we use Azure Front Door to get traffic in and then pump the traffic over express route to the Azure Stack Hub so that the very important high secure data is stored on the hubs and the CDNs and the static data and the websites are all hosted in public Azure so you have the cost and technology of best the best the best in bread of both worlds so you keep the costs down by using this easy service in public cloud and you have the high secure self-hosted platform of Azure Stack Hub but both are used in the same way with the same CI CD pipelines the same source codes and the same processes like you would use the public cloud you can just switch the private cloud with a slip of your finger by changing some settings in a JSON file and that's it I think I love that you just you just brought like all that together what I love about hybrid like when it comes down to like leveraging hybrid crowd environments right you get you take the advantage of the cloud services which you can run in Azure but also as you mentioned like on Azure Stack Hub itself and then you build them together and take it once okay let's say hey these services can actually run in the public cloud and we can take advantage of it but then at the same time the data which for example we have regulations in place where it just doesn't allow us to store data somewhere else we can actually keep them inside and we can still use the same tooling with Azure DevOps the same the same experience with Azure Resource Manager templates so I really think that is fantastic and again you brought it to the point how awesome that all of that can work together now maybe you have time like to explain me a little bit how you help with that solution and what your part was to working with with Julian on this and what you actually still provide like as a service for them yeah sure so as of today we run both of the Azure stacks day and night and 24-7 shift model with our team so we take care about the whole platform like alerts like patch and update marketplace management but even when it comes to resource planning for the future so we look with every customer that we onboard or with every project we onboard what other resources we use today we run the numbers what other resource usage maybe in 6 or 12 months from now capacity planning with the customer also as a service we help when it gets to finding problems in ARM templates debugging these but as well as we are trusted advisor and jump in every time Julian has a question about what is the best way to store my arm templates how should I structure my repositories for infrastructure as code should we jump to Terraform instead of ARM for example and we do very cool stuff around this and and we also have a lot of third party solutions like for example we had a business need SQL server cluster in a high availability setup and and SQL clusters produce a lot of traffic if you set up them wrong so we already we also put in some data platform MVPs and help got help from there to set up these clusters behind these medical systems and some other systems which don't have the technology for high availability we use it as a as a third party solution called availability and replicate data from one stem to the other and can switch over in a couple of minutes from one region to the other region without almost any data loss which brings like some kind of hyper view replica or Azure cyber cover free functionality which is not there yet to the stems by third party solution so my job is also to watch the market and always have the best solution so if there's no first hand solution for Microsoft we use carbonite if there's tomorrow for example as a side recovery I go to Julian say let's evaluate it make a proof concept and switch again back to the Microsoft way of doing it correct on this product and I see myself as a how do you call it maybe as some kind of guardian angel for the customer as a customer success manager however you want to call it but I'm responsible at Julian can can get his job done and I bring in a lot of experience about this and I would another part of this is also the friendship you spend so many days weeks and months together that also this relation here ended up in a very deep friendship by working over one and a half years on solution and this is what makes it very cool so I love the project I love the employees I love everything around it and when you work together that close and and you get friends in this that the quality that gets out of such a project is just amazing and by pain from operating the infrastructure platform and and you know it as well as me like what a magic as just a cop is yeah we come from days where we had hyper B with a system center and a Asia pack on top of it so this was a full-time job for many people just to orchestrate and keep the platform up and running and this is just out of the box by Microsoft at almost no cost compared with earlier days with much more functionality so it's just amazing to have this turnkey cloud in your own data center now I love what yours is saying again this is this is I think it's really key and you brought up the word trusted advisor I really like that I think if with with the world getting increasingly complex right and especially it environments getting increasingly complex to deal with all these traditional applications but then also very modern applications and then across different environments you need to have some sort of a trusted advisor which can help you to solve your challenges and I again I really like fascinating we all the work work you're doing and now this is right really I was thinking about this for a long time to ask you that question is what would be like your key to a key takeaways when you're working or when you start working with a customer what is like what are the most important things when you do a project and not just an Azure Stack Hub project but like an Azure hybrid project in general key takeaways actually are that in such projects when it comes to Azure Stack Hub and bring the application to Azure Stack Hub if you do it right you should be on the side of the customer and helping the customer take good decisions like for example the way Julian came to me was like hey we have this requirement for on-prem cloud platform and we had offers from various OEMs and OEM one says we need HDDs the other says we need SSDs the next says we need a hybrid and I'm more than confused I have the budget and I have to keep budget very structured and low as always because everyone wants to spend as little as possible but he was just not in a situation where he knew what is right and what is wrong and who is telling the truth and who just wanted to say it and so it was very clear that I sit beside my customer and take the decision like I would have the money myself and then from there on we took over and jumped to all these challenges we had with the project and everything was from day one from the feeling like I'm one of Julian's employees or colleagues so my key takeaways also on second part of it is that you need to bring all the people on the table so like when you run traditional infrastructure you have the silos you have the networking guys you have the storage guys you have the fire warning guys you have the infrastructure guys you have the whatever guys the SQL server guys and in these hybrid projects when it comes to infrastructure as code and software defined networks you need to have everyone on one table and you have you need to break the borders between these teams because it's not five single teams and silos anymore it's one big team and it's very very important that they speak with each other so you have to break out these these silos and help the people to talk actually to you're in some way from every perspective you're in the middle as the trusted advisor for almost everyone and in our case it was very helpful that we run data senders for more than 15 years now so we can be friend with almost everyone of the silos which which makes it very easy then to get a connection to the people and and get their concerns because a lot of people that run classic infrastructure they have no idea how how this cloud platform works but to to get that vulnerable and and and from from some concerns they have they don't they don't even try to ask how does this work is it secure and whatever they just say I don't want it I I'm just afraid it's new and I don't want to learn something new and for this they think it's no good choice of technology and when you then get into talks with these people and you hear for example like the networking guy says this is dangerous so it most is just I'm concerned to lose my job you need to talk to the people make them clear that the jobs change the roads change you get the opportunities you give some work to others but you get more work from others so there's enough work for everyone but probably the role changes somehow but there's nobody kicked out of a job because of a cloud platform this is nothing I've ever seen and and this is what I want to say you need to put on the trusted advisor in many cases and you need to communicate communicate with all the silos and people and then you do a great job. I think many times where you have these silos right and you can just not do a project with having these silos and different teams which are not really collaborating together and I think that is a great experience also what you can bring as with your personal experience when you work with a customer actually to tell them how not just in technology wise they need to bring stuff together but also how they should organize their teams to create an important part of your company. Yes.  Yeah actually the COE is a very cool project that Tibi and I thought about and where it comes from is very simple so you have in many cases the ISVs with their software they want to bring to the Azure Stack Marketplace as well as you have customers who want to do a proof of concept and to get speed in this project and to not spend a lot of time on structure requirements that I just mentioned we talked to Dell Technologies and we've built this Microsoft send off excellence power by Dell Technology which is basically a full blown state of the art eight node Azure Stack Hub system with the large configuration with whatever you can we run 100G network on top of it we run 40 net firewalls and we are directly connected to the Frankfurt Internet D6 carrier which means we have a super fast connectivity when I say super fast I'm also talking about peering to Microsoft Azure so we peered the German Azure regions with two to three milliseconds of latency which is almost like they sit beside our Azure Stack Hub and we use the COE now as validation as a service for a couple of very cool software vendors to validate their solutions to bring workload like VDI to Azure Stack Hub not only to Azure and me and Tibi help them actually to get their software validated and get the marketplace filled up but we also help others to solve their problems so we make it very easy for those people to onboard get a subscription and test their solution and this is what we want to achieve to say Azure Stack Hub is an awesome product believe us and test it touch it try it and love it this is the idea behind it and that said Microsoft has a lot of things to make this process even better and like I said with the operator role already so even Microsoft now has cool projects for ISVs and IS in place and we are one of the first five of the first wave of these in the pre-ferred partner state so this gives us also for Microsoft a very cool label like saying modern our IT in Germany is one of these pre-ferred partners with all the requirements like dedicated stuff on Azure Stack Hub and with we help others bring the solution to the Azure Stack Hub system in our data center which is the COE and if someone wants to test the solution everyone is welcome contact me, contact Tibi, contact Thomas we can help you overnight. That sounds really great I think we will need to have a separate discussion on this and I think we definitely will. So with that I want to quickly ask you so if people want to know more about you or about the project itself where should they go? There are two places to go one is our personal website which is spotonaro-it.com the sendoff X is in the works which everybody also gets his own website and my own personal blog is also in the works which is dino.blog which is super easy to recognize so get there ping me on LinkedIn on Twitter so my Twitter hand list at dino botonaro the same for LinkedIn the same for Facebook just contact me on every channel you want I'm always glad to help as MVP community is my world so if someone needs help has question about Azure Stack Hub we also have our own German meetup which is called hypercloud meetup you can find it in the meetup platform you can find it on Twitter announcement so I'm glad for everyone who is interested in the technology and not only about Azure Stack Hub even Edge and even hybrid this is why we call our meetup even the hybrid meetup to build awesome solution on the Edge but not only on the Edge it can be hybrid Edge and public cloud combined. So with that I want to say thank you Dino thank you Julian for joining this video I really appreciate what you talked today and how you showed your solutions you built with Azure Stack and the Azure Stack Hub so all of you thank you for watching and hopefully see you in the next one thank you Thomas take care