OK, so I'm Max, and today I'm going to talk about what we've learned operating Bluemix. The sad part is that a lot of the people operating Bluemix are in the audience, so they probably could join me and give that talk. But I'm here representing them, so I'll do my best. Another person that is not here that I've done this talk with before is Matt Zuchiu. He's actually part of the IBM China Lab, and he's the lead developer for our CPI group. This talk kind of covers what I've learned indirectly being the open source liaison with working with my colleagues here, and then the product manager on the back here, which is, well, the director, Fabio. Doesn't have the long hair, but that's him. All right, let's get going. This should be fun and interactive, so feel free to stop me. Obviously, I've got a deadline to stop, and then you guys got to go to lunch, so if the discussion gets too much, then we'll move it. But ask a question. I've given this talk before, and it was interactive, and you'll see why. It has this format that helps it. So let's get going. So first thing is, what's Bluemix? I'm not going to bore you too much, but it's important to set the stage of this. It's a certified pass. Well, we're getting there. I mean, there's other aspects of the certification that we still need to get, but it's a moving process, and our intention is to definitely support that. I'll share some public statistics that Simon Moser, my colleague, mentioned today in the keynote, is that it has, well, first thing we did mention is that we have three kinds of deployments. So one is public, dedicated, and then the last one is private. Public is what you get when you go to Bluemix.net. Dedicated is essentially a slice of a software environment for a particular customer. So we have different customers. So the customer gets its own, but we manage it for them. And then private is on-premise, so very similar to Pivotal PCF, in some ways, and other companies have that too. So as Simon mentioned this morning, we have over a million registered users. So these are the public numbers. If you want to know more, talk to Fabio. I'm not allowed to say more than that, but it's getting pretty big. We have 500-plus-thousand running apps and hundreds of services. And we use software. Now it's a source of strength and pain, but let's leave it at that. We've also looked at OpenStack as a potential alternative. But OpenStack has its issues too, so don't get me wrong. So everybody has issues when you start operating at scale. And it's a worldwide team, right? Developers and also DevOps. So a lot of our colleagues here in Germany and Bublingen are essentially doing a lot of the development. So they're in the room here. But we have also people doing DevOps, like for instance in Roma. We also have people in North Carolina as well as in China. So it's all over the world. It's a worldwide effort. So what have we learned? All right, so in the US, there's this guy called Dave Letterman, who is now retired. But he used to love doing top 10 list. And those work really well, especially when you have a short amount of time and a very focused set of things that you want to talk about. So that's what we're going to do. So we're going to use top 10. And that's why it's sort of interesting for you if you have questions. We're going to go from 10 to 1, OK? So number 10, I call this change. So Fabio and his team and a lot of other people essentially have this tight change request process. It has its bad parts. And you'll now see kind of the format of each one of those top 10. So I start with the bad, I talk about the good, and I talk about the lessons. So what's bad about this? Obviously, it makes it slow. We have to go through a tight change process where people like Fabio, Richard, and others have to essentially approve any change to the system. That can make it a little bit hard. But we're a huge company. Remember, we are 300 something thousand people. And I don't know how many works on Bluemix, but it seems like all 300,000 work on Bluemix sometimes. And then because it's worldwide, well, people in the middle of their sleep, what do you do? You have to deal with it, right? So that's a problem, but there are some good things. So what is good about it is that if somebody in Woma decided, well, they're going to make a change, and they figured it out, before it actually gets approved, it actually has to go to the US. And then also the guys in China have to take a look. And then people are traveling, right? So before it actually happens, there's this process that essentially sort of coalesce it and make sure that it can actually make it good. So what it does is it limits any problem. It makes it slow to adopt new changes, but it can limit things. And there are more eyes reviewing it, right? So that's sort of good. So what we found is that definitely if you're doing a large team like we do, if you can't afford not to do a large team, obviously do that, right? If you can have everybody in San Francisco like Pivotal does, more power to you. IBM, we can't, right? We're I with International for a reason. We are global, and we are a very large company. So we have to go worldwide. That's just the way it is. Yes, Chris? I just want to right now. Yes? I would generally cut. Yeah, and obviously there is going to be that too, right? There are customers that like this. They are happy to slow down. So that's number 10. Number nine, audit checklist. So what we do in some ways is we audit things a lot. Obviously it comes with the bad part. So what it is is that some of them are manual, and it can be a long time to sort of check those. But the good part is that using canary-based deployment, which is sort of a feature of Bosch, it helps because then you can sort of run your deployment and verify, oh, things not quite working, and then you can backtrace. And we do some audits for just at steady state. So in some ways, it's operating the system and then verifying as it's running that things are working. So the lesson here is that we've actually ended up creating our home-grown tool called Doctor. It comes from the China team that essentially allow you to, in some sort, it's getting there, do a continuous monitoring and then audits of all of the deployments. So in many ways, you can think, I can't reveal the numbers, but you can imagine we have X number of customers that have private, and then X number, maybe X square number, dedicated, and then public, which is a smaller number. And all of these go to the same tool so that the teams all over the world can kind of get a view of it. Number eight, checking, log checking and monitoring. The bad part of this is that because we are such a large environment or such a large set of environments, the log rotates very quickly. So you end up with a problem, and if you don't catch it, you might lose it in some sense. So a large amount of data, it's in the terabyte. How do you keep it? How do you manage it? Obviously, better cloud would help. Things like S3, for instance, you can keep large amount. But over time, it becomes also impossible. So what happened two years from now? It becomes an issue. Bluemix already has operation about a year and a half now, so it's getting close to that. And then how do you separate? How do you query over those logs? So that's a huge issue that we've sort of experienced. The good part here is that the way it's sort of designed, it doesn't always work. So don't get me wrong here. But when it works, or at least the design point is to allow streaming and allow different syncs. So when you do that, then you can use that data to your own set of tools to do the kind of query that you would need. So I've seen a pivotal working with them, for instance, that we've used Splunk as an example. But at IBM, we have our own grown solution. So it's not really a solution to you, but it's a lessons learned that you definitely want to start introducing some level of log tooling early. Because as your system grows, it can become an issue. How do you debug? It becomes a problem. All right, so moving along, number seven. I actually like Drake a lot. I don't think he's always kind of like sad, but that was a sad phase of Drake that I thought that was cool. So what is this? It's Boschinate rules. The reason I say this is because I have actually contributed a third line of code in Boschinate. You can just go to the repo and you'll see. So I'm not complaining about it, but I'm saying there are some issues with new stuff. And for instance, in the case of Boschinate, it can become, and I think we're solving this now, to recreate a director. And certainly because it was in development, we've had lots of frequent updates. So keeping up with that was an issue for us. Obviously that may not be an issue for smaller companies, but as a large company with a history of being very good at keeping, we deliver something to a customer. And then 20 years later, as long as they are still paying the bills, I mean, we'll support them. And it's a great thing, but it's also a source of pain. So when you're in that mindset, it can become an issue with frequent updates. The good part is that in general Boschinate has been great. I mean, it's moving towards the new Bosch CLI. If you haven't looked at that, it's going to be a single binary. And it really works well. And we've also, as part of this, introduced external CPIs, which, as we mentioned today, is essentially the source of growth for this movement, which is Cloud Foundry. If you think about all the other clouds, Paz, none of them have this one feature that we have, which is we can work across different clouds. And as far as I know, it's close to 20 now. And then today, I just, right before this, came from a talk of multi-CPI, where we have a CPI, or meta-CPI, to talk to multiple clouds. So it's a great source of strength for us. And we definitely love that. So the key is to deal with better planning for things like changes. And it's important for you to do that. Otherwise, you run into some issues. All right, next one is what I call custom software. This is actually a super important one. What that is is you want to make sure that you have custom releases. So in other words, if you want any piece of code in your organization to be part of the environment, whether it be installed on a stem cell, on all the VMs, or you want to have them as a separate deployment, like, say, for instance, a service, don't try to, I would say, use Bosch, adopt it, and embrace it. So what I mean by this is, like, some people at IBM, we've created our own hacky stem cell. And then we put a little agent running in there. And then we said, well, maybe some of those deployments, we should have our own job. And it doesn't really work with Bosch. When you start doing this, you start breaking the abstractions. And it becomes much harder for you to upgrade and to keep the system healthy. So create your own custom software. So first thing is, custom software is going to happen when you're a large company. Like, for instance, we have a security release at IBM, because the security guys think they know better, and probably maybe they do. I don't know. But they have their own release. Who am I to say I'm not an expert in security? So I have to just deal with that, right? So they create their own release. So that's going to happen. You can try to bake things into the stem cell. Usually a bad idea. Create your own release, add it. These days, Bosch has this thing called add-ons, where you can easily run time config and deploy those custom bits into all VMs. So definitely want to do this. So of course, with external CPI and co-located releases, you kind of solve this problem, right? And of course, there's a lot more information these days about creating releases, which was an issue. So Maria, who's the top essentially committer for Bosch at Pivotal, she's written really great presentations about how to create releases. So I urge you to go there. Towards the end, I have a couple of slides to point you to where to go. But if you go to Bosch in it, you'll see on the top of Bosch in it, there are some pointers to our slides. But definitely you want to create your own release. It's not easy, but the point is there's more information now. So simple lesson. Do not have your own custom SEM cell, custom Bosch, use the extension point. So the CPI is an extension point. The release is an extension point. But don't try to do more than that, because then you can survive with the rest of the system. Number five, this is an important one. It's kind of still work in progress, is that power DNS is actually an issue, right? So we highly recommend you not to use power DNS. But we don't really have a good alternative yet. We're still working on it. But let me talk about what's bad about it. So first is that it's single point of failure, so it's very hard to do HA on it. And we don't really have a solution. So if you're deploying on a cloud that has a DNS solution, then obviously you use that. Otherwise, you're going to have a bad day. The good thing is, as I mentioned, some of those IaaS support it. And we're also working on a power DNS test solution for Bosch where we're essentially using local DNS. So all of the VMs will have all information, in sync of all the other VMs. Now, that doesn't solve a full non-DNS deployment of CF because CF uses as part of it like discovery, especially Diego, discovery of services. And they use console. And you can't really replace console. But there's various discussion in the open source community to see how to do this. So you may still be able to use those other DNS solutions. But maybe the platform will have a power DNS less solution, basically batteries included. So the lessons learned is you really do have to think it's a general lesson of any node in the system that is not designed to BHA. Because when you do that, it will prevent you from growing in the future. So be careful of that. And it can become very hard to remove. I mean, this is based on our experience. We're still struggling removing power DNS usage. Number four is update. Security update are important. And they're painful. But you got to do that. Why is that? Well, obviously, the internet, as you know, unlike some orange person in the US, is not a source where you want to go and think that trust any information. It's actually exactly where a lot of bad guys, bad girls are coming and trying to attack your system. So this is nothing new. You should all know that. So what happens is that there are CVEs pretty much weekly. You're not all aware of them. But I can tell you, Chip and the rest of the Cloud Foundry Foundation and various folks at Pivotal and IBM and various other companies, we know about when there is a CVE. And those happen pretty much weekly on average. I mean, sometimes you don't get them. And what happens when you do get those is you have to be able to get the security in. But they're costly. But you still have to deal with it. Because the last thing you want is to be hacked in. And then you be in the news that your system, like what happened to Yahoo, it's unfortunate. It's like you're struggling. And they just got hit by this massive security bridge. You don't want this to happen to you. So what does the Bosch team do about this? Well, they essentially release new stem cells and new releases frequently. And recently, they've been going to a hardening of the entire system for security. And obviously, as I mentioned at IBM, we have our own security release. So you definitely want to take this very, very seriously and follow the guidance. So if there's one thing you want to remember from this talk, is definitely make sure if you're going to go in production, being able to deal with that is very important. We did come up with a way to essentially make it faster to even update the OS, we call OS reload. It's also a source of various pain. But this is one maybe advantage of that feature. Number three, multi-bosch deployment. So this is still something we're struggling with, but it's certainly an approach that you should think about. So what this is, is that when you start with one Bosch director, it becomes essentially your bottleneck and potentially your spoff as well. And when you start going, especially when you have multiple environments, that becomes a problem, right? So having a way to do multiple deployment with Bosch, that's definitely, so this is essentially, you do one Bosch and then you create other Bosches for your other deployment. So that when one fails, you don't lose the entire system. We've seen the director fail. I mean, we've seen pretty much every failure that you can think of in the book. Some of it self-inflicted, but some of it also because the software has problems. It's a big piece of code, right? So there are issues. So divide and conquer, right? That's the obvious solution. So definitely reconsider your deployment strategy. There are different sort of best practices. Certainly Dimitri is a source of great knowledge. He's seen a lot of it because he talks to everybody. So you wanna sort of hone into what he discusses on Slack, for instance. But experience with other things. So as you have your deployment, try other things.  There are other ways to do things that may be better for you. Number two, this is something we've learned from our colleagues in Italy. And it's kind of interesting because you see cultural things from different places kind of filter into a worldwide team. I know talking to Fabio and the rest of the team is that a lot of the deployments never work 100% of the time.  And then you think, well, when was the last time the whole deployment actually did work? Can't even remember that. So what do you do? All right? So you have to address the failures and then obviously continue. So Bush allows that model. But the point is that it's never gonna be 100% successful. And the guys in Rome, like maybe with their cool attitude, yeah, fails, just restart it. Don't stress about it, right? And that's the key is that the system is so large and there's so much, especially for big environments, there will be failures. Don't expect a fully pristine 100% just work perfect. All right? Like my camera is perfect German engineering. Doesn't always work like that. It fails and then you kind of deal with it. So Bush will help you with that. And even when the system is failing halfway, it's still usable. So don't stress too much about it. So embrace failure in some ways, right? And that was the point that Simon made today. This weirdness, it's gonna happen. And you just have to adopt it as part of your culture. So trust the tool. Failures are gonna be part of the system. And sometimes they don't work perfectly the first time. And it's okay, right? As long as it works in the end. And number one is to back up your director DB often and a lot. Okay, why do I put this at the end? It's an obvious one. Well, because I've experienced it personally. So last year, trying to go on vacation, Friday, this is a story I'll finish with. I think it was like Thursday and I took Friday off. I get a call like 4 a.m. in the morning. Chris Ferris tells me, we need your help, buddy. The whole blue mix is falling apart. And I think it's Bush. Now, Michael Franco on the back actually does a lot of that sort of firsthand. And he's very good at sort of identifying where the problem is. And I guess maybe, I'm not sure if he was the one identifying where it was in Bush. But anyways, it got down to me. And I know, of course, I'm going on vacation. So what do you do at 4 a.m.? Chris calls you. Well, I deal with it. So I'm getting ready to go to the airport in San Francisco. Got my laptop. I'm going to the code. I've got very little bits of logs to go by. And then on the plane, I kind of have the aha moment to see where the problem was. And of course, I get back and identify two lines of code in Bush where we're not checking for an error in the go code. And potentially that's what happened. The symptoms was that disk on the director where the director keeps all its data disappeared. So now we have a director in a deployment. And they don't know about each other because the data just disappeared. So first is identifying it. And of course, what do you do about that? So in the end, the disk got lost. So when you disconnect the director for its deployment, there's not much you can do. Although we kind of came up with some clever thing to do, the guys in China. So I'll talk about that. Obviously, how do you solve this? Or how do you prevent this from happening? Becoming a problem is to back up, right? You back up your director database. So when something like this happens, then worst case scenario, you can always go back to the old backup. But any VM that was created since the backup, you lose those in disks and so on. So that's a real issue. The problem with backup was that it was very slow for us. So when we have a large system, the backup took forever. Before the first backup got done, we were already doing the next deployment. So it's used this, right? So what happened is we worked actually the team from Rome work with them, with the Bosch team to make the backup come in faster. And partly it's because we're not backing up any information about blobs because they're kind of in a separate place. And you can recreate them, which is good. Now, some IaaS, for instance, AWS, has a way to snapshot disks. So if you can, the best thing to do is to take your disk on your director and snapshot it so that way the IaaS is doing the backup for you, in addition to maybe Bosch backup, right? So one solution we came up with, and it's kind of a clever one from the guys in Beijing, is to modify the dummy CPI and replay the deployment. So think of it this way. You have a director who no longer knows about the deployment, but you have the deployment. And when you have the deployment, you can always go to your console and get all the IDs for your disks and all the IDs for the VMs. So now you can create or modify a dummy deployment. You have your manifest, pretend that you're gonna deploy that one again. And instead of deploying, it gets the disk and VM ID from a list that you've created of what exists.  So we did that. That was over my two weeks vacation. The good thing at IBM, and I'll say this, is typically we have very, very good managers and I happen to have one, Alex, and he's super cool. So those two weeks I got back, generally. So I'm trying to get a couple more weeks, but anyways. So it happens. So with that, I wanna thank you and take some questions. So questions. The OS reload. Okay, so the guy that owns the OS reload is sitting next to you. That's Fabio on your right. So the problem with this is that in software, this is kind of a unique software problem, but it's actually an interesting solution we came up with that actually probably works other places, is that there are some data centers that have limited VM capacity and software doesn't have a first-class reservation system. So if you have a large deployment with VMs in it and you try to say, for instance, update or redeploy or add to your deployment, you might lose the VM. So if you release some VM, you might lose it. And since the system is out of capacity. So instead of releasing VMs and getting new ones, we essentially keep the VM and we reload the operating system, which is in some ways what you need to do because the VM hypervisor is fine. It's just the OS that you have to reload, right? So to do this, you actually have to modify the CPI to do this. I've talked to Dimitri a long time because at Pivotal, they actually have a similar problem for their data environments where instead of having created a new VM and then reload large amount of data, they'd rather update the stem cell by reloading the operating system. So there is some discussion about maybe potentially making this a first-class citizen. Any other questions? Before you go, I've got number 11. Oh, Chris. So he's the hiring manager. Thanks for taking my talk and doing a little advertising. Okay, so number 11, let's do this. Always seek knowledge. Why is that important? Because it's large and it's hard.  That is very true. Not everybody that are great developers at Bosch know how to use Bosch. I was at Bibli again and I mentioned this. I love cars. I'm an engineer. I love the German cars, but guess what? I am a terrible driver. I think most of you probably fit in that category or one of those, right? So it's not because you're a developer, you know how to use it. Bosch is complex, so large learning cycle. So the good thing is there are lots of talk. Dr. Nick has tons of information. Maria is adding stuff. Dimitri, anything you need to know, just ping him on Slack. Pretty much you'll get an answer. If you can't get an answer from him, he'll point you to somebody that can give you an answer. So Bosch IO is your source, right? And Dr. Nick, as I mentioned, so seek knowledge, okay? So that was in case nobody had question. Anything else? Okay, we have 15 seconds. No? Thank you.


What's up everybody? Welcome back to the YouTube video. My name is John Hammond and this video we're looking at level 17 of the Natus war game from over the wire. So on the tails of level 16, we just got the password for level 17. So let's run the script to check out what this web page is and what we're actually working with here. Here we have the form that we've seen before just for logging in it looks like we're given a username input field. And the button will check existence. So kind of the same SQL injection attack we've seen before, but maybe there's something different. So let's actually take a look at the source code and let's see what they're doing here. Just copy that link to the index source. And again, we'll have to de-entitize this and we can remove all those break tags because we don't need those. Alright, so we have the same schema as we've seen before just a user's table with username and password have the same test if we're actually posted to the page in the PHP code. And we run the same query looks like we have the same SQL injection attack vector because they're just concatenating in the username. But it's commented out the output as to whether or not it got a result. So no longer can we determine a yes or no binary dichotomy thing. Does this user exist? Does this user not exist? So we can't use that for our typical blind SQL attack. We won't be able to determine via that yes or no one or zero. Do we have the correct character and what we're trying to leak out? So we have to go through a different attack. And this will be pretty neat. This is when I think one of the coolest things that you can do with with SQL. But let's go ahead and start to post to this web page and see what we can do. We want to change our method to post same URL data will be username. That's the variable that we've got to work with an HTTP and we'll say subscribe blah blah blah post run this. No outputs just like it told us but we know we have our SQL injection. So if I were to change these two single quotes and the original quote the double quote that they're using in that SQL statement we can or one equals one and throw a comment in there. Run this and looks like we're again no real output but we can do some other interesting things. We can actually because we have that SQL injection we can run SQL commands and SQL functions and do SQL things. Let's actually try and sleep for a certain amount of time. So this function will do exactly as it says it will wait or sleep a certain amount of seconds that we pass in here as a number and it will actually operate because we're using that and function. So if something returns and we get this sleep function which will return one if it succeeds or it'll return successful. But the and will mean it will execute if the user that we're checking for according to the application code exists. So let's try this if I hit control B to run it. It returns immediately because obviously the user subscribed does not exist. But if I change this to Natus 718 Natus 18 is the next password that we want. If I hit the go button here the build output there's no response for at least five seconds because we've ended that sleep function. So this gives us the building blocks for a timing attack for time based SQL injection because we can actually leak out the password right. We can say where password we actually don't want to use the where clause because it's already being used for that where username. So we can just say and password is like and use the same methodology that we've been using before. So add in what we've seen of a scene password join of scene password because we'll keep track of that in a list and a character that we iterate through. Don't forget our percent sign here so we get the wildcard and let's change the sleep to two. So let's get that character pool that we want to work with. So let's say characters can equal let's get everything from the string module so we can use lowercase uppercase and digits as we've seen before. Sorry for this duplicate code here guys but now we'll do a while loop or actually we'll do a scene password again to determine the length. Length of scene password is less than 32 because we know that was the length for it. We'll say four character and characters try and get this response here and let's actually just print trying and let's get what we're looking for. We don't need the plus there. We don't have to print that out but I'm going to do this in the command line. Let's change that preference so it looks. Let's run that pipe on not a 17 script and I forgot my colon my bad. Okay so now we're going to start to iterate trying all these characters but X we hang a little bit and we got a test if we actually got a hit if it slept for two seconds or if there is a real difference in the time between one execution in the next. Then we know we've got the correct character because that and password like successfully executed and then sleep will successfully execute. So let's change this and let's start to get a time to determine how long this code takes to run. Let's import time. Let's actually do from time import all so we can just use the time function real easy. So for every character we're looking at we're going to try and send it and then we'll determine what the time currently is start time. We can just do that for debugging purposes and then we can say and time can be the new time after we've already made that request and let's say the difference can equal and time minus start time. So we can get an idea of how long that took difference. Now we can print these things out and while we're watching it difference is maybe 20 tenth of a second but I'll scroll up here. Once we got to X we could tell the difference took more than a second because we slept for that second there. You might have different values on this depending on how your internet connection is because we're literally timing how long this request takes. And that sleep function whatever integer amount of seconds you pass into the sleep function in SQL obviously that will vary. So if I were to run this with two you'll see a different difference because now you're sleeping for two seconds. But since you're going to automate this and weaponize it to actually leak out the password. You want to do something that will give you relatively quick speed but still let you be able to determine what is the threshold for this actually being a successful hit. So we can say if the difference is greater than one then we know we actually have a successful hit. That's the correct character that we've seen in that position of the password. So let's add that to our scene password and break out of this for loop so we can keep moving seen password dot append character we're looking at in the loop and then let's break cool. We don't need this start time notice anymore. Now let's start to loop. What did I do wrong. I think that was happening because nothing was buffering this output here. Now let's try this. We actually have that print statement will tell us the trying and it will determine that X is the correct password. Okay cool. So now we have an attack looks like we're leaking out the password but we forgot the binary notion in the password here so we may be missing capital letters. Let's make sure we include and binary password like so we get case sensitivity when we leak out that field in the database. Now we can let the script run and by the end of it will have the password for Natus 18. All right, I'll let this run and I'll see you in a little bit. So it looks like the script did finish and we have a potential password. Let's head back to the original script and save it as a new one Natus 18. Paste the password in here change the username and let's see if we can just get that page see if we got the correct password and we're ready to move on. Let's control B run this and here we are Natus level 18. Sweet. So we did it. That was the successful loop and pretty much a good Python attack for actually implementing a timing based SQL injection exploit if you want to call it that. And I think that's super cool SQL map does some stuff with that if you haven't seen that tool totally check it out. But I like to consider that methodology pretty good for a timing attack and you'll see those in a lot of catch the flag competitions. And when you don't have explicit SQL injection and you can't get a results easily to determine your blind SQL injection. You can still leak out pieces of the database just by taking a little bit of time using a loop like this. And you can just run with a while true if you don't have this criteria about what you're leaking out about its length or whatever. You can just run forever. And once your loop starts to act weird and gives you like random bites, you know, okay, I pretty much reach the end. So that's it. That's how we can do a timing attack and time based SQL injection and Python and some web hacking. Thank you guys so much for watching. If you do like the video, please click that like button. Please let me know what else you're thinking, what else you'd like to see, what else I could do better with in a comment. If you're willing to subscribe and thank you so much. See you in another video.


This 2017 Ford F-150 SuperCruz 6.5 foot box Lariat is stock number 11930ZA. We're here at Summit Automotive in Fond du Lac, Wisconsin, you're new and used light-duty truck headquarters. This 2017 Ford F-150 has the 5-liter V8 engine. Pumps out 385 horsepower, this truck has been fully safety and inspected by our service shop. Has a fresh oil and filter change, all the fluids have been checked and topped off. And this truck is 100% ready to go. Shadow black is the color we shoot all of our videos in 1080p, 60 frames per second. So if you have HD capabilities on your computer, tablet, smartphone or television, turn them on right now. Because it is definitely your best way to check out the quality and condition of the truck. And options before seeing it in person, we're going to go through everything on this vehicle. And if you like the video and want to subscribe to our YouTube channel to see videos that we do each and every day here at Summit Auto. And the lower right hand part of the screen is a subscribe button. Click that and then click the bell notifications and you'll get updates every time we do videos here at Summit Auto. Comes with the 20 inch painted and polished aluminum rims. Has Cooper Discover AT tires. These are 275, 55R20s. And these tires are brand new. We put those on in our safety inspection. You can still see the little wear knobs on there. Front fender is in excellent condition. I didn't see any scuffs or scrapes or dents or dings on there. Headlight lenses are nice and clear. The front bumper is in really nice condition. I didn't see any dents or dings on that. And you do get the chrome trimmed grille. The hood is in excellent shape. I didn't see any major dents or dings on there. And the passenger side front fender looks really good as well. Passenger side rim. No scuffs or scrapes. And that tire is brand new as well. As you go down this side of the truck, take note of how clean the body is, how reflective and mirror-like that paint is. We take these HD videos so if you are far away or even if you're close by and just cannot make the trip down. But you're still interested in purchasing the truck. You can see the truck, hear the truck, and have confidence in the vehicle that you're looking at before you even get here. So that when you do get here there's absolutely no surprises. And you can make a smart and informed buying decision from your own home. Very nice down that side. This back rim is in excellent shape as well. And the back tires are brand new too. Frame and underbody looks really good. And it's like that all the way underneath. The rockers look good. And it comes with some nice chrome step bars. FX4 off-road suspension package coming around to the back. Rear bumper looks really good. Has the backup parking sensors. Full towing package which includes a receiver hitch four pin and seven pin wiring. The tailgate is in really nice condition. It has a back flip fold up tonneau cover. And that's in really nice shape as well. This one does have the tailgate step assist. And it has a Ford spray in tough liner. So that's a factory spray in bed liner. The bed itself looks like it's in nice shape. Does have LED bed lighting. Definitely brightens it up. When you have the tonneau cover closed. Really nice back there. Tailgate shuts nice and solidly.  Just as clean as that passenger's side. No major dents or dings on the driver's side box. For full disclosure this back rim is in excellent shape as well. And down the rest of this side the truck looks really good. You get the heated power folding mirrors. Directional signals. Blind spot mirrors. Has the driver's side door code entry. If you want to figure out how to find your door code on your F-150. For 2017 to 21. And the upper right hand part screen is a video on how to do that.  No rips or tears on these seats. I believe both these seats are heated and cooled. All weather format in here. Comes with the auto headlamps. Power pedals. Tilt telescopic steering wheel. Power windows. Power locks. Power mirrors. Memory driver's seat. And like I said these mirrors do power fold in. But I believe the vehicle has to be on for that function to work. At least the accessory position has to be on. You get the built forward tough animation on there. And get the digital speedometer. This truck has 61,798 miles. There are those power folding mirrors. I always like showing both sides that you know that both sides are working properly. You do get a compass display there as well. And we'll start it up in just a second to show you that there's no check engine lights or anything like that. Leather wrapped steering wheel. Audio and Bluetooth controls on the right. Cruise controls and information center controls on the left. This one has AM FM and Sirius XM radio capabilities on the sync 8 inch radio. You have your climate controls on here as well including your dual climate controls. And then you have all your different apps and stuff in here including Apple CarPlay, Android Auto. So that's a lot of good stuff that you can do on this radio. And you have all your other different apps there too. So what you can do with Android Auto, Apple CarPlay project your cell phone to the screen via one of those apps. So if you have a navigation system on your cell phone project it up there and it's like you have nav right on the screen. Stability control, downhill assist control turned out four wheel drive.  Factory brake controller, dual climate controls has the heated and cooled seat buttons. And it does have the sync system to connect to your Bluetooth cell phone. Keyless entry, remote start, power drop down tailgate. This one has the ambient lighting in here as well. Passengers side format and seat are in excellent condition. No rips or tears on there. And the headliner on this truck is in excellent shape. You didn't see any rips, tears, major stains or anything like that. Map lights and power sign rear window button right there. And this is also where your backup camera shows up and you can see that that's working nicely. And that you can zoom in right over your receiver hitch to get hooked up to your boat camper trailer the first time, every time. We'll take a quick look at the back seats and then we'll start it up and take a look under the hood. The back seats are just as clean as the front seats. No rips or tears back here. It does have the latch child safety system for any child car seats you may have. Power sliding rear window with the built-in rear defrost. These headrests do fold down for extra, for better visibility when nobody's sitting back here. These seats also fold up and there is a little bit of storage compartments back here. These are your tie downs for your box. They go on those inserts back there. And you get that nice flat floor back here, all weather floor mat. Has two USBs, 110 volt 400 watt plug-in and then another 12 volt power point right there. Very nice back here. You do get the child safety locks on the back doors and these back bottoms of the doors are in excellent condition as well. We'll start it up and take a quick look under the hood. Starts right up. No check engine lights or anything like that. I would personally like to thank you for checking out the video today. And hopefully from this HD video you'll be able to tell just how clean this truck is all the way around. Inside and out. Under the hood we have the 5 liter V8 engine. Engine maze is very clean, runs very smooth. Once again this truck has been fully safety and inspected by our service shop. Has a fresh oil and filter change. All the fluids have been checked and topped off. The truck has been gone through mechanically 100%. Has four brand new tires. Is your mission sticker. And this one is 100% ready to go. See more pictures of this truck or one of our other 450 new and used cars, trucks, SUVs, minivans, Wranglers, half tons, three quarter tons, one tons. You name it we got to go to that website right there www.summitauto.com. Full pictures and descriptions of every single vehicle from two locations all at www.summitauto.com. And if you'd like to check out more HD videos you can go to www.youtube.com. Remember to like, subscribe and share. Click the bell notifications to get updates every single day. In fact in a second you'll see a link to subscribe to our YouTube channel on the upper left. A link to more F-150 videos like this one on the upper right. A link to this vehicle on our website on the lower left. And a link to one of our latest YouTube videos on the lower right. Click those, check us out and we're super excited to help you with the Selture Clean 2017. Ford F-150 Super Crew 6.5 foot box Lariat FX4 in shadow black clear coat. Thanks again for checking out the video.


happy to be here with you today, because I would like to talk a little bit about a very important topic that concerns not just specialists, but all of us, because everybody think we use internet at home and we have our router to provide our Wi-Fi at home, so I think it concerns all of us. This is a campaign that FreeSoft, that FreeSoft Foundation Europe has been conducting for a long time, and I've been involved with that since the beginning of this year, and I think that is very important, well, for a number of reasons. As Matias already said, I'm, Lukas Lasott, I'm part of the legal team, if you have some questions you feel free to write me or talk to me here, as I said I like to talk to people, so yeah. So our presentation will be why router free does it important, first we needed to know what is router freedom, and the panorama, the legal and political panorama in Europe, and how to make a stand for router freedom, okay, so we recognize the problem, but we need to do something against it. So yeah, let's see why router freedoms is very important for each one of us. Yes, router freedom basically is the right that each one of us have to choose our own equipment. So if you want to buy and to use our equipment that has free software inside of it, we have this right, and the problem is that some internet service providers in Europe, they don't comply with this right, they put into the market their own routers with some, sometimes with proprietary software inside of it, and they don't allow the customers by contract to use another device, another equipment. So our idea, our stand is to enforce the right of people to choose their own equipment, and the grounds and the reasons for that are pretty convincing. I put just four here, but there are more, this is the most important. So the first one is the freedom of choice. This is the basic thing that everyone has to choose their own equipment. However, think about that. The router, although usually it's very hidden inside your house, and nobody takes a lot of attention to this small piece of equipment, but it's extremely important because all our data, internet data, backups, our business interaction go through the router. So if our router is not with our free software, our digital sovereignty, as Alex mentioned before, it's not secure. So the router has a very important role, having the control over our router has a very important role to keep our digital sovereignty. So the third argument is free competitions and compatibility. This is not in the individual level, but this is a social level, so this is a little bit higher. So having a lot of router providers, router manufacturers. This is good for the price, for the marketing interaction. So this is also very important. And of course, compatibility. So you have the right to choose your own equipment. You can use the equipment that best suits you. And of course, this helps also with even the environment arguments and so on. The security issues is very clear because when you know what is inside your router, you know how to better implement the security measures in order to protect your information and your data. I would like to tell you today that router freedom, it's not an abstract idea for the future. It's in fact a right that has been protected by European regulations since 2015. In Germany, FSCP had a very nice job. I wasn't there, but I knew and I've been working with the results of this effort in order to implement the European regulations in Germany. Since 2015, the National Reality Directive already established in the article number three that everybody has the right to choose the equipment of their choice. However, these general regulations must be internalized in the countries. And sometimes there are some problems and there are some interpretations about that. In 2018, there is a new regulation saying that all the definitions that the countries are implemented on their internal market must comply with the National Reality Directive. And they said that the national regulatory agencies must define some positions in order to comply with that. This 2018 regulation established that BEREC, the Board of Electronic Regulators in Europe, should issue some guidelines in order to establish all the parameters that the national regulatory agencies should adopt in order to comply with the National Reality Directive. However, although it looks very nice, there are some problems. And as Mattia said in the beginning of our meeting today, some internet service providers are raising some discussions in order to not comply, but in order to do something different in the interpretation of these European regulations. So what is happening today in Europe? Today there is a discussion about the network termination point. What is the termination network point? Is the discussion that the router is part of the public communication network or part of your own private network or the end user network. If the router is part of the public network, so it's a piece of property from the ISP. So therefore we have no right to choose our one router. If the router is part of our private network, so we have the right to choose it. So therefore it is very important that the national authorities consider that router is part of the private network because in that case we have the right to choose our own equipment. And what is happening today? So there is a lot of discussion going on saying that router is part of the public communication network, right? And this discussion arrived, achieved the European level and the BEREC is now proposing a public consultation in order to establish the guidelines to where to determine the network termination point. Of course there is very objective arguments that say that having the router in our private network is better because it complies to all those arguments that we heard a little bit earlier. It's better for security, it's better for privacy and in the end it's our right to choose our own equipment. Well in Germany we did a very nice job in the internalization of these rules. So since 2016 there is a law saying that in Germany ISP must allow the users to use their routers. It means in Germany a router is part of our private network, private communication network. Well but FSFE work is about community and of course we are very interested in other countries too. And we have conducted some internal research in order to say, to see what is the panorama, what is the legal and the political state of the art in Europe to see users and consumers in other countries, they have the right to choose their own equipment and what is going on? However, and we need your help, we need the help of the community to refine the data that we collected. And how we did that? Well it's very important for us that we receive the feedback from the community but also to hear that you are also participating in order to defend this very important right to choose in your own equipment. So we have prepared a wiki where we condensate argument procedures in order to talk to the national regulatory agencies in order to raise this topic among your national communities and your national authorities. Please if you have the time check our website, this is the link and of course we have the results of this research that we conducted and feel free to talk to us and say look, in my country the situation is not like this, we have the right or in fact we don't have the right, so how we can do that? So in the wiki there's a lot of material and it's based in our experience in Germany so what I can say that it works and we can made a stand for router freedom. So this is my short presentation about this very important talk and I'm very glad if we can talk later about it. Thank you very much.


Hi everyone, Joe for jaspyscasebreaks.com coming at you with 2019-2020 Panini Chronicles soccer team all addition So it's China exclusive Team all is a big e-commerce site in China So I guess Panini did a exclusive set for them. We've seen team all stuff before I think we got select football team all Edition if you want to check that out, too And we got another break of this up as well jaspyscasebreaks.com two box Random club number one we sold 20 spots. There's only 39 teams So the 40th spot is going to be a $50 a break credit everyone gets two random teams There are going to be an instance probably a few instances where they're going to be Cards that are not represented right here on this team list We'll re-randomize the original list of 20 at the end of the break and then name on top will get any if any They'll get the non Team represented cards if that makes sense. All right big. Thanks to these folks right here. So actually let me Put that original list of 20 into this blank list right here. Let's double up This list right here And you can see the $50 a break credit right on the top So someone is going to get one team and $50 a break credit everyone else will get two teams Let's roll it. Let's randomize it Four and a six ten times for each list. Good luck one two And tenth and final time after ten times We've got Jeremy Agno all the way down to Joe cook All right, four and a six ten times for the 39 teams in the break credit one There's 40 items right here perfect ten times one two And tenth and final time after ten times We've got Westham United Down to man city All right, so Jeremy with Westham Salta de Vigo for Jamie Jeremy with Atletico Madrid Roman with Torino Jeremy with Valencia Carlos with man United Bologna for Ramon or Roman Juan with Barma Calcio 1913 Joe cook with Watford Mike Tower with Napoli Carlos with Newcastle United Joe with spa and Latio Jamie you got my club Liverpool football club champions Juan with Sevilla John with AC Milan Jeremy with Bartholona John with Chelsea Mark with Burnley Would Nessie for Joe real Madrid for Kayla Pedro with Uventus Robert with Southampton mark last spot mojo Cagliari Juan with Bournemouth and Roma. We got mark with Tottenham hot spur Josh with Genoa Pedro with the real bettas bilompi bilompi a I think maybe Josh not heard of that club Everton via villa real for mark You also have Leicester City Robert with inter Milan Phil with Arsenal Mike Tower $50 a break credit Phillip with a few aren't Dina mark with Wolverhampton Joe with Crystal Palace Caleb with Sheffield United and Joe cook with man city. So there you go We're gonna pause the video when we come back We're gonna see if there's any trades and then actually let me alphabetize this by club right here. There we go When we come back, we're gonna see if there's any trades and then the two-box break itself So there you go. Joe for Jasmine's case breaks.com. I'll see you in just a little bit All right, welcome back folks a little bit of trade chatter, but in the end no deals were done. There is the Final list right there Mike, I'll send you that break credit right after I'm done with these two quick boxes here All right Let's pop this open. Let's see what we got There you go. So remember the the exclusives Purple laser. I don't know what those look like. We'll find out exclusive red parallels as well All right, we're gonna start in Italy by the way with the city of I Like this concept by the way the mini boxes in the different leagues. I thought that was a really good idea Like that they're they're adding some some of the rookie card labels here, too. I think that's a good idea There's Latin Ibrahimovic Francesco Totti Kevin lasagna You're an Atalanta fan. Yeah, they almost beat PSG today I don't know if any of these Syria agais have a huge amount of value in the secondary market, but As you saw in the description all cards will ship. I don't think he's quite reached his potential Yeah, like who who is like just has really good technical skills But never really shows up in the game when he needed and and then just always complains about fouls Andrew Wiggins All right, we're in La Liga now turns actually pretty good though David Garcia Kind of hard to see the text against here. Those red parallels are actually the way we've not seen purple lasers, right? But I missed red parallels in this That look red because these red parallels are exclusive to this set Let's just make sure those are taken care of really quick. All right, so let's just kind of recap the first couple boxes really quick Tell oh No, that's on our list. So cards like this will end up going to that lot. That's a Parma, right? Was Parma. I know Parma's on this list. There it is Juan with Parma. You'll get that one The Italian club. I'm not too familiar with there's Genoa That'll be for Josh Proust Sevilla now we're in La Liga Sevilla will be for Juan We know I don't think we have Espanol on this I don't think so. So that'll be part of that that group as well back to Italy Torino I know that's on the list Roman with that and Zlatan where AC Milan will go to John McCall These reds are really hard to see That's that's for Villa Rial. You can see the red there in the text Getafe, these are red too Valencia this will be for Jeremy Agno and the Jose Luis Morales for Levante Who is I don't think is on this list Probably part of that that not the not represented club list Must have added a million clubs in here. All right, Premier League. It's the league that I'm most familiar with Let's see what we got Every time I flip this around it's gonna go the other way. We got Tammy Abram Chelsea we got some red coming up. There's there he is again. So that'll go to John McCall and Chelsea There's our first purple laser parallel. We've got a bunch We got Digible Sidibe for Everton my rivals. That's for Josh proofs We got Aaron Connolly for Brighton Hove Albion Which is not on this list. So that'll be Part of that non numbered or non represented group We've got press proof Antonio Rudiger for Chelsea. It's another one for John McCall We've got Nathaniel Shaloba for Watford. That's gonna go to Joe Cook And we've got Tim Kruel because these are all press proofs Right there. That's for Norwich canary. This is what they call Norwich is not on the list. It'll be part of that group The Norwich stuck in Norwich stayed up, right? All right box to second box And break two of this is in the store Flying to Italy first going to Syria. There's some red right here. There's Thomas and that rookie ticket autograph Stratkosha for Latio. I know Latio is on this list somewhere That goes to Joe Cook We got in that prestige design. We got Andrea Pia Monti for Genoa. It's gonna go to Josh proofs other red Got more red right here Francesco Vickari For Spal, that'll be for Joe Cook There's Stefano Sensi for Inter Where is international? There it is Robert Runkle with Inter Milan And then we've got Andrea Consigli for Salsullo Which I don't think is on this list. No, it's not Cards Here all card ship to ladies and gentlemen, so just reminder on that. Let's go to La Liga So from Italy, we are flying to Spain It's the only place I've been to in Europe is Spain really beautiful Barcelona and and Pamplona as well That's Luis Juarez right there Matias Vargas We got Figo Luis Here are some red parallels right there There's Garrett Bale Where's Garrett Bale gonna go ladies and gentlemen? There we go for get off it. We got Nemanja Max Simovic I thought get off it was on this list. Maybe it's not I don't see it might be on this list right there There is a Ines Kunal Ford for Valadolid, Ria Valadolid We got another red parallel. We got Rafinha Midfielder for a Celta de Vigo. I know that's on the list That is for Jamie There's Garrett Bale any idea where he's gonna go. He's definitely out of favor at Real Madrid him and Zidane don't seem to get along Real Madrid goes to Caleb, but I think he's only 33ish He may find some life somewhere else That's atletico Bilbao I don't think it's on the list, but that'll be part of that group and there's Mallorca Takefuza Kubo I don't think Mallorca is on this list And then there's Luis Suarez for Barcelona that will be For Jeremy Agno to the Premier League. I don't know if I don't know if Real If he goes back to the Premier League does he? Could Might be expensive that that might be the problem Who's gonna pay for Garrett Bale? I think he has some years to give. He didn't play a lot this season So he's still has some years left But at what price point? That'd be the interesting thing for Man United. Here's a red parallel Declan Rice for Westam United. It'll be for Jeremy Agno And here are your purple lasers. We got David Silva I think he's moving on right from Manchester City. That will be for Joe Cook We've got Cesar as pelicueta for Chelsea. That's gonna go to John McCall We've got Morgan Schneiderlin for Everton. That will be for Josh Proust David Silva going to Lazio. That's what Joey's saying Hugo Lloris World Cup winner Tottenham Mark Bissette that and finally we got Isaac Hayden for Newcastle and that will be for I know Newcastle's on my list here There it is Carlos We with Newcastle. There you go, ladies and gentlemen as typical with With most team all sets No hits. No like relics or autographs. I think those are awfully rare if there's any at all But it's those parallels that we're really chasing. We haven't seen too many big names in this at least from the English Premier League side of things I don't think we saw too many big names But we can keep trying. Jaspyscasebreaks.com. We've got more. I'm Joe. I'll see you next time for the next one. Bye. Bye


Hello everyone, I think it's time to start our talk anyone is here Thank you all of you for coming here or Listeners in this talk. It's really nice to see faces not in virtual And thank you the organizer on the CNCF or a 13 hour or talk to be To be here. We are to talk about how attackers use exposed Prometheus And server to exploit coon at this class there My name is Miguel and it I work as a security researcher for the last five years in different topics like cloud security or Sintending content to social engineer or also for detection and I work with David a Torres is manager of Prometheus team at 60 and we're here to try to explain this man It's like we put more and more layers more and more software But if you fail in one point everything is Broken and with that is the sports matrix. Okay, it's a typical Insecurity is very typical to to talk about this, but I think repeat and again and again and again in the next in the next time well I Lost The first thing to mention here is the disclaimer We don't like to release any new vulnerability or CVE to Kubernetes or Prometheus here is for common sense to educational purpose and we don't like to Provoke any attack for real customers and also we Recommend to follow always the secretive practices in Kubernetes and also to use Prometheus to monitor everything But please don't open the door to attackers or expose your Credential for free because it's very easy. You are an attacker to get to get access Why it's important or why someone likes to do Kubernetes fingerprinting. Well, if you are a Pentesting or ethical hacking or you're working security The first thing to do is when you have a target is to gather all information as you can Okay, you need this information to choose the best tool the best technique to get access or exploit this these things and it's normal to match with CVE or Non-vulnerabilities with the exactly tool and you get access is the first step to in detail access You need to gather information and we try to explain in this talk is possible with the metrics of Prometheus to gather this information and Create the plan to to attack a cluster of coronaries And this is not it's not something new if in the past this problem with Tesla is Forced because expose our Kubernetes dashboard. It's something I don't know if you you like to protect everything and you follow the secretive practices and you I don't know we use a lot of security techniques or Application if you expose your credential in clear text doesn't matter everything you Don't you access for free to your cloud provider and start to Crypto mining for example, but this one example of the things you can do and I like to remark this because in Spain No, but on the part that said this aquel que no conoce la historia está condenado repetir like you you work in the Incident response team or digital forensics the most important thing to do when you create the report is to to know what happened with this In an incident and you need to learn from the past to don't repeat and again in the future And this is important because what happened when Prometheus you expose again and Like the credential or not? well Also, we like to mention that this is not the first thing on call the first guys that Talk about this you do a quick search in Google you can see an issue from this shop that discuss this the same topic It's like if you use Prometheus and expose the the dashboard is Like your credential or you have a problem in the future or not? Also J frog Write this amazing Article about this problem, but more focus on the secrets not in the pivoting on gathering information, but this is really nice and Finally, we are not the first a speaker talking about hacking monitoring. This is something that is cool, but As I mentioned before The history repeat and repeat and they doesn't matter if you explain once or two. That's a matter The first thing to do is it's a real scenario to Access to a Prometheus exposed to the internet well, you need to configure is not by default the dashboard open to the world But the configuration is not enabled and if you configure the dashboard and sure Joe Configure the authentication or is free to the attackers to make queries or access to the Prometheus metrics and if you do a simple to Google dorking you can see some several results with real Prometheus service post And also if you are in different security conferences, this is very typical use other sessions engines like soda and then she's off To see the impact is a real impact because there are a lot of servers exposed or not With sudden is most simple because you use to filter the Fabricon but as you can see a lot of server are exposed with this introduction I like to share the hands to my friend and I OK? There we go. OK. We are under, and you guys know the ways of open web service, which is published, and we will get that way from there. But in a way, we don't know when we will be able to do it in time. So, the ground are not on the basis of this. But I'm not on the basis of this. So, let's start looking for where to use the code. This is not a database caused by an old spoiler. It is not a spoiler. It is not. It is not. It is not. It gives us information about even the chassis sometimes, the board vendor, but most important, it is giving us information about the system vendor or the product name. Cloud provider, it is. So, we are starting to have some information. We know that after behind this example.com web page, we know that maybe it is AWS or iShare, we know already what is the provider. And we are starting to add things to our map. This will be our map that we will use later on to make a route for attacking. We can have more information. This is another node explorer metric called node underscore network underscore info. And we can filter by all the labeled device like F or net. So, we have all the physical interfaces of the machine. And this is giving us some important information, like not only the physical address of the node, but also the availability zone of the service provider on the VPC. So, we will have also information. Also, these are from the cube state metrics supporter. We have also information about the hostname with the cube underscore node underscore info.  As you know, these load balancer services can be found at the service provider implement behind something to expose to the public. And we can also have information about the ingressers that we have. This is interesting because the ingressers are telling us which reverse proxy we are attacking. If we are attacking slash api slash cartdb slash product, it goes to one service in the cluster and if we are going to make a slash catalog slash id, we are going to another service in the cluster. So, we are starting to make a map of networking inside the cluster. We are starting to have more information now. We are actually starting to make the most important thing is the root of attack. We know that behind these example.com slash api slash whatever, it goes to as ingress and that ingress go to a service provider. And we can do this with different points of the api. We have also information about the nodes and even the name spaces. And we are starting to add new things to our map. But let's go further. With KSM, we can have the name spaces, we can have the workloads, the deployment, the stable sets, even the pods, the container inside the pods. We are starting to have real information of what are the names of the applications, what are the name of the workloads that we have, what are the name of the containers, what are the names of the cron jobs that are being executed. And this is giving us a real map of what is behind. But not only of the user workloads. As we can see in this metric, this is from the Kubernetes components metrics that they expose natively. We have the Kubernetes underscore build underscore info. And this is giving us information on the version, except version, not only major, minor, even the git commit and the build date of each of the components on the Kubernetes control plane. And this is important because now we are having more information, but we can start to add vulnerabilities for the components of the control plane. We cannot access that now, but we are starting to add more things to the map. Let's change the dimension. We are talking about Kubernetes, but let's talk what is behind the machine. We can have information about both the image that the node is running and the kernel version. And this is important because we are adding more known vulnerabilities, not only of the Kubernetes cluster, but also from both the kernel version and the image of the operating system that it's using. Let's go further. Let's go for applications. We are having Kubernetes metrics is giving us more information about the image and the tag of the containers running in the cluster. So we know which image is running, which tag they are using, which hash it is, and even we know also the registry that they are using. If we even want more information some applications are even exposing the more specific information about the version or even the configuration options in some custom metrics like Prometheus underscore build underscore info. And now it is when it is starting to get interesting because we are adding even more known vulnerabilities for each application that we can find in the cluster. And last but not least, we are also having information about the secrets. The name of the secret. Yes, but we are also having information about the annotations and the content of the annotations of the secrets. Why this is important? I don't know if you know that in some versions of Kube CTL Kube CTL was adding an annotation of the current applied configuration that was exposing in plain text directly the content of the secret. So all the roles and all the things to service accounts no sense because the content of the secrets was being exposed directly in your KSM Prometheus metric. So we already have a lot of information and we already have a nice map of the cluster that we want to attack. Do you want to know something funny? We gather all this information most probably without anyone noticing because Prometheus has the capacity of logging all the queries made by the API on the frontend but unfortunately this is disabled by default and you can even check if this is disabled or enabled with this metric like Prometheus underscore engine underscore query underscore lock underscore enable. If this is a zero we are in Ninja mode and no one realized anyone locked anything that we did to gather all this information but I'm not the hacker one I'm just the Prometheus guy and I will call now my friend from the secure part Miguel you drive from here okay nothing to do okay we have all this information it's quite simple to make queries, gather all the information map everything you don't need to initialize to know if this component is vulnerable or not, if you can escape or not because you have the information we like to expose three different examples on how this is possible or this fictitious but not it's possible to achieve the goal of the attacker it's like I like to leak data or I like to crypto mining or I like ransomware it depends on the goal of the techniques or you can perform different paths to get the access and I think it's interesting to see what happened with all this information the first scenario is with like data okay this is an example of a real scenario everything is fine we follow the secretive practices we don't have any CVS or known CVS in any of this part the components are fine the nodes are fine the applications are fine and we have an IPI to make queries but I like to leak the data inside without authorization the first thing to do is to use for example a SQL map to try to do some injection in the IPI and it's possible to do or not it depends but in case you can't what is the registry if it's a third party or private I know the images and I can perform some social engineer to get access to a third party registry for example because I like to upload a malicious images in the continued delivery to get access because the other way is impossible to initial access in some cases in some scenarios we craft the imaging with a vulnerability to get access and perform the other steps to do a privilege or something like that but in this case you need to focus on the supplication because the other way you need to 0 days or something like that but in this case we need to do by supplication attacks and you can use beef to create a phishing attack or you can do something similar to tipo squatting to upload a new container with similar names I don't know but this is the first scenario the next one is possible to be a register or not you have an application and you know we are promising you that the ingress send the command to an application that is vulnerable to love4j and you have initial access and you know you are inside a container but this container also have vulnerability in the node that allow the elevate privilege to what is the goal in this scenario the crypto mining you like the AWS credential and you need to do these steps to get access to the node and following or seeing the config files or in the environmental variables if the AWS key or the cloud provider key are in this place like the example of Kubernetes dashboard it's a clear test in this case we need to do these steps this is a long path if you like to do very short you can do the same example with Kubernetes it's possible if you use AWS and you configure your secrets it's exposed by default in the dashboard and you make the query and get access to the credential and it's fine you don't need to perform any need to access or any other step because it's unclear ok this is example but is in the documentation please be careful if you expose the Prometheus it's possible that expose your secrets and it's something known for the for the community and again don't open the door ok and finally another scenario run somewhere it's a bit strange in Kubernetes cluster it's much for traditional environmental database that you cipher the data and ask for a rescue and in this example is a bit different you know an application have a vulnerability in the initial access via spring cloud you can inject a command via htb and access to the to the container and also you detect with Prometheus that one component are vulnerable at one tv and you can use to change the name space or get access to the secrets and access to the data in the database the problem here is to ask for the money ok I don't like to give any advice or anything to know who is possible to ask for the money in this case because you need something to I don't know deploy a pod with user interface or something like that but it's something similar you get the information from Prometheus and prepare the journey to get access ok this is a summary of these three attacks or three scenarios and I like to summary that is for security it's important to protect all the information you are exposed ok that doesn't matter if it's a metric or it's a credential or it's a server application I don't know but it's important because in security it's a long battle it's red team versus blue team and all the time we are fighting or cat and dogs it's a different idea but a long battle if you expose your credential or you expose your Prometheus or expose the dashboard you change to a speedrun it's very simple and we like to finish with the normal recommendation in each step ok all this talk is about follow the security practices in Kubernetes follow the security practices in class in the code, in the deployment you have these layers and sometimes we forget the metrics and the security metrics in this talk we don't like to explain any vulnerability to Prometheus itself because it's not the scope but it's also something to keep in mind and we like to share this recommendation and I think that's all thank you so much if you have any questions


Hello everyone, I'm Gabriel and I'm a Group Product Manager for Bookend.com and today I will be presenting some ideas and experiences we have had when developing platform products. I want to thank the products school for the opportunity to have this space and share some experiences with you and yeah feel free to connect via LinkedIn if you are interested. So I bet you have never heard about technical depth, non-functional requirements, velocity or reliability and performance and you have never struggled with these kind of things having slow velocity or low reliability for products that you have built and then understanding how to address this while delivering functionality. So today what I want to share with you is over a period of two years we have been dealing with these situations and we have developed some strategies when building this security platform at Booking that have resulted us to have some benefits along the way. What I say we is because I didn't develop this alone. This is a constant work with team members, team leads, engineer managers, engineers. We have tried different things together, documented the benefits, adjusted and continue to operate so we can find out what works for us and we have been learning over time. The idea here is that if you can get some of these ideas and use it to understand how to address your particular challenges maybe you don't have to take two years to achieve this but in less time. Again every challenge is specific to your context and circumstances but who knows maybe some of these can help you to navigate your own challenges. My name is Gabriel. Again I'm Group Product Manager at Booking.com. I've been there for three years. I've been in the product management area for more than five or eight years. Don't remember now. I have a technical background, electronics engineering and I'm really happy to work on platform oriented products that enable developers or analysts and security engineers to achieve their objectives. So the problem we're going to be looking today when working on platforms is very important to have to clearly determine the return of investment. So if we have a negative return of investment because the users of the platform don't trust it and there is low adoption we are in trouble. Having low adoption means that we cannot rely on the network effects of the platform. The more users the platform has the more return of investments there is because the value is distributed across multiple teams. The root cause of this sometimes is focus on delivering fast functionality and trying to please multiple stakeholders, users, multiple teams, management while sacrificing quality. This then results in velocity degradation and losing the capacity to adjust the platform to continue evolving it to satisfy more problems or maintain its current guarantees and therefore reduce its utilization and facing out. Without booking we wanted to understand how to avoid this but let's go back a little bit to platforms and non-functional requirements or quality characteristics. Today the platform we're going to dissect is going to be the transport ship. The transport ship is a platform for maritime commerce. It can be used by companies to move goods from one place to another. And the platform has several kinds of systems. It has navigation systems that allows the ship to be routed to different areas. It has cargo systems that allow the ship to transport frozen goods or liquids or dangerous materials and to distribute it properly to utilize the space better. And it has propulsion systems that is very straightforward. So let's talk about the quality characteristics. You may have heard this as non-functional characteristics, non-functional requirements and there are many kinds of quality characteristics out there. Some of the most known ones are in the screen, security, maintainability, reliability, performance. I like to refer and to use as a reference the ISO 2500010. There you have a list of quality attributes and sub-characteristics that you can use as a reference. But every team, every product will define what is more important for them. So let's put these two together. So we continue with our ship. We have the navigation systems. For example, it is important for the navigation system to be reliable, to be secure so it cannot be tampered, to be extensible so we can build more on top of it and then make the ship able to navigate across other kind of routes. It has to be maintainable. It has a good usability, therefore allowing the operator to reduce the amount of error, for example, they can input there. And similar to the cargo systems and propulsion systems, as you see, the quality characteristics can be different for the different areas of the platform. However, working on quality characteristics is not going to increase the capability of the platform at a functional level. This is very important to distinguish. Increasing functionality could be, for example, making sure that the ship can navigate through bad weather or at night. This means that maybe the navigation systems have to be extended. Radar has to be set up, lights needs to be put across the ship. This is separate from the quality characteristics. Quality characteristics keeps the current functionality, but makes it work better. For example, a calculator that can produce the result in seconds versus a calculator that can produce the result in minutes. They both produce the correct result, this is the functionality. But each one of them do it in a different amount of time. This is performance. So this is an example of quality characteristic. So let's talk about the challenge again. Remember, having a sustainable platform development so we don't fall into the velocity degradation. Our structure at booking initially was the classical team approach. We have three teams, navigation team, propulsion and cargo team. Each one has a group of developers, a product manager, a team lead, and each one has its own concern. However, we try to create value through delivering outcomes that impact the business. For example, we have the increase revenue by 15% by having the capability to route the ship through bad weather routes. This is something that will impact both navigation and propulsion teams. In order to achieve this, we need to align their backlogs and therefore guarantee that they both align and work together and then produce each one's part so we can deliver this outcome and the ship is able to navigate through bad weather routes. Anyway, outcome oriented backlogs is a topic that is very well documented and discussed in the Silicon Valley product group and I recommend a book that is called Inspired by Marty Gagan, the founder of Silicon Valley product group is an awesome reference to read. We're not going to enter into details about outcome roadmaps here, but I just want to present that this has a challenge. Takeholders wants us to deliver functionality fast. It's difficult to align teams. So in the end, we favor functionality development over quality characteristics, maintenance and development. We create knowledge silos by having teams distributed across different areas. And there may be a problem with motivation of team members. Imagine that one of these teams contains components that are very boring to work with. Team members feel they are stuck there. They have to maintain these kind of systems that are always breaking. It's an old technology, not easy to evolve. So this creates team rotation, knowledge departure, and a lot sort of issues. In summary, and very summarized way, we lean towards delivering fast functionality, creating quality depth. This creates a problem that we have issues to fix constantly, which steals time from development functionality. This is a vicious cycle. It continues to increase, the number of issues steals time from development, and therefore we see velocity degrading. So we thought, okay, what can we do about it? Some new approach, different approach, let's think out of the box. So we said, okay, let's reorganize our teams. Let's refocus on this problem from a different perspective.  Task forces mean that all team members are part of the same team. We are the platform team. Navigation propulsion and cargo systems are part of the ship of the same platform. All the team members contribute to all of these systems. But not at the same time and all the time. We identify the outcomes that we want to achieve, and we form a particular task force with the required amount of team members and the skills and also the motivation from the team members to work on this area, and therefore allow them to solve the problem within the fine set of constraints. So for example, we create here three task forces, one for each outcome or maybe two task forces and then we leave the other outcome in the backlog. What improvement did we see here? Now we have one team, there is alignment. Everybody understands this is a ship, it's not navigation systems or cargo systems or propulsion systems, it's a whole ship and we have an impact on the business. We can think about outcomes. We can work focused, so increased focus on delivering this outcome, regardless of which components we are addressing. We don't have backlogs to align. We have increased motivation because we are learning and contributing in different components. So yes, there may be components that are very difficult to contribute towards, but I'm not working all the time the whole year on these components, I'm shifting. However this created some other problems. We identified that the ship, yes, it has to be extended. We want to continue to evolve the platform and work towards achieving outcomes, business outcomes, but we have to keep the ship running. The ship, and in this case, needs to interoperate with other ecosystems, and these ecosystems may be changing. So interoperability is important, libraries that we use change. We need to make small adjustments. There are concerns regarding high availability, vulnerabilities, business continuity. There are drills happening like data center outages. This requires our team to dedicate some time. The whole team was distributed across task forces. Therefore, we didn't have focus on this, and this started to still focus from the task forces, therefore reducing one of the improvements we achieved. So we said, okay, what else can we do? How can we organize to this? And this is our second attempt. We introduced the concept of continuous operations in addition to task forces. We discussed together as a team and we said, okay, let's try something. Let's put a person that or a couple of persons that for a week or a particular period of time that you decide is going to be working only on these small tasks to guarantee continuous operations of the platform. While the rest of the people continues to work on task forces, focusing their time there, focusing their attention. And then we have, yeah, we have all the benefits of the task force and also we can guarantee that the platform continues to operate. However, we found out that these challenges sometimes are bigger than what maybe one or two persons can handle in a particular amount of time. Also, the platform has multiple components and not all the components are entirely known by one or two persons the whole time. So now these persons needed to, it was very hard to be on this weak rotation to consume the continuous operations backlog. And the team didn't start to look at this in not a very nice way. Nobody wanted to be in this rotation. Although everyone understood that this is beneficial. So we needed to say, okay, what can we do about this? Let's think of, this is in a span of months and maybe years. What can we do? The third part of the attempt is to create bases and do basekeeping. Okay, what does this mean? Maybe we're now going back to the classical team approach where a base is a team and holds certain components and has members that are dedicated to work on these components. But no, we made a complete turn and now we have a different mentality. We have a different mindset. Bases owns components. The component distribution and how we achieve something balanced is something that is outside of this conversation. It can be a complete different topic. But think about, we have these three bases. Navigation base, propulsion base, cargo base that has an administrative function. Base members report to the base lead or the team lead similar to the classical team approach. But now they also are owners of these components but not on the functional development, not on the functional evolution. Bases are responsible for the quality characteristics of these components and they have a backlog as well. They have a quality characteristics backlog or quality outcomes, right? Let's see how this is combined with everything else. While a person is not part of a base, sorry, while a person is not part of a task force, they are working on their base, on the base tasks, on quality outcomes and small tasks that needs to be addressed constantly that come from this task that we're not able to be addressed immediately by the continuous operation person on rotation. This continuous operation rotation continues to be there. It continues to address some small tasks that can be handled within hours. But now it serves as well as a triage layer. It can send tasks to be dealt by the specific base that owns this component for things that maybe are not urgent and are more complex. In addition, every base member can participate in task forces and task forces will be contributing, will be working with components that are across multiple bases because their task is to achieve an outcome that requires some functional development, some evolution to increase capabilities of the platform, same outcomes that we have said before. So for example, there is a task force that wants to achieve outcome one, the revenue increase by routing through bad weather. This requires two touch components on the navigation base and the propulsion base. And this of course can be dealt with by team base members from the three bases, based on their interest, based on their skills and based on the complexity of the problem. Whenever the task is done, the outcome is delivered, this task force is disbanded and the team members return to their own base and continue working on their backlog and maybe part of a new task force later on. So what did we achieve with this? Now we have, with task forces, we reduce silos. We have more focus and motivation because team members can work with other team members, other base members. They can increase their knowledge in a particular technology. They can just work in something that is more interesting for them as well, that are not the components their own, is refreshing as well. They can share knowledge. The bases, instead, create more ownership for these components. We know who can handle specific incidents on the platform. We have a grasp on the quality of these components. We increase the knowledge and understand what academic part needs to be studied in order to guarantee that we can continue to evolve these bases and these technologies and these components by creating sense of ownership. And finally, the operations rotation, which continues to be very relevant now, is a task that is more relaxing in a sense. Assesses information, assesses tasks that need to be performed, understand these tasks can be performed immediately, so not to remove still focus from bases or task forces and creates a triage, so tasks that require more complexity, more specific knowledge, but are not urgent, can be addressed by the bases themselves, keeping our users happy with the delivery guarantees, the quality levels, and functionality development. So putting all together how this operates with us. Remember, base members are in a base until they participate in a task force. When the task force finishes, they return to the same base and continue working on the base backlog. The task force lead is the keeper of this backlog. They guarantee that some time is dedicated to the base for base keeping purposes, adjusting the quality characteristics, achieving quality outcomes, and they together work with the product managers to identify, based on a particular set of time, a quarter or a semester, how we are going to deal with both worlds, how much time are we going to dedicate in a semester on a quarter for base keeping, and how much time we're going to dedicate for task forces. In our case, for example, every semester we decide, every, for example, for H1 in 2022, we're going to use every four weeks, we're going to use one whole week for base keeping. This is everyone stop doing what they're doing in the task forces and go that week to their bases and work in the base's backlog. After that week finishes, they return to the task forces. Some members that are not part of a task force at the time continue working in the base, but this gives some afterburn, let's say, to the base keeping assert every period of time. Product managers understand which outcomes we are going to be delivering. Task forces are working on these outcomes and they know some time from this semester is going to be dedicated for base keeping, so they can adjust the estimations. So this way, again, what we achieve? Sustainable velocity. We have task forces to dedicate on capabilities, evolution and functional characteristics, bases to guarantee a certain level of ownership and increased quality characteristics management and the operations to guarantee that the delivery guarantees, sorry, to guarantee that we can provide the users with some assurance that the ship will continue to operate constantly regardless of what happens in the context situation in the ecosystem. So here it is, a very summarized version of everything put together. As you see, each base has defined their own, the most important quality characteristics from their perspective. Navigation, for example, very important reliability, security extensibility, maintainability and usability. Cargo, for example, has other concerns. They are concerned about safety, resource utilization efficiency and interoperability. And for example, propulsion base is working on a particular outcome, quality outcome, to increase the profit in 2% by having better fuel efficiency in their propulsion systems, increasing the fuel efficiency by 10%. This will give the business a 2% increase in profits. This is entirely quality characteristic, non-functional. Task forces, on the other hand, are guaranteed that we can navigate through bad weather by evolving navigation and propulsion systems or maybe increase our market share by consuming other kind of products like dangerous liquids. Therefore cargo systems need to be addressed, but also this has an impact in propulsion and navigations because of compliance, and therefore these systems need to be evolved. As you see, everything put together resembles the classical team approach, but it's a complete different mentality that has allowed us to have less uncertainty on dedicating time to quality characteristics and therefore guaranteeing that we can continue to evolve the platform sustainably over time. Again, another recommendation. There is another book that is like a sequel to Inspired that is called Empowered, also by Marty Kagan. And Marty Kagan in this case talks about this kind of organization, how to guarantee that each member of the team is empowered to deliver the whole capacity they have in their role. Very interesting read as well. And this is the end. Thank you very much. Maybe it was too fast, but probably you'll be able to rewind and see it again some of the parts. I hope it has been interesting, not too boring, and that you can use this as a reference foot for thought to maybe have new ideas on how to address some of these challenges that we're constantly facing when developing platforms. Thank you very much and hope to see you around. Bye-bye.