It's great to have you here in your free time in the evening. My name is Andrea Pizzoni. I work for Accenture here in Singapore. I know Accenture is usually not a company that is famous for doing deep learning or machine learning in general. But if I may say so myself, here in Singapore we actually do have quite a strong deep learning and innovation group. And really to put the point across, again, consultancy companies are usually famous for slides. I have no slides today, zero slides. I have a Python notebook and we're going to go walk through some code together. So the idea here is you want to deploy some models on your mobile phone. And that's what I basically do for most of my day job. When I tell these two people, they usually tell me, that sounds quite crazy. I usually know that mobile cannot really support deep learning stuff. If you have inception, if you have big models, they're going to take hundreds of megabytes of space. They're going to take gigabytes of RAM on your computer, so you need a big powerful cloud instance to run them. And that's actually not really true. Again, as Martin said, you have a lot of weapons now at your disposal. First of all, architectures have evolved to make things smaller and faster. And now even you want to really be crazy and use a very big architecture. There are methods such as quantization and weight rounding that allow you to actually do it. So we're going to take it from the top here. We're going to take it very slow. And I know here in this meetup usually we use Keras. So let's say we want to use Keras and we want to deploy our Keras model on mobile. The idea here is that unfortunately, if you know how Keras works, after the Keras training is done, it usually tries to save your model in its own format. And we need to try to convert this into something that then the mobile is already sort of equipped to read, which would be the TensorFlow Protobuf format. That is the TensorFlow sort of vanilla TensorFlow way of saving models. So fortunately we can do that. Now Keras supports TensorFlow backends. So we can do some magic and try to make Keras models saved in TensorFlow format. So what we do is, again, we try to build a very, very simple networking in Keras just for show. We're stacking some convolutional layers, some relus. If you really have a good eye, you can see something weird is happening here with some lambda. Don't worry about it for now. We're going to get back to that later. Again, very simple model. We're going to train it. We're not going to do it right now on this very small MacBook as the CPU will take a while to train a good model. But anyways, the point is, while you train your model, you want to be able to save whatever you're training, whatever parameter you're creating in TensorFlow format. How to do that? Fortunately, Keras has a very powerful tool that are Keras callbacks. Keras callbacks, I think Sam talked about Keras callbacks last time, are something that get called automatically whenever a special event happens in Keras. In this case, for example, we can create a callback that happens every time an epoch of learning, an epoch of learning is done. What we want this callback to do is, once we have set up our backend in Keras, which is the TensorFlow backend, and we have sort of got the session out of this backend, we want this session, the TensorFlow session, to save the model, to save whatever the Keras sort of front-end has trained so far. So this is very, very simple to do. We create a class to create a callback. We initialize this class with a saver and a session. And on epoch end, so every epoch, we're going to save whatever Keras has trained. And this save model, this save function is simply a TensorFlow function. You can go online on the TensorFlow documentation and find it.  Checkpoint files are the way TensorFlow saves a model in progress. So you have not finished. You have checkpoint files. You cannot just use these checkpoint files on mobile, but it's something. You're basically already converted your Keras model to TensorFlow. So now that we have checkpoint files, let's say we finished our training. So we have a lot of checkpoint files that were saved at every epoch. We want to take maybe the last checkpoint and really freeze the model in TensorFlow format so that we can deploy it on our mobile platform. Okay, there is a few ways we can do that. It's actually not too difficult. One way is actually we can start writing the definition of our TensorFlow graph in Protobuf file just by calling writeGraph from TensorFlow. That's it. Very simple. Since we already sort of extracted the session from Keras front end, we extracted the session from TensorFlow back end. So we can use this session to just say print out in TensorFlow in PB file or PBTXT file, whatever it's in the session. So this is very powerful and can be very funny as the names that you use to sort of choose your models, your modeling Keras, whatever name you use for your convenience. If you save your model in PBTXT file, these names will still be there. So you can actually then, if you choose to save your model in PBTXT file, you can open double click on this PBTXT and really see whatever layers, TensorFlow vanilla layers, the Keras model created once you call it. But say, even simpler than that, we don't want to sort of save the TensorFlow graph definition separately. We just want to create our Keras model. We want to train it and we want to freeze it in TensorFlow format all by using Python. We don't want to use any sort of Google tools to do that. You can use Google tools. There is a very famous tool called Freeze Graph. It's a C++ utility. It has Python bindings. Link is over here if you want to check it. This notebook file will be put on GitHub. Anyways, what we try to do here, we don't want to use too many tools that we don't know exactly what they do. We want to try to do things ourselves. So let's say we trained our Keras model. We now have a lot of checkpoints. What we want to do is we want to prepare our model for freezing and then we want to actually freeze it. So when I say prepare the model from freezing, there is a few things that we might want to do. We want to maybe clear out whatever remain there is left and we want to maybe, if we have some assignments of specific devices assigned to every node, we might want to remove that. So for example, if we train the model on the GPU and for whatever reason we really assigned this node on the GPU, then we don't want these assignments to remain there when we actually deploy on mobile because we're going to use CPU to do inference on mobile. So this is very simple again. You get your model definition. You get the checkpoints and this is again another TensorFlow method to get the checkpoints.  And again, what you do is you set up a TensorFlow session and you link it to Keras and you restore your checkpoint in memory after you finish the training and you try to write it down in PB file. So here basically what you can do is you can load the checkpoint, remove all the assignments of hardware or such and then save it back. Once you have done all your pre-processing, you can freeze the model. In order to freeze the model, you have to actually specify a few things. You have to tell TensorFlow which layer is your input layer and which layer is your output layer. There can be more than one output layer. For example, object detection models sometimes have three output layers to say where the objects are, which objects are those and how certain I am that these objects are there. So that's not a problem. You just specify multiple sort of output layers but in this case we created a very simple model again and the only output layer that we have is a simple softmax node. So nothing we are happening here. So what we do, again, we have our checkpoint that we saved through our Keras callback. We load our checkpoint and we specify what is the input, what is the output and simply at this point we can load the weights that we have been training through Keras and save them in proto buffer format. In order to do that, we simply have to, again, use the TensorFlow playing method, convert variable to constant that really frees whatever you have created. So once everything is converted into constant, it can be put into a pb file through the method of write serialize the string. And again, all these methods have documentation and we can link it to the GitHub and you can go and check it out. As you can see, it's very, very simple to actually get a Keras model, save it in proto buffer file and just deploy it like that. So at this point what we have is a sort of standard vanilla pb TensorFlow model file. What we want to do now is we want to try to understand exactly what happened. We're not really just happy to save it and say hopefully everything is going to be okay. Unfortunately Google provides a lot of tools to do that. First we're going to use is the tool called Samurai's Graph. This is a C++ tool that comes integrated with TensorFlow. When I say it's a C++ tool that comes integrated with TensorFlow, it means you actually have to go download TensorFlow source. You cannot just do pip install TensorFlow on your Python. Unfortunately that's not enough.  There are very detailed instructions to do that. And you have access to all these sort of nice utilities such as Samurai's Graph. Samurai's Graph allows you to actually take a pb file and really inspect it to understand whatever operations you have in the file, which ones are your input layers, which one are your output layers, even which one is the estimation of the power necessary to run this graph. So you have an estimation of exactly how much computational power you need to run this graph based on basically the input size and the operations that go through. So this is, for example, let's say we try to summarize the previous graph that we created. And as you can see, we get one possible input, the image input layer, not a surprise, and one possible output that is the class result softmax layer that we created before. And we have just a few operations, right? We have some constant operations, some relu, some convolution. Everything seems fine. Actually, there is some catches, but we can talk about this shortly. Again, this works for every kind of model. If you have, for example, three output layers instead of just one output layer, it's still gonna tell you. For example, this is the standard output of a summarized graph for an object detector. So you have still one input, image input. You are trying to create an object detection model. You pass an image to it. And you have three outputs. So again, the bounding box, that is where the object stays. The probability score in this case means how certain the model is that the object is there. And the class index, which means which object the network thinks is there. And again, you have your operations and everything seems fine. But actually, for both of these models, if we tried to deploy those on mobile, it would not work. We would have some bugs and the application would crash. So what I would like you to understand is you can actually use this C++ tool to actually diagnose what's going on in your model and understand if you're gonna be able to deploy it on mobile. So, for example, on the first model that we have right here, we can look at the operations and we see there is a random uniform operation. And I just made a big mess. And there is a random uniform operation. We think, okay, why do we need a random uniform? Random uniform is an operation that creates a random number with the uniform probability. So it doesn't seem something like we need for our inference. And in fact, TensorFlow doesn't ship with this operation on mobile. So if you compile TensorFlow on mobile, by default it will not have this operation inside because it's not used during inference. So then when you go and deploy your model on mobile, it will not work because this operation is not there. So either we can try to recompile TensorFlow with this operation inside or we need to try to understand what's going on and why we have a random uniform in our model if we're not gonna use it for inference. Turns out random uniform is used for dropout. So dropout means you have a layer on which with a uniform distribution you're gonna turn on and turn off some of the nodes. So, for example, in order to avoid overfitting, you might want to turn off 20% of the nodes every time, but you don't know which nodes because you don't want to turn off always the same nodes. So you have a random number generator that tells you which nodes you want to turn off. So, of course, dropout is only used in training. It's not useful in inference, so that's why TensorFlow doesn't ship with it. So what we would like to do now is we would like to remove the dropout layer from our network. So let's think about it. We want to remove our dropout layer, but is that safe to do? So, again, by sort of just a simple understanding, that would not be really a safe thing to do. You are taking out 20% of the nodes every time you are passing through the dropout layer and now that you are in inference, you're not gonna take out these nodes anymore. So, after this layer, whatever comes after is gonna be in a different shape. It's gonna be different because before 20% of your input was cut off and now you're gonna have 100% of your input coming through. So, actually, what historically has been done when you remove dropout is you substitute it with a multiplication layer so that you actually say if you remove 20% of your nodes in training, you want to maybe sort of multiply all your inputs by 0.8 so that they are still at scale when you do the inference. However, this is not really convenient. So, fortunately, TensorFlow actually doesn't... doesn't actually do dropout. It does something called the inverse dropout. So, without you knowing, TensorFlow is already rescaling things at training time, meaning that whatever if you deploy, for example, take out 20% of your nodes, the inputs are already scaled by 1.2, meaning that then when we have our inference time, we can just safely remove dropout without any issue because the weights were already scaled before we started. So, again, we know we have TensorFlow. We can remove dropout. How to remove dropout? Again, there are a few methods to do that. Some of them are very funny.  Or maybe you just parcel the nodes, you see whatever has dropout in the name, you remove it. These are sort of not really orthodox methods to remove dropout. The safest way to do it is if you have your model, you already know you're putting dropout over there. Simply write your function so that you can have a boolean when you write the function to get your model. If you are training, you build the model with dropout. If you are not training, you build the model with dropout. So, let's take a look and see. And in this case, we are building, again, a very simple model with a boolean. If we are training, dropout gets put in. If we are not training, dropout is excluded. This way, actually, when then we build the model and we freeze it, dropout is going to be out. And we're not going to have any problem with random uniform. We don't have to recompile TensorFlow with random uniform. We are just safe to use it. Here, again, nice bonus. We're using lambda here, meaning we want to use the dropout layer directly from TensorFlow vanilla. This is just because you want to really make sure of what's going on when you try to remove layers and you don't want to just assume how things are going. So, if you are sure TensorFlow is using inverted dropout, use the vanilla layer. Actually, Keras also uses inverted dropout, but for dropout, Keras does something a bit weird as it splits the graph in two. One direction is for training. One direction is for inference. So, again, you can still work with that, but you might want to avoid this weird sort of dual graph going on. So, you might want to use the TensorFlow vanilla layer, which just has a single path and just rescales the input, and that's fine. Again, now the dropout is going to go out in inference and we're safe to use in mobile. Let's take a look at the other network that we had in our inspection tool. As you can see here, it says 306 nodes assigned to device GPU 0. So here, whoever has built this network actually didn't really follow the advice that we said before. We have to remove the device and we don't have to have anything assigned to the nodes. If you try to deploy these on an iPhone, for example, or an Android, it will say, I don't have any idea what GPU 0 means, so I cannot actually run this model. So let's say, okay, you don't really want to re-freeze the graph because that would be a lot of work. Fortunately, again, Google comes to rescue and they have another very interesting tool that can really help you deploy your model on mobile and suggest if you really want to deploy your models on mobile, always use this tool. And this tool is called the graph transform tool. So Google GTT, which again you can find here, then on GitHub you can go and explore. The Google transform tool, the graph transform tool, really allows you to do a lot of operations to optimize and sort of change your graph so that then you can deploy your mobile without any problems. So it allows you to, for example, fold constants, fold constants means if you have, for example, a set of nodes and these nodes always go out as a constant in the end, it's going to just compress them all and just going to output this constant because it doesn't matter to have many nodes. So you're going to cut out nodes. You're going to give it the nodes that you're going to want in the input and the output. He's going to go through all these nodes. He's going to check, for example, which nodes are not parsed here. If you have some nodes, for example, you don't want these nodes to be there, so you just cut them out. And this tool is amazing and can do everything for you without you doing anything by yourself. So again, it's a C++ Google tool, so you download TensorFlow from source, you build it with your Bazel, and then you can use this tool. You just have to compile this tool specifically, and then you can use it. How you use this tool is very simple, so this is a sort of sample comment. Which one is your graph in input? Graph is just the model that we saved. Which one is going to be your output? Which means if you want to overwrite this graph or you want to give it a different name simply. Which ones are the input layers and the output layers? And which transforms we want to apply to this graph? So in this case, for example, I'm stripping away nodes that I'm not using. I'm stripping away identity nodes because why would I want to keep identity nodes over there? And again, I'm compressing constant, compressing the batch norms and removing the device. So this is sort of the basic stuff that you really want to do to deploy the model on mobile. GTT is so awesome that it can do even more advanced stuff. For example, it can do quantization and the weight rounding that Martin was talking about before. So for example, if I actually want to transform my graph and I want my graph to be quantized automatically, I just have to add one transformation that is quantized weight. And now if I run this command in C++ I will actually get my original graph transformed, take away all the constant, all the identities and such, and quantize all the weights. Just to give you a sample of what happens here is I have sort of a few models. So for example, this is your original model and let's take a look at that. This is 12 megabytes. It's simply a sort of a squeeze-net based model so it's very small, 12 megabytes. But it's possible that you might want to have a bigger model or you might want to have many models in your phone. You have a quantized model. A quantized model is 3 megabytes. So it's the same model, quantized. You just have to go and add the quantized transformation and it's going to go down to 3 megabytes. Unfortunately, sometimes the accuracy of the model can also degrade a bit. There is something called fake quantization that you can do during training to sort of improve the quantized result. If you really are interested in fake quantization, it's a bit extreme topic. You can go check out a Google employee called Pete Warden who has a wonderful blog. I will put the link on the GitHub notebook. And he's trying to teach a bit about how to use fake quantization. At least he's explaining what it is and how to use it. So that's very interesting. But say you don't want to be so extreme. You don't want to really lose accuracy. Then you want to find the compromise. You still want to sort of reduce the space occupied by your models. Then you can use weight rounding. Weight rounding, if you look at this model, it's still 12 megabytes. So it seems applying the weight rounding transformation did not do anything. So when you go and compress this model, so we try to compress this. And now this is 2.8 megabytes. While if I go and try to compress the original model, we have still 11 megabytes. So actually weight rounding works very well, especially if you think that when you deploy a model, for example, on an application, the application is going to get archived before it's sent to any app store. So the model is always going to be compressed. So you don't have to zip your model and then extract your model and so on and so forth. You have to do that if you want to actually keep updating your model without updating the application. And then you might want to have your model in a download folder and you keep it zipped and you unzip it when you want to use it, possibly. But if your model is already right there and you're going to keep it constant, and whenever you want to upgrade your model, you send an update to the application, then the compression really comes out of the box. So let's go back to our Python notebook here.  Say you really don't want to take out Dropout because you don't trust taking out nodes. So you can add Dropout and recompile TensorFlow for both iOS and Android. So the links are here. Basically what you have, you have, I don't have internet here so I cannot show you, but what you have is you have two files that basically list all the operations that TensorFlow is going to ship with for your platform, Android, iOS, anything. If you want to add operations, you can do it. But on the other side, if you really want to be extreme on the space, right now TensorFlow occupies about 20 megabytes, you want to be super extreme on the space of your application, you can remove all the operation that your graph is not going to use. For example, if you know your graph is not going to use Concat and it's never going to use Concat, and if you're going to start using Concat, you can update your application to change your TensorFlow. Then you can go and safely take out Concat from TensorFlow and recompile it, and it's not going to link all the Concat files and it's going to have a smaller footprint. So you can have a TensorFlow that is actually much smaller than what gets shipped by default, and Google actually suggests you try to do that. Again, it's at your risk, but if you feel very brave, you can actually do that. You go get those files and select the operations that you want, and you can actually check the operations that you are using through the summarized graph tool that we checked before. So you can use the summarized graph tool, have a list of all the operations that you're using, and just keep those operations and delete whatever else is you don't need, and the TensorFlow is going to be much smaller. So again, bonus, if you really want to be brave and... Okay, now we are finally ready to use TensorFlow on mobile, and there are some examples that Google provides. For example, this is an iOS example. We can take a look at this example, but you can already see this example here is in Objective-C. Actually, it's Objective-C++. What we really would like to do is we would like to do... We would like to write our inference engine in TensorFlow so that we can only have to write it once. We write it once, and then we deploy it on Android, we deploy it on iOS, Windows, whatever. We don't really want to rewrite all the sort of session operations for every single platform. So we can do that. We just have to use the underlying TensorFlow-C++ API again. So let's first take a look at what Google is doing here. So to be simple, they just... If we take a look at this Run-CNNOnFrame function, they are taking an image from the camera. So the idea is always you're going to have a few components in your mobile app. You're going to have your UI, where you do all the normal mobile stuff. Then you're going to have a wrapper in between the UI and the TensorFlow engine that can actually talk to both components. And then you're going to have your C++ underlying. So what Google is doing here, they're going to take a photo through the camera. They're going to sort of pass it through the wrapper, convert it into a format that TensorFlow can understand. In this case, if you take a look, they just try to get the raw data and sort of reshape it in a format that is nice for TensorFlow, and then try to run a session through this data. And we're going to try to sort of mimic what they're doing, but we're going to try to do it in C++. So in order to do that, what we're going to use, for example, is OpenCV. You don't need to use OpenCV. If you're going to use words, for example, not images, you don't need to use OpenCV. If you go use computer vision, OpenCV is a nice tool to have. It's easy to do preprocessing. Maybe you already know the tool. So we're going to have a quick demo of, in this case, a soda object detection model running on an iPhone. So we have our iPhone here. We're going to get some images. Soda photos, yes. Wow, this is very small. Okay. So in this case, I trained my model to actually recognize soda products on a mobile device. So what I'm going to do, on the left, you have the sort of streaming of the mobile phone and on the right, you have the photo I'm going to take. So I'm going to take this photo. And you see, in half a second, you get all these colored rectangles around these objects. This is a good sign. It means the objects get detected. We might want to check if the objects are really what they seem to be. So let's just, I mean, you have a list of these objects. Let's just go and check. Yeah. It seems quite all right. You have some cans in the top two levels, Coke normal, Coke light and such. I'm not partnered with Coca-Cola or any affiliates. So, yeah. No, you can go and buy Pepsi. That's fine. I'm on video, so both companies are great. Anyways. So as you can see, in half a second, a sort of task, like object detection that is thought to be even more difficult and more complex than object classification can be done in just half a second by an iPhone. The iPhone of three, four years old, you're going to do it in three, five seconds, but still, if you had to do this task on the cloud, you would have to take the image, and maybe you need a sort of high-rich resolution image to be able to detect all the details. You're going to have to push it on the cloud, so depending on your internet connection, it's going to take a while. All the processing is going to happen on the cloud, and if you are deploying at scale, it's possible that you have thousands of users sending you images concurrently. It's going to be queued and processed when the cloud is ready, and then the result is going to be pushed back. So it's not going to be as fast as 0.5 seconds. It's going to be probably even three, five minutes. So actually deploying on mobile, especially if you are an enterprise, can both save time and cost, as you don't have any sort of cloud infrastructure to build, you don't have any cloud computing to maintain, and the users themselves will bring the infrastructure to you and you're going to have a decentralized TensorFlow engine, an inference engine infrastructure that works cross-platform and you just built it in C++ once, and you can deploy it on Windows, iOS, and Android. Okay, I think I'm done very simply. Let's just summarize what we learned today. We built a Keras model and we saved it in TensorFlow format. We froze it using the TensorFlow C++ API, and we checked it through the summarized graph tool to be able to understand exactly what was going on. And again, we tried to use this tool as a diagnostic to understand exactly what was not going well with our models. And we deployed and improved our models through the graph transform tool, and then we created a simple sort of cross-platform inference engine in C++. And I guess in 40 minutes, that's enough stuff. If you want to follow up, again, if you see whatever you have seen here is cool, you want to join the team, this is my professional email. If you want to follow up on anything else, you need advice or just need to chat about TensorFlow because it's awesome. You have my personal email here, feel free to contact me. Don't spam me, please. Okay, I'm done. So if there is any question, I'm happy to reply. Yes? Okay, this is crazy, yeah? If you want to recompile TensorFlow, you can add all the operations. You can actually do training. I don't know why you would want to do that because you can only use CPU unless Sam is going to teach you how to use GPU. But again, you have to understand, usually for training, we use maybe an NVIDIA Titan X, which is as powerful as the 10,000 iPhone. So... Oh, yeah, I mean, if you... Again, it always depends on how brave you are. I would not suggest it probably in 2017. Maybe in a couple of years, who knows, the mobile phones will get so powerful that you can train any model. Maybe you have models small enough, who knows. But definitely something theoretically you can do, you just have to recompile to include all the sort of operations and then whatever you get on the phone is the sort of vanilla C++ API that can do basically everything, okay? Yes, one question from the back. Yeah. Okay, so weight rounding, basically, you just try to wiggle the weights in a way that then... Okay, basically, if you have normal weights, by the way, the compression algorithms work, like Z, you cannot really gain a lot of space because you cannot do any checksum or anything because these weights are all over the place. But if you actually wiggle around these weights just a little bit so that they look nicer for a compression algorithm, you actually lose very little accuracy, but you gain a lot of space when you compress. Just because, by the way, these compression algorithms work, they can actually check some a lot more and through the checksum then they can go back to the original weights. But actually, this is a very good question and why a reference from tool can do weight rounding? You're really serious about sort of getting smaller models. I would advise you actually do something called deep compression and I think I have the reference here. Deep compression is actually very simple and very similar. I have a reference on the Python notebook of a deep compression repository where you can just call your Python and you can compress your weights in the same way. It's going to take half an hour instead of five seconds, but the accuracy result will basically be the same as the original model, but when it gets compressed it's going to take less space. So again, I'm going to push up the... I don't know who can check all the references to deep compression and such. Oh, excellent complication for mobiles. We are experimenting with that. I think the performance are actually quite great, but again, I don't want to say too much because it's very experimental. Even TensorFlow is suggesting not to use it in deployment for now. If you, for example, are deploying to clients, as, again, so far it's something very new and that they are still improving. By the way, if you are building TensorFlow from source, you can decide to include the XLA compiler during your configuration script. If you buy build from source, one of the steps is you have to run a configure script, which is going to ask you a lot of questions. For example, if you want to build with CUDA, if you want to build with OpenCL support even, and one of these questions is if you want to create the XLA compiler shipping with TensorFlow. So if you want to try to experiment with that, that's the way to go. And I think it's very interesting. Maybe in one or two years it's going to be very great.


Hi everyone. I'd like to take a look today at sociological analysis, which is another step in our exploration of critical media studies. Much like reception analysis, sociological analysis has a lot to do with the audience and their interpretation of an artifact and the role that the audience plays in hoping to both define and interpret and then eventually be critical of that artifact and understand it at a different level in a different way. So let's get started. For sociological analysis, examines the recurrent patterns in media, how those patterns influence the interactions of people, and how an audience assigns meaning to symbols used in an artifact. So as you can see, it has to do with the artifact in the media, of course, but it also has to do with how our society influences people and how that in turn then influences the interpretations that people might have of a particular artifact. It kind of started the origins of sociological analysis, apart from strictly sociology of course, have been grounded in symbolic interactionism, which is the character and content of people's social interactions and how they're influenced by the symbolic meanings that people assign to objects, events, other people, and social contexts. That's what we mean by symbolic interactionism. So really what we're getting at here with social interactionism is that people have different interpretations of things. Somebody on an island sees a boat, gets very excited. Somebody on a boat sees an island, gets very excited. So they have different meanings of the same construct, the same context. So what this leads us to is the belief that meaning is relative and personal. Meaning that as we've talked about in previous videos, meaning really rests within people, that the symbols themselves, the words, the objects and things don't have meaning so much as that the people observing them and interacting and engaging with those things have meaning and apply meaning to those things. So meaning is relative though and personal. It's different for each person and it varies from person to person. It's very subjective. Meaning is also built via social construct. We are of course influenced by the world around us, especially as adolescents, but even all throughout our lives really. We are influenced by where we're at, who we are, who we're with, the things that we've learned and experienced and all that kind of stuff and what our culture expects of us, what society expects of us. Meaning is built and influenced heavily by these social constructs. And in the end though, because meaning is relative, it's personal and all that, meaning is interpreted. Meaning is interpreted. Meaning that each person assigns meaning to a particular event or symbol or anything else that's going to be assigned specifically by that individual and it's going to be interpreted. Meaning we make meaning out of these things essentially. That meaning is not automatically just assumed or lined up for us. We make meaning ourselves out of these things. Meaning is interpreted. And what this looks like sort of in a real world sense is that there's really a very, very much a loop here and it starts with that symbol. Whatever that symbol is, is that symbol a word? Is it an image? Is it an object? Is it a particular social context? Whatever that is, we have that symbol. Whatever it is we're going to interact with. Then we have that individual meaning. When I look at a tree, I see one thing and you look at that same tree, you may have a different feeling about that tree or different meaning related to that tree, right? Depending on your perspective, your experience, your frame of reference. So each of us has an individual meaning for whatever symbol it is that we're taking in. That individual meaning is of course built on social influence and affected by social influence. We are creatures of our environment. We are influenced by those things that surround us and the people who surround us and also again our own past experiences and things. But we are socially constructed beings really. So there's a lot of social influence that goes into that individual meaning, right? That hits us with that individual meaning as it's developed by the environment around us. And then though our interpretation, our sense of whatever it is that symbol is, is constantly evolving. We rarely just stay the same. Now these don't happen a lot of times when we talk about persuasion sometimes and persuasion is incremental a lot of times. It's very rare that we just have a complete about face, about a topic, or about something all at once. These things happen over time. But over time, if you look back to who you were 10 years ago, 20 years ago, even even five years ago or a year ago depending on where you're at in life, you'll see that you see things differently. You interpret things differently based on where you're at. I can tell you, for example, becoming a parent significantly alters you for you the world. You see danger in places you didn't even dream it existed before, right? And you have worries that you never thought about before when you become a parent or when you move out of the house for the first time, your world changes because now you're responsible for the bills. You're responsible for remembering to get groceries. You're responsible for all these things you weren't responsible for before. That changes the meaning of these things for us. We are constantly evolving and so are our interpretations. So then the next time we see that symbol or come across that symbol, we may have a different individual meaning and that cycle continues, right? So to put this in context just a little bit further, we could take a symbol like this is a sort of a controversial one, but that's why it's a good representation here, I think, but like the Confederate flag, right? That's a symbol. It's a symbol and there are a lot of feelings about the Confederate flag. People have a lot of different feelings about it and interpretations of that, but it's important to remember that those interpretations are individualized. Each of us has our own individual meaning behind it for some people who grew up in the South and really have a strong affection for that flag. They would tell you that it's about Southern pride and that it's about independence and that it's about individual rights over corporate federal rights and so forth and those types of arguments. Other people look at it and say, well, that's a symbol of secession. That's a symbol of people who wanted to leave this country and we didn't let them, but that's a symbol of secession and division within the United States. Other people look at it and say that's a symbol of slavery. It's a symbol of people who wanted to perpetuate slave ownership and so forth. So we all have these different individual meanings and they'll have these different shades as well. I mean you can talk to some people who, you know, if you get a groupie who'd be people together who like the Confederate flag, just as an example, who appreciate the Confederate flag, then within that group though you'll have all kinds of different reasons. They'll all be on that end of the spectrum in terms of we like the flag, we appreciate the flag, but their reasoning will be different. The strength of which, you know, some people are like, this is the greatest flag ever and should not be, you know, done away with it, whatever, and other people I like might say, you know, I like this flag, but is it really something I'm ready to fight over? Not really. I don't want to have an argument over it or get into a big thing over it. I like the flag, but it's not that meaningful for me. So you'll have all these different people across the spectrum, all these different individual meanings related to that. A lot of these will be, of course, constructed by our society, by, you know, if you grew up in the south, you're much more likely to have some special connection to this flag than somebody who grew up in the north, for example. It's not to say everybody who grew up in the south approves of this flag or whatever, but odds are if you grew up in the north, you're not really going to have any special connection to this flag at all in that way. So it's kind of, for the most part, that kind of subgroup grew up in that society, grew up in that culture where they were taught a different meaning of that flag or taught to see it in a different way than other people might be. And then we take it not only just growing up, but what's the current climate around where we live? Do you live in a community where the Confederate flag is flown proudly, or do you live in a community where, you know, it's kind of like, that's a thing of the past, let's get rid of it.  So it's not just what we grew up understanding and how we established that frame of reference, but what our society says now, what our culture says now, and how that influences us, right? And of course we're constantly, our idea is going to be constantly evolving over this. Maybe that as a kid you thought, well that's a cooling flag, I really like it. Or you know, you've heard all the stories about Southern pride and independence and so forth, but then maybe later on you get to college, you get to a point where you're learning about the history, some of the history of that flag and thinking, boy that's not so great, I don't really approve of everything that it represents. And maybe I'm not as much in favor of it, so your idea of that flag may evolve over time. And then the next time you encounter that symbol, who knows where you end up with it, right? Again it's not like you're probably just going to hear one person talk about the flag and automatically decide, oh well now I don't like it, or now I do like it. Change your mind entirely, but the next time you see it, it's going to be a little different, right? And that cycle continues and we go back and forth and we go back and forth on these things. So depending on our individual meanings and societal influences on that, so anyway, so this is kind of all grounded in symbolic interactionism and the way that we engage and interpret those symbols as an individual. Some of the major premises, there are a couple major premises really at work here in contemporary terms for sociological analysis. One is what we call dramaturgy. Dramaturgy in, you know, kind of layman's term so to speak, I guess he's not a layman, but Shakespeare once said, all the world's a stage and all the men and women merely players. They have their exits and their entrances and one man in his time plays many parts, his acts being seven ages. Really that last part there is what we want to look at. One man will play many parts. What does that mean? Well that's what dramaturgy is getting at really, that dramaturgy approaches communication approaches, analysis approaches society as though it is a drama, that we are actors on a stage in essence, that we are constantly engaging in what we call impression management, impression management, which is just that we put on these different sort of faces. Not that any of them, I'm not saying that we're lying all the time, but you know we're slightly different at home than we are at work or than we are when we're out with our friends. We put on these different kinds of personalities in some ways. That's what we mean by impression management. Now there are a couple of different ideas at work here as we define dramaturgy. One is stage, which is just the you know the overall idea that the world is the stage that this is constantly happening. Another is the setting, so the specific context that we find ourselves in and how that affects and shapes how we're going to behave, what language we're going to use, all that kind of stuff. The part refers to the individual, refers to you if we're talking about us in terms of our own impression management, then we are the part. This is the part that we're playing. This is the part of us that we are that we are choosing for this moment and thinking that this will provide us with the best way to achieve whatever outcome it is we have for that situation, we desire in that situation. Then the team, there's a team aspect to all this too. You know it's pretty rare to have a one-man show really, so they happen but they're not all that common right. Most theater takes place in a group context and that's what we mean by team. It requires other people to kind of buy into what it is you're you're trying to sell, right, to approve of and to kind of participate in your little drama, whatever it is, the impression you're trying to convey there. So basically dramaturgy just sets out life in society and in conjunction with others as in the midst of our play here. So we have the idea of dramaturgy as one type of sociological instance and so it's not difficult to see, sorry, it's not difficult to see how dramaturgy really affects the media as well. Of course if we talk about first of all social media obviously we're very much engaged in dramaturgy there. We're selecting the pictures that present us in the best light and we're very specifically choosing what to post and how we frame it, how we, what language we use and all that kind of stuff, right. And then in the media as well the media convey this ideal framework of social identity and we found that some critical analysts would tell you that that the media really is working to influence who we should want to be and telling us who we should want to be, right, setting out all these ideals and and setting out this is who, this is what's normal for life, this is what's right about life and in some ways giving us some pretty false expectations about these things. So, but that's how the media factors into all this then too. They provide this framework for us to look at and say oh that's what I should be doing or that's who I should be, that's what I should be holding out for, right. And that's, so, for example that's a regular comment in my home, my wife loves hallmark movies. And one of the regular comments when she's watching a hallmark movie and I always know this is coming is why don't you do that for me, why don't you, these guys are doing all this stuff on hallmark movies and say why don't you ever do that or why don't you ever say that to me. Because we don't live in a hallmark movie and I'm not as smooth as those guys and I don't have a team of writers writing for me, right. But I mean that's the short answer but but it really sets out these expectations, right. And I know she's kind of joking around but but in some ways it does affect our sense of reality and and set forth this sort of in intenseness in our dramaturgy in the lives and though in the role that we're trying to play because we feel like we have to live up to these things or this is what I should be doing and who I should be or who I should at least want to be. It sets that framework for us a lot of times media does through dramaturgy then. Another premise that we look at and another idea that we can look at and dramaturgy approaches things from the perspective really of the individual putting on the show, right. But then there's also what we call frame analysis which has to do with specifically how things are framed and will then interpret how a person will then affect how a person interprets it, right. So how something is explained to us or how something is is set forth for us will interpret how we view that how we engage with that information. So for example if we look at you know personal reality we're talking about you know personal reality and frame analysis. We're talking about you know the way our frame of reference is developed. We have these experiences, we have our culture, we have our behavior attitude and values system. These things shape who we are, shape our perspective of the world and how we see the world and how we engage with the world, right. But connecting this then to media and to artifacts in the media we can look at what we call media frames or what are known as media frames and these media frames really have to do with selection, emphasis and presentation, right. So when we look at the media we understand that somebody is putting this together. We touched on this in reception analysis but somebody is creating this stuff and specifically if we look at the easy example here is the news. If we look at news broadcasts or news channels somebody has to decide what's going on the news, right. That's selection. Somebody's making a choice about what to put in, what to keep out. Probably based on what they think an audience wants to see but it could be that they have an agenda as well so they're making selections here about what's going to be talked about, what's going to be shown in the news, how it's going to be presented, what language we're going to use, all this things. So but initially the selection is just what are we going to put in, what are we going to keep out, what are we going to choose to have in this broadcast and tell people about, right. Then we can look at emphasis as well. When we choose a story, first of all that, I mean the selection process is an important one but then once we've decided something's going to be in what emphasis are we going to place on it? What emphasis are we going to place on this? Are we going to present, you know, if we look at a newscaster, a news program reporting on a trial, for example. This is an easy example here. If you're reporting on a trial what are you going to emphasize?  I mean, they aren't supposed to do that but it happens, right? We have to make choices about how are we going to present this to people and that's the last one presentation, how are we going to present it but what are we going to emphasize? Are we going to emphasize if the defendant is a young person or are we going to emphasize their youth and their naivety and kind of almost set it up like, well they didn't really know what they were doing. We're not going to say that out loud necessarily but are we going to emphasize, look at this young man. He doesn't really know what's happening in court, didn't really know what's happening in that situation or are we going to emphasize, you know, the plight of the victims, for example, of whatever this person is on trial for. What are the plight of the victims? Are we going to emphasize that? Are we going to side kind of with them and really tell their story more and then that impacts how we present this information. What are we going to choose to about that story? What are we going to choose to present and how are we going to present it? What language are we going to use? What imagery are we going to use? All those types of things. How much time are we going to give this? So the media goes through this process of selection, emphasis, and presentation and we are exposed to this. We're exposed to all these things so between the two of them though we still have our personal reality but we also have the media framing things, right? And so that agenda setting function that we talked about in reception theory but the two of these then combine to create what we would call an individual frame which is that person's unique perspective on things. So we can look at things on okay what's this person's background? What media are they taking in and how does that impact how they're viewing this artifact? We need to get to know the audience member a little more so that we can understand their perspective in that regard through frame analysis and that's what we're talking about there. Okay this has just been a brief rundown of sociological analysis. I hope it helps you understand a bit more in the different aspects of sociological analysis. It's really a broad topic and a lot that we could look at in it but I just wanted to give you an overview for some for some basic shared understanding of that topic. If you have questions about sociological analysis or anything else related to critical media studies please feel free to email me. I'd be happy to chat with you and answer any questions that I can via email. In the meantime I hope you'll really get out there and consider the context in which all these things take place and how our individual influences but also the the sociological influences of the media then impact how we view an artifact.


But yeah, let's get the show started. I'm here, built to hack, hacked to build. Who am I? My name's Chris. I'm a security engineer at Heroku. We're pretty much remote code execution as a service. If you don't know what I'm talking about, go look the platform and you'll see what I mean. Find me on Twitter, Brompony. But yeah, I like hacking stuff, I like building stuff. I am lazy, so I build stuff that helps me hack stuff quicker, whether it's web, Windows, Android, and recently I've been looking at containers. They're a lot of fun. So this is what we're going to look at, problem, some existing research. I'll introduce to you Bob. So it's Bob with the silent T. And we'll look at some stuff that Bob can do. And yeah, this slide's more for me just to actually remind myself what it is that I actually share with you today. So we might deviate a bit, but I'll try to bring it back in. So what are the problems? So in a modern cloud, DevOps, SDLC, agile environment, there was one big question that I had to ask myself quite often was, how do we identify and exploit container vulnerabilities? So for a pen test or if you've got a shell in the box and you want to break out, you want to point some stuff, how do you identify and exploit container vulnerabilities? But then in an engineering environment, how do we test, secure, and monitor our containers? So I come from a red teaming, consulting background, and now I do more engineering. So I'm constantly trying to break our platform. And when you've got thousands of containers and you don't always want to pop shells and stuff, how do you do security at scale? So these two questions are very much aligned, but they're pretty much asking the same thing, which is, how do we basically identify and point stuff and break out of containers? So in terms of existing research, these are just categories of types of tools. This list is by no means complete, but if you're in a terminal, you've got a shell, and you want to find out if you're in a container, and are contained, identify as the sec, com, caps, LXC, or Docker environment that you're in, and it will tell you, very important. Then there's a whole bunch of Google container tools. I highly recommend you look at their repo. Great stuff in terms of static analysis for container images, be it a Docker or an LXC image. Clay is really good. If you want to look for vulnerabilities in images, it's a really great tool. And then when it comes to benchmarking, Aqua Security's done a lot of good work there. If you actually want to benchmark your images and your security controls that are in place. Now you'll see that from this list, there's not really much there in terms of actually breaking out and poning containers. So, I came up, I had this problem that I experienced. This was the existing research that I had. So generally when I like to solve a problem, I like to just hone everything if I can. So, that's why I did Bob. You can find it on GitHub. It's written in Go. Yeah, the hipsters were doing it, so I figured let's see why the hipsters doing everything in Go. I'm gonna know it's slightly ironic by my hat in beard, but you know, I'm in denial. So basically, like I said, I'm lazy, and I also wanted something, when you're writing the same exploits over and over again and doing the same breakouts, you kind of want to automate that stuff. So, Bob will give you the ability to auto-pwn. So, skids are gonna love that. Sorry I've had to say, but it's definitely a thing. Also helps you do not only breaking out containers, but things, Linux, post-exploitation. When there's a lot of things that you want to do and we'll get into it, you want to automate it. So, Bob will help you do that. And of course, you can do all these things in a pen test, you can't always. If you want shells, you get shells. But if you also want to help your engineering environment, so set things like return codes and plug into your CR environment, Bob's also designed for that. And I'll show you what I mean by that. So that's Bob. Like I said, we're gonna be pawning stuff. When you're pawning containers, the hollow world of pawning containers is exploiting docker.sock. It's a Unix domain socket that docker uses to orchestrate basically containers. If that thing is ever exposed, you find that it is game overs. So basically, it comes from people wanting to use docker don't do that, use that. There's better ways to do it originally. But basically, this allowed us to do a local privates to root and basically break out of the container. It's a really fun way to get into container exploitation. So how do you do it? What are the manual steps? So first, you've got to find docker.sock. So it's a Unix domain socket. It's speaking HTTP. It's basically a RESTful API on a domain socket. Then you want to interact with the socket with curl or docker clients or net cats. Once you find it and you can interact with it, you then want to create a new container and mount the host's file system in your container with the dash view command. You then run that new container and then you chroot or chroot or chroot, as I found this week. There are multiple pronunciations for that. And there you go. You are then pretty much root on the host and you have access to the host file system. And now you know if it's Linux and you access the file system and everything's a file, it's game overs. So that's the manual step. Now you don't want to do that all the time. Excuse me. But I think it's a little bit tricky in containers. Because when you're living off the land, especially in containers, the commands that you need, for example, the find command, might not be in the container. So if you're using really minimal images like the docker image or the alpine image, that's the thing. You might not have the tools to actually do the stuff that you need to do, like what if there's no, I have config or IP adder. Sure, you can read from ProcFS, but this stuff can slow you down. Especially if you need to find the docker socket. It's not always that var run docker.soc. It could be at moo slash bob. If you don't have the find command, you don't have recursive searches, how are you actually gonna find the socket? So this is where syscalls come in. So on the left there, those are the syscalls that the find command makes use of. And on the right there, that's what bob does in go code to do pretty much the find command. The nice thing about go is that when you start interacting with the operating system, go makes use of syscalls. It won't actually make use of actual shell commands. Some languages do that. So the nice thing is, is that if the command isn't there, you can use the syscalls. And the syscalls will be there to allow you to perform the actions that you wanna do. So that's what we do with bob. And I'm gonna show you a demo now. These are prerecorded. I'm done doing live demos because I just ran out of sacrifices all the time. So what we're gonna do here is, at the bottom terminal, so I know it's a little bit small, but basically that's our host. And we're just gonna watch a directory on the host. So you see there's one file called boblinux. In the top, we're gonna run it in a container. So we're just saying a Docker run and then we mounting a socket. Turn off the light. Turn off the light. Yeah, can you make it bigger at all? Yeah, unfortunately that's, can we turn the lights dim? Can we, no? Yes? Can you turn the lights? Yeah. Fortunately, I can't make it bigger because the resolution will go off the board. We can just break the lights. Ha, ha, ha. Ah. Ah, there you go. Ooh. Ooh. Ooh. Ha, ha, ha. The room full of hackers and you just turn the lights and you get that reaction. Ha, ha, ha. We didn't actually do that. Ha, ha, it's magic. Someone in the audience did. Ha, ha, ha. So we're in the container now and we're gonna run bob and you know, if you're inside a container you wanna find sockets and we're just saying dash socket equals true and what bob's gonna do is gonna go and look for all the unix domain sockets. Now you found a unix domain socket what now? What do you do? So of course you wanna auto-pwn this stuff. So I'm now providing dash auto-pwn equals true and what bob's now going to do is it's gonna look for the domain sockets that you can find and it's gonna try and pop it basically. And what you'll see now, bob is now giving me a TTY. So this terminal that I'm in now is not actually a terminal on the host. You'll see that that's a OSX host there and I'm not gonna create a file inside a 10-fold and that file is now gonna appear at the bottom terminal which is on our host. And we'll see that appear in a bit. So now I'm in a TTY that bob has provided me and you'll see that file has now appeared on the underlying host within the container. Now once you're done with that you can now exit bob and now you're back into your container's terminal. So that's one way that instead of doing those steps that I mentioned earlier, you can just basically say auto-pwn and boom you're root on the host. Of course if the docker socket is not authenticated or protected often it's not. So automatically just auto-pwn all that stuff. So it's quite nice to get a TTY on the host. If you want think-pen test you wanna get a shell on the host. Also if you wanna test maybe in your CI environment if there are exploitable sockets in your pre-production or staging environment you can tell bob to do this and we'll make use of return codes. So what we'll do here is we have a CI environment and we're gonna run the same thing. So what's happening over here is at the top right I've configured my CI environment to run bob and I've provided the auto-pwn but you'll see that I've added the CRCD equals true flag. So what that means is don't drop into a shell but if you successfully exploit that vulnerability set the return code greater than zero because if you're familiar with CI environments fail tests are marked as any test that has return code greater than zero. So if your CI environment runs a command a test and the return code is greater than zero the test is failed. So bob will do that. So this is very useful for engineers and for CI environments. And if you look on the left the test is running and we'll see that bob picks up a couple of UNIX domain sockets but because they're not exploitable they're not Docker sockets you'll see bob's found them the test will actually pass. So this is quite a nice thing that you can do if it doesn't make sense to try to get a shell now like if you're doing thousands of push requests pull requests a day you're not gonna want shells in this context but from an engineering environment you wanna test are they exposed Docker sockets is this thing ponable? So you don't want your engineers focusing on understanding okay now I've got to exploit Docker sockets like just run it automatically and then you can find out if that's the case. So it's very useful in CI environments but of course there's more to prone there's more to more than Docker sockets. It's called privileged containers. Anybody raise a hand privileged container in here? Cool so thank you for putting your hand up but it's the worst when you're at a talk and someone's like can you please put your hand up if you have seen X or A or B or C. It's always the worst but privileged containers are pretty much the worst thing in the world of containers why? Cue awkward silence. The case for that is I know I just try hard behind comedy and it's always an epic fail. So privileged containers are basically a I was gonna say another word there but it's not a good thing. And it's very dangerous when it comes to your syscalls that you have access to and the Linux capabilities. And this basically means very very easy container escapes but now you're in a privileged container. How do you break out? So I'm gonna show you a new feature that I released this week. What this does is Bob makes use of C group controllers and we're inside a privileged container and what we're gonna do is we're gonna abuse the C group Linux release agent capability and what that means is that we're gonna get command execution on the underlying host. This is how C groups work because we have access to the mount syscall and also because the Linux capability sysadmin has not been dropped, we can abuse that.  So you'll see now I'm using the Pone group command, I'm saying hostname and I've got a hostname of Docker desktop.  So at the bottom I'm on my jump box, standard netcat listener, the great thing, if you're using Docker on OS X, be aware that your VMs, your containers are running in a Linux kit VM, that VM by default comes with netcat enabled, of course it's got busybox, but of course it's got the dash E flag is enabled, the evil command, you do not want netcat on your boxes with that. So it makes getting shells really easy. So we're gonna sell it, Bob, Pone the C group, so we're in the container, run this netcat command, connect back to our jump box and what's gonna happen is that we're gonna get a shell on the underlying host, which happens to be the Linux kit VM and in the OS X environment that's basically game overs. So because we're running in a privileged container, this is really easy, trail of bits that are great right up on this, C group, Pone Edge, it's really interesting and you'll see now we've got a shell at the bottom and you'll see we're on a host called Docker desktops, if you see that hostname, that is the VM that Docker for OS X uses and now we're in the host file system and if you ever wanna find out where your container is living on the underlying host, look at the mount command and if you look there you'll see var lib Docker overlay two, that's where your container is living and what I'm actually gonna do now is I'm gonna create a file in the Linux kit VM and you'll see that it'll appear inside my container. So I'm in the container and you'll see the OS does not like it because there's some Docker weirdness but you can see hello is over here and we created it on our jump box. So basically, Pone Edge C group, you break out of the container and you basically root on underlying host. If you're running something on a Debian system, it's running bare metal, you'll land up actually being root on the host. So of course, if you added the CR CD flag to Bob with this, it would exploit this but then set the return code. So you wouldn't be getting shells or you can just execute a command that you want. So pretty useful, privileged containers, they are out there but when it comes to containers, there's also environment variables, very, very good place to look when you wanna Pone containers, you'd be surprised what's in environment variables. So there's good old ENV, if you type that in a terminal, that's what you'll get but there's also the PROC FS in Linux. PROC FS, it's given me a lot of this hair loss that you see over here today. But basically if you read PROC and then the PID of the process environment, you'll get access to the environment variables that have been set for that process. Now this gets interesting because ENV is not always cleared and things might be living in PROC PID and this is when ephemerality becomes a problem. So you might do like a PS orcs and see okay, there's a process with PID-99, you go read PROC-99 environment, there might be nothing there because the process is dead. So you gotta be really quick when you're doing this. Bob does this really quickly via OS environment and Bob will go ahead and check the ENV, OS.ENV so like typing ENV in your terminal and go in the PROC FS and analyze that because sometimes there's the case where the ENV in your terminal is being cleared but the PROC environment mapping has not been cleared. So sometimes you can find a whole bunch of stuff because of course environment variables, it's the next general secret storage mechanism. So that's pretty much what the go code looks like if you want to process PROC FS. It's really nice and minimal. I'm a big fan of go as you can probably hear or see but how do we actually go about doing this? So we're gonna run a container and I should probably press play. That's a good place to start. And we're gonna add in an environment variable. So it's called touch of approval and super secret password. So what Bob will do by default, if you do the recon flag, if you don't pass a word list, Bob will look for password and secret. And you can see now Bob has said, okay, we've now found these PIDs that have this key term in PROC environment and Bob will then look in the ENV for the process that it exists in. And that's by default. But what you can also do with Bob is you can also supply a word list. So you can tell Bob, you know what? Analyze all the processes that you have access to and look for these words in this word list. So we're gonna provide a word list now and inside that word list were the terms password, URL, secret, find me and Bob went ahead and looked at ENV and in all the processes inside there. Now this is where the return codes also happened. So if Bob has found a keyword in a process environment that you're looking for, the return code will be greater than zero. That's what I'm gonna show you now. So now you're gonna use a word list where there's nothing inside. And you can see Bob didn't find any of the entries inside the word list inside any of the processes. And then you'll see the return code now will be zero because Bob didn't find any of those key terms. So that's all fine and dandy. How do we also do this in a CI environment?  So in this case, and these configs are all the same on the CI environments that you're using. So we're using a shell script here and we're gonna be running two Bob commands. So we're gonna be doing the socket auto-pot. So we're gonna try to see if we have any vulnerable sockets and we're gonna be scanning environment variables and Bob will go ahead and do that on every get push and every pull request or however you've configured your CI environments. So let me just fast forward this. This is in true Chris fashion, I made too much content for what I have. So I'm like, let's get to the really cool fun stuff. And of course you can't double up speed with the videos that I make. I'm just challenged in that way. So in every CI environment, Bob is gonna look for these key terms. It is regex as well. So you can say, okay, look for S3 tokens or whatever. It's regex, the world will definitely burn at some point. So we can see Bob has gone ahead and run those two commands. And the test will pass because from the previous tests, we know that there are no exploitable docker sockets. And in the container that we are running, I remember now in this CI environment, all our tests are running in containers. That's how we configured it. And now we know great stuff that test passed our containers in pre-production do not have exploitable docker sockets and the environment variables that are there do not have any of the key terms that we specified. But of course, clouds, clouds, yeah. Clouds love to host containers. And of course we've got metadata services. So the <IP_ADDRESS> address, it exists in GCP or AWS Azure. And basically if you can get access to this, you can do a whole bunch of stuff. It depends on the platform. So for example, that cool request in AWS, there's an IAM role associated to that instance. You get the credentials for that. It'll change, it varies on GCP. There's a whole, yeah. That is a whole world and talk right there on what you can do with metadata services. Excuse me. So you rarely want to find these things. But also if you think Kubernetes, there's also APRs out there. Metadata service is basically an APR. And you wanna look for these things. Also you wanna find out what is on your network ranges for your containers. For example, EC2 classic, still around. If you use EC2 classic, and this is from the documentation, you're all the hosts. Every client that uses EC2 classic is all on one big, flat network. Yes, I know. What year is this? So if you are running anything on EC2 classic, your host is exposed on that 10 range. So if you're in container environments or any post-exploitation environment, you wanna find what APRs or what stuff's available. But if we go back to the container quirks now, you might not have Nmap in the container. You might not have Wget or Kool. You might not have the tools that you need to do the footprinting that you wanna do. So of course, Bob has a useful feature. So you can tell Bob, go ahead, and look for metadata services if you don't have Wget or Kool and all the interesting stuff. So you just add the metadata flag. And Bob, by default, will look for two endpoints, Kubernetes control plane endpoints, and the 169.254. So you can see now in this container, those two endpoints weren't there. But now we're gonna tell Bob, use the word list and see if the container can hit Heroku.com. And we're gonna specify the endpoint list, and then Bob's just gonna process that list and see, okay, can I issue a successful HTTP request to that host? And there we go. We see, Bob says, we've got a response from Heroku.com. And if we look at the return code, it's one. So you can use this in your CIO environments as well if you wanna check what the egress is. You wanna see, okay, I know that my Kubernetes or my control planes are sitting on these RPs. Tell Bob to scan those RPs in your pre-part or in your production, and Bob will tell you, oh, by the way, I can interact with this endpoint, which I shouldn't. And of course, when you're on a host and you wanna break out or just pawn more stuff, it's pretty useful when you don't have Wget or Kool. So there's also binary hijacking. This is awesome because a lot of orchestration environments explicitly trust the commands that they're provided with.  It just runs LS. So if you can control that container, you can override LS command to write your command. And we can do some pretty cool stuff with that. An example is kubectl or kubectl. I heard it's another pronunciation. You learn something new in Vegas every year. So with kubectl, it executes the containers or the pods tar command. That was recently exploited and used in an interesting chain by TwistLock. I highly recommend. But because as an attacker, you had access to the tar command on the host, you could mess with Kubernetes and basically break out. So pretty interesting. Important to remember that at some points in a container's lifetime, a binary might be executed. Binary I mean command. So maybe the power off command or the RP adder command might be executed in your container. So what do we do? We hijack all the binaries. So be careful. You can really break stuff here. You're overwriting binaries. So it's not too friendly for production environments. So I've been told. So if you wanna do this, what do you do? So these are the manual steps. So you gotta find out if you have read or write permissions to burn or S-bin or user S-bin in the container.  So either a webhook to verify execution or you wanna find something. You can go ahead and do that. You then wanna replace every binary that's an S-bin or bin with your malicious executable.  So in terms of payloads, Bob will do this by default. We use webhooks. Why? Because sometimes if you execute, if you put in a shell payload and then you set up your handler and then it could be one minute, it could be 24 hours until you get a callback. Webhooks are useful because it will confirm exploitability. In this case, we're gonna say, okay, tell me which command was called and what parameters were sent to it. That's how you would do it in Bash if you were on a Linux host or in a container. If you were to write a Bash script, that's what it would look like. So how do we actually do this and what do we get from this? So you could get a callback in one minute or five minute or next week. It really depends. What I've successfully achieved with this kind of attack is getting tokens provided to call commands. So some platforms will say, okay, this client's running a container. It's been 24 hours. Let's back up the content of the container and let's push it to an S3 bucket. And of course, it's not an open S3 bucket. They authenticate it as they should, but because they're running the call command inside my container, I've overwritten the call command and they're actually calling my command and then because they're using a token, they have to give the token to the call command that they're running in my container to back up the contents. So you can find some interesting stuff over there. You can also prevent logic. So often client environments, in your hijack binaries hijack the power off command. A lot of orchestrators say, okay, this person has had two hours of compute, turn off that instance. Okay, cool, but now you've just hijacked the power off command and that orchestrator thinks that they've executed the actual legit power off command, but you've hijacked it, so you might get more compute time. And general mortgage, you can really break stuff with this kind of stuff. So hijacking binaries, it's fun. So enough of me talking, let's actually see this in action. So on the right, we're in the container and basically I'm telling Bob, go ahead and use the hijack command. And I know that text is really small, I apologize. You can throw stuff at me later about that. But we're telling Bob, hijack all the binaries in the host, in the container and execute this command. So Bob is gonna configure the host such that any command that's been hijacked, it's gonna execute a call command and that's gonna hit a web endpoint, which is the logs that you see over here. So I executed the ID command and you'll see that I got a callback over here. I was told, so I'm gonna say ID hello and then my logs on my web server say, okay, someone executed the ID command on the container and they sent the parameters hello y'all. So what does it look like if we were to run maybe docket exec on this running container? So that's what we're gonna do now. So this could be kubectl copy because they don't execute the talk command. So we're gonna run the ID command via docket exec and then you'll see on the left, we just got a callback because we've overwritten. We've hijacked the ID command. So that gets a lot of fun, that gets pretty interesting. I enjoy doing that. You'd be surprised as to what you can get. So Bob automates that whole task and it's a lot of fun. But of course there's more. What about all the CVs and all the lovely stuff that we have in docket? Yes, you did just see shell shock and bash bug and you probably like what year is it? It's actually quite a useful thing to actually load containers because you'd be surprised how old some infrastructure is. And if you're in infosec, you know, people don't patch things. So that's a good payload to know, especially container environments, especially for LXC containers. There's a whole bunch of stuff. That's some really useful CVs. But one of my recent favorite ones is the Run C ponage that came out really cool. Basically similar to hijacking binaries but here you are hijacking the Run C binary on the underlying host. That's what the vulnerability allows you to do. So basically from your container, you overwrite the Run C binary on underlying host and then you run in the context of that and because it is owned by root, you run as root. Now this is slightly destructive. So yeah, I always call this a last resort exploit because you can actually break the underlying runtime because you're breaking the Run C binary. For Docker and LXC it gets a bit tricky. Sometimes you need to be a privileged container. Sometimes you need root. This is because LXC and Docker treat privileged containers very differently. I won't get into that but it's really interesting. And yeah, what you're gonna do in this case and Bob is pretty useful is that if you wanna verify if a host is exploitable to this because you can't actually trigger your payload being executed. The payload is only executed in this vulnerability when someone runs a command inside your container. So if someone's running a kubectl copy or if someone's running a Docker exec. So you could wait. So in this case, we're gonna give a very nice webhook so we're gonna tell Bob.  Sorry, excuse me. And I should probably place play or to play. If only someone invented that by now. So we're in a container in the top and we're gonna tell Bob, try and exploit CVE blah, blah, blah, blah, blah, blah. And the payload that we're gonna give is a code command. So we're gonna tell if the successfully is exploited we should get a callback on our host. So that will be the terminal on the left. So that's our code command we're saying, execute this. So now we've run the payload. Now you gotta wait with this payload. So now to trigger it, I'm gonna run a Docker exec and that's at the bottom terminal on that container. So you can't trigger this.  So I'm gonna run binsh and you'll see Bob will exit at the top. So it says finished and now we've got a callback in the bottom left. So now that tells us, okay this particular container was vulnerable to that CVE and we now got a callback. So now you could go ahead and get another container on that platform and then give your shell payload. So pretty useful. If you wanna do that exploit manually there is a bit of work to it but Bob will take care of that for you. Awesome, conclusion. Yes and I haven't had my time cut yet because that always happens. So Bob can be used by both pen testers and engineers. As you can see, I can use it to get shells or to verify exploits ability. You can also use it within your SDLC environment so security at scale is really, really hard. And one important point here is that we are testing for two kinds of vulnerabilities. Immediate, so like getting a shell immediately or we are also getting delayed. So remember the hijacking boundaries or the run C that is delayed. I have 10 minutes. Was that 10 minutes Jeff? Awesome, this is like, this never happens. I apologize, your ears must be, I'm going really fast as well. Caffeine, I blame that. And of course it being 10 o'clock in the morning. It's like, get everything out before everyone falls asleep. So you see, I digress. Now I need to come back. Yes, Chris, what were you saying? Feedback, yes, that thing. So you're gonna be able to get feedback on two kinds of vulnerabilities. So if you hijack all the binaries, you have a web endpoint, you can get it at some time and also with the run C stuff. There are a couple other features that I have not touched on today. So there is some stuff that like, if you're on a hosting, you wanna push stuff to S3 buckets for data X-fold, it's pretty useful. I added that recently to Bob. There's also quite a cool way that you can scrape metadata stuff. So if you've ever gotten access to a metadata services, that's a real pain to get the actual content. There's a nice trick that you can do on GCP legacy metadata services. You just basically connect on port 80 with netcat. Don't send the host header, enter, enter, and blah. All the metadata stuff is yours. So I built that into Bob to do it automatically. So yeah, there's a couple other things. It's all on the GitHub repo. So all the documentation is there. I haven't gone through all the functionality, but yeah, it's all there. You can find it. All the binary releases are there. And yes, I thank you for your time. It's early on Saturday morning. And yeah, I'm gonna end it there. Thank you so much. Thank you.


You got nine million dollars, Corbin? Uh-huh. Cool. Then you can afford to pay Rihanna to show up and perform at your wedding. Oh, wait, you're already married, forget it. Damn. Now what am I gonna do with my nine million? Well, you should probably give it to, we should probably give it.  I'm Rihanna, you can follow us on Instagram, and Twitter for a more juicy content. Thank you to everybody who supports us on Patreon, and follow us on official Twitter account, and subscribe if you haven't, and hit that like button. Hit the like button. Hit the like button. Hit the like button. Hit it with your nipples. Hit it with your tongue. Hit it with your, and you're. You fill in the blanks. Today we get a, this is called Seeking India's Forgotten Vegetables. What? That's, I never imagined ever we'd be reacting to a video about India's Forgotten Vegetables.  Here we go. The United Nations estimate that more than 90% of the farmer's varieties have disappeared from the farmer's fields in the last 100 years. Rabbits, they took them, probably colonialism. It affects every aspect of our life. It affects the climate, it affects the economy. Bringing this awareness to the people means changing the way they behave today for a better future. Eat more vegetables, that's for sure. Stop eating processed foods. You want a cookie? I have had a passion for food and cooking it from childhood. As a food designer, I study the interaction that humans have with food and the cultural influences that surround it. So for us to understand what our food landscape could be in 10 or 20 years, we would have to go back and trace our past and to do that, we will have to look at recipe books. One such recipe book that I treasure are the books that my grandmother left back. What sparked my interest was the fact that it had a lot of ingredients and vegetables that I did not know of. I hadn't seen them in my lifetime. I had to find out why these vegetables were not present in the kitchens or why these vegetables were not present in the market. There has to be a reason to it. My name is Sangita Sharma. I work on sustainable agriculture. I travel the length of the breadth of India to understand where the food equation was gone wrong. Why were we having to consume pesticides? And this search took me across villages, farmers and that's when I understood that wherever I went, the farmers did not have access to their own legacy of seeds, their heritage of seeds. Seeds is the missing link. Seeds do not belong to the farmers anymore. They belong to a handful of multinational corporations. How can we lose these indigenous varieties that are a heritage? For us to find out the answers to these questions is very important. Only then we can prevent from such patterns repeating in the future. We have over something like 29 varieties of lettuces and about eight varieties of basil. Most of these varieties will not be seen in supermarket shelves because this food heritage is getting lost every day. This whole chain is controlled by a handful of corporations. It would be sad to see such vegetables being presented in a museum saying that these species went extinct. We must safeguard the indigenous varieties of not just India, but across the globe. We are blessed because we are connected with farmers across the nation who are growing crops without chemicals. And here today, we support them by providing them the know-how we help them save their ecosystems seeds so that they can propagate them and safeguard those varieties that are being lost every day. As I speak, 200 species are getting lost. Who's held responsible for that? For me, working with food is extremely satisfying. Some of these traditional vegetables, they can be shown how good they can taste when they are cooked in the right way. I am extremely inspired by the work that Sangita does. If we don't have such seed walls and banks, these seed varieties would just get lost. But to bring these seeds to use, we will have to start cooking with them. I discussed with my mother and we said, okay, let's do a hundred days challenge where each day we cook with one of the under-recognized vegetables. There's a lot of engagement and they started contributing. The biggest surprise that I had was that a lot of them were right around us, like the pumpkin flowers or the moringa flowers. You know, if you forage for them, you can find them in your street, in your garden. The notion that anything that you cook with has to come from the market or it has to be packaged in a fancy packaging. You know, it's hard for us to accept that anything that grows around you that is edible can be cooked with. So here we have frucuma and onions. These are all the avocado, 12 varieties of brinjal is sown. We have chilies, fennel, basil, 15 varieties of tomatoes that are sown here. This is carrots, parsley. There's rosemary there. Everything is here. In just two acres, we showcase our diversity as rich as over thousands of varieties. Oh wow. And if today we are reaping abundance, it's because we respect the soil. We respect our planet, our earth. Seed cannot belong to a handful of multinational corporations. Seed is a gift of life to life itself. She's the Indian Bernie. The seeds cannot belong to a handful of 1% of the corporations. That's right. Give them back to the farmer. And let the farmer have them. I agree, Indian Bernie. Burn down all the factories and replace them with farmland. Bernie's the best, man. I love Bernie. That should have a different title, man. Also, I would love for them to go over some of the vegetables that aren't in...  Specifically, he had mentioned, he was looking at his grandma's cookbook and she was mentioning vegetables and recipes that he had never heard of. What are they? Yeah, I would love to know. What are those recipes? How do they taste? This is a wonderful introduction to this issue. Didn't know there was such an issue. I did. I live with my wife, who is obsessed with plants. They have an indoor garden. Yeah. She gets it from her father and he gets it from his father. They're both very agricultural. They're not farmers, but they have farmland. Green thumbs. Yeah, green thumbs. She grew up. Her dad would walk down the road and be like, ask Stephanie and her other sisters, but Stephanie really took to it. Like, what plan is that? And so she was always able to recognize different kind of things. And her mom always tried to cook healthy and all that kind of stuff. So she's always been about that. And so my wife's always been about native plants, not bringing anything that's not native to where we live into... She'd do good in Australia. No, yeah. It's so funny because I saw a TikTok. And it was about this guy talking about how detached we in America are from our food. And because he saw somebody...  And the guy was like, why? And she's like, it's not from the store.  They don't even realize where food comes from. No, it's true. And how it just grows in the tree and that is probably better than what you're getting at the store. Or even you have mentioned this before, where I remember when I was teaching because I was around teenagers all the time with these kinds of topics of conversation would come up.  And I said, well, we eat the meat, we don't do it for sport, we eat the meat.  So the majority of them raised their hands. How many of you've killed an animal? One raised their hands. And I said, you guys have a disconnect. And you think the anesthetized pink, you know, wavy meat in the plastic thing, you think that's meat and missed the fact that what it took, same thing. I sometimes, and John and I, I've always been this way, but I'll get, depending upon the day, I can get emotional walking into the grocery store to see the overflow of produce that's just sitting there at my whim to just grab and take. And also, we're really dumb in terms of not just what we eat, but what we think is healthy because we just assume that something says it's organic, that it's a better thing to eat. Yeah, no, we're very disconnected to what our food is. And we do have food here as well that has gone extinct, really. And similar to this where it's just not produced anymore. Well, and we do things like strip white rice of its husk and I think it's better because it's brown when you actually have a dietary profile of it is actually worse now. There's just so many things. And this, I love, I can tell how important this is to her.  She looked like the famous Indian actress, what's her name? She was in Tenant. Oh, yeah. What's her name? Dang it, I know who you're talking about. Very famous, been around forever. We've seen a few of her films. What's her name? I don't remember, but Dimple Kapati. Yeah, she reminded me of Dimple. Isn't? Is that Nessia's wife, I think? No, I don't. Am I crazy? Anyway. Anyways. Yeah, I would really love to know what those vegetables were. And I would also like to know, I assume based on what she was saying, that industrialization has hit pretty much every continent. I would assume it's an issue there, like it is here because it's an extraordinarily huge issue here. Oh yeah. Of who controls the food. Here, it's Bill Gates. Okay, you didn't know that. Largest owner of farmland in America. There's basically, it's Bill Gates. I think there's, it was five and it's now three companies that control 90% of America's food. It may still be five. No, Bill Gates is in fact the largest owner of farmland. The other thing that boggles my mind is the foods that are popular and they're just nothing but poison for you.  You look at them, and packaging as well, the waste in packaging is off the charts. Are you saying America's a waste roll? Wow, rude. Anyways, let us know your thoughts on this if there's other videos, if you know anything about this, obviously please let us know. Yeah, I would love to learn more. And my wife already probably knows about it. Let us know down below.