Welcome to my session. I'm going to talk about Chef and M Collective and how you can use those tools to build your own clouds and control a lot of infrastructure. First, a couple of words about myself. I work in Berlin. I do a lot of infrastructure stuff, a lot of work around deployment. That's how I got drawn into my project called Scalarium, which is a configuration management tool for controlling clouds on EC2. The talk will be part of lessons learned on building such tool and the lessons that we took while working on a big infrastructure project for a client with 1,000 servers. What all of those have in common is that we talk about cloud computing, controlling clouds, so what is it? In my context, it's basically infrastructure as a server, so mostly Amazon EC2, but for a lot of people it also means in-house capacity like VMware, Eucalyptus and stuff like that. So servers via API. The problem is if you use one of those, what you end up is you get a wild mix of servers. You can boot a Linux server, you can boot an Ubuntu server, but it's basically just plain infrastructure. There is no coordination, there is no real system. And what you really want to have is some kind of a system that I can deploy my app to. I want a structured system that is fully configured where the individual components know where the other components are and what they're doing and where you have a lot of automatic configuration and so on. So what you really need on top of those plain APIs is automation. Automation for configuration, for scaling, for deployment.  Of course there are commercial automation solutions like engineers that I mentioned, right scale. They do this mainly on Amazon EC2. But what I'm going to talk about is how could you build this for yourself. Which I don't recommend unless you have a very good reason to do yourself. But there could be reasons. So for example one of our clients they had 1,000 machines in-house that they already used to deploy a very large real estate application with millions of users. And they had 1,000 machines. And you cannot just like say, oh let's go to complete it to Amazon and just write off those 1,000 machines where we have like 10 full-time admins working on those. We have like very clear security guidelines and so on. Another reason as I said, governance security policies, if you're working for the government, if you have very sensitive data it's very hard to give those away. And mostly flexibility. If the commercial solutions don't fit exactly your need or mostly it's the other way around that you are so inflexible to come up with a good solution. That's, in this case what the enterprise company had to, like they were convinced they had to build their own because nobody covers our requirements of course. So what do we need if we want to build a system that automates infrastructure with thousands of machines? What are the basic ingredients? And those are like the core components that I want to focus on. What you do is you, of course you have all your thousands of machines that you need to configure and individual classes, individual projects. And you want to somehow control them. Controlling means you want to deploy your software. You want to start a new service. You want to replace failing systems. And mainly there are three components that you need. The first is, of course, this command and control infrastructure that has all the logic, how the instance should be used, what kind of configuration you want to have and that executes the commands where you go to in order to deploy something. Then you have the communication channel. And then you have on the individual host the actual configuration. How do I turn a fresh booted instance into my red application server into my process database? So those are the three components you have to implement. As I said, the first one is probably the one where you that is mostly individual to your constraints that is very dependent on your processes, on your protocols and so on. So the main idea is you talk, this component actually talks to the infrastructure APIs. So in the case of Amazon, it talks to EC2 to start service and it handles all the interaction with the user. So either via an API, via the command line interface or via the web UI. And it stores the configuration as I said, what do you want to do on the machines? I'm going to show you an implementation based on fork, which we use to do all the interactions with Amazon. A simple rate web app and a call to be repository. But more on that later. The communication channel is in theory very simple. It should just relay the messages, relay what you want to do to the individual machines. But the problem is you want to do build it in a scalable redundant way. You have thousands of machines. So you need a scalable solution. Just logging via SSH doesn't work. Because it doesn't scale across multiple machines. If it doesn't scale across data centers, you need some kind of an asynchronous scalable solution. And what I'm going to talk about is how you could do this with M Collective. Which is a tool that I think very few heard about, who already heard about M Collective? Okay, okay, a couple. The host configuration is probably the most obvious stuff. How do you actually turn the machine? And how do you actually execute the local commands? So mostly this will be Chef. I guess pretty much everybody already heard about Chef, right? Who has actually experienced what was working with Chef? Okay, because I have a part where I do an introduction to Chef. But depending on the time, it could be that I have to make it really short. So those are the three components that you need. And let's start at the bottom if we're going to talk about those. So the host configuration. Yeah? As I said, Chef. Chef is a Ruby DSL to describe how you want your server to look like. And then Chef does the magic of turning it into the machine that you desire. And the great thing is that it's a implemented Ruby. And the configuration is also Ruby. So it's a Ruby DSL. It's very easy to extend, very easy to look at, especially if you're a Ruby developer. But also, we've seen Java, Perl, PHP guys using it to build their machines. Because it's very, very high level and very easy to read. The basic idea is I have a blank Linux machine. Then I have my Chef executable that takes cookbooks. Cookbooks are in Chef. The general definition, how could a server look like? How do you achieve installing a patchy? What are the necessary steps? What are the recipes in Chef terminology? What do we have to do in order to achieve this? So your cookbooks are your complete repository of possible configurations. And then you have the configuration file that just tells it, okay, for this specific machine, I pick this script and this script and this script and I have those four configuration values. And then Chef goes and changes your machine to look exactly like you specified. In an operating system independent way, which is very, very great about it. Because you can have Linux machines, you can have open Solaris machines, you can have BSD machines. Pretty much every flavor of Linux and Chef will do the right thing. It will choose the correct package manager. It will know how to read and create files depending on the operating system. And at the end, you hopefully have a machine that looks exactly like you specified. It has rails installed with your specific version of Ruby, your gem sense on. If you look at Chef, there are two ways how you can run Chef in a big scenario. The first one is if you pop up the Chef documentation at the website and you read about it, you read Chef server everywhere. And Chef server is probably the way Chef is intended to run. So the idea is it's a client server architecture. I have the Chef server on the one hand, which stores all my cookbooks, all my global configuration, and I just have to upload it and change it there. And then I have maybe hundreds of notes that periodically connect to the server and download the configuration and apply the configuration if it change. The good thing is you don't have to maintain the configuration on individual machines, but it has a couple of drawbacks that I will talk about in a minute. And the other way how you can run Chef is focusing on the single server view. It's called Chef Solo, which basically means you push the configuration down to the machine and you just keep the part that does the actual bootstrapping, that does the actual logic of transforming a machine into what you want to look like. And those are the two possible ways how you can run it. And I think that you should focus on this way if you're running an architecture like this for a couple of reasons. So for once Chef server is very complex compared to Chef Solo. So everybody who started to look into Chef has to know about Rabbit and Queue, has to know what couch to be, it used to be that you have to use stomp and so on. So it's very complex. You had at the beginning they used OpenID for authentication. So it's a very complex piece of software. And what you want to achieve is pretty simple, just push configuration. The problem is that for all this complexity, you don't gain a lot. They don't have a concept of different environments or stages. So the new version of Chef is going to get at least a different environment. So you can say in production I choose this configuration variable and staging another one. But it's very hard to scale this approach if you have 1000 machines, 10 projects and each project has like staging production tests in QA. Because you have only one global repository of cookbooks. So everybody is going to step on each other's toes. The other problem is that there are no life cycle events. There's only one global Chef run. And you cannot really force it. You can only wait until the client connects and then applies the configuration. And it doesn't work really nicely with deployment, for example. If I want to deploy a new version, I want to do it now. I don't want to wait like half an hour until every client connects and applies the configuration. And while doing so, also checks of Apache is installed. But I just wanted to deploy the new version of my code. So the Chef itself or Chef server at least is more aimed at, I'm pushing out configuration changes. But if you want to handle like a complete life cycle of a server, including deployment, it's probably not the right choice, at least for now. So what we ended up choosing twice for like first for Scalarum and then for the big client project is an approach where we use Chef solo and push the configuration via, so either via Nanite or in this case via mCollective, which I'm going to talk about shortly. So the idea is that I use my communication channel to just push the correct config files to push the correct configuration and cookbooks down to the machines and just let Chef execute it and do what it can do best is apply transformations in order to make my machine look like it should be. So yeah, because of the time I'm not going to talk too much about Chef, I just like going to give you a brief intro probably for those who don't know how Chef roughly works. So the main idea is that you have cookbooks for the different things that you want to achieve. So in this case, I have a MamcashD cookbook that I use in order to control MamcashD, in order to install it, start it, stop it and whatever. And the general workflow is I have attribute files. So I have a folder where I put attributes files, roughly like variables that I can extract or same defaults. Then I have the actual recipes. So the recipes are the part that actually has the logic of how do I install MamcashD. And I have templates which I use in order to create files on the system. And like all Chef recipes, all Chef cookbooks have this basic structure. There are a couple of, there could be a couple of other things like files and metadata and so on. But this is the core of Chef. I have a cookbook that consists of attributes, recipes and templates. Yeah. So an attribute file just declares defaults. So in this case, we declare some defaults like what should the user of MamcashD be, what the port where I want it to listen and what should be or how much memory do I want it to have. So this way you just extract variables so you don't have to use the same value everywhere on your recipes. And by doing so, you can nicely also override it. So attributes have a way of overriding them, which I'm not going to talk about too much, but you can override them, which is great for having defaults and then have the user decide on different values. If we look at a recipe, so this is the actual gist of Chef. In this case, we're defining a service. So we're telling Chef, there is a service on your system or maybe it should be, but I can control this entity called MamcashD and I define how do I start it, how do I stop it, and what we want to do with it. In this case, I don't want to do nothing, I'm just defining it so I can reuse it later. And so this is the service. So I have multiple recipes. In this case, this is a service one and then we have the install one, which includes the service one because I don't have to really declare the service everywhere. And then I'm actually installing a MamcashD. So I'm calling out to the package resource in Chef. So it's a Ruby method that installs a package for me. And so I'm installing two packages and then I'm creating a template. So I create a file on the file system dynamically with the input that I have. And the template is just an ERB template, which probably most of you are familiar with. And then the ninth addition is the notifies on the bottom which tells Chef if this template changed while you created, or if the version you created is different than the one of the file system, fire this event. So in this case, we're restarting MamcashD if the configuration file changes. So if like I install the system once and a month later, I change, for example, the memory value and I rerun this, sorry, I rerun the Chef recipe. It will restart MamcashD only by changing the config file. So it's a very nice twist in order to be able to declare dependencies but not like being restarting services all the time if nothing changed. And the last thing that is missing is the template. Pretty simple, just a very simple ERB template where I'm accessing the default values. So in this case, I'm accessing the memory and the port and the user definition. Very simple. So how do I run it if I have those lying on my file system? I first have to have a configuration file. So this is just a plain JSON file that the most important part is the run list part that tells Chef what do I want to execute. In this case, I want to execute the MamcashD install recipe and I want to access another recipe that we haven't seen. But the nice thing is I can override the memory value as you see. So it's very easy to have defaults and then in your JSON file override them. And once I have this JSON file, I can just call out to Chef Solo which is the part that takes the config file, takes the cookbooks, changes your system. And this is how we call it via MCollective. So we have just Chef Solo installed in the machines and we push new configuration values down and call Chef to in order to execute it. You can do a lot more with Chef than just installing packages and starting services. So you can create users, you can create routes, you can create file directories, delete them, execute arbitrary calls. So Chef has a lot to offer. But I hope that you got just how Chef works. Probably the important question is how do we do deployment with Chef? There are two ways. One is you can use the Chef deploy recipe which is also a resource where you can specify please check out the code from here and put it there and then maybe fire callback which works nicely for like source dependent deployments like a typical Rails process where you check out code, touch temp restart on passenger and restart it. And it's compatible with the Capistrano way of deploying. So it has the same file system structure. So you can like deploy via Capistrano once you've set it up with it. And the other way is to have just an arbitrary script because we can, everything that you can script you can do with Chef. It just has like nicer capabilities to express yourself. So you can call, you can like download the VAR file from somewhere, you can like do a code check out, then do a configure make make install. So the only thing is you have to write those steps yourself. But you can do pretty much everything with Chef. And it's then nicely wrapped in error handling and notifications and so on. Yeah, this is an example probably not important too much how a deploy resource looks like. Yeah, so this is the component. How do we actually change the local system? We use Chef for it. So then the next question is how do I, how do I propagate the messages to the instance? How does, I'm a rates application server know that it has to deploy now? And we do this via mCollective. mCollective is, if you look at it website it says it's a framework to build server orchestration or parallel job executing system. So it's, it's designed to propagate messages reliably across probably thousands of machines and allows you then to implement the actual logic of the orchestration. So it's just a means of communication. It works basically by using ActiveMQ as a broker, as a message broker. And the idea is that you have agents running on your systems that you want to control. And then you can talk to the, to those either via the Ruby API. So there's a Ruby gem to do it. Or there is a small command line interface where you can do pretty much the same thing. And the idea is that you just call agents on remote systems.  So how does it work? The general idea is first you have your client and you ask the broker, I want to call us, like all agents that have a certain condition. I want to know, I want to call a chef on all of those. So what mCollective does it first it discovers who is online, who is listening to those messages. Because it could be that a couple of machines just shut down or crashed or you've put it up a couple of new ones. So first it has a small discovery phase where it finds agents that respond to your message. And then you can call those actions and mCollective will then return the individual results to you. So you can inspect and see, oh, the call failed on this instance but it succeeded in those 10. And by using ActiveMQ it's a very, very scalable and fall toned away and asynchronously. And it scales nicely over thousands of machines because you're not like waiting for the individual responses. You're just publishing messages and it's very easy to scale this. How does an agent look like if I have to implement one? So this is like the simplest agent that just echoes the data back to you. So you validate the input where you can say I only respond. So in this case we have the Hello World agent with an action echo and I validate that I get a message that is a string and then I just reply with it. So very simple. And if I want to call it there are two ways. The first one is via the command line. So mCollective RPC. So I'm calling the Hello World agent with an action echo. I give it a couple of arguments and you can see at the top it's firstly determine how many agents are listening, how many are reachable and it has a timeout of two seconds by default. So in this case it found one agent listening and then it gathered the responses and at the end I've printed out the actual output. The other way how you would call it is out of Ruby. So this is a very simple script how you could call it via Ruby. So you just require mCollective and you find that I want to have the RPC client for Hello World and call the echo action on it. It's very simple. Nothing too fancy yet. So what is great about mCollective is because it's using active MQ you can use like broker possibilities like fan out and topic broadcast to find out who is listening to your message and who should respond. So and mCollective those are facts and filters. So the idea is that not all my agents are equal. Some of those should be application servers, a couple of others should be database servers and then I have load balancers, I have caching servers and what have you. So I can have user defined value so those are usually classes which the agent loads on booting so you could have just a simple configuration file on every machine that tells you you are web server, you are load balancer, you belong to project A, you belong to project B and then you can filter by those. Another mean would be facts. So facts are things that the agent can introspect about the local system like what packages are installed, what services are running and so on. And the great thing is that you can extend it via Chef in Ohai so you can get information that Chef knows like what packages are installed in order to use those as fact filters in M-collective. So how would you do this? It's very simple again via the command line you just say I want to find all hosts that have a certain fact so in this case I'm not calling an agent but I'm just finding what hosts are available and I only want to return those that have the fact country is UK. Just an arbitrary fact. If now no agent has it nobody will respond to me. And the other thing I testing for a class. I can do the same in code so this is how I would do this in Ruby, how I would filter what kind of agents should be responding to me so in this case I'm only interested in dev servers and only in servers that have the country flag set to UK. So this way it's very easy to leverage the capabilities of a broker to find out specific hosts like I want to deploy to all my production machines that belong to project A. And those are just filters and M-collective make sure that it only calls the correct hosts. So in our case this is an example of the logic in the command application that called Chef on a couple of machines. So we just have a client that we instantiate and then we call the actual RPC method on the Chef agent and gather the responses and then return it. And of course somebody has to introspect those and see did everything succeeded or not. And the other part is the agent running Chef or calling out to Chef which basically the most important part is just the call to Chef solo. And the other part is extracting the JSON configuration file that you're giving it and making sure that you can log it and so on. But the basic idea is I use this, I use M-collective to call out to Chef and I always give it a JSON config file for example. And this way I can scale it across hundreds of nodes. So how do I can ensure security in such a setup? If you have a lot of machines, if you're so big, of course security is a big problem. How can I make sure that like nobody can hack it to my box and then control 1000 of machines? That a couple of different means how I can do this in M-collective. So the first is at the client level. So of course the client has to connect to ActiveMQ so you can have an ActiveMQ username and password. Probably more secure is using AIS and RS8 plugins where you can encrypt and sign every message and then every message and then the actual agents can verify those and say I have the certificate of this user and I can verify that this message is authentic. A simpler version would be just using SSL which gives you the signing part but not the encryption part. And of course you can use TLS for all the communication with the client. So that's a broker. On the broker side you can have ActiveMQ permissions to say what clients can talk to what topics and to what exchanges on the broker. So you can say that you have different versions of your client and one can only talk to production machines and the other one can only talk to staging machines. And at the agent level you have everything that because the agent is basically just another client. So you can have authorization ACLs and you can have auditing. So you can just have a very small plugin that logs every command and what certificate. So basically what user called what command and log it somewhere. Those plugins are included by default. So it's very easy to have an auditable system where it can make sure only certain users can call certain customers. How do we scale such a setup? How do we make sure ActiveMQ or M Collective works on thousands of machines? As I said it's not too difficult to do because your only problem is scaling ActiveMQ which is a solved problem. Probably not always very nice but it's you can have a network of brokers. So you can have for example per data center one and you can talk to each other and relay traffic. You can have clusters so that the client always tries to talk to multiple ones and if one fails it tries another one. And you can have master slave setups. So it's not too difficult to have a very scalable and reliable infrastructure with ActiveMQ. Of course in practice it's always more complex than that but at least there is a way where you can do this. The nice thing is that ActiveMQ is very very good at handling thousands of messages. So even if you have hundreds of machines you don't need like 20 brokers or something. So what we did in this case was having like one broker per data center and then just one aggregation broker where you can then fan out to the individual data centers. And then you can handle the command and control part. This is probably the most individual part so I don't have like too much to show but more a couple of general concepts if you want so. So the responsibility is this is the place where I store the configuration. I store how should my clusters look like. What instances are part of what setup and how it interacts with the actual infrastructure provider. So it starts and stops the instances. It replaces failed ones. It allocates IP addresses and so on which is probably the easiest part and the boring part because it's just a couple of very simple API calls.  So I think queries with every call what service are responding so it doesn't notice if an instance is failing. So if you configured I want to have 100 instances in this cluster and you say please deploy and only 99 are responding and collective doesn't care and because it always does like an on-demand discovery it will not notice anything like an event like an agent disconnecting or something like that. This is tricky to do this right. But the main responsibility is in the end to generate those configuration files for Chef and depending on the role, depending on the configuration. This part is probably similar to all cloud management software that are out there and this is the first part where most projects agree on the custom part is at the lower end so user management authorization like the business processes how should you restart things so this is where it gets complicated in big companies and this is the main reason why they want to have usually their own custom solution. Because they say we always like before we deploy an application we have to get like an acknowledgement from somewhere and so on so it's usually very complicated and usually very custom. Not very efficient but those are enterprises. So this is the part where you have this custom logic like who should be allowed to do what kind of functionality do we want to offer to what users and all the deployment recipes of course are usually very very custom. So if you look at for example one configuration how it could look like so this is a very simple JSON representation of the role which is like the core concept of how do you want to how do you split your clusters so in this case we have a rate application server role where we define multiple chef recipes that we want to have and as I said this is one drawback of Chef server that it doesn't know events so you only have one global chef run but in this case we have a set up event, a configure event, a deploy event and undeploy and shut down where we can find very granularly define what do we want to do on certain actions and those would be the things that you configure for your customers and of course you can add what kind of instances should be running in this configuration. So in our case the architecture looked like this so we had a lot of so we had a very simple rail tab that is the endpoint for the user either via API or via the web which just manipulates this storage that I just showed you in this model of a role of cluster. In our case we stored it in but it's basically just a means to manipulate a model that you have of your system and then you have again on the back inside a lot of internal agents that then go out and call EC2, call the actual agents on the system and push to respond to the changes, update the model that you have in the database and then the user sees oh deployment failed or whatever. So in our case all those internal agents are either also M-collective agents or there could be just simple risk worker or something like that. But the general idea is that you use again the message bus in order to publish what did we or what from our experiences what works well and what doesn't work well in the system. We made the experience that Chef is probably a great tool for doing configuration but it's also very easy to write spaghetti recipes. So it has a low learning curve but it's very easy to get very very unreadable recipes and especially in bigger enterprises the idea of having to execute the configuration always and always again is not very appreciated. So what we started to do is to go more and more into the packaging step so to use Chef to only execute package managers for example configuration but for the boot strapping. Another thing is that that Chef has some annoyances and if we have a couple of minutes I can probably talk about those two so one is that Chef is not really idempotent and the other is that the two phases and what I mean by write once and test everywhere is you write one recipe and then Chef executed on thousands of machines but the Chef run is dependent on the local machine. So for example we had a case where like one of 10 servers will not compile PHP correctly. Nobody knows why, it's just like some weird case and the better solution is of course to have a package version of PHP and then where but then how do we get this package from so you have a Chef script that then creates a package and so on but it would be nicer to be able to have a binary output out of Chef run that you then only propagate to machines which is also a lot easier to verify if this machine has all the correct configuration and has something that Chef is missing. From our perspective MCollective worked out great as a tool. The only problem is as I said the missing presence and event notification so you have to have some kind of an internal agents that do keep alive checks and heart beats and so on. Nanite which is a different agent framework that also it has those but it has a couple of other problems that we can talk afterwards probably and a very nice information or a very nice change in MCollective is that the guy who wrote MCollective works now at Puppet Labs so there is going to be probably a much tighter integration with Puppet in the future with MCollective which means that MCollective probably will have built an agents for doing the same thing that I just showed you to do with Chef but we don't have to reimplement it where you can already just call out to Puppet. Yeah so we have a couple of minutes so are you interested in why Chef is not item potent and why two phases of Chef suck? Yeah? Okay so the first annoyances of Chef is that Chef looks item potent but in reality it isn't or at least you could again say it's a bug feature so in this case we're looking at a directory resource of Chef that should create a directory for me in this case I want to create data logs and it should have the mode 0644 it should belong to Mike and it should be of the group users and if you don't have this directory on the file system Chef will create it for you exactly like you specified it what happens is if the directory already exists Chef will say oh the directory is there I will just skip this step at least what I would expect is that Chef will then see if the user and the mode and so on is correct or not but Chef doesn't so if you have like a server where the directory is already created and another one where the directory is not existing and at the end you expect the permissions to be the correct ones on both machines you're wrong if you want to you could say is it a bug or is it a bug in the directory thing so there are a couple of other places where it's like this but especially if you have an undefined state of your machines beforehand it can be tricky because you have to then to ensure that the permissions are correct and so on another thing with Chef is this one would be easy to fix just to make sure that all the resources also to make sure that all the sub attributes will always be correct the other one has more to do with how Chef itself works so if you have a more complex Chef recipe like this one I want to touch some file on Red Hat and then I check if this file exists I want to restart Apache does anybody who maybe has done something with Chef see a problem so it will the second part will never be executed even if we are on the Red Hat machine the reason is simple is because Chef has two phases the first one is the compile phase and this can be very annoying if you are playing with Chef and you don't understand why it's not working because this is totally valid Ruby right the problem is that it's not doing exactly what you expect so at first Chef loads all your recipes and compiles them to its in-memory representation and you want to do and change on your system and then it goes on and executes it so the result of the execute method is not that the file is actually touched it is that the in-memory representation of Chef would change to know that you want to execute to touch some file on the file system so when the if statement is executed it wasn't created yet so at first it loads everything and then it executes it all at once of course in the same order and so on but because of those two phases you have to to use a couple of other helpers in this case the correct solution would be to use the only if there is an only if and not if helper that do the check during the runtime and not during the compile time and the other alternative would be to execute it right away so during the compile phase to execute the resource and yeah, so this is very important to know if you're working with Chef like nearly everybody runs at some point against those two bugs or features of Chef and it's very annoying yeah so I'm closing with a couple of bad remarks on Chef any questions? yes please yeah so we do this by out of the recipes to call out to different web services and to see of those already so a typical thing is to that you only want to restart your application service if the database server are accessible are running and we do this by also supplying information about the cluster state inside the configuration file that we put down so an instance always knows what the state of the other instances are and then inside the recipes you have a check that for example the restart recipe of the web server would then check are there any database servers available or not and if there aren't it would fail and you would get the exception back upstream where it can then introspect it or you can of course automatically handle it and retry for a couple of minutes or something like that yeah so the question is how do we bootstrap Chef on the machines so we on EC2 we use the user data script for it so on Amazon EC2 you can give machines arbitrary user data so metadata if you want so and inside this we give it a simple shell script to install the like the agent and Chef and what most Linux versions on EC2 will do when the init script detect that they are on EC2 they will check if there is this metadata available for this machine and if it is they will download it and if it starts with a bin as H or a shebang it will execute it so this way we bootstrap the machines on EC2 on the in-house systems with VMware we had the agent already on the images but with EC2 it is very simple using the user data script so Capistrano is definitely a much more simple tool so if you have only a couple of servers that you don't bootstrap on a daily basis Capistrano is a lot simpler, a lot easier and I would recommend using it if you have a lot of machines it is not very easy to do this by Capistrano because Capistrano uses Synchronous SSH connections which means you cannot do a lot of those because usually firewalls will prevent you or network congestion so you cannot do 1,000 SSH connections push a lot of data over it a lot not very fast and not simultaneously if thousands of things should happen to your servers so what we ended up doing is porting all things that we did with Capistrano to ChefDeploy the nice thing is that ChefDeploy also supports the callbacks of Capistrano so everything that you can do before Update Code Chef supports the same callbacks but you can still because of the file system layout is compatible for one client they wrote a simple gem that then takes those callbacks and export your CAP file so that you could still use CAPDeploy from the command line and I think they were also thinking about writing a Capistrano plugin that actually only relays all your commands to MCollective yes ChefServer has data bags which is basically arbitrary data that you can put in that you can then use in your recipes or cookbooks and access and we do pretty much the same so in the JSON representation where I store information for this role I have like arbitrary JSON that I can put into too and then we of course give it all yeah so if you look at the architecture so everything is basically stored in CouchDB and then the agents look at the database and generate a JSON configuration on the fly and push it down and it's very easy, very fast and this way you can in the database we store what systems do you have, what representation should they have, what data bags do you have and then generate the JSON and put it down to the instances yes so because you have now a central place where you can query all instances and store monitoring data for example so we store a lot of monitoring and metering information in Redis and then have an agent that every couple of minutes looks at this data and decides should I start and stop more instances and it's very easy because you can query the instances and then you can just start once and they will pop up automatically bootstrap via Chef and be available so thank you very much


The next 7 Star Terrarium event has been announced for Pokemon Scarlet and Violet. We're going to go through all of the details in today's video. So as you may have noticed, I am in a little bit of a different location. I'm visiting my parents at the minute, so normal headquarters is not here at the minute. So we're making best do, but the next 7 Star event has been announced.  The new event has been announced, as you can see here, will be Typhlosion off the back of that Samurai event ending last night. Typhlosion will be the next one and completing the trio of those who Sui and start a Pokemon, which is, in my mind, at least a big hint to what will be coming soon in being home compatibility for Pokemon Scarlet and Violet. So we'll keep our fingers crossed for that one. But if we can take a look at the details of this event itself, as you can see here, the event will be running from the 14th of April this Friday until the 17th, and that'll be the first phase of this Typhlosion raid event. And it'll be running the second phase from the 21st of April until the 24th. So we're going to have those two phases as normal, back to back weeks. This Friday is the starting of these raid events, and it will be a Ghost Terra type, which is a big hint, big throw to the Hissui and Typhlosion, which has the fire and ghost typing. The thing is, other than that, it's going to have the Might Ismok as usual. You will be only able to catch one of these per save file. But the details are pretty sparse other than what we've got here. It's just going to be seven star, strong ghost terra typing, and there isn't much else to go on at the moment. But those are the details. And this is the Pokemon that we're going to be getting this Friday. Very excited about it, and I know probably a lot of you are as well in the run up to these last three Pokemon that we've had. The Hissui and Starter Pokemon are the ones that we're looking for. I think the Typhlosion is probably the most popular out of all of them. So hopefully you're looking forward to it. Of course as well, we will be doing some best builds on the channel for this event in particular. So we'll get that up either tomorrow or Wednesday. But let me know down in the comment section below what you think will be a good build going into this one. First impressions for myself. Something like Hydragen going to be good with that dragon that resists the fire typing, and the dark typing as well that resists the ghost. Things like Roaring Moon as well, going to be another option there that are going to be good. You've got to bear in mind that Typhlosion probably will have a Tidnability flash fire. So it's going to resist any sort of fire type attacks. It's probably going to play on Sunday day as well as a setup for boosting the power of something like Eruption. And then you're going to look at things like Willowisp for boosting Hex Damage if that's something it plays off for the ghost typing. So there's a bunch of options that we can look at and we can really drill down and get some really strong builds put together going into this event. So it'll make it easy for you to do when it drops later this week. But thank you so much for tuning in. It is a very quick one today. Obviously the setup and everything. We can't really go into too much detail in this one. But like I say, I will drop the best builds video later this week. But I would love to hear what you think of this new event, this new announcement. And I will see you all in another video very soon. Do drop a like, do subscribe to the channel and I'll catch you all in another video very soon friends. Until then, take care. Bye bye.


Hello everyone, so as you know that the speaker couldn't come so we will be taking up question and answers like Q&A on OSINT and no more about it and it's going to be an interactive Q&A so that I'm going to be asking questions to them and then we will be popping some questions for you as well so that it will be interactive it shouldn't be like a speaker is speaking and then you're listening that's it it'll be if you have a question in the middle just raise your hand will make sure that it gets answered so the first question so I would actually like to give it to Sudhanshu. Hi guys so basically the idea is to have a casual discussion a lot of times we do talks and then we listen what the speaker has to say but a lot of times you know the audience would like to ask some questions which is not related to the talk they're actually attending so you know this is the opportunity I mean not just for you for us as well I know are there a lot of things which I would like to ask and maybe some of you have questions some of your answers to that I would love to answer that and to ask questions all right so I would like to ask one thing like what kind of OSINT do you guys do is it more into security or offensive security or defensive security or you guys do investigation or it kind of makes for you guys I mean I'm just trying to ask this because we want to get more talks which are you know which the audience is more interested in so for the next time when we select the talks we'll keep that in mind that you know what kind of audience do we get me I've been doing pen testing for a long time right so I'm always a little biased towards that side but I did in my last job I used to do OSINT for investigation and due diligence so I know a little bit about that side as well. Sudhanshu has done a lot of pen testing he has always been an attacker so you know if you want to not just do OSINT but also get into security you know bypass antiviruses and all those you know malicious things he's the guy. Jennifer has been head of research in multiple organizations and she generally do OSINT in investigations on the people side and she's really really good on that. So I want to ask Jennifer that how do you do human hacking? Okay so I managed to think that I wasn't going to speak beside introducing speakers this year and look we'll be lo and behold I'm speaking again. So yeah so I kind of was here last year I spoke at the Diana initiative last year and then I spoke on the stage I had massive imposter syndrome and it's the reason why I didn't apply to speak again this year. I can't code in the slightest I can't hack what I do everything is behavioral science I hack people I follow the way that they behave I follow the way that they act I don't need to be able to break down things to be able to tell what their passwords are because guess what if they like their kids if they like their dog that's most likely their password if they're kind of simple about things then they're going to have leaked things on everywhere else. So I basically just follow people's behaviors people are inherently stupid and people like to follow patterns so if people have got like patterns of life behavior people like to but we're all tired of a morning so people use the same routine people get the same trains of a morning so people complain about trains the number as a London commuter we all sit there and we complain with as a London commuter Southwest trains why is my train delayed well thank you for telling me exactly what train you get on every single morning and then I sit there and I'm like I'm then sitting next to you on the train and I'm having a conversation with you about the local sports team because guess what you're following them on Twitter before you know I've just happily socially engineered myself into your life you don't even realize it but I know everything about you because you've put it all on your dating profile I know your dating profile sitting there because you've used the same picture as your LinkedIn picture why because you like the headshot that your company did for you which is also on your company profile like people are so so simple so that's kind of how I look at things so when I kind of worked alongside these guys in different ways and shapes and forms I've kind of looked at things in a really really different way and that I kind of come up for it from a well that's great what these guys can do and they can massively massively help me in different ways but actually from my perspective people are like you hear it so many times people are the weakest link but people are inherently stupid and whether it is no matter how good at security anybody else is there's somebody in your family who is stupid and it might be your 12 year old kid it might be your 60 year old grand who's just decided she wants to get an Instagram account to see what is going on but people don't understand I've got new bosses because I've just joined a new company and they've all decided we need to learn Twitter no you don't please just keep off it but they've all gone on there and they've all done randomly weird things like follow their kids great thanks you've also all joined your housing associations and now I know exactly where you all live no that's kind of what I end up doing so it's in a really roundabout way is what I end up doing and a really long thing this is why I don't talk because I can't stop so that's what ends up happening all right so I will throw a question to Sudanshu Sudanshu so when you you do any kind of security exercise be a pentester or a team what is your approach you know on how you start OSINT you know figuring out a company and then how do you go about it and how effective is this process I mean how is it different from doing just pentest not doing a lot of recon and versus you do a lot of recon so what are your thoughts on this thanks room yes so I've been doing pentesting for quite some time now and I identified that recon is one of the most important process not the exploitation or the privilege escalation or the persistent one if you know your recon if you identify the organization or your target you can do a greater amount of damage than running just exploits on them right so we do it all the way so the first thing that I would like to mention is if you have a name or target or a domain whatever you're gonna start with just identify anything that is associated to it right now it depends on the type of the set that you're targeting right if it's a domain that you can enumerate subdomains if it's an IP address you can see what are the IP addresses associated to it near to it or neighbors to it right so we can do different things around different technologies different assets if it's a website the first thing that I would try to do is to identify the open ports to identify the technology stack instead of just running a sequel injection or trying to run a burst scan over it right so identifying that anything that is associated to your target type that is the primary thing that I try to do once I have that information then I go into deeper like I say if it's a web application then I try to crawl that application see what are the keywords that they are using if there's any directory that I can enumerate once I have that information then I'll go into the exploitation or identification of the vulnerabilities so that is how we approach about vulnerabilities in general anything specific feel free to answer so do you have any interesting case where you find someone has done a really really big stupidity maybe you're leaking some information or leaving some misconfiguration in the system and if the you know stories you have around this these stories like sound really really stupid or simple sometimes like I was doing a pen test for an organization and they had just they just gave us a domain and said that okay enumerate the information identify the target attack surface and then in the next phase you're going to do the exploitation and testing of the stuff but in the first phase itself we identified some credentials being leaked on Github and some on Pastewin and even before the test was going to start we had the admin level access just because the credentials are lying out there so it might sound really stupid that it was a really simple assessment we didn't do any hardcore exploitation on RCE over them but just because that information was out there anyone could do it right so this makes them a really lucrative target because it's really really easy for anyone to do it you don't need to do know how the tools runs how to use command line or how specific technology works but just because that information is lying out there you can directly use and exploit it to gain that admin access and once we had that access it was really easy to get into the network and to pivot into the domain admin and it was game over right because parameter is one which people really care about once you're inside that parameter inside the internal network it's usually a DA or the credentials or the secret card details so that's usually how it goes. Well I'm surprised how relaxed in how relaxedly you say you just a DA you know it's not a joke. Yeah so from what Jenny said and what Sadanshu said I remember one of the story which is into security and as Jenny said people are inherently stupid so one of the pentas I was doing and then I was doing a little bit of forcing on the target I was already inside the machine trying to do some brute forces on the windows boxes and then searching the name of the company I find a good repository the person actually revealed the password the password was summer 17 and then I was like alright let me try that that didn't work so I went back and I was thinking hey so the first one actually is using this password and should work somewhere so I realized I went back again to the GitHub repository check the date of the repository it says 2017 so the next thing I was trying was summer 18 because I was doing the test in 2018 and bang on this work I was like what this shouldn't work man I mean it was it was as stupid and that day I realized that you know a lot of times people when you say that you should not use same password across multiple websites what people do is they will come up with the patterns right so the patterns I've seen is password you know name appended by you know the first three letters of the domain name or maybe the first three letters of the form fields some weird patterns and the problem with that is once you once someone figured out that pattern it is as easy you know as trying the same password everywhere so yeah so I would like to ask these you guys how many of you use the same password and at multiple websites or if someone would like to volunteer to share the same password yeah I mean yeah give a demo of it here yeah all right so now we've talked about the use cases we have talked about human hacking attacking but how do we do defense again like defense for OSIN what should we do to make sure that our data is safe all right so there is definitely overlap when you do OSIN for defensive security few things which you have to do is when you do attack you just do it once but when you do defense you have to do it in a recursive manner so you know you obviously you have to attack somewhere so you'll you know write scripts which will scrape get up and look out for your password sensitive information about your organization but you won't just check it once you will check it maybe daily or weekly and figure out you know as soon as someone goes out about your organization on on any of these websites you should get an alert notification of Slack whatever similarly you should have a really good track of your assets because when you are in cloud you don't even know if your box only has a private IP address or you know someone just a pen in a public IP address as well or maybe someone change the security group so you know you have to keep an eye on how many assets you have keep constantly checking them similarly if you have some vulnerabilities in terms of you know any of the any of the user civilian information or if you would like to keep a check if someone talks about your company in security manner for example someone talks about the company's you know I just hacked your company and someone tweets about it so before the whole world gets to know you should definitely know about it so you should have a check on these keywords maybe you can use the streaming API's of Twitter and keep an eye that as soon as someone talks about your company you should get an alert and you are the first one who should be taking action on top of it maybe it's a false positive maybe it's a negative it doesn't matter because it's in terms of reputation you should be on top of it right similarly you can use a lot of threat intelligence feeds for example if you have SIEM systems and you get an IP address hitting you and hitting your website again and again you can maybe check the rank of the website and assign the score to the IP address now if the IP address is sending multiple requests based on your score you can actually block the IP address a little a little earlier than you will block other IP addresses so you know you can and there are so many things about it you can use Google alerts you can use detection page detection kind of things to make sure you know what kind of parameter you are leaking out alright so I mean it cannot be summarized in just one answer of course there is no specific way to talk about it but overall these things should be checked in a recursive manner especially if you are doing OSINT on your own organization you should keep an eye on these things in a recursive manner. Thank you so can we have questions from the audience? Sure. So this is my question for Jenny right? Yeah. Have you ever ran into an individual or personality who stumped you? Can you speak to that? Maybe there's something interesting about their personality that makes it difficult for them to be hacked. Can you use this? So yeah. There's a couple of cases. And there's one that I'll kind of mention. People who have split personality disorder are really difficult to follow because which personality do you follow? So somebody that I was actually trying to map out had multiple personality disorder. Now I was able to map out 12 of their personalities and 12 of them all interacted and you could and the problem was they didn't all interact. Three of them interacted together and the rest of them were all separate. Now you could see there was a clear delineation of timings that occasionally the person ended up dealing with. Some of them ended up using similar kind of variations of phone numbers. Some of them used the same phone number others but it was really one of those that some of them as soon as the personality of the individual changed they used a different mobile. They used a different device. They went from a Windows device to a Mac device and it was one of those of a this is impossible. And it's one of those of a okay what is the purpose of what I'm trying to do here and what is it? And that's what it always ends up coming down to. And it's a case of how am I trying to map it out? When it comes to large criminal organizations so I work with a number of different charities. I work with a number of human trafficking charities and trying to track those organizations tends to be really really difficult because they are so much better and advanced than we are at monitoring them. So they know they know on methodologies better than we know our own methodologies in how to track them. And that's why we struggle because they know how to hide better than we know how to find them. So they're the ones that I normally find the difficulties for. So where I work with stop the traffic and I'm I'm kind of the Nigeria lead with that.  I will be able to tell you the people on the ground. I will be able to show you the picture and can I have their mobile number and can I give you anything more than that? I'll be able to tell the police on the ground here the details. And then I can't piece the final elements together. And they tend to be the weirdest cases. Sometimes I just can't find people. The less information I find about people the weirder it tends to be. Kind of bringing into what you were saying about how can you kind of reduce your the risk. I work with a lot of high net worth people because they're like oh help me reduce my footprint or people will turn around and go I don't have social media I'm safe and I'm like oh hello challenge accepted because I don't care if you've not got an active footprint I'll find you passively because someone's taking a picture of you if you've walked along this corridor today I don't care if you haven't got Twitter somebody's put you on their Twitter today and you've got if you live anywhere you're on an electoral roll somewhere your birth records are somewhere somebody's leaked some information about you so the passive records are out there about you if I can't find passive records to do with you and I can't find active records to do with you do you exist have you change your name if you change your name how have you done that because records to do with name changes are also kept and that's when I'm really then starting to question and if I can't find anything to do with you that's really when I'll start getting a little antsy and that's probably when I'll start rooting towards these people with a healthy find what I need and there you go generally speaking how would each of you hide from yourselves to you any other questions guys just one mic so for somebody that's just getting started with investigative recon what are some of the most valuable tools or services available that you found in your experience and there is no magic tool my boss wishes there was a magic tool there I there are certain things that depend on what you want to do people is a great thing I'm not going to advertise anything but people is a great tool if you have any form of detail like goes with a phone number or an email address it's a great starting point and I honestly from there on in would use a lot of stuff that you'll end up finding on github beyond that honestly the rest of it is just kind of be searching around I wouldn't go paying for a lot of the tools that are out there and that's simply my choice and I would recommend the data split tool that the boys standing next to me develop because it's amazing and yeah definitely check out the data split tool that these boys use and have developed and they don't use it they develop it what am I talking about but yeah take what I would honestly recommend is all of these tools have trials use them see if it's useful for you don't ever sign up to anything unless you do use the trial because that's the worst thing you can possibly do and most places if they don't offer your trial it's because they're trying to trap you immediately into something and you're just going to end up getting trapped never ever get trapped into that oh we've got all this amazing data if you love linkedin information rocket reaches your one forwards that will give you your email addresses from linkedin I would like to add to that something so there's no such tool that can help you in anything that you require right so because every time your requirement would be different sometimes you're trying to identify the domains of a company next time you're behind a person right so the best tool that I would recommend is your methodology right if you understand what you're behind and you want to how we want to achieve then you can achieve right so like Jenny mentioned we created a tool of our own because you're not finding anything out there which held our case it might not help you you can give it a try definitely but it's the methodology and understanding of what you are behind and how you can identify that that is going to help you and if you know that you can just hire anyone to code it or you can code it yourself so we can take one more question before we wrap up so when you're conducting some investigation for a red teaming exercise to participate in the reconnaissance phase and of course there are many aspects in this kind of investigation so there's the organization itself it's financial position many of the important employees affiliates you name it so you have a shit ton of information and you really need to correlate them in one way or another and maybe your brain is sometimes too little to do that as smart as you are do you have any tips and tricks on data management in terms of this kind of correlation yeah so as far as I understand your question you're asking about when we do any kind of investigation or OSINT or red team we find a lot of data could be in terms of important people or assets of financial acquisitions or whatever how do we keep track of that data is that what you're asking yeah so there are definitely a bunch of phases in this so one is that you document everything of course that goes without saying where you might use whatever editors you have or you know CSB files you should always record these kind of sessions maybe you can use browser add-ons or some screen to order software so that you know what you are searching for and if you find something you always have an evidence once you have the information the most important part is to connect the dots as you asked and none of the tool as far as I know will automatically do it for you so most of the time I've seen analysts using some graph software where you visualize these things you can obviously do it manually you know but you can use utilities like Lumify or you know case file you can use case file you can use multi-go where you know just pump data and then connect the dots those kind of things especially in the kind of exercises I do we don't really have to join a lot of dots it's not more than one or two layers of you know the information we have but especially when you are doing investigations or especially around people that's where you find some information on Facebook and then you know you reverse image that do this, do that so that's where a lot of layers are there so and I think in red teaming at all you don't need a lot of joining the dots kind of thing of course there are a few layers and it depends on the kind of size of the organization but honestly I've not encountered such kind of tests so far okay that's what we'll wrap up there and we'll move on to the next session thank you very much thanks


which is green flag site, and then there's rice with woods, which that's car top, which is also a green flag site. So, from an environmental point of view, we have to make sure we do the right thing. Almost, of course, it's over 700 acres of woodland there. It was on previous visit, we've parked up at Man Pest Wood and done like a sort of three mile walk and sort of finished off with a train ride. It was quite a nice time. I'm quite active, I like to come to the station. Yeah, it's very scenic route and we're lucky to have a railway in such a lovely setting. Yeah, I mean it's hard to believe you're sort of being great at London. Both it's a reminder of what was rural middle things. Yeah, it is. So this is 13 and 3 quarter inches, and you don't cross a public road, then you shouldn't bike from the health and safety objective rather than the office of road and rail. I see, yeah. So it's a slightly lighted up regulatory environment. Because I see you're really slippery. Particularly on the first run of the day. Yeah. This is where you're likely to start running in the traction problem, about zero. Yeah, as you can see, when the line terminate at Eleanor's, we're tracking in from Eleanor's down to Wellington about three years ago. Yeah, I've been on one of the last visits you were doing. Yeah. So this always was a passing loop once it ceased to be the term. By now we've been back onto single track. Right, yeah. Cheers. But it was wide enough to fairly easily accommodate the two tracks. So we're double track now all the way back. Yeah, from here all the way into Woody Bay, yeah, that's right. That's quite nice to have up here. Yeah, oh yeah, it's been marvellous. Particularly when we took them working, it's made it so much easier. And the great advantage for us at token working is clearly we don't need a radio controller. No, we can do it just with normal weekends when we don't want to go on radio controller. We don't have a controller anyway, but we can still run two trains on token working. That's massively helped because otherwise we just couldn't get enough people on the trains. No, the queues were becoming very difficult to manage and we had to stop selling tickets and it all became really rather... Yeah, I've seen rather different similar things happen. It's great that maybe we want to visit, but... So now we don't have to worry about that, we just keep selling tickets and we can always keep telling people, even if they just miss a train or the train is full and has to depart, another train will be here in 10 minutes. And people are happy with that, you know? Yeah. Worse before, another train will be here in half an hour. It wasn't quite so easy. Which will be taken down soon. There are some proximity sensors on them. So as you drive towards them, after about 10 meters or so, they switch on and start blowing artificial snow all over you. That sounds good. Children love that, don't they? Yeah, I can imagine it. All these cables here, clearly, are all Santas. Yes, I see. We haven't had the opportunity to pack it all away yet. No, well, it's okay. It's slightly uneven on here. If you come out wrong row, then you notice it. Yeah, Chris was saying starting on 140 grade.


In this discussion, we will discuss the discussion question of, what are internal controls and their purpose? So internal controls, it's going to be a big topic as we grow our firm or as we grow our company, we typically need more internal controls, which are typically kind of a form of bureaucracy that we're kind of put in place. But one that has a goal, a system, there's a reason for the internals growing up and the goals are going to be what we will discuss here. Note that as we have a small system, if we have a limited amount of people, if we have a sole proprietor, for example, then we're going to have less internal controls. One because it's not feasible given the number of people we have to have some of these larger internal controls and because two of the cost-benefit analysis. We always need to be considering how much benefit are we getting in terms of these goals we'll discuss as compared to the added cost that will be involved when we make a system that...