All right Hi guys, welcome to tails from the trenches. Oh, there's water. Thank you Welcome to tails from the trenches the good bad. I'll give over stack operations the good news is we are the only thing standing between you and the end of the summit So Unfortunately, there's a one of our one of my colleagues or one of my friends was not able to make it John Dewey He was supposed to give this talk So one of my current colleagues Jesse Keating is gonna help me give this talk today I am not the John Dewey you were looking for Oh, that's me. So I'm Jesse Keating. I have been for the past couple years the deployer of the public cloud We like to call our stuff the Kraken we release it frequently before that I was a release engineer for Fedora for about seven years Been a contributor to Ansible the orchestration engine quite a bit and to many other open source things Besides that And my name is Craig Tracy as this slide points out you may remember me from the failed keynote at the Portland summit But I've been involved in open stack now for probably a little over three years I've been a contributor to open stack contributed to a whole bunch of ecosystem type projects around open stack as well as other public public public cloud providers So basically day-to-day soup-to-nuts open stack Okay, so One thing when that we discovered when we're Coming up with the content for this talk is that each of us had very different perspectives. So right now I'm in the company called blue box the Business that blue box is in is selling private cloud as a service Which means we stand up very small or small to mid-size open stack clouds for customers Completely dedicated to them whereas Jesse's coming from a rack space a public cloud Which is an entirely entirely different thing, right? So I'm also a blue box employee, but a few weeks ago. I was a rack space employee in a rack space We dealt with a small number of very very large clouds. In fact, they were all pretty much the same cloud They all had the same off-end point You could move stuff or you could launch stuff and one or the other the other But essentially they were there were six mostly independent very large public clouds So just this is my own curiosity and we were talking about this just before the talk I'm curious to see how many people are our operators of a public cloud or open-sack cloud today Okay, and like let's say on is that multiple clouds if you're managing multiple clouds How about greater More than one Do you have yeah, how many things do you independently deploy to that you have might have different versions of open-stack running on Okay, and how about you like maybe more than 10? Okay, if you alright that also adds I think a bit to the perspective of what people are doing So the first thing we want to talk about is installation and this is really tales from the trenches but as an operator one of the things that you're going to have to deal with is installation and There's a famous quote. There's this famous quote. Does anybody know who said this? It was me It is not a solve problem. I maybe about a year ago. I thought that this was a solve problem and I One of the projects that I contributed to quite a bit was the open stack I'm sorry the Stackforge chef cookbooks I was a core contributor for quite a time quite a while and saw a ton of churn in in those cookbooks that that churn continues today There are tools out there that you can use you can use chef puppet salt Ansible and our tool is ansible base. It's called Ursula. There'll be a link for it at the end But one thing that made me really realize this is when I looked at the source base that we have on Ursula today It's 15,000 lines of code There are over 1500 commits that happen all the time and last week I submitted a pull request that was 1500 lines itself So is this is definitely not a solve problem yet? When I when I joined rack space, I thought it had been a pretty much solve problem, too They they were fairly well established I was coming in to help them out with a lot of the automation a lot of the the more interesting things that I thought of at the time But in reality as we got better and better at doing our rollouts and doing our installs We realized that a lot of the decisions we made and how we did our installs We're having profound impact and what we were able to do with our cloud later on and so before I left we were in the investigation process of Rediscovering and revamping how it is that we install and manage our open stack So installation by far not a solve problem. Yeah, and No, I think it's installation is a very Personal thing or personal to your business and and how you deliver it So these projects like the the open-source open-source chef cookbooks or even our code is very opinionated We welcome contributors, but and we'd be happy to make it as Unopening it as possible, but it still ends up being something like that. There's also a a relaunched effort at beefing up the Open stack playbooks for Ansible for creating an open-sac Environment it's going to happen on stack forward It's going to be a reference model for how to stand up and we're going to attempt to make it as Unopinionated or at least is to start with as unopinionated but with the hooks in there to make it as opinionated as you want to So as we as Jesse and I talked more about our different experiences, you know Again me me delivering lots of small clouds, whereas he's delivering one large cloud One thing that was that became clear to us is that despite our differences in the way we install despite our differences the way We operate it Especially around installation. There is some kind of convergence in the community It seems that people are starting to deliver things in in a in with artifacts Containers is seems to be a very popular thing right now There's a whole slew of new projects around delivering open stack with containers. Yeah I mean if you looked from the past couple years everything sort of starts off as dev stack You know, it's this great thing that you can do or in drop and it's a working cloud and everything's great because it worked in dev stack But once you take that into production, you realize that you can't really manage that But a lot of the production deployments particularly for the smaller clouds look a lot like dev stack We have all of your services running on a single system or very and if you want any sort of ha you just duplicate that over and over and over again But as as we Moved on it became more apparent that trying to run more than one service in the same environment was kind of a hassle particularly as each of the open stack services sort of became a little bit more independent started became becoming more opinionated about their dependencies and Operating at different schedules. It really really started to become apparent that each of these things are are loosely coupled which get to in a moment Service microservices that sort of interact with each other at a API level So the move to take these things out of being a monolithic thing that you just stamp out ten of them Into little individual pieces of services that you put in as many places as is appropriate for that service Is starting to really become clear at least to us and to other people in the operator community? Yeah, our job again spitting out large numbers of small clouds to make sure they look the same all the time So artifacts for us is key And things that we've learned along the way. So the first off was that distro packages. We're not going to cut it for us There are many many many times that we are going to want to add our own Changes to open stack open second is an internally evolving thing And we do not want to be beholden to the cadence of any kind of distro You could you could do your own packages, but as a guy that did my own packages for seven years That's not really a job that you want you want to be focused on getting your changes integrated with your code and out Into your cloud you don't want to be dealing with the the intricacies of how do you build a deb today? Or where did my rpm dependency go to build this thing and where do I put it? And how do I update the apt-cache and how do I make sure that all these things are in sync? That's just not interesting problems. So what's what's the better option here? Okay, how about another poll of the people who are deploying open-stack? Are you guys using distro packages? Maybe raise a hand Wow How many of you have made customizations to those distro packages? Okay, good pretty much So at the same time for us deploying from source is not really an option We have done this and this is proven to be difficult the problems that you run into are right off the bat dependencies so The the Python communities as well as the system level the system love dependencies that you have are constantly moving So I cannot tell you the number of times that we've been bit by some upstream Python change that's happened Or some system level dependency that's kind of moved on where we're not locked down enough The end for anyone that's accustomed to the way open-stack is delivered the requirements that text files are not hardly versioned They kind of say hey, we want this package greater than equal to one dot four and that doesn't really cut it in production Yeah, along with with the problems of deploying the source I mean, that's it's kind of the model you see a lot of of New-age developers or you just throw up a vagrant you get pull your source and off you go but Often you'll wind up with different things running in different places at different times and Craig, you know, he has a really strong desire that all of his clouds look the same Even though they're getting deployed at different times and as a the large huge cloud deployer I want all of my instances that are running the cloud to all look the same because I don't want my my operators and my Admin's having to figure out This version of this library is working here But this version other version of this other library this on this other node in the same cloud is not working And so we needed to have consistency across the board of what we were putting out into our into our environments so with that this was kind of a sore spot for me, so I Worked together with John Dewey and we started a project called gift wrap and gift wrap The whole premise of gift wrap is to take open-stack Which has always been delivered as an integrated release not a product or set of packages Into product into packages that you can consume so what gift wrap does is it goes off to? Garrett and basically decides what kinds of packages it has to use it uses Garrett And as well as DevSec to determine its dependencies out the bottom will land either an rpm a Debian package or now We will we haven't already working in test but not checked in yet a Docker container and that's the direct that's the direction that we're moving in terms of shipping our bits We intend to ship all of our bits in Docker containers So lay down a base OS with ironic because again, we're using images something that we know Then we will lay Docker containers on top of it again something we know and that's easy to upgrade We this was a really good convergence point kind of like you can see on the train map that Craig and I realized that Within rack space. We had an internal tool the public name now for the striker. It's a gonna be thrown over the wall soon I hope but it was doing a lot of the same things a slightly different way It would it would pull down get from upstream open-stack repositories use a tool called And I'm drawing a blank, but it integrates the localized patches that rack space might have into each of those different repositories Do an examination of the requirements files plus anything that we've hacked in that we've realized is a Requirement for this particular project or a certain version requirement and build individual virtual environments for each of our services We wouldn't go down the path of building a package or a container yet What we've done is just build a large tar ball that is all the different individual virtual environments. We toss in our configuration Manifests to configure all those services and wrap all that into just one big tar ball that we toss on our payload server One thing I didn't mention about give wrap give wrap also use the virtual line Technique for ensuring that the services isolated in terms of its Python dependencies So let's talk about operations. How many of you get well a lot of you were installers Do you also consider yourself like long-term operators? How many operators we got? We already asked that question Same hands. All right, cool. I have a short-term memory issue so one thing that was critical for us in stamping these things out is how we get to our initial architecture and We so we announced our our offering for the first time back in a year ago in November of 2013 at the Hong Kong summit and we went GA in May one of the things that got us to where we need to be almost immediately Was we unified our architecture meaning that we could Take any single one of our nodes and run any one of the services on those nodes And in many ways well we in in many cases. We're actually co-locating compute as well as controllers on the same nodes This Becomes problematic when you get into certain types of workloads and scale scenario. So for us We we made a very clear decision Recently to start breaking these services out and make sure that these service are services are composable by themselves So if you have a heavy let's say you have for instance a very heavy Workload in terms of image writing So if you have a 200 gig image every time you write that image to the same controller node that's serving up your API requests You're gonna take a hit so having these these services be composable in such a way that you can break them out into different nodes if you have to is it is a great thing to do so I In my estimation I'd say Going with the unified architecture to begin with is great It's a great vehicle vehicle for getting you started, but it may not be where you want to land long term At the same time you want to make sure that you're not building snowflakes, right? It doesn't every stack again We're in the process We're in the business of building or stamping out these stacks one by one We don't want snowflakes that we can manage just like we don't don't want to manage these crazy dependency chains The same thing goes for how we stand up the stacks snowflakes are not allowed in what we build And with that comes some kind of improvisation, right? So if one thing that I always think about like in terms of the way we built our stacks Because some of them are so small that we're not including things like Swift We're doing things behind the covers that make it look like Swift. So it's what like look like Swift is there So for instance if you think about glance glance can back right into Swift and then you have you can have multiple glance services that are talking back to Swift for Sharing up images because we are private cloud dedicated to one customer They might they might not have the capacity for a Swift cluster in this case We are doing things like our syncing images across different glance notes to make sure that the image is always available for a glance request and My last Recommendation around architecture is that is to always make sure that you're separating your control plane your data plane operations so You will undoubtedly no matter how big or small you are be asked to add more capacity to your cloud and By separating the control and data plane paths When you get the new node addition or quest You will be able to operate on that single note instead of it touching the entire stack So this is a less pretty graph a less pretty picture of very small text and then I could find earlier today This is more of a picture of what a Rackspace public cloud looks like so Craig has a very interesting constraint in that His cloud has to exist entirely for his customers We stamp out a whole control plane for each customer and it can't be co-located with anything else Which means that they have to utilize some of that customer capacity for their control plane Rackspace had something of a Big benefit here in that we had an internal private cloud that we could use to run our public cloud So essentially we had an infinite number of resources at our disposable to to launch our our control plane for our public cloud And the only physical constraint was the number of hypervisors going to plug into it So essentially from day one or at least from the day that I got there all of our services were separated out Every single Nova API was its own virtual machine its own instance on our internal public cloud the The Nova cells the console glance API all of those things if it was a service That ran out of a virtual environment. It was 99% likely to be its own VM We might size it down size it up, but it's going to be an entire VM We also broke things into cells so that we had we could treat a Set of physical hardware on its somewhat on its own, but break apart how much traffic was going on so We had a lot of sprawl of control plane Consumption or control plane instances no matter how small our cloud was But it got us around a lot of the problems that that Craig was having and we could easily target Just Nova API to do something or we could easily target just Nova compute to do something But even this is not quite the best because if for a very small cloud like safer a dev environment We're going to be consuming 15 20 VMs just to manage to hypervisors, and that's pretty darn heavy so again, we're Not we but rack space was going down the road of trying to shrink this down We didn't want to go back to the world where our services were stacked on top of services because that gets really ugly on the system But what we did want to do is take these things that were consuming a full instance Shrink it down into a container, and then you could stack them on on each other But they're separated out enough so that they they almost feel like different operating systems So again, if you can if you can afford it if you can design for it Isolating each one of your services down into the smallest thing you possibly can is Really really powerful and what you can do with the orchestration across your cloud All right, so let's talk about upgrades You know installation that that problem that we solved earlier that's that's the easy part once you get it installed You then have to upgrade it otherwise you get You know lots and lots of bugs pile up features go missing And it's it's kind of a hard problem the longer you wait the the harder it is to upgrade You have to rent that forklift lift everything up drag it over here set it down and pay all the penalty along the way So Upgrading early and often is highly recommended, but upgrades can be kind of hard particularly when you start looking at How to do upgrades at the same time that your customers are using your cloud so In a lot of craigs clouds They're single-tenant. They have one user for that cloud It's kind of easy when you're working with one user on a smaller scale to come up with a good agreement on Certain times of the day that you can go in and disrupt their services to do your upgrades the rack space public cloud had has Something on the order of hundreds of thousands of users. I couldn't get ten of you guys to agree on where to have lunch together There's no way I'm gonna get a hundred thousand users to decide on a good time to upgrade the cloud I couldn't even get that many people decide on what a time is because time is across the globe It's somewhat meaningless So we always have to operate as if there are constantly live users Who very quickly turn into angry users when the cloud doesn't work? That said, I mean it's still the still the goal still is near zero downtime for upgrades So so in order to get zero downtime You have to start diving into things like how do I run one Nova API with version a and another Nova API with version B in the same cloud and that gets really tricky when you start diving into the way that Nova works and the way that Nova does RPC versions for their messages, but also changes the content within the message without any sort of versioning along with it Good times. We also have to deal with database migrations and when and where to restart our services We had a really fun deployment About a year and a half ago that was maybe a month of change didn't think it was that big a deal Rolled through our pre-production. It's just fine We go to deploy to one of our large productions in the database that I think it was the Nova database migration starts and goes and goes and goes and goes And it goes some more Eventually took about four and a half maybe five hours to to migrate our production database in the way that Nova works when that Migration is happening. You can't really have any services running So our cloud was just nope. Sorry go away for four hours. That's really really long time to have an outage and so that hurt a lot and started us down the path of actually checking how long migrations are going to take when changes are made upstream and Checking how long they're going to take on our backup Content so that we have a much better idea come deploy time as to how much outage We're going to have or whether or not we can do any adjustments to that migration ahead of time to make it hurt less The other takeaway here is prune your data, right? So yeah, so over seconds notorious for soft elites and if you can Please prune your data before doing the migrations Yeah, one of the big reasons why the migration hurt is that you know We had the very large cloud that had been running for a year and a half or more at that point with Hundreds of thousands of customers doing whatever they do with the cloud Which is lots and lots of creates lots and lots of deletes and we didn't really have a policy for pruning the instance table The instance data table. So that thing was huge. It was like billions of records or something odd like that so if we had Had a policy in place of when we can prune how much we can prune and where we would do long term storage of the Stuff we pruned out it would have been a much less of a problem So there's there's a lot of headaches you can you can fix Beforehand if you really plan and think about the fact that once you have it installed you're gonna have to upgrade it so one thing that That also kind of touches operations to some bit is is how does how do your user interact users interact with the cloud that you've built being Someone who stamped out a lot of these small clouds. This is something that for a long time was landing on my lap and and now lands in support In our support team slap, but things that always stood out to me are these are the types of questions that I would get from users all the time is I Actually can't believe how big this one is is how do I create images? So a lot of people who have a red hot red hot base release. They don't know how to create an image They're coming from legacy it shop They want you to kind of help them to help usher them along So you should use tools like packer or image disk image builder show them how to do that and make itself service for them I think this is probably a frustration for a lot of people running operate operations both from an operator's perspective as well as from a user Which is the CLI so CLIs? are They are there's no consistency a lot amongst many of the CLI so For instance, I'll do a nova list to show instances, but I'll do a glance image list to list images Why is it not nova instance list? Some of those small things as well as if you think about if you've ever used quotas before quotas use a demand a tenant UID but they will accept a tenant name and the tenant name does not work for adjusting a quota Along those those are just some of the customer-facing ones that that do get people angry on mailing lists And there's a lot more users than their operators Under the covers. There's some operator like CLI things that are just just awful So in the in the cloud that I was helping deploy we had five major databases that we'd have to migrate during a deployment There were four different ways of migrating those databases So every single database every single service was a special snowflake in our automation Which made it very difficult to do some of the fun things we wanted to do when every single thing had to be different Had to be an exception and it just added more and more and more lines to our automation tools It just made it harder to make any changes to it At the same time we have a lot of users again coming from a legacy legacy IT background that are they want to use Horizon as their interface to open stack Horizon Also exposes things, you know if you're you'll expose things that they maybe perhaps shouldn't see because they're there, you know They shouldn't be messing with networks for instance or it doesn't provide some functionality that maybe they want to have I once had a customer that We were doing a migration for and he I said well what we'll do is we'll just move those We'll take those instances from this host and move them over here or retarget them to a different host He said well, I can't do that because it's you know, I haven't I've never seen that tab in Horizon I said well, let me show you the CLI and we can actually do this and move these instances over And then the other one that we get all the time is this one is no valid host who's seen no valid host before and Who's ever figured out exactly from just looking at that what the problem was? Really can we hire you Because you know your cloud very intimately, right Yeah, yeah, yeah, so this takes debugging but for from a user's perspective, this is a terrible user experience People will try to spin up, you know, we have customers who use static IPs or they try to target an availability zone that's not present and they'll get this this one error wrap must wrap dozens of actual underlying errors and the way we deal with this is we're using A log aggregation through log stash to expose this through Kibana So a support guy can can look at this and say oh this request for this instance. This is exactly what happened But even at that that's not really that great We plan to extend that to users so that they can have a dashboard in Horizon or in our own Custom UIs that will show them the lifecycle of it of a VM But again, this is something that we should fix an open stack the rack space solution wasn't quite as elegant We just shunt all of our logs to the same server and we have grep for instance ID as a thing that we do or for reference ID but of course those those request IDs they're Unique for a particular service, you know Nova has a request ID when you want to build a server But it's gonna ask for a network from neutron neutrons gonna have a different request ID So again, you end up having to play grep the rabbit down the hole to find out what actually happened There's no real real easy way to instrument a process from customer request or end user request all the way to fulfillment or error so One thing that Opusseq has taught me is that these things when they fail they will fail dramatically and For certain definitions of fantastic, I guess There's been There was a talk earlier that was what was it it was many hours of Many hours of boredom punctuated by moments of sheer terror Which I think sums it up completely Most times things are working really really well, and then all of a sudden something happens into like what yeah in this case train arrived at the station great and You know, I think it one thing that was also pointed out in earlier sessions, and I think this is true is that You need people who are who are very who are very broad and very deep So the we're dealing with lots of lucid couple times loosely a couple technologies that you know You need someone who can dive in and figure out where the problem is across the entire stack, which is Typically a very hard thing to do So let's talk about one of the failures that we had So this was one of the one of the minor ones, but this is still fun so when we were dealing with Havana neutron Havana Specifically we decided that we were going to take a stable update So we were going to move forward on our stable branch and we were to deploy neutron and things would be great Because we would pick up all these changes one thing that was not clear to me is that apparently in stable branches we changed the fall configurations and What that meant for us is that when we changed the fall configure configurations for neutron all the agents Would actually start flapping and this is only because we were using one of the options not both of the options the two options are Agent downtime and report interval and for me. I don't know as an operator I don't think we should ever change the faults in a stable branch, but this is what we do So another great example of broad but deep right So we have to have broad knowledge of how to deploy lots of differently independently working services and how to manage you know packaging and and Upgrading and database and all these things, but we also have to have deep knowledge of things in neutron Like what does agent downtime mean or what is instant or the check-in time mean? What real impact does that have if those two numbers change right? So how do you have that very specific knowledge, but also have the knowledge to do everything else above that? And more neutrons I'm not trying to pick on neutrons here, but so first of all No matter what configuration dealing with an open stack It's typically complex and the lesson that we've learned here is that you only ship the config that you need Not the ones that can get in the way and like Jesse said really understand The config options that you're using so this is one that bit us really really really hard We decided that we were going to move the VX land some time ago And we did all the work and we got it all working and we went to go deploy to one of our first customers And things went horribly wrong What we had fantastic fantastically wrong So there are two options in the neutron config and you can see this is now fixed I pulled this from the latest source code for I believe it was this was from my source I think so there are two options one was enabled tunneling and the other one was tunnel types And you could select VX land GRE whatever so we were in the midst of wanting to deploy VX land So we said okay, let's Let's start doing the right in the code and we wrote tunnel types equals VX land and at the same time We set enable tunneling equal false. Well, you would think that that would do nothing that did something much different than that What that ended up being was 45 minutes of entire data center downtime and for our hundreds of customers and For hours for that one particular customer it ends up creating VX land tunnels and we were in event Essentially looping the network over and over again All right, so I'm not gonna pick on Neutron, but I'm gonna pick on a few other things So in our in our wonderful world of scale We have a couple of problems that we ran into and had to figure out how to address So Nova compute is this wonderful cool thing that runs on all of your hypervisors and what it it's kind of stateless but kind of stateful It gets away with being restarted in that if you stop it It forgets all of its date and if you start it it wants to remember all of its date And the way that it remembers all of its state is it goes to the database and says tell me my state and tell me the State of everything that happens to be sitting underneath me Doing that on one Nova compute that's got five VM sitting underneath it. Not a big deal We were doing it on you know 7,000 Hypervisors that had hundreds of VMs under many of those and so In our ever-going effort to do our deployments faster We got better automation that led us to do this on more things at a time And we end up killing our database because all the Nova computers would come up and say tell me everything and the database Would just lose its mind So that was fun that we had to to learn about how to do a much better job at bringing up our services We also had a lot of fun with glance glance has a few interesting properties to it first off glance API acts as intermediary between your Image storage land and the hypervisor that wants to use that image and every time a hypervisor through Nova compute asks for an image Glance figures out what that URL is and then Fetches it somewhat from that source and passes it kind of through so your your API ends up being an intermediary For the data that's coming across as well Which means that if you're fetching a whole lot of images like say for 10,000 hypervisors you have to scale out your API nodes through some some interesting reasons we end up having to List out all of the API nodes in each Nova compute config file So that if one failed it would find the next one and find the next one, but have a specific address to get to it That meant that anytime that we scaled That we needed to scale out our API isn't had a new one We had to go and touch configuration files across all the Nova computes C problem a A little bit more fun with glance We thought it would be a good idea to rotate our passwords for some of our services one of our services being swift Swift being where we were storing our our images So we updated the the swift password, and then we updated the password in glance Everything looked great to start with and then we started getting a few build errors for some of our older images We weren't quite sure what was going on weren't quite sure was going couldn't figure out couldn't figure it out eventually figured out that Swift and it's there. I'm sorry glance and it infinite wisdom was storing the path for the images in The database where it stored them that path included the password So You either have to trawl through your database and change all the passwords for all the images or you just don't rotate your passwords I think this is still an open bug It was kind of fixed and then somebody said but wait, what if I have multiple places with different passwords and we just sigh and then finally the The other thing that you have fun with at scale is is the cost of introducing a new feature In our efforts to make our upgrades better We wanted to make use of conductor for compute an external conductor shields the computes from the database rights all the computes instead of talking directly to the database they send a message request to the conductor and the conductor will Talk to the database on its behalf. It also shields it from Internal object version changes if compute gets an object from say Nova version Five, but compute is running version four compute can ask an external conductor to please translate this into something that I can understand And send it back to me as long as your conductor can talk both five and four that'll work really really useful feature for doing Rolling upgrades, but there's a bit of a cost It takes all of your computes that were independently writing to the database and instead shunts them all to the same thing to write To the database so you should probably have more than one You should probably have more than two and when we're dealing with 600 some odd hypervisors per cell You should probably have like three or four or more we think When I went to go stand this up in our production after making sure it worked really really well in in pre-production We ran into a problem that I kind of mentioned earlier every service gets its own VM and we needed to isolate these per cell so that Only the computes in a particular cell would talk to this particular Conductor because they all use the rabbit that's per cell I mean I needed to spin up at a bare minimum two conductors per cell which meant that in one of our larger environments I needed to spin up. I believe 50 new instances in our internal cloud With at least two CPUs in those instances Unfortunately, it'd been one of the clouds that had been around for quite a long time It hadn't seen any capacity increase for a while because hey cloud is infinite, right? Of the 40 some odd that I needed to create of the 50 that I needed to create I think I got four before I ran out of capacity And so now we have to make a significant or somewhat significant Investment in the under cloud in order to roll out a new feature that Saves us a few minutes here and there in some of our operations. So as you scale up You have to kind of worry about some of these things of how will you scale out? The thing that you use to run your cloud as you scale the capacity of your cloud So then I tried to come up with a couple of gotchas that we had run into along the way One of them that we ran into was DNS resolution hostname. So make sure DNS resolution and hostnames are always correct Your stack will look like it's working and it will it will behave like it's working until it doesn't We so in particular we had it one pretty way out of bug with our ansible code that was actually Just mildly or writing the wrong hostname into the Etsy host file All kinds of things on the back end will be affected by that. So be aware of that make sure that's right We had our own little fun with DNS and that a while ago We decided we wanted to start using DNS as a way of addressing all of our control plane instead of just using IP addresses DNS is a little bit easier to speak a little easier to understand what's going on And at the same time we were we were building up the automation for standing stamping out our new cells and our new environments And part of that automation was launch an instance in our under cloud And then once you get the IP address that it got to create a DNS record for it Well fun little thing about DNS and a records is that you can have more than one of them so we sometimes got more than one of them and Part of our launching our instances, you know We embed an SSH key in it and there wasn't really a whole lot of other Validation that I'm talking to the thing that I want to talk to other than it accepted my key so occasionally we ran into some really really fun things where a Our automation ansible would try and connect to a particular system that it thinks is maybe a nova compute looks up the the DNS record picks one of the maybe and it got back And logs into it sometimes it wouldn't work and that was okay We always tolerate systems that we can't connect to because you know when you're at the scale that we were at We always have some things you can't connect to But sometimes it would connect in and it would be great until we started trying to do some somewhat destructive things And then we realized that oh, I'm not really talking to a nova compute. I'm talking to a database node Boy, it's a really good thing that we have database backups So after that we kind of went down the path of let's do some audits and make sure that we only ever have one a Record for each system and that that a record is pointing to the system that we think it's pointing to so really important That DNS is doing what you think it's doing Another one that we've run into is service ordering so because we're not using distra packages We are crafting our own upstart scripts And if you don't have those right you will not know until you reboot that host So in one particular instance, we were actually starting up We're launching Restarting the instances after reboot before the networking came up and as soon as you do that the instances are instances are unreachable And probably have to be rebooted anyway. It's just it's something that's fixable But at the same time something that's not optimal Service ordering is also important not just when you're rebooting a machine that has a whole bunch of services on it But when you're orchestrating how you upgrade your services in a rolling manner we Recently introduced grace will shut down of computes and in doing that we wanted to Start addressing each of the services in somewhat of a logical order and we thought it would be great if we on the way to doing a migration we shut down All the API nodes so that no new requests come in and then we shut down all the computes gracefully so that they could finish Whatever they were doing and then we can move on with our day So we shut down all the services and then we asked all the computes to gracefully shut down and all the computes took a really long time We couldn't really figure out why that was until the light bulb went on and somebody said hey Don't those computes that are doing things need the control plane to do them So yeah, we should change that order So ordering how you bring down your service and bring up your services does have kind of an impact and what's going on There as much as everybody likes to say services are magical start them all up and they'll sort themselves out And it doesn't really work that way order is important Yeah, and the next two. I think we've we've already talked about logs I think everyone knows that logging is a problem. In fact, there was a session on Monday to rethink how we do logging I would recommend using some kind of log aggregator at least At the same time actions are hard to trace across the stack We talked about that before as well RPC failures So this is this used to happen to me much more not so much recently But it's very difficult to find the status of your services when RPC is not working correctly One technique that I've used there is just to use Nova Manch because you're going directly to the source of truth Nova Manch service list or something like that One thing that bit me recently Is that it is database backups? Who here thinks that backups are a good idea Every hand should be in the air, right? So we're all gonna have something like this So we had three weeks in a row of Friday at 9 p.m. My time in Boston For this one customer that was constant like key someone just fall over in itself And I had no idea what was going on turns out that we had this errant cron job on there because it was our first customer hadn't cleaned it up and We just had backups going on top of backups on top of backups and as soon as that happens all bets are off If Keystone's not happy ain't nobody happy So we had our own fun with with backups it was similar But we had an environment where and I'm gonna pick on it But neutron wasn't able to deliver IP addresses in a reasonable amount of time And nobody could really figure out if it was sort of the environment that was the problem Or if it was the service is the problem because it seemed like the problem would follow with our Versions as we bumped our versions, but also we wouldn't experience it in some of our other environments So we had a really to hard time with you know Validating the environment validating the new code because sometimes it would fail and sometimes it would work And we really couldn't figure this out until we started doing some real deep analysis and we didn't have a good We didn't have a very good correlation system Which would have pointed this out pretty quickly, but what was going on is that we had a scheduled database backup That would hit every hour And in this in particular environment We didn't have a slave database set up for that backup to hit so our backups were hitting the production database And when the backups would hit the production database neutron would get unhappy with its connection to the database And it would just slow down and not give out addresses So any builds that were happening like in that hour or the 15 minutes after that hour that took for that backup to Happen it would just have a bad time, but only one environment didn't have that setup And that's the only environment we could see it and it only happened on the hour So finding those types of things is is really kind of difficult But since we all agree that backups are good You have to be mindful of what that impact is going to be on your backups And you have to be mindful of not doing you know four backups at once of the same database And the last one that that kind of annoys me a bit is database modification So there are many operations that you undertake as you're operating cloud where you're going to have to modify the database the one that that we ran into most recently is We deploy sender with a default back end and we don't define multiple back ends typically But in this particular case we were adding an additional back end As soon as you do that you there's no easy way to migrate to two back ends instead of just one If you look at the sender database, it'll typically be host name at sign and the name of the back end So what that meant for us is as we added the new back end in Sure, we have the new back end working But we have to go back and retrofit all of those volumes that are already assigned to the previous back end So something that's not optimal Fixable certainly, but not optimal so who here thinks it's a really good idea to have lots of people with direct database Right access to your cloud databases Great. We're all on the same page here And Then we just so these are the references actually the second link is wrong So you can just tweet at me if you would like to get the correct link there. I just realized that now As much as we've kind of crapped on open stack I I did mention to Jesse earlier when we were doing this talk That I was having trouble coming up with really really really bad problems that we had it was You know When we did this abstract I was like, okay in the next six months like or the next couple months We're gonna come up with more stories about things that have gone wrong But for the most part I've had sleep. I've been sleeping. So that's a good thing Yeah, I mean we on on the rack space side of things a lot of our pain was because of our success You know that we had problems from having a cloud this really really big But we wouldn't have a really really big cloud if it wasn't a really awesome thing that people wanted to use So even though we have a lot of problems, we're still really really grateful that we're working on on pretty awesome stuff We just got to make it even more awesome, right? I just said that a second the the Haha That's an engineering solution right there. So I'm gonna move that over It was in a different repo and I plan to move it over and I totally forgot to move it over So I'll get that for you But it will be there And the obligatory we're hiring slides. So If you want to help us make open stack awesome for small or big clouds come see us So this is the last session of the day We are standing between well We're no longer standing between you and beer, but if you have questions feel free to come on up We'll we'll talk them out


Nader should be able to see me and hear me. Ah, here we go. Let's try and get some loot. Let's try and get some loot. I'm going to try and let me load up the game. I should be able to just tweet at it. What is it? Yeah, I can guarantee though. Is it? It ain't going to get something higher every single time, are you? Billy J? Let's tweet this out. Copy all that. Badoom! Might as well put it on my Discord as well. Hopefully the game holds out and won't kick me. Nah, it's the servers yesterday with dodgy, man. Kicking everybody out. OK, so let me send me an invite. Oh, no, it's obviously me. It's all right, it's no worries. Yes, crump! The icon's looking sexy next to your name. Sniperscope is from Coventry just around the corner from me. Yeah, the servers are a bit dodgy, to be honest. I'm going to try and join you, Joe. Join, friend. Well, he ain't killed him already, have you? Ah, that won't let me join you. Send it wherever. Yes, Josh! And he spoke to you in ages. Uh, OK, send me an invite. You got to tell me when you have, because I don't have notifications on. Let me try and do it to you. Invite. Get that. All right, see if you join me. Yeah, I'm on the main... Yeah, there we go. OK, so what story point is it we've got to go to? Yeah, yeah. Yeah, yeah, we'll do it. Not a devil's tell, it's my trickster. We just got... ah, say... Say mama. Bridge parties, turn up anything about... See, watching stream people, don't forget to hit that Like button as well. It really helps me out, and I appreciate that. I really do. OK, so we just going to look for someone to loot. I think we have to go to that back... that back door, don't worry, that lifts down. I did this, and I got absolute battered. Then the servers went off. Yeah, yeah. The insurgents took everybody...   No, he's here again Yeah, we killed the boss. Might as well just die. I mean you can get loot, you can get legendaries from any enemy, but You may as well. You may as well Take a few out when you're going down. I can't even see the boss. I could see bright lights Mate, we already over here. I don't know why you're hanging about with we ain't messing about mate You ain't got time for that Come back boss.  We actually needed you over here. We ain't took off Anywhere near as much of his health as we should have by now Come on.  I Agree Josie Intense fights lots of stuff dying spamming talents tons of loot. What is not to like exactly? We don't need cake. Well, we do like cake, but that's not lie and the bus.  Hey, all this loot is useful to me. I Don't care. You know, I actually know his golf. I'm pretty sure his golf is a club sport addition I'm asking that if I'm pretty sure it's a I'm pretty sure it is and it's still only a two liter Yeah, they're two leaders. Yeah, we see mine said a 2.5 five parts sounds like a beast Spits out more than flames spits out cookies mate. All right used to it.  Oh, I thought I was a purple thing. I thought I was purple. It looked purple For a second that looks so purple No, but I thought I did Mine's a one-off and he knows it you didn't get any a license back mate Fingers, well, we've only got access to four abilities and there's four more Maybe I can only get access to two pecs and our skill trees as well. There's tons more Come on and kill me That's the thing Joze if you get duplicate legendary as you can dismantle them get the mud Applied to another legendary that one legend echo up to legendary intrigue intrinsics basically bad ass Okay, so I've got room for ten more primaries ten most. Okay. Yeah Wait, I've got anything new here. I really want to put an SMG on now.   There we go Where's the captain as a captain?  We're standing in which don't help Yeah, I can see everything Progresses on to either main game. So you loot Yeah, yeah mods So I've so I've been told anyway, so Derek Jason has got two voodoo matchmakers.   I Didn't even see an epic yet. Never mind a legend. Oh, he just backhanded me May we ain't getting any legends and legend those anytime soon. I'm telling you these these drops are all true Let me find me not all moxie fun for like five hours before you got one Yeah Wow Yeah, a lot of people saying they got a shotgun as well. I've seen the shirt It's got like a kind of like an old slow steam on it Pretty cool Mate, I just want anything. I want anything. I just want any legendary to be honest not gonna lie Yeah, I'm gonna have to have a words with people come fly. I think Man, this is just Crazy Where is he? Okay, so Let me see what I've got. I must have something a little bit higher now My weapons are good. I'm happy with my weapons at the minute Dead Yeah, I understand that but the thing is amongst the All these enemies though where you getting peppered you ain't you ain't Well, if you saw who can sit back, I guess No, ain't come on. Sure. I can say sure. I got higher legs Made my armors rockin now Almost a thousand armor Jaws Aaron had no epics or legend those yet mates none Communo yes, they do drop this ultra Ultra where's big dude come on big boy? Oh He's how he's coming off now Let me get us Out just head by my mic head by my mic Sharp corner of it as well. I have it right next to my face in it I need a I saw someone else stream with a much better mic. I'm gonna get Yeah, it's just like more clear And you see the road you have to speak into it at certain angle to get the best out of it This one was like any angle man. It was six. So I'm not sure what it was to be honest Yes, people three hundred and three people watching more way more than I ever expected especially on the game They ain't even really out yet. So yes, if you enjoy in a stream do smash that like button I do I appreciate that support really do I Really do and this won't be the only time I'm streaming today. I will be streaming destiny.  It's just all over man Make this bus ain't having it now.  Oh That white looks so yellow then it was just ammo That white looks so yellow then Yeah, it's true envy. I haven't seen any one of the legendary armor piece just weapons I think there the armor pieces will come definitely from the higher tiers though Yeah, I don't think doing the demo Beeman, I actually really love it. I love the game sounds good to me Yeah, it's just new in it. It has those destiny one vibes of farming This is farm. I'm the go over and over and over and over and over trying to get that grasp of Malak May we just mounted him then Told you the LMG is a beast All right, just die Good luck and baby get anything else. No Yeah, people compare this to the destiny to we have now which has been out for how many years This is a brand new game man, and it's still it's still good. It's still Yeah Don't tell about the captains will walk. Where's the captain? I was here Yeah, it's walked out my bubble like it went there He's walking around is trying to get away the little sneak I'm pretty sure if we keep going one of us has got to get one one of us has got to get a legend Oh one of us has got to get a legend though So I even if even if the dude we must get this one is still it's still a legend. Oh Aiden his name is Jay by the way Yeah, I bothered if I see one dropping for me or you or him I know it's here, and I will sit here all night for a minute.   You don't need just make sure I haven't got anything here higher For the same Yeah, it's what it's all about You farm your loot you get the materials. Yeah Go buy better loot.  Yes, yeah, right front of us Mate look at the bullets slow and down. It's so cool. Where is it? Oh is here? It's hard. It's so hard to keep up when so many enemies here. My inventory is full. We've got to go back My inventory is full.  Oh It's moved to my stash. Oh, so yeah, if you die, we have that mold on it goes straight to your stash That's pretty cool Really you can't open your stash in a demo. All right. Hey, then we're going back to the hub How do you get the idea get back from you would you have to go back? I'm guessing.  Let's do that then. Let's kill him And then we're just back out to main menu and go to the hub that way. Oh, he's going to the crossroads Yeah, let's do it now. Yeah, I think he knows what he's doing Jesse can dismantle but do you get? What you can sell them for because you get materials if you sell them Communities should definitely tweet about me. Why not? You definitely should Send out a tweet about me. So if you dismantle so I'm yeah, how do you? Oh, yeah, hold why I have to mark it first Oh, so you can mark multiples and dismantle them. That's cool. So you get You get iron for dismantling but you get these kind of you get screwed up for selling them and he gets a lot of it so If I need iron, I guess I'll do that as well. Yeah, if you dismantle a weapon, you'll get iron So where are you? You are here? You can go to any of these can't we which one of these sells the weapons? It's this one and it might get a better weapon while I'm here. Oh What we've got here firepower. Oh, it looks looks good.  Oh, you can mark a low. That's cool. I love that shit Do you know I saw as well, you know within that destiny update the drop yesterday by the way when the best additions I didn't see nobody pick up on You can highlight all and apply one shader to everything Yeah, could put a shade on multiple items at the same time Obviously it would take multiple shaders, but instead of going in and out of each armor piece cool cool cool I'm not gonna get rid of every weapon. I'm just getting rid of the ones. I'm not gonna use Um, yeah, keep them two I'll go down to the armor paces. I'll keep my second highest of each. I don't need this I'm just gonna go. I'm just gonna kind of say to get more and more of each anyway So there's no point in keeping any of it if I'm honest 75 click on it, mate. I've got so much Shit I got three thousand six hundred eighty five, but I've only been here once before that was yesterday before I got kicked I'm trying to come back on here last night again Xbox when having any of it man 10,000 you got 10,000 the scraps cheese By a weapon she got weapons for sale or armor Now make actually I'm gonna go and have a look at the armor like that The problem is you can't upgrade it in this because there's no crafting is there So there's no point in wasting my resources on the shotgun upgrade Okay, so sell items 71 how much we're gonna get for this I Think I just brought my game I brought the game trying to sell 72 irms at once it sold one and then again go I crashed. Oh my life Yeah, well 18 isn't 71. Okay, so that didn't work. So let's try that again Geez man.  I got kicked for em. I tried to dismantle too many things at once and froze the game So that's a bug. I'm gonna have to report Nice one, Jose. I will see you later man May my internet's the best It's not my internet. It's a service. Well the game just couldn't deal with trying to dismantle that many items at once Cory Morgan, are you playing? solo or in a Group in a group you get way more drops I'm not sure if he you get more of a chance for legendary because neither of us have had one Kamuna.  He's having a son He's calling it exotic Dave after me. My legend Aiden get ready to join back by the way. Oh, are you send me an invite then I'll just join you's I don't know If you are you're paying online, I could probably join you and I'm the best course I remember my my watchdogs videos Course, I do I used to love that game man most underrated Multiplayer game there is enjoying you fuse. Why do I always switch between games for videos? It's because I made a mistake With destiny one of covering the same game For many years always wanted to branch out never did So now I just want to cover multiple games that I like simple as that Yeah Thanks again for getting the guy who killed Yeah, I'm gonna try this again and hopefully We're gonna go ten at a time If it would let me click it. Oh my life So let's see if that 13 Yes, we good. We're going 13. We're going with 13 people Oh my lord. I want to delete everything here. I'm gonna keep them two up top need myself a sniper Yes, I'll keep you to Dixie. Yeah, they're super rare Super super rare. I mean legendary People are getting in with the soul Look at all that beautiful currency This is what you gotta do when you're farming you gotta get that gotta get those materials for when the full game drops You can purchase that loop And upgrade shit because like I said the crafting system Isn't right now available?  That's 15 items. Let's go with that Okay, so let's do this again. I'm only a few more times people.   I got a legendary on the floor. He goes to my stash I won't be able to use it in the demo and I'll be pretty upset 16 Almost done almost ready. I'm almost ready.  I'm almost ready. I'm just dismantling I can't can't be going back into farming with a full inventory Give me a second silent and I'll show you Yeah, I'm ready. We are farming this one That's what we're farming silent AJ anything Q dislike about this game. What I dislike about the game is I'm not getting any legendaries. I Don't like the motion blur when you're running, but they're taking it out. It's there's an option in the full game. That isn't there Other than that, I'm loving it.  It's It's a much different game to destiny another game I love but it's got enough of the elements of destiny for me to love this just as much Oh Jay You ready click the thing that legendary dual pistols, I don't believe you I Do not believe that legendary dual pistols. I need to see them Yes, Cory. We need to see poor Yeah, monster.  Is this better than borderland tree? Borderlands is borderlands and it's a completely different game Borderlands in my opinion is the original looter shooter the original looter shooter and no one will take its crown But you can't really be comparing games like this to it. It's just too different True you got the thunderbird.     We got these people. We got these people. I got a weird buzzing sound. I don't know where it's coming from He just scheme. Yes So who's hosting this is it you Jay? That's all good Yeah, it's trying to watch the flow in it. I'm so down.  I am. I don't know. It depends how it goes. The golems and limb looks so good Man, I got a horrible buzz in my headset. I hope you can't hear that on the stream if you can let me know I'm working on the car mate. I've got a feeling that one's coming One is coming. I've got a feeling chef one is coming I didn't see just tweet, but I'll look at it now.  This dude's just got that looks Absolutely scandalous. I haven't seen that before I Have not seen that weapon before Okay, so just give me a second. Give me a second Saves it there. There we go. We've got it. We've got the image there. Let me just quickly People drop this in so you can see it because you gotta see this shit You've got to See this It is unreal Where are we there is Check that out people check that out I'm gonna zoom in a little bit so you can get a bit there A bit of a better picture of it.  You'll see it. By the way a crump.  You are so lucky. I Need that in my life. I Need that in my life. Let's go again. Also massive thanks crump. I appreciate it. I really do for the super chat We go in let's go Now I'm trying I can't do much more.  I didn't get nothing good there. Let's go again.  Mate, I'm sure I'm shooting at you here. Okay, so where is he's round here?  Oh, okay, so we've got a bit green then special Nothing special Yeah Yeah, do they all have the potential and I just let him let this one kill us They just you kill us Can we can we open the chest up here? Oh cool submachine gun and a Automatic shotgun, I will take you. Thank you very much. Yes. He don't matter what I mean I don't I just don't see how people can compare this anthem. It's just that's just like Daft Just daft There's more loot in this demo using the whole of the anthem game Because anthem wasn't a bad game to play. It was a great game to play the mechanics. We were unreal It's the the point of it was bogey. It was Kind of bad content no loot at all This has got everything ain't bogey at all. It runs more or less fluid for a day to demo to serve is a pretty good holding up Yeah, that's what it is But but but but but but but is damn bitch bitch Tom the legendary dropout random just farm the game Hide a tear supposedly the better But Some dude in my chest Ballish and there's not you can just just random mate just random Yeah, they ain't canceling that right there's not a chance Yeah The thing is as well The server's game Well, basically crashing because there's so many people playing it shows you how good the game is people want to play It's seen people playing get want to jump on it Yeah, right man.   Yeah, that is if yes true slice slice It definitely feels a bit bit laggy though on somebody else's host, but it's not too bad Mate, I'm gonna die here See enemies from behind or just malty me.  Where's the bus? I was there I'm just chopping everybody up. Oh, there he is. Right. Yeah I can see doing here think this is a demo look at all these ads on the screen right now Look at all the bullets look at the audio blitz is going on And I ain't noticing the frame dropping. I ain't noticing lagging or anything like that So, yeah, it's good people man.  That's the bad way of looking at it Now, yeah, no one I mean as good as a what's his name is that solo and shit Yeah, that's a toke as good as a turkey is probably one of the best PvE players in the world He ain't so low in anything con tier 15 Not until he's got that ultra ultra gear. Why do you say that he probably will? Yeah, he's getting it now I'm gonna put on that fully auto shotgun actually Give me a second. Let me just quickly put this shotgun on.   Oh Yeah, this shotgun ain't that good to be honest Not compared to this bad boy This thing takes his health off man I'm down I just gotta watch the fire. I'm just standing fire. Just looking at the boss It's messing me up Oh, no mods to unlock him my fine item and dismantle it. Yeah Oh that looks like that there looked like a legend though and it was in fire. It was a green I hate this game I got so excited then as well My palms started to get sweaty Now this is just like farming strikes from destiny one. You see that grasp of magic falling on the floor and you're like We see a legendary angle. I'm falling before you thinking.  And yeah, he does look like a cross between the division and destiny People are saying the same about gears of war and destiny too.  I can feel it. I can feel a legendary coming May I be happy if you got one I'd still be happy Matt, I can't even see Can't even I thought I had one then as well There's like a quick Yeah, that's what it is It was a green the blue come up into the air It looked like a legendary Now I just want to see one chaotic wolf How did you get that are you doing this farm here? You got that, uh Assault rifle. I just shared on my stream as well Yeah, yeah No, no, no from the bird the one I showed on my stream that weird looking thing. Is it a Come on. Well, it's called some some Very looks gaze Oh man, I need that in my life.  I was there No, it's not him That is him He's had healing light on him. I thought there's someone else. Oh, let me up That look like a legendary that looks so yellow then Shit You can King Dre. You can get legendaries. Yeah, this one here someone got Um Check that out Check that out He got that Irritable there or yesterday shared it with me on twitter But there's a quite a few I've seen about 10 15 people Can so You can definitely get them Definitely Just ultra ultra Some people are getting too Some people are getting free man RNG is on their side man. Yep captains Yeah, isn't it? Man, I'll probably be in the bed next year where this is going.  Let me know that goes if you're getting a legendary from him Um, you can't end up in the same team as me at the minute because I've got a full team But if I come on solo when I stream later We can play together. That's no problem Man, man, I can't get There you go. There you go. I'm down. What do they get there? I don't feel got anything today No, I'm just a green Just a green He's left up Okay, let's go. Okay Yep I got no reason to check my loot now.  That man was killed me. Jeez Oh, he's dead. He's dying.  I must have been Yeah Here we go. We killed him People to the stream if you've just dropped by do smash that like button I do really appreciate that support helps spread the stream out and about as well and I appreciate I really do I really do people so smash that thumbs up Derek Jason, I got the thunderbird to drop from one of the breaches. Well, there's breaches here There's one right here. Mate, you're going down Here they can they can drop from any enemy. I'm not getting off on these dudes. Let me come across to that boss Yeah, I'm someone over there. There he is Hey kids over here doing it by himself I can't even see the boss. Where is he? I was right there It's here I'm hitting somebody. I can't see who or what it is Like he's there adds a shield in him Tell you look so yellow, man Anthony, how'd you get that? Was you farming this too?  It's pretty quick It's pretty quick in doing so. That's a pretty cool mod on that Uh I'm hunting the prey. I haven't got that selected, but What else we got here? Oh, that's pretty cool. That's a cool mod Oh Nothing higher there. What about my helmet? What we've got on that? That's good speed up or what we're going this bad boy. Yeah, cool All three to help me out with my uh, I'm changing shotgun as well Oh jeez That's much better Okay, I'm ready. You ready? Let's just bad boy shoot like Yeah, I'm going should have super super same bullets now Where's this boss? Let me see what these things doing Yeah, we're doing some way more damage now Way more damage now Then muds on that armor I had muds on my armor that helped my super same rounds. Where's he going? He's here It needs to be like a a mud that increases loot drops. That'd be pretty cool, isn't it? Black fire, you can't do that.  Come on. Give me that loot Give me that loot.  I thought he hit me off the map then Mate this shotgun's a monster And that without my super same rounds, man Let's try that on an enemy Just attempt just link to his video It would be much easier because we're going mate. I ain't hanging about This is a looting time.  I'm so done here Oh, I noticed He's jumping all over kind of things as well Where is he? I was in the corner there Give me them legendaries Yeah, that's enough. That's enough.   I'm just giving the breaches.  I know Devastate is pretty cool.   How are you? Doing Okay, I was running and then I wasn't I'm over about none of that. Oh, what happened? I lost connection to squad leader Shit Recreating the game. Please wait Yeah, last connection that ain't a good sign Huh, it's the image. Oh Yeah, I lost connection to squad leader Ben Edwards Yes, you will be able to uh transfer characters between uh transfer weapons between characters You will have like a vault like system like you see in destiny Yes, red ricks. Thank you very very much for the donation Absolutely love you Oh, yeah, um, it says it's lost connection to squad leader, you know, I'm just gonna do Tardis or try not to spam please Am I back? I'm back. I'm back bum bum myself Should I see what I can do see what can do solo watch me get a legendary away imagine that that'll be it May I ain't surviving this is no chance. I'm surviving this Oh man The captain died so much quicker solo man, I got I got a green in the blue straight away Man, I'm killing these dudes I'm killing these dudes Oh no, I'm just gonna die. All right, so I'm going back.    I'm done People who have got legend days in my stream so far. Um, have you been doing it solo or in a team? What do you think the drop rates are like? because In a team we've just been doing it for what how long is this streaming going on two hours one sec Jay Uh, this stream's been going on for like two hours. Oh got some with me. No, I haven't so I'm just wondering Is it more efficient? I mean, is it more efficient? Uh, in terms of loot drops if you're in a team I don't know if it would I mean, I'm just get I'm getting equal amounts here Ben Edwards to be honest to I said the drops ain't low when you're doing it in a team, but air and uh The legend those are ultra ultra ultra rare Well, bitch the thing is, um I can get this done way quicker doing it solo I can probably get it run two or three times then run in a team If you see me shooting at the wall or my aim going off people it's because I'm looking at my chat I'm still farming it. I'm just getting as many drops soon as solo.  I'm not even joking. What if that chest up there's um, there It just drains to be honest, uh For ice, I think that's a pronunciation name. I haven't had much in terms of great weapons I'm just using what's the highest level Some of the blues got great muds on them, but I don't know what we've got here. We can put here I haven't got a blue Big daddy with Eclipse size that big so I'm just gonna keep you on for now I mean, he just dies so much quicker like this, but I think I've had like a rare every time I've killed him Like when we was doing it in a team That definitely weren't the case I weren't getting a rare every time I killed the boss More or less. Yeah every time I kill him. I'm getting a blue and he just dies so much faster, man So much Okay, I got two greens and we can team back up if you want team back up if you want Just invite me to your game. I'll join you again. We're in it again. Wait for that invite So let me know you've done it Let me know when you've invited me and I'll join you If I died and try and invite you See what happens Okay, so let me do this again I have Sent the invite Get it If someone wants to join our game as well if they're a high level, whatever you're playing on There is my game code so you can join that p254 f7 through 7 got a space for one So whoever joins first joins first You yeah People i'm just gonna quickly run to the toilet.  That's his name already Man, I'm not sure these These breaches have got so much health forget it I'm ready Mate look at that beast on his back man Go auto looted Come on one of us has gotta get a legendary sometime soon Yep He certainly is Yeah, these breaches do just keep coming out of that door I think that's about it and it Anymore like that I can see So might as well just let me kill let's get get rid of this dude Get out of more Someone just said that in your chat. Someone just said that exactly in mine. Yeah blank blank He can just kill the captain you're right, but the loot has a chance of dropping from any enemy So he may as well kill the other big dudes that are here Make sense makes sense These big guys come back out of this door. I don't know Man, I'm in flames. Yeah No more breaches.  I was gonna say May the breaches are based no matter how if it's unreal Yeah, these breaches drop so much loot man. It's all green, but it's still loot I'm gonna blow up there That's enough, isn't it? It's enough I am we two. Um, actually can get the same exotic score legendary can get me the demo Do drop there's a little well.  We're playing with now as you can see here He has two he's got one there On his back.   One of you motherfuckers has got to drop me some good That's it That's it. That's it I haven't got no legendaries yet. Unfortunately No, that's all But we'll get one. We will get one people. We will Come on Come on Two or nine on what one armor My inventory is full. I'm getting a good sign What we've got here Okay, so let's die do that again.   Am I good to go? Oh, you've already gone Mom Didn't even wait for me. Look Didn't even wait for me.   Yes. What I thought That's what I thought bitches up.  You fools. What are you doing? No, I was in my menu.   I'll tweet out saying We need we need loot We need the old streamer perks everyone talks about because we ain't got none of that My health is coming down now Whoo, how'd you come out behind me?  I need that luck. I need that luck stream.  Give me that luck. Shit Good job. You can't run off the map. Where's the enemy?  I don't know what we're doing wrong, but Look at that bad boy and he's back though. That's what we need That is what we need I don't know what he's using but it's a monster Uh, I have had none drop yet zoo and not a single one Yep, it's definitely a farm It's definitely a farm I don't think we need to do that. I think if we respawn we get full ammo Yeah, teeny is going down Going down Come on. Come on big boys The boots ain't coming out this door no more. What's going on here? I think we killed him too quick Yeah, they were on that They had problems with the game and they sorted them out relatively quickly The problem last night was xbox had problems So even though people can fly fixed up their issues Xbox was suffering.  I'm burning people. I ain't got nothing Nothing Yeah, you can Motion blur will be You will be able to turn it off on the full game. It's the only thing I hate. I can't take that motion blur That's cut scenes though, isn't it? Is it really? Oh It's unfortunate, isn't it?  You want some of this? You got what?   Shoot it.   That's my turn now stream. Please please Give me luck stream. Give me luck stream. Give it to the wrong person You give it to the wrong person You can't believe it. Can't believe it Can't believe it Can't believe it Can't believe it.  Where did you get it dropped? Was that from the captain? Yeah, see see stream.   It's a scumbag.   I hate everybody's got a legendary and I ain't got one It's going great mate Yo, that's filth That is disgusting man So this legendary he just got the amber vault every quick shot it basically don't use your ammo So if you get in constant headshots, you're not losing the ammo.  That's what we're doing because the breaches drop loot as well Michael mine too Mine too I ate a full tub of Ben and Jerry's last night watching Lord of the Rings My toilet my toilet was not great for this morning. Let me tell you I can feel it coming I can feel it coming. Where's his boss? Come on. Okay.  I'm almost killing everything here.  My RNG is just so bad. I must have some higher here Cool got anything higher there Why'd you say your loot's going up to J 209? Yeah, I got 208. I got 208 pieces here. Yeah Come let's go. Let's go. Oh Enemies can be out mate.  I got an extra chop Yo, I got an extra chop What Why have I just put on then gives me an extra. Um, I can't remember what it's called What's it called? Temple blade I get two of them two of them It's a piece. I'm one of the armors. I just put on get two charges, man. So I can chop twice. Yeah, that's sick What's that on actually? Why is that on? What was it the legs I just put on? Yeah, yeah slasher mud on my trousers. That's sick Matt chopping twice is the best is the best My looks so bad I'm getting slapped from every angle here Yo, it charges so quick as well That looks so purple then I don't know if epics drop here. Did they only drop from vendors? I'm sure someone in my stream will let me know I think epics drop from vendors I'm pretty sure as well. I've got another mud that I've just put on. Let me have a look.  I've got my chest piece So that means on this LMG two hundred bullets of soup or two. Yeah, 200 bullets of super same rounds.   I must have been standing on fire. I must have been standing that fire I'm I'm all right now. Well I say I'm all right now. I'm about to go down again. I think Yep, I just got Had nothing there anyway Had nothing there I'm telling you if you get another legendary before I get one. I'm not going to be a happy man Where's the boss? Is he dead already? He is any cheese Make my super same rounds last so long Oh Imagine as well people super same rounds 200 bullets start two clips And having a mud where we playing ammo with headshots imagine that I'd have infinite super same rounds basically That'd be pretty cool Yeah, we need to die. There's not many ads here left Come on Let's sit on that grenade Michael York that merit attack. He's just an ability on the The trickster the mud. I've just gotten a random blue piece from farming this Where are you? There's the destiny emblem. You're literally sitting on the destiny emblem Michael pull and we're dying because what we're doing here is we are killing the boss Then killing the other enemies around him and then dying and it respawns us and they they respawn So he's constantly do this forever It's the fastest way at the minute. We know of it in farming legendaries So we're good Man, I'm getting bullied here. I only get jack then That's a green. I thought that was a yellow So what do you um fingers look like j the legendaries? They're bigger Which is the same looking she look the same just yellow Every every like slight yellow color. I'll see it's getting me excited These ads are locking onto my face But Give me loot breach. I give me loot. Give me legendaries as a blue.  This repeat get that loot and you are good to go That's if you get loot. I'm getting loot. We're no legendaries Captain's dead, mate. We aren't messing about here. Come on. Bowie chairs Come on.  I need that lock from us.   You've got that choice Two or two is the cap for what? Can't be I've got two all right How to go down then She crazy ass game Man, I need health so bad My second I thought it jumped off the map Even though you can't Nothing for me there anyway Nothing for me. My luck is so bad There is no pvp in this game And that is not a bad thing That is a not a bad thing I'm gonna try this a couple more times if I don't get a legendary. I'm taking a break.  I'm seeing man. How did I go down so quick then? What even happened? I think I've adjusted my armor. I put on these amazing mines, but they're I'm doing I think it's pretty bad Come on. No, I don't want blues man What level are they and I'm gonna lower That gun he's got raises them up into the air and it that's so sick That's so sick Nothing for me again Two more tries Trevor I'm pretty sure legendaries Have more chance to hide the difficulty But because under demo we are Limited to a tier five.   It don't hit anywhere near as hard as this weapon. I'm using now The thing is what I've got is I've got uh, the Super Saiyan Vans. I've got a mod on so I'll get two clips of them And because I'm using that well, it's basically an LMG. I get 200 bullets of them. I'm not seeing nothing here But in saying that about the difficulty, um People have reported getting legendaries on tier threes and fours. So They will drop. I'm guessing they're just rare even rather than what they are on tier five, which are super So Yeah Highway, I've got no legendaries yet, but mate. I'm playing. We've got one here About 15 minutes ago.  I had a single one Yeah, mr. Ghost. This is out on all platforms. It goes to the playstation store download it and you are good You don't have to pre-order. You don't have to part of any money. You don't have to purchase the game to download the demo and play And you can literally get to what we're doing here farming for the full game Because your progression transfers over and literally about an hour and a half probably not even that Yeah, I'm just letting everything Take my weapon. I don't care anymore You're standing in front of me dying No, nothing nada again a couple more tries and that's it a couple more tries and that's it. I'm taking a break Yes, mr. Ghost the demo is out right now. It released yesterday You can play it all up until the first of april when the game officially releases. So yeah I can yeah, that's cool Mr. Captain give me a legendary, please on my last run Give me give me give me give me That game chopped in the back. Yeah I'm so dead.  Thank you Yeah, I'm not seeing nothing Nothing Yeah, I'm gonna take a break. I have something to eat come back to stream later That's what I'm gonna do Yeah, so Yeah, I'm I'm gonna take a break. Yeah So stream, thank you for stopping by unlucky today. Obviously. Um Mill gaming.   I'm playing with he's got two but me Nah, I'm taking a break and I will be back later either streaming this or this and this all turn those notifications on people And I want to thank you for stopping by I really do but I will catch you on that next one Peace out