So, my goal here today is to get you thinking systemically about security. Not thinking about the what, you know, cryptography and hashing and password security and using template auto-escaping. You know, I'm not talking about the technical details, although Kelsey gave a talk earlier today which covers a lot of those technical details in depth, so if you want to know more about that, watch the video. Now, what I'm going to talk about is how does your organization think about security? What is your security program? Let me get a quick reading. How many people here work for an organization that you would say has an established security program? Wow. Okay, that's awesome. That's actually more than I thought. And how many of you do some form of security in your day job but don't really understand exactly how it fits into a larger picture of your organization's security posture? All right. Okay, cool. So, I'm speaking to the people who raised their hand the second time and I'm speaking to the people who didn't raise their hands at all who don't think that they have a part or don't understand their part in an organization's security. You know, really I want to answer a simple question here which is what would a minimum viable security program look like? You can't, you know, I work for Heroku which is part of Salesforce, right? Salesforce, 15,000 employees. We've got a security program. We've got lots of security programs.  And remember when we talk about minimum viable products, we're not talking about, we're not talking about just one part, right? I love this image because it really describes like exactly how you should think about what minimum viable means. It doesn't mean let's just build one part of it, it means let's build something that satisfies. So in this example, I want to tell you how to build a skateboard. So the conceit of this talk is, let's say you've got one week, you're going to sprint on this for a week, you're going to sit down with your coworkers and at the end of that week you want to have an established, defined, measured, successful security program ready to be iterated on over the next five weeks, five months, five years. And that's what we're building. So here's what we'll do. Monday, we're going to develop our training program to make sure that developers understand what building secure software means. Tuesday, we're going to develop an SDL, which is a fancy version of saying what is security here and how does it work.  Thursday, we're going to talk about what a lot of people think of as sort of the boring parts of security, governance, risk and compliance, formal security programs. And Friday, you're going to tell the world that you've just done some awesome work. Let's dive in. So train your staff. So security is a shared responsibility. A system is only as strong as its weakest link, and this means we need to strengthen all of the links. Every single person at your organization is in some sense accountable for the security of your organization, whether they are a developer who needs to write code that protects against SQL injection, or an admin who needs to not fall prey to a phishing attack and share corporate calendars, or a janitor who needs to keep the doors locked, or a manager who needs to not let someone approve a change that would be a bad idea for the company's overall risk profile. These are all actions that people need to take to ensure that we're doing our best job protecting our organization and, most importantly, our customers. So you really need to have holistic security awareness training for everyone at the company. This isn't optional, and I'll talk a bit about why in a minute. So what I suggest you do is focus on some very basic security hygiene practices. Good passwords is the easy one. Luckily there are several good password management utilities, LastPass and OnePass. They are not hard to use. I hesitated a minute. LastPass is a little hard to use, but it has some good features, so it's kind of worth it. You can make a decision there about UX versus features and decide which one you like. Train your staff for using a password manager that will dramatically level up the sort of organizational security posture of their platform. Shared password reuse that is using the same password on one side and another, and then site A gets compromised and attackers use it on site B is an incredibly common exploit vector. There was an interesting breach a number of years ago of a company, what was their name? They were a MongoDB as a service provider, and the way that they were compromised was that the password that was used was used by a staff member on Adobe's website, and Adobe was compromised. With the compromise to the Mongo provider, a shared password used by a user of that service was used to compromise another service, a continuation integration service. You have this chain of the attacker moving from platform to platform, harvesting passwords and trying them across other systems. Training out password reuse through the use of technology is a really good way to cut that down. Multi-factor auth is a thing, it works, you can train your staff how to use it. Basic training in customer privacy procedures is something that you should probably be spending some time writing down and helping your staff understand. This will differ from organization to organization depending on who your customers are and what privacy means to them. This is worth the investment. Let's talk a bit about phishing because that's the biggest threat that you'll probably face to this sort of general population of your staff. Phishing, this is from the Verizon does a yearly data breach investigation report where they compile data on security breaches from hundreds of organizations and do a bunch of analysis and grouping of the types of vulnerabilities. They find that more than two-thirds of incidents that follow this pattern of trying to steal data feature phishing. Most attacks start with either a targeted or an untargeted phishing email. What's really scary if you're in the security field is that almost a quarter of recipients open phishing messages and 10% of them click on attachments. Which means that just 10 emails gives you a 90% success rate. That's pretty scary. That means as an attacker, I only need to send you to your company 10 emails to have a fairly good chance of successfully phishing someone. What can we do about this? This is a big threat and it's really hard to address because it's people. So there's some technology tools, good email filtering helps, Gmail's great. Being able to store and archive all of your organization's emails so that you can determine the scope of a phishing attack if one occurs is another great technological stuff. But really training is the main thing that you have to do here. The same study, the DIBR also found that the best early warning system for phishing attacks is your own staff. A properly trained staff, the average time to respond to a phishing attack was 20 minutes. So if you have a staff that knows what phishing is and understands what it is and how to report it to you and how to sort of tell you that something's up, you have a better than average chance of catching an attack early on. You can do this yourself. You can also pay for this. FishMe.com is really good. They'll run phishing attacks against your staff for you. You give them your staff email lists and they do targeted and different types of phishing attacks. And anyone who files for one gets taken to a special customized training specifically for that style of phishing attack. So it's a good service. I can recommend them if throwing money at the problem is something you can do. So the other part of this is writing code. We're at a programming conference so I'd be remiss if I didn't talk about code a little bit. So who do you think should be responsible for writing secure code? Whose job is it to write secure code? I heard someone say everyone. And it's the same question is who's responsible for writing tests? We used to have this idea about software testing that you had test engineers and they would write test code. And then the other engineers would write the code code and then they'd like, I don't know, meet with pistols at dawn or something. And like, yeah, safe to say that didn't work. One of the main innovation of test driven development isn't necessarily the acronyms and the styles and the functions, but just the idea that testing and coding are this inextricably linked cycle. And it's the same way with secure development. Prisa Tabriz, who runs Chrome Security, has said that one of the key factors in her success at Google has been to push decisions around security as far down the chain as possible. And this is kind of counterintuitive because you might intuitively think, these are company wide risks. We need to have the director of security making all the security decisions because then we'll be really secure. And it doesn't really work that way. The further you get from the hands on keyboard, the people writing the code, the less context and information you have and the less able you are to really assess a risk. And so, if you, like me or someone in management, your main job should be to empower and push those decisions as far down the stack as possible. Sure, in the middle of an incident, you need command and control, right? You need top down crisis mode leadership in the middle of an incident. But for the day to day bread and butter of writing code, it's got to be people, average developers writing code day to day. And so there's some good news here, which is that actually writing secure code is easy. Now, there's this idea that security is really, really hard, so we need to leave it to the experts. And I want to push back on that. Yes, there are parts of security that are hard. Cryptography is impossible. I barely understand how sort of prime factorization crypto works, and now we've got this elliptical key stuff. And I mean, you left me behind a long time ago. But most, most coding does not involve that, right? Most coding is day to day, fairly easy basic security hygiene. And basic security hygiene can get you a surprisingly far away. When you look into breach reports, those that have happened because of vulnerable software, they're always basic stuff, SQL injection, cross-site scripting, cross-site request forgery, the basics. The stuff that's in the OWASP-POP10 is rare for a truly novel and hard security vulnerability to actually lead to a real world compromise. So by expending a small amount of effort in basic training, you can get really, really far. And there's a lot of good resources out there. These are four of my favorite. There's probably quite a few others. OWASP maintains a sort of top ten security risks with information on how to address them and what they look like and pointers to different languages. Mozilla's secure coding guidelines are probably the best publicly available application security guidelines. They're somewhat language-agnostic, although Mozilla writes a lot of their code in Python and Django, so it'll play well to this crowd. Microsoft's guidelines are a little more focused towards compiled software as are Apple's. And so depending on what environment you're writing for and what type of software you're writing, one or more of these may make a good secure coding guide. And your company's secure coding guide could literally be go read the OWASP-POP10. That would already put you many, many feet above average. All right, so Monday is complete. You've developed a basic security awareness training program covering phishing attacks, multi-factor off how to use password managers, and you've picked a secure coding guide. Maybe you've customized it a little bit. Maybe you've just taken one off the shelf and put some resources out to your staff. So Tuesday, we're gonna build an SDL. Okay, so it's a buzzword, I'm sorry. An SDL stands for Secure Development Life Cycle and depending on the level of formality of your organization, this could involve lots of fancy flowcharts and diagrams that look like they're off a government slide. But really an SDL is an answer to a very basic question. Okay, we've told people how to write secure code. How do we make sure it happens, right? We know what the best practices are. We know that our staff is smart and wants to do them. What is our mechanism for making sure that they do? And for me, the heart of an SDL is figuring out this virtuous cycle, is figuring out how we take the things that we know, we translate them into best practices for our organization. We follow those best practices in development. Things happen, either successes or failures. We analyze them and that builds more knowledge. How do we build this program so that we are continually feeding in the results of the things that we learn about security as we develop software continuously? So for me, the minimum SDL needs to answer three basic questions. When do we do security? At what point during our software development life cycle do we think about security? Who is doing that thinking? And what does doing security even mean? So you could answer these questions in several ways. I have a suggestion, this is how I've answered them. I think doing security throughout development as much as possible is the best way to go. We have an internal security mailing list. We have a chat channel. We use GitHub comments. There are multiple ways at Heroku for staff to get in touch with us and ask us questions ranging from very simple like, hey, what library do we use for OAuth again? To, I'm designing an entirely new crypto widget and I need a lot of help. Any size of engagement when we're there. So having experts available for questions and building a culture where it's cool to ask that stuff and where people help each other out with it as much as possible. It's probably a good idea to have an explicit security step when you're sort of planning and building a new product and probably again a review just before you ship. These are hard when it comes to agile because we don't have as many explicit design upfront steps and shipping might be something that we do tens or hundreds of times a day. So I don't have a great answer here. This is still one of these really hard problems. Sort of figuring out how to integrate security and agile is an ongoing problem and something that's sort of pretty tricky. But I still think you can identify moments where you can identify sort of touch points. If you're about to launch a new feature, if someone's going to take the time to write a blog post about it, yeah, maybe at the same time you might want to take some time to do an explicit security review. So who does this work? Again, you should be pushing security decisions down as far as possible. So we really focus on giving engineers tools and documentation and authority to make decisions and taking a default position of trust. Our basic assumption is that everyone is reasonably competent at their job and trying to do it well and that the decisions they make are more informed than the decisions that we make and so we should start from a default of trusting what average staff do. If you have dedicated security staff, if you're lucky enough to have a dedicated security team, I think the best position for that team is sort of a consulting role, right? You're not necessarily saying this is the architecture or yes, you may build that or no, you may not.  We have a thing we talk about on the team about, we try not to say no, we try to say yes if, right? So don't just say like, security's role shouldn't be you can't do that, it should be there are risks, how do you plan to address them? And the decision about how far up those risk decisions make need to be based on some fairly good understanding of what risk means for your organization. There is such a thing as acceptable risk, right? You're going to come up with a situation where you've got a known problem. But if you don't ship tomorrow, it's going to cost your company $40,000. And so you have to make a decision, is this security risk so bad that we need to pay the money or do we have a plan to remediate it in a reasonable enough period of time that it's worth the risk? And the greater the risk on either side, the higher up you probably want to push that decision. That's a company-wide risk decisions what probably needs to be made company-wide. We build tools, this is one that we have, we have a little, we refer to this as the twine game, it's like a choose your own adventure, sort of point and click to sort of figure out what level of risk a particular project is going to be and sort of self-service the decision about whether you might want to involve our team or not. So what does doing security mean? This is the last part here. What are we talking about when we say doing security? So I think checklists are the greatest invention since sliced bread. Probably better, I would give up sliced bread if I got to keep checklists. The good introduction to checklist is in Tull Gawande's book, The Checklist Manifesto, great book. He's an amazing writer, really good to read. And one example he gives is so there's a doctor at Hopkins called Peter Provenos to sort of the invention of using checklist, inventor of using checklist in medicine. And they were having a big problem with central line infections. And so he designed a checklist, I think it had five items on it, it was really simple, like do you have clean drapes? Is the needle clean?  Like it was really, really basic stuff. So they gave this to all the doctors and nurses doing central lines and monitored the results. The ten day infection rate went from 11% one in ten patients were getting infected to zero. And in fact, they were so surprised by this result, they thought they were doing something wrong. So they measured it for another 18 months because they thought that they had messed something up. And so all in all, in that two and a half years, there were two central line infections after the introduction of the checklist. Down from one in ten to two over two and a half years. Gwande notes that there are these three kinds of problems in the world. There are simple problems, it's like baking a cake. Once you know how to bake a cake, you can do it repeatedly over and over again. You can give someone a recipe who's never baked a cake and they can probably do an okay job of it. There are complicated problems, like sending a rocket to a moon. The recipe is much, much longer, but there's still a recipe. We know more or less what all the steps are. We get it wrong more often because it's more complicated. But we could write a checklist for sending a rocket to the moon. It would be a lot longer than the cake, but it would be a thing. Then there are complex problems, like raising a child. There's no one way to raise a child. You can't give someone a laminated checklist on how to raise a child and expect that to work repeatedly every time or even like any time. And the key observation is that we are besieged by simple problems. Life is full of fairly easy things that we just don't know how to do or that we just don't do consistently. And so checklists are how we solve simple problems. We hand someone a checklist that reminds them how to do the simple problem. So these are a couple of ours. Let's see what are these, this is an initial. This is sort of a project level assessment, like this is a self-service checklist that a developer would walk through when they're getting ready to write a new component or reviewing one. We have a checklist for vulnerability management. That's when we find out that there's a vulnerability in open SSL, just hypothetically, and have to decide what we're going to do about it. We have a lot of these, we think they're great. If you want to dive into the checklist world, there is, yes, there is a checklist for writing checklists that will tell you how to write a good checklist. And of course there's Gwande's book, which is pretty fantastic.  When do we do security, who does security, and what is doing security? And if you take only one idea away from this talk, please let it be checklists. They are great.  Okay, so you know your staff is trained. You know how to write secure software. Now it's time to start thinking about when things go wrong. As Bob Dylan said, everybody must get owned. I think I have that right. The fact is though that this is sort of unfortunately true. Bruce Schnauer observed that we're starting to sort of view breaches as a fact of life. And this is depressing because it shows just how bad we are at our jobs and how much we need to really level up here to stay ahead of the black hats. But there's some silver lining to that for people involved in security. Because it means that more and more we're judged on our response. Our time to respond, our ability to contain and attack, our transparency, our security practices. Really interesting point, this is a tangent, but this just happened. There was just a circuit court that ruled that the FTC has the authority to find companies for violating security practices under the idea that it's deceptive trade practice to offer your customer's privacy and not follow basic security practices. This is a really interesting decision because it implies that the FTC might actually be getting into that courts and regulators are getting into the business. Not just of regulating PCI and HIPAA and those sorts of things, but actually like do you hash your passwords? You don't hash your passwords, that's an accepted best practice. We're gonna find you for doing it. And so suddenly I think we're reaching a point where companies security practices are being critiqued, certainly in the court of public opinion, as we've seen with Sony and Ashley Madison. But I think we're actually shortly going to start seeing company security practices critiqued in the court of courts. And I think that's nothing but a good thing, right? I think that will drive much more adherence to the things that we already know are our best practices. But so what this means for us is that we need to get our house in order before anything happens. There's no way, if your incident response planning starts when you get that phone call telling you that something's wrong or that there are attackers on the network or that there was a login from someone who left the company a year and a half ago, I mean it's just gonna be disastrous. I know one company that was breached last year that I spoke to a person who introduced themselves as their CISO, their director of security. I found out later that they didn't have a security department. And when the breach began, the CEO called this person and said, I'm promoting you to our chief security officer. Deal with this. Yeah, don't do that. So it's hard for me to be more prescriptive as I've been in previous points here because I think the details of what an incident response plan will be are going to be very specific to your organization and your risk profile and your regulatory exposure and your customer base and your product and et cetera. So I just want to give you some questions to think about and a framework to structure your incident response plan. And the work in writing an incident response plan is answering these questions. So I break down IR into five steps. The first one is initiating a response. How does someone report a breach? How do we track incidents? Do you have a bug tracker do you use? Do you have a white board? Do you use Trello? Where do you track that stuff? What are the, another good point. What happens if the thing that you use to track is the thing that's been compromised? Good question to ask. What are the roles and responsibilities during an incident? As you move into actually managing the incident, you need to understand how communication is going to be happening. Who communicates? Where does it happen? Who's involved? How often do you send situation updates? An important question for people at the sort of senior or management level is at what point do I need to wake up the CEO, the executive team? How severe does it need to get when I need to bring in lawyers? Those sorts of questions. We then need to figure out what's even going wrong and how are we going to collect this information? Who's going to follow up? This can be fairly lightweight. Our preferred tool for assessment during a breach is just a Google Doc. We open a Google Doc and everyone just writes in it.  But now we've got a nice, complete time-stamped log of everything that we did during the response. So this doesn't need to be heavyweight stuff. But knowing that that's what you're going to do, it saves you those five minutes of arguing about where you're going to track your work. So once we've figured out how severe a problem is, we need to know what our response SLA needs to be. I mean, the reality is that not every incident is everything's on fire, all hands on deck, stop the presses. Some things are, yeah, this thing is bad, but worse would be waking up the team responsible. Let's get them in in the morning to fix it. And you should have some system for determining that, so it's not a seat of the pants decision. Once you fix things, a really common problem is to sort of knee-jerk and fix the one thing that happened in that situation, even though it actually maybe that's not the root cause or maybe this incident exposes a lot of other long-term things. So how are you going to ensure that any long-term remediation tasks are actually followed through on?  There are likely going to be both legal requirements and then there are, I hope, also ethical requirements around when you tell your customers that something happened. And finally, all of this is useless if you don't learn something from it, and so you should understand how you're going to reflect back on the work and explore the causes. And what do we need to collect? What sort of information do we need to know? Again, there may be legal reasons for this. We have some requirements around the information we have to collect around incidents for being able to sort of talk about this to our customers, but you may also want to be able to collect metrics on root cause so you can go back in five years and say, hey, five years ago half of our vulnerabilities were network related and now only a quarter of them are, we should focus elsewhere, or whatever it is. So a lot more reading, I'll give you a few pointers. We wrote about incident response at Heroku, the pattern that we use for handling. This blog post in particular is about production outages when a service goes down, but we use the same system for managing security incidents. And it's based on the incident command system from search and response teams. And then this last one, Magoo, Ryan, last name is escaping me, he ran security at, he worked in security at Facebook, he ran security at Coinbase, he's got a pretty good pedigree, and he kind of wrote like a, oh shit, you've been owned guide, and you could do a lot worse than just starting with that as your IR guide. So Wednesday, this is the hump day, this is the hardest part of your week, creating your incident response plan. So Thursday, governance, risk, and compliance, this is the awesome part. There is an absolute alphabet soup of governance and compliance regimes out there for companies and for organizations. These are just the ones I know of in the US, I'm sure that there are like 37 million more across the world. And for the vast majority of at least small organizations, none of these are worth your time. Now, this may not be true, if you're a health information startup, like that HIPAA spec, all 800 pages of it, it is your best friend. If you're taking credit card payments, you better know PCI, if you wanna sell things to people in Europe, Safe Harbor's gonna be pretty important, but for most smaller organizations, you can skip right over these things. But you ignore formal risk programs at your peril because sooner or later, you will want to take credit card payments, you will wanna get into the health information market, you will wanna sell to people in Europe. And it's gonna be very important when that happens for you not to have shot yourself in the foot in the early days by completely ignoring this work. So you can save yourself a ton of effort by laying a very easy and simple groundwork right now for a formal risk program. And really what we mean when we talk about GRC is documentation, right? We're at the Django conference, I don't have to sell you on documentation, like we know it's a good thing, so document your security work, right? Have you made a decision about company policy? Write it down. Really, really easy stuff, but surprisingly, a lot of company policy decisions stay in email. Hey everyone, from today forward, you must use multi-factor auth with your GitHub account. And then you don't actually track that anywhere. And so someone, when an auditor asks four years later, like, what's your policy around multi-factor auth for GitHub, you can't produce anything to show them. Whereas if you had just taken that email and put it on like a wiki somewhere, now you've got a policy. You don't need to worry about formal language. There's this idea a lot in compliance like you need to use this very stilted, legalistic business language. The audience here are not judges and lawyers. They can be very informal. Our official password policy requires that you use at least two of letters, numbers, uppercase, lowercase, and emoji. And nearly every auditor that we've showed that to highlights the little emoji line and then gives us a big thumbs up about it. The other part of this is just tracking as much as you can. So someone asks you for access to a GitHub repo. Reply back with an email, yes, I'm confirming, I'm giving you access to this repo. It seems a little formal and weird, but it ensures that you have a paper trail. And again, when you get into the point of being audited, this is what an auditor is gonna look for is a paper trail of what you've done. Even better, most of us are engineers or work with them, write a system to track access, control, and access requests. We wrote one, it's great, auditors love it. I also would suggest that you write three documents to become the skeleton of your risk program. A data classification guide, what data you have, where it's stored, who has access to it, what controls are around it, what category is it, is it PII, personally identifiable information, is it payment data, is it customer data, you know, how do you classify and think about your data and control access to it? The second are checklists for access control. Think onboarding and offboarding, right? When someone starts or leaves your organization, you need to make sure that their accounts are turned up and turned down. This is important for formal audits, but it's also a common way for people to get breached is someone who used to work there three years ago still has access to GitHub for some reason and their email account gets taken over and you can do the math from there. So having a checklist of who gets access to what, when, and how, and tracking that, and then you can go through and uncheck items one by one as the person offboards. And the third thing is a weird thing to document when you don't have much process already, but I think it's one of the most important things you can have is what is your exception process? There are always going to be exceptions. There are always going to be situations where you need to break a rule and it's so much better to know how to break rules than to pretend the rules never get broken because if you just think that everyone always follows this policy always, then you never know when someone's not and it bites you to come later, but if you spend the time to decide who approves exceptions, at what level do they need to go out? How are they tracked? Those sorts of basic things. Again, auditors love it and it's gonna level up your company's security posture. So document everything, write some basic policies. So you've done most of the work, now it's time to tell people about it. You know, the fact is if you actually work through this checklist, you will be better off than most of your peers, let's say. No, I mean look, like again, we only have to look back over the history of data breaches over the last few years to see that people are getting owned through some pretty basic stuff. So if you've taken the time to address the basic stuff, you are doing a really good job and I know that this stuff is scary and I know that security seems like a battle we can't win and maybe it is, but we can win it most of the time and we can win against most of the attacks and if you've taken the time to address this basic stuff, you're in pretty good shape. You should feel fairly good about this foundation and you should be comfortable and happy bragging to your customers about the work that you've done. So I'd suggest three things. You do need a privacy policy. This is a legal requirement if you're taking any sort of personally identifiable information and that should live at your site slash privacy. I'd suggest a security page as well that talks about what you do about security, document some of the stuff that we've talked about earlier and you should maintain a security knowledge base. Talk a bit later about whether this should be public or private. That's an interesting question. So the privacy policy is necessary. It's legal requirement and a lot of companies just won't even do business with you. Okay, I work for Salesforce again. When I wanna buy a thing, the form that I fill out to send for the initial legal review, there's a link for where I need to put in the link to their privacy policy and if they don't have one, like I can't even get the request started. Like our procurement team, our legal review team just won't even think about buying from you if you don't have a privacy policy. It's just a non-starter. If you have lawyers, they'll write one for you. If you don't, Automatic has published a couple of templates that are worth starting for. Weirdly, Automatic and WordPress have different privacy policies even though they're the same company. So I don't, but I haven't figured that out. Maybe they're just two different versions of the same template, but they're both good. So you could steal and attribute and share alike and use them. Your security page. You should summarize your security practices. This can be less formal. The best way to think about this is sort of, this is where you tell your security narrative, right? Like this is where you talk about, at a high level, what your security program is trying to accomplish. You know, you can kind of explain there what the things that you do, what your program looks like.  And you know, you can talk about all of this stuff in somewhat bragging languages. If you have any formal adaptations, if you've done PCI or HIPAA or et cetera, you should list them here. And the most important thing that you should have on this page is tell people how to report vulnerabilities. You should probably have a security mailing list. You should probably have a PGP key that's kind of like a Brown M&M test, like looked it up in Wikipedia if you don't get the reference. The idea is it's sort of a sniff test, like a way for people to tell that you're serious is if you have a PGP key. I'll tell you in like two and a half years at Heroku, I think one person has sent us an encrypted email. So there you go. But if you tell people how to get in touch with your security team, they'll be much more likely to actually do that and not publish something publicly about how they tried to report a vulnerability and you didn't listen. So the last one is the security FAQ. The way I think about this is every time a customer asks you a question about security, or an internal person, a product manager, a sales person, a marketing person, a non-technical role asks you about security, write down the answer. And over time, you'll discover there are some natural groupings. You know, at Heroku, of course, there are a lot of questions about containerization, like how do we separate one dino from another dino? This comes up a lot. And so there's a lot of questions in there and we've kind of grouped them all around like container security. You'll notice that we don't publish ours publicly and this is an interesting point. Transparency is a really important value but there are also some good reasons to limit this information. There may be confidential information in there, there may be things that disclose information about other customers that you might not want, there may be information about the level of your security readiness program that you want to be transparent with customers but you probably don't wanna share with non-customers because they're not, you know, because that's a much broader group of people and they're not under NDA. So my litmus test for publishing security information is transparency going to make my customers safer. If it is, publish it even if it hurts. If it's not, if it's going to make them less safe, don't publish it even if it hurts. So that's your day five, privacy policy, security page and a security FAQ. So to recap, your minimum viable security program is train your staff, develop an SDL, a virtuous cycle to ensure that you continually develop and learn from your software development practices, have a plan for incident response when something goes wrong, be ready to do something about it, lay the foundations for a formal risk program and tell the world, good job, you've built a security program, thank you. So I'll be around in the halls in the rest of the week and there's contact info there so you can ask me questions in any of those formats but I won't be taking them here. Thanks y'all.


Hello and welcome to this session, let us understand the following problem today.  Now let us write the solution we have delta is equal to 1 x y 1 x plus y y 1 x x plus y.  Now expanding it through minus x so eliminating this row and this column we are left with minus x multiplied by x minus of x plus y. Now which is equal to minus x multiplied by x minus x minus y this minus x cancels with plus x so we are left with minus x minus y which is equal to plus x y. Therefore the required answer is x y. I hope you understood the problem bye and have a nice day.


Sparking because they got the double Sunday. That's never a good sign. Get out of there UI Goku. No way! How's it going everybody? Dodd, did I hear back with some more Dragon Ball Fighters rank matches? This time playing the suggested team of Gojina Blue, Dragon Ball Super Broly, and UI Goku. I don't think the team had a theme behind it necessarily, but it was just a team that the commenter played. And you know what? It's comprised of three characters that I really enjoy. So, oh Broly C-Assist! So I'm actually really looking forward to using this team is what I was going to say. But Broly C-Assist, that is a classic. I usually use Broly C-Assist a lot. I actually ended up going back to his A. Am I going to get mixed? Am I going to get mixed? Oh! Oh my God! I got hit trying to go for a DP. Tried to invisible reversal my way out of the corner, and you know what? Didn't really work out for me that time. Hopefully in the next... Okay, I'm not dead. I should definitely either vanish or Spark. I'm down for... You know what? I'm going to go for a Spark. Quick Sparking, dash in, overhead. Let's go cross him up. Are we going to be in the corner? Oh my goodness! Barely! I'll take it though. Okay, we made a mistake. That's fine. We can recover. We can recover. Oh, he's getting hit by that? No! The dive kick! It would have been sick! Alright, I can't believe he got hit by the UI Goku A-Assist though. Not going to kill with that. We have to switch. Go! Broly! Put him in the corner. I don't see how that's fair, but you know what? Broly doesn't... I don't think Broly cares about what's fair.  Do I just go for a 2M? Would he wake up level 3? I'm going to stomp on him. Okay, okay. At least I didn't die. Alright, that's fine. Oh my goodness! Okay, he mistimed that a little bit. Otherwise, I definitely wouldn't have been able to block it. I forgot I didn't have Sparky anymore. Trying to hold Vanish just ended up putting myself in the corner. Wow, he's got combos. Okay, you want to look for that? If a baby player does that combo, you know they got combos. Oh, okay. Every time they do this setup, it's always Cross-Up. Watch, he's going to cross me up right here. Yep, see? Honestly, it's a good... It's good knockdown pressure, but it's so good that they're always going to go for the mix that they get off. Like, nobody... The point of that mix is that they could go left or right, but nobody goes left. You're not going to work that hard just to go left. It's like... I don't know, dude. It's just... It's one of those things. I definitely... The psychology of it just is very funny to me. No! I can't believe I got hit. That was sick. Okay, it's just going to come down to UI. Oh, man. Yeah, but just working so hard in a mix to go right. Who wants to... Like, he earned the option to go right. He's not going to want to go left. All right, let's see if I can't... Oh, barely. You know, if anybody... If any character would make this comeback, it's UI Goku. Hold on. We got the flip. I'm mashing, but it's okay. Mashing is just... Mashing is just skill. I'm falling. All right. Do I want the corner? No. I'd rather just take the kill. Let's see if we can get it down to two bars. I didn't really work too hard to get that down to two bars, but that's fine. It's still going to be two bars. I'll take it. Okay, Super Baby 2 is also relatively low. If we can get him out without getting hit, this game is really even. Unfortunately, we'll have to face the boss character that is Broly. But at this point, you know, we've put ourselves in the situation. Oh, I got to stop going for long-winded buns. I got to start throwing out more jumping mediums. He got combos. We already saw he had combos before. Sea assist extensions. Okay, now this one, I don't know where he's going. This one, he did not set up the right left. So you know what? I'm UI Goku. Do I care? Why didn't I do a level three? Please kill level two. That all looks smart. No, my IQ is showing. This is so embarrassing. I want it to look smart. I should have level three. All right, we were UI Goku player before, so we can't beat one now. Flip. Oh, that was sick. Dodging the bro. Man, UI Goku is so cool. Okay, we're probably not going to win this. But I'm UI Goku. I'm just going to mash the lights. The lights will do it themselves. All right, come on. Come on. We can win this. We can win this. No. Oh my goodness. Why did I? Why did I do that? I don't know why I got forward Keyblast, but that was the incorrect choice. Got Keyblasted out bro. It was just on site, too. As soon as he saw those Keyblasts come out, he was like, all right. That was a great first game. If we can get more out of Gojira blue, I'm feeling, I'm feeling good. We need a better start for sure. Okay, good clash. There we go. Beautiful. We're probably going to have to switch out here for sliding. I don't mind though. Throw this man Broly. Let's go. Up tech. Unfortunately, we missed time in the up tech. But again, it's fine. We can get a Broly combo off this. Oh, are you going to switch out maybe? No switch. No switch. That's fine with me. Two bars. I don't know if this is going to kill. It's probably not. Come on. Oh, that's fine. Beam. I'm looking for it. No, I was looking for the switch. Wow. Nice. I can't believe you confirmed off that into the quarter that man, that was a great switch. I had to set my feet for that. Darkness spring shot combos. We've got combos. Whoo. Why am I mashing? Alright, I'm switching. Oh, see, it's just nice. Broly spawns a little bit behind him too, so it's hard to get like, I don't know. It's hard to time the defense, I guess. Combos again. Loops. Loops. I'm going to down tech mash. Oh, but like I said, I'm just going to go with the theory that he is going to try to mix me here. So just hold back. Oh, see there you did it. Nice. The second time is not as guaranteed as the first. But they do have to, I guess they do have to establish the mix up in the first place. That was nice. I'm glad he went that side. Now you can see what I was talking about. See, it all came full circle. I did have to get hit for it, so I'm not sure how good the demonstration was. I did have to suffer for it. But hey, I'm glad it got showed up. Alright, now down. Nice. We lost UI Goku too. No, wait, no, we didn't. We lost Broly.  I was just thinking, I was like, wait, we lost UI Goku? Aw, man. The comeback. Why do I always forget about Broly? He just charged at me. No, Broly! My own Broly against me. He has two Broly assists. Two Broly C assists. This is bad. No. Stop it. Stop it, dude. He keeps going for my Broly. Why? Alright, I'm just trying to make the most out of this.  So we've successfully gotten rid of one of the Broly assists. That's good. That's good enough, I think. At least, well, not maybe not good enough, but it's less scary. I don't have to go against both. Alright, just take the kick there. That's fine. Oh, I wasn't going for a drag rush. I was just staggering. The pressure. I'll take it. Alright, now the boss monster. What's up, Broly? So I heard you're pretty scary these days. Come here. Dive kick this man. I regret it. I regret it. At least you feel any better, Broly. I do regret it. Alright, fair play. Check me into the corner. Okay, to be fair, these are both strong characters. I will literally walk through the Key Blast. I'm bad at the game. I'm bad at the game. Oh! Alright, I'll spark two. Just barely missed the six-end. It wasn't in range. Huge misfire. It's getting close. Nice dash up. 3 bars here. I would if I was him, yeah. Is that the slow one? Does that combo into the slow? It is the slow. Alright, don't UI attack. It's too risky. It's too risky. Do I do it anyway? Oh my God. The sparking. I forgot it was in sparking. No! The pressure. Oh, we couldn't escape. We're going to lose the first set to 0.  Great games. Unfortunately, we couldn't get it done. Okay, we are back with the next game. Unfortunately, we didn't get a rematch. I was actually really hoping we would just because we're playing early. I guess usually you can get rematches during the morning. Let's go! Alright, early hit. We need to see more out of all of our characters, honestly. We need to see more out of me. Can I get some Broly combos started up? Why am I... I don't know why I keep thinking that combos. Unfortunately, alright, tech out. No! He was swinging from the get-go. Alright, I see it. 17 B-assist? Is that his B-assist's right power charge? Yeah, okay. I mean, if it was C-assist, I think I would have gotten a combo down. Like that. Alright, I'll take it. Super jump. Get up there. Broly combos. One orb. Can I get two orbs? Two orbs into the assist. Grab him. No! Drop the get-go. G to come out. Save me. Just tag out. If you're ever in danger, just tag. Alright, speaking of tagging, I'm going to go ahead and get Gohan back in here. Goku assist in the air. Love to see it. Get back down there. That was my bad. Broly! No, stop. I can't let you hit Broly like that. That was actually pretty nice. Please tell me that hits. Okay, good. One bar is going to be enough there. I wasn't sure if that was going to combo. Alright, we are really struggling. We are really struggling with combos today. But when are we not struggling with combos? You got to remember, you got to embrace the drop. Okay, reads. Nice. Love to see it. I'm going to switch into Goku here just so I can... I don't- I just want to land this grab. How did- what? How did we walk away with the hit there? Damn, that was close. I literally heard the God slicer sound effect. I actually flinched. I closed my eyes. I thought it was over. Alright, Broly level one. That's going to be good enough. And now it's just down to 17. Do I make the read? 17 players like to be a bit aggressive. Jump in! Oh, he didn't do it. Jump in! Oh! The Super Dash came in, but it was a little later than what I was reading for. Okay, I'm worried. Orb? Yes, no, can't Super Dash through it. Oh my God, that was smooth. Barrier? Yeah, I'm not messing with the barrier. Mixups? Whoa!  He's insane. Android 17 players are crazy. No! I got to play Android 17, dude. I forgot that this character is cool. Like, this character is so sick. The follow-up doesn't, we were a bit too far out of the corner for that. No! Nice! That was also really good. 17. Oh, why does... Oh, yeah, I guess they were both in tournament of power. I was going to say, the game has so many voice lines, I'm always trying to figure out why they would give voice lines to a certain character. Also, this should be it. Exit just so we get an extra hit. Still have three bars? Yeah, yeah, okay. I exited because we had an extra half a bar and if we needed all three, it would have been nice to have them. But didn't really need them. So we're going to look at the camera, turn away. Let's go. All right. Feeling good, feeling good. First game felt really nice. I'm hoping for a bit more out of the second. I'm trying to, I'm just trying to go crazy with these characters. These are all characters that if you get momentum, you can really just run away with the game. Okay, do I, do I, do I just 2M? Okay. I can just throw me in. The 2M being that, that much range on the 2M just makes me want to use it. Fake the Kamiami, I'm overhead. No, he slightly got up a bit after. No, that could have been so cool. All right, bar and a half. Is he going to spend it to switch out? Probably not. Oh, just get the first level up. I kind of agree actually. Oh, corner. I'll take it into the air. I don't really have a lot of assist up. I'm not going to mess with that. Go optimal. No, get back in the corner. There we go. All right. I think he might actually be done for it. We'll see how much damage we can get out of this. But with a sea assist. Yeah, that's, that's going to be it. Three bars I think because I just got nervous. Man, that is a lot of damage though. Big bang, Kamiami, I'll put some away. We now have the character lead and it's looking like this might be the momentum we've been looking for. Hold on. I'm going to read the jump. Oh, the DP was still right. Even though I was not reading that. I was just trying to catch up in the air throwing a button. Ah, nice switch. Got to admit, good tag. 17 going off power charge. Oh, wait, he had power. It's built up already. Key blast ran right into him. Oh, the kick. That was clean. No, I don't know why he's both my assistant panic there, but I was definitely panicking. All right, Broly comes in. We're not going to wait. Wow. Okay. That time he backspun. I thought he was going to like back up a little bit. I was going to go. I was trying to adjust to what he did last time for a new read. Oh, that's bad. See, it's just don't go on. I'm getting calm for that. I'm getting calm for that. Damn 17 is so. If that would have worked, I would have been shocked. I would have been so impressed. A mix. All right, throw him back into the corner. Love to see it. Whoa. No, yell at this man, Broly. That was clean. No barrier. You can't escape the corner. Broly doesn't want you to leave. All right. Level three. Do we read spark? I'm going to say yes. He didn't spark. You want to go away? The hate stun is still so good. I reacted super late to that. Nice. All right. I know this Gohan already has a level and he's limit broken, but I still think, you know, there are others. They're scarier characters to see in limit break. I definitely, I definitely think that I was, I missed one too many. Oh, I'm just waiting, dude. I'm just waiting. Yeah. Let him to reflect being just a scream is so. Yell at him. Yell at him. Broly just screaming out of here. We're just screaming. My goodness. You I go to again. Three bars. I'm going to do it one more time. The damage boost coming through, even though this is a relatively scaled combo. Damage boost. I'm just going to grab him this time. Broly players. Oh, why did I do the medium version? What if you have techno? I would have looked like a genius. Yell at him. No. Well, we couldn't scream forever, dude. Sometimes you got to rest your voice. Take it easy, Broly. I think hopefully we have it from here. He could just raw level 3s though. Oh, he's nice. Oh, I'm crazy.  Oh, no, no, no, no, no, no, no, no, no, no. Is that okay? Is anybody scared? Is anybody? Boom.  That was unfortunate.  That was cheap. I have to go another game. I actually feel like, oh, no. Why did you take work like that? That was just unfortunate. Okay. We are back with the final game up against a great breeze rank who's using a pretty interesting team. All right. I just need, we just need to wrap up strong. We started, we started, we got bodied.  That was nice. We started, got bodied. Second game was relatively clean for us. We got lucky with the UI tech at the end. Now it's going to be up to see if we can't walk away with it. A positive. Oh, no. The two inches. Don't get three bars. Oh, three bars. Oh, nice dude. Andrew, that move from Andrew H21. The, like the palm is so sick. Unfortunately, we are going to lose Broly relatively early. All right. That's fine. Clean up the play. Clean up the play with Gogeta. As long as Gogeta, we need performance out of one of the leader Broly. The UI Gogu has been, oh, level one. I hope I timed that correctly. Oh, no. I thought I timed it just right. Nice. It's done. Wasn't enough that time. Oh, no. I might 21 get away. Man. Why did I think that would work? That's not how that move works. Oh. Today's two H. All right. We're going to spark here. Yeah. Definitely need to spark here. Momentum was getting away from us. Oh, nice vanish. Oh, that was almost sick. Luckily, we did just keep swinging. At that point, I mean, I really didn't have another option. I couldn't do it. Not with that combo. Anyway, that combo was bad. Only one assist. Let's go. All right. We got to get Trunks back in here. I need a character down. We can't allow them to have access to this many assists. Trunks. Oh, missed time. That's fine. We're going to get it. Just level one. Don't do anything too risky. I've been dropping a lot. We get this game. It's going to be through sheer will. Because the combos are definitely not holding up there. Part of the bargain. All right. 18 comes out. Do I 2M right away? I'm heavy. All right. It was risky. I admit it. Dragon rush. Yes. The reflect read. We were right that time. Five bars. Oh, no. No. Another bad drop. 21, two and a half bars. There it is. One. Okay. That's bad. We're going to have to make another comeback. We've been asking a lot of UI Goku. But you know what? He's always been there to answer the call. Let's go. I didn't think you'd get hit by the jab. Okay. That made the subsequent grab with him pretty futile. Nice. I do regret it. Wow. We got lucky. Oh, nice 2H. Level three here seems. Okay. Never mind. Going to go with the level one. UI UI tech. Or is that crazy? It wouldn't have been crazy. I think it would have worked anyway. Go. Oh, really? That time? Wow. I don't know. Maybe they were set for something else. I cannot believe they saw the UI tech for the second one. Okay. All right. We're down a game. We're down a game. We could still come back though. Oh, no. Okay. This is looking bad. This is looking like it could go bad though. This is looking like it could very much go bad. Two inches. These two inches are crisp. Okay. They came out just right. All right. In the corner, we got that going for us. Use my jump. I'll take it. Oh, the barrier. Nice call. I was definitely looking for like a cheap pressure. Sparking because they got the double Sunday. That's never a good sign. Get out of there UI Goku. No. Oh, and it hits both. Oh, no. The Androids. Why? Oh, God. Oh, we're alive. Let's go. Defense. I'm back in the game, baby. You thought you got go G to two. Never that. I think at this point you can tell it's fake confidence. We've people lost. Oh, no. Broly. I've asked a lot of a lot of characters today, but I need you. It's just, I just, I'm having, I'm having, I'm having a bad day. Broly. I'm so sorry. A level three right here would be devastating. Okay. Good. I wouldn't have made it, dude. I wouldn't have been able to please. I almost got myself in by level three. All right. We're all changed the future. We're out. The confidence is bad. The confidence is bad. The momentum is all off. All right. Hold on. Down tech. No, no, no down tech. Keep lost. All right. Go, go, go. Just go. Six, seven. Oh, nice finish. I thought mashing there would be the one. Oh, no. It's over. Maybe it's not over. Two bars. Two bars. Oh, it is over. Wait. No drop. Oh, beautiful combo. It is over. Oh, no. We never got to set our feet again. Devastating, dude.  I'm a cupcake. Not wait. Well, that's going to be the end of this video. Hey, that was devastating. Hopefully you enjoyed it. If you did, definitely let me know down below in the comment section while you're down there. Make sure to like button and subscribe and check out some other videos that should be on your screen right now. If you want to catch those other than that, I've been Dr. Doi and I will see you in the next one where hopefully we won't get beat down like that. That was crazy.


Okay. So, hello guys. I'm really grateful to see all of you here today and today we are going to go through what OpenStack Performance Team is and one very specific experiment we have done during last OpenStack development cycle. So, about 1,000 nodes and we of course will cover who's OpenStack Performance Team, who's taking part in it, some quick round of introductions and I hope this session will be really interesting to you guys. But first of all, let's start with some round of introductions. So, my name is Dina Belova. I am a development manager at Miranti, currently focusing on performance and scalability researchers including performance and scalability researchers of OpenStack. So, Alex? So, my name is Alexander Shapochnikov. I am a distinguished engineer at the Mirantis and my main focus is working on the making, OpenStack working on a pretty big scale, like 1,000, 2,000 nodes. So, Matthew? Thanks. So, I'm Matthew Simona. I'm working at Sineria, part of a project called Discovery and this project aims to study OpenStack to see if it fits with the Fog or Edge Computing requirements. So, we are very much interested in scaling OpenStack and doing some experiments with multi-site deployments. Cool. So, first of all, our agenda. We'll go through what OpenStack Performance Team is, when it was kicked off, what we're doing on daily basis and what can you find useful for you guys. That will quite switch to this very specific 1,000 nodes emulation experiment and Alex and Matthew will talk about methodologies, test environments that were used, the observations and the conclusions that we made. At the end of the session, we'll definitely have Q&A session time and we'll be really glad to answer all your questions. So, regarding OpenStack Performance Team, originally it was kicked off during Mitako Summit at Tokyo and technically we are part of large deployment team working group. But comparing with large deployment team were focused not on running tests, not running OpenStack on huge scale, not on the operators type of questions, not on the development type of questions, not on the deployment type of questions, but on the OpenStack evaluation and defining methodologies of how can you evaluate your OpenStack, define if it's scalable enough, is it performant enough and if it can fit your workloads you want to run on it. So, during the time we exist, we run tests on various cases for various OpenStack components and they're underlying technologies and we did it against several test environments. For instance, at Miranto, we mostly are running tests on two test beds, one is 250 physical machines, the second one is 500 physical machines. Maybe we'll go through the environment shortly and of course the tools that we chose to run tests depend much on the test nature. So, if we're talking about control plane testing of OpenStack, about measuring API latencies of OpenStack components, the most obvious solution to PIC will be Rally, basically because it's configurable enough to cover complicated cases and even extended cases like density testing where we're trying to push as many OpenStack resources as possible to the cloud and check how very primary workloads behave between those resources and how is your control plane and data plane still behaving. If we're talking about reliability testing, we can use combination of Rally and OS false library that allows us to emulate issues with OpenStack like OpenStack services availability, hardware issues, networking issues and check how your OpenStack continues to behave on both country plane and data plane layers. If we're talking about density data plane testing, right now we're mostly using shaker tool that allows you to measure and evaluate OpenStack tenant and working capabilities and measure such values as latencies and throughput and all that kind of stuff. As said, we're not only focused on OpenStack itself but on the underlying technologies like messaging bus, database and for instance container cluster systems like Kubernetes. For instance, if your OpenStack is running on top of Kubernetes, you probably would like to know how your Kubernetes is working. So, in that case, of course, we use other tools more specific for that kind of tests. In short, OpenStack performance team is trying to evaluate OpenStack, define methodologies of how you can do it and we share our experiments and our experience and going to provide in performance documentation. So, this is basically part of upstream OpenStack documentation where you can find all our tools, methodologies, test plans, test results and I suppose you can check this link and find something very interesting for you guys because we have very cool researchers regarding reliability tests, regarding neutron features testing and regarding, of course, 1000 nodes testing that guys will be shortly talking about. So, Alex, please proceed. Thank you, Dina. So, 1000 nodes experiments. What is it all about? So, first of all, 1000 nodes means 1000 of compute nodes. That's the first thing. Most of the current companies are thinking about going with pretty big clouds like 200 nodes, 500 nodes and sometimes would be even 1000 nodes. It's not a big deal previously to build a cloud of 100 or 200 nodes but usually it requires a lot of the customizations, properly patching and at some point you're figuring out that you wouldn't be able to go with the newer version of OpenStack because you have to backport some patches and you stuck with some kind of very, very customized cluster or your cloud which you're not able to upgrade and you have to migrate workload to properly newly deployed cluster. So, at some point it's pretty tricky to go with a lot of customizations. So, the best way is to go with the original upstream cloud and during our evaluation process of current, which was during the Liberty release cycle. So, we basically checked out all the testing results that we have in Miranti Scale Lab and decided to give it a try, a code of 1000 compute nodes with standard settings without any additional patching of the core services. Take one and every service by one instance. There is no HA proxy. There is nothing additional for all balancing service. So, each and every service, only one instance and 1000 of compute nodes. As you probably know, there is common approach existing in OpenStack community to put a lot of the control point services as long as core service like RebitMQ and MySQL in some particular nodes called Controllers and that will lead to some performance issues on the scale of 200 nodes because these nodes usually are a bit overloaded even if it's very, very kind of top-notch hardware used. So, the best approach is go in Granova service and put them on some separate nodes and as soon as cloud starts growing, that's pretty good. But as long as you have everything initially combined, it's very, very hard to go and move that from the working cloud. And if you start to putting every single service initially on the different nodes, that will be pretty big control point over here from the beginning of the cloud life cycle. So, we decided to change the paradigm how we deploy it and we put each and every OpenStack service into the container. So, we use Docker containers. Well, everyone knows about the benefits of the containers and it's very kind of simplifies this ACD pipeline. It simplifies development. It simplifies upgrades, updates and even downgrades and rollbacks in case of failures. So, we put each and every service into the container and how it's helped us. We probably kind of, well, we get a chance to monitor each and every container for the resources consumed by this particular service. So, it's kind of Neutron, Nova or anything by that. And we also aggregate all the computer-related services like kind of Neutron agent, Nova Compute and LiveBird in one containers and basically scale that out on top of Mesus plus Marathon to 1,000 of these container instances. So, because we put everything in the containers, we got the benefits of using the pretty simple and the pretty commonly used toolkit of CAdvisor and InflexDB and Grafana to monitor and measure all the stats. And we basically, after we deployed everything, we run the rally pretty simple case. It's called boot and list, which actually spawned the VM. After that, it will pull for its status and when it's ready, it will just do the list of the VM. So, it's kind of mostly no API request. And do that in the 50 concurrent threads for the 20,000 of the VMs. So, that's pretty neat about the testing environment. So, you could properly switch to the describing of our Mirantis lab for doing that. So, it was initially it was Mesus plus Marathon and Docker for the tasks, which are OpenStock services. That everything was run on 15 nodes, which kind of have two sockets with E5 zons, 2680, one quarter of terabyte of RAM and almost one terabyte SSD. So, that was that we took the liberty release of OpenStock because it was during the Mitaka Summit and we decided to test and do that with the kind of, you know, stable branch without going with the development stuff, not to trying to play this kind of property, some features which wasn't completely developed at that moment of time. And the one thing we modified, the libvirt driver, we didn't use the fake one because the problem with fake one is it wouldn't retrieve glance in which it wouldn't do the kind of spawning of any ports, any preparation. So, we modified the original driver and just removed from there and mocked up the part which run the QM or QVM, but still reporting active state to all the VMs which they at some point should be run it. So, it actually do all the ports for the neutron retrieving glance, I mean retrieving image from glance and stuff around that. So, that's pretty about our deployment in Mirantis. So, made you probably produce their setup and their initial results. Thank you. So, yes, at Inuyah we were using pretty much the same idea as Mirantis. We deployed OpenStack on approximately 30 nodes of a platform called Grid 5000. It's actually a small fraction of the resource that can provide the platform. So, if you want some more information about Grid 5000, you can check the performance documentation weekly and you will find all the useful links to the resource we got and how to get an access if you want. We used also a containerized deployment. We were based on the Mitaka release at the time of the experiments and to be more precise we used COLA to deploy OpenStack on the platform. And actually we just modified, we just added some new features. We used experimentation on our platform like we need it, we really need to test different topologies of service. We configure services on the fly and so on. So, we had this in augmented COLA tool which is available on GitHub. You have the link on the bottom of the slide. As Alex mentioned, we do make use of the fake drivers on our side because it was easier for us to get this cluster of 1000 nodes deployed. But, as he mentioned as well, there is one drawback to do that is that, for example, the no-effect driver doesn't implement all the neutral operation on virtual machines. So, that means that evaluating neutral with no-effect drivers may be not accurate. So, that's why I think you use the modified driver to do that. Before we dive into more details on the results, I would like to add some more elements on the experimentation process we have. We measure all the resource consumption of the OpenStack services deployed with 1000 compute nodes. So, we made a set of experimentation with this deployment, with this type of deployment. And for each deployment, we consider three distinct phases. So, you have on the left part of the slide phase one, which corresponds to a phase where OpenStack is freshly deployed. No resource has been created yet on OpenStack. So, that means that no API calls have been made to create VMs or create tenants or create networks and so on. So, that means that during this phase, OpenStack is just idle and empty. So, measuring the resource consumption on this phase is quite interesting because you will see, you will evaluate how much it costs for all the OpenStack services to maintain the state of OpenStack. Next, if we move to phase two, phase two is basically the phase where we are injecting some loads into the deployed cloud. So, as it was mentioned earlier, we use Rayleigh benchmarking tool to do that. For the sake of simplicity, we use boot and list scenario with these two key parameters, so 20k iteration and currency of 50. So, basically at the end of the benchmark, you will have 20,000 VMs created by the benchmarks. So, if we move to phase three, phase three is a phase where all the resources have been created by the benchmark but not yet cleaned by Rayleigh. So, in this phase, you will measure basically how much it costs for OpenStack to maintain all the resources created by the benchmarks. There is also, we could have added phase four which will be the cleanup phase but it's out of the scope of this presentation here. You will find all the details in the performance documentation wiki if you want. Okay, so we decided to make this presentation on a pair service basis. So, we will go through different services like RabbitMQ, MyADB and so on. And we will present you some observation and conclusion we got on these services. So, here, we consider in this slide RabbitMQ in the phase one, so where OpenStack is empty. What we can observe here, basically we made a set of experiments with different number of compute nodes. So, we increase the number of compute nodes and we measure the resource consumption on the RabbitMQ server to see how the resource are varying with the number of compute nodes. What we can observe here is that the resource consumption regarding the course usage and the memory usage is increasing linearly with the number of compute nodes. So, we started from 100 compute nodes until 1,000 compute nodes which this kind of increase is quite kind of expected. There is also another key metric to check if you, when you evaluate RabbitMQ it's the number of open connections you have on the RabbitMQ server. So, here, this graph depicts the number of open connections during the idle phase, during the phase one on the RabbitMQ server. So, as you can see here, for 1,000 compute nodes, for 1,000 compute nodes, you will have more than 15,000 open connections at the same time on the RabbitMQ server. So, here, you must pay attention to the fact that you may hit some system limit on basically the number of open files you have on your system. So, you may have to tune your system to handle this kind of load. So, if we move now to the, yeah, if you move now to the phase two, so RabbitMQ under load. So, what we observe during our experimentation, so here RabbitMQ was deployed in a standalone mode, so we use only one server to handle the load of all the messages. The load that we observe during the rally benchmark was large enough, was big enough, sorry, but still it was tolerable. So, for example, we observe a CPU usage of, a core usage of 20 maximum if you look at the graph, you will see that in average it's more like 10 core use and RAM was increasing until 17 gigabytes of RAM. During phase three, after the benchmark stopped, we can see the effects of the periodic tasks. So, the periodic tasks are basically tasks that are made to maintain the states of OpenStack and all the resources created. So, the cost of this periodic task is evaluated here to be something like between three and four cores and 16 gigabytes of RAM. If you now move to the same evaluation for the database server, we just present some highlights. We won't go into too much detail here. So, what we observe for the database server, was deployed as a standalone server as well, like Rabbit MQ. We observe that for 1,000 compute nodes, the database footprint was small even for this number of compute nodes. So, you have the number on the three first items. So, few cores usage, less than one gig RAM used, less than 200 open connection at the same time. What is interesting to note here is that you can see on the database, you can see the effect of the periodic tasks. So, we measured the effect of the periodic tasks that OpenStack is doing and we evaluated it to approximately 650 requests per second on the database. So, this is the effect of the periodic tasks. If we check now the database on the load, the conclusion is that the database is behaving correctly. So, there is something, some observation to be made nevertheless. For example, if you check the graph on the top right corner, you will see the number of connection open on the database at a certain point of time. So, here we hit something like a little bit more than 2,000 connection at the same time. So, similar to Rabit MQ, you may need to tune your MayaDB instance to accept this number of connection at the same time. Memory is increasing the same way. It's expected because connection and memory are tightly coupled. Regarding the cores, the cores usage was quite stable with an average of 0.7 cores used. So, if we now move to the OpenStack core services, so I will start with the Nova scheduler and Alex will continue with other Nova services or Nova OpenStack services. So, if you deploy the 1000 nodes cluster, it is easy to hit a bottleneck regarding the scheduler because of the nature of the scheduler. So, the scheduler service is a little bit different from the other Nova services even if they rely on the same code. The difference resides in the fact that, for example, for Nova API, Nova API is able to leverage multiple workers to handle all the incoming requests. So, as you can see on the graph here, Nova API is able to use something like 10 cores at a certain point of time to handle all the incoming requests. But for the scheduler, it's quite different because the scheduler instance is stick with only one worker. So, that means that this worker has a lot of job to do like accepting the incoming request. It must take the decision, the placement decision for the VMs and it must also continuously update its internal view of the OpenStack compute nodes. The view is important because the view reflects the state of the system and the view has to be up-to-date to take accurate decision on the placement. So, here the scheduler is only using one worker. There is one worker around to overcome this issue is that you can start several scheduler instances. So, by horizontally scaling your scheduler, you will be able to accept more incoming requests. There is also one drawback to be aware of if you start another scheduler instance is that placement decision of two schedulers may collide on a single physical post. So, you will maybe increase the number of retries you will have to do to place a VM. So, this is the observation for the Nova scheduler. So, we'll take the mic to Alex. Thank you, Mathieu. So, let's switch to another conductor. So, that is one of the most loaded services in OpenStack. So, it's actually half almost no idle timing as long as you have compute nodes, of course. So, as you could probably see, in phase one there is almost kind of close to five-course load. So, it's only about regular reports coming from the compute nodes which have no instances at the time. And during phase two, a lot of workloads place it on the compute nodes and you see the kind of pretty interesting graph which consists of small spikes and pretty big huge spikes. So, small spikes, it's kind of periodic. It's kind of, it's maybe kind of a little bit kind of slightly kind of flat, but it's basically every one minute periodic. So, it's kind of reports from the compute nodes about the resource available since kind of overall stats. And the huge spikes, it's about ten minutes' periodic which kind of reporting the states of the instances. So, it's a shutdown, active, something like that. So, as you probably see, as soon as all the workloads place it, there is nothing changed in the conductor because it's still servicing the same periodic test, there is nothing, nothing got changed. Our resource consumption is pretty huge. Sometimes it's up to 30 cores, but usually it's somewhere about kind of 10, 12 cores if it's kind of, could be averaged. And let's switch to the second service. It would be NoAPI. So, NoAPI is pretty kind of predictable. So, it consumes a lot of resources during the workload placement. RAM is growing not so significant. So, kind of, it's pretty expectable. And as you probably see, it starts a little bit, kind of a little bit shaky at the end, but it's only because of that. As we mentioned in the rally scenario, we're using Novaboot and LIST. So, LIST is, all the time, LIST are basically returning a pretty huge list of instances. So, for NoAPI, it's a little bit tricky to handle this stuff. So, only because of that, the load on it's a little bit increases. But at the end of the workload placement, you see this pretty stable load, same periodic stuff, and kind of stuff like that. And in phase four, as you could probably see a little bit, that's very, very critical part. It's when the rally starts to clean up all the workloads. And because of that, that's such huge spike. I mean, that's the kind of confirmed thing. That's, well, you should care not about only workload placement, but kind of active workload putting and removing. So, if you have pretty active usage on your cloud, you should take care about the kind of NOVA also, because it requires a lot of resources. So, that's it. And let's switch to the kind of mostly kind of hungry service, Neutron server. So, most the consumption, as we see, from the Neutron server, it was about 30 cores. Sometimes it could be, as you clean up phase, it's 35 cores. And that's pretty interesting all the phases. So, first phase, as you probably see, a lot of the drifting and with pretty high loads. And it's basically the time when all the compute nodes got added to the cluster. So, during this time, all the compute nodes are reported, all the DVR routers got created, and stuff like that. And then, just before the starting of phase two, there is a spike that is the moment when the rally test prepares the environment and creating users, tenants, and most important networks. So, that's mostly API load. And then, the second phase could be divided into two parts. Second, this kind of growing, and after that, it's falling down and going pretty steady. So, first one, it's when the Neutron, I mean, basically, we're trying to fill out all the compute nodes. It's placing one VM per node because it's kind of pretty free, all of them are available. And it's basically kind of Neutron trying to tune up each and every DVR and DHCP on each and every server. So, only because of that, kind of initial load is pretty high because it has to do that first time on the each and every node. But after about 1,000 of VMs got placed, it becomes pretty steady. And it's kind of growing significantly. And after the overclothes got placed, you see that stable load about 10 cores. It's mostly about the RPC stuff from the agent about the status. So, it's kind of clear periodic. So, that's everything about the Neutron server. So, let's switch to the conclusions we have. So, what you would like to properly to do for the big open-stack quote based on liberty or new release and if, of course, you go in with the containers. So, as you probably know, you could limit resources in containers like cores, memory. And in this case, you probably would like to tune up API RPC workers because by default, service will grab number of cores from the containers that wouldn't work for you. Or just not to limit cores and know approximately how many cores you will have from some particular system. So, you probably would like to set up some particular numbers because it will be clear at least how big the scale for some particular services. Next, MySQL and RebitMQ wasn't a bottleneck at all. At least in terms of the huge resources, I mean consumption. There is a rumors about MySQL, RebitMQ instability. There is a rumors that maybe a lack of resources or something like that. It's maybe true if you're using the clustered solution with not enough performance between the, not enough usually network bandwidth and CPU or resources for the cluster of the kind of MySQL or RebitMQ. But for MySQL, as you could probably see, there is a very, very small impact on the resources. It's consumed, I mean in our experiments in Miramarantis, it was about three or four cores in the kind of pretty stressful scenarios. For RebitMQ, yes, it took a lot of resources. Clustered solution will take even more because it's overhead for the cluster, but it should work fine still. And main issue is the scheduler performance. So, as Meteor already mentioned, one scheduler is not enough and multiple schedulers wouldn't work for you because they will have a lot of the race condition in the putting the VMs on the workloads. As far as I know, in the Ocata release, there should be some improvements for the scheduler which kind of allow it to go and be at least working on the multisread course in some, there's a little bit different approach in the workload placement and scheduling in-between inside of the scheduler itself. So, just properly, you know? Yeah, so, before I go ahead, thanks Alex and Meteor and there is one thing I really want to emphasize that such kind of experiment can run on really modest type of hardware lab and if you have your own interest in checking some specific services or some specific workloads, not necessarily the ones that we were taking, you can do it yourself, use the methodologies that are written and described on the performance documentation. So, the first link is about 1000 nodes. We have our weekly meetings of our performance team on Tuesdays at 3.30 p.m. UTC and there is some set of sessions I really want you to emphasize. Check this week on OpenStack Summit. Later today, Red Hat guys are going to have really cool session regarding BrowBeat. This is a testing tool that can check OpenStack scale on performance using Rally and several wrappers around it. Tomorrow, there is a really cool session made by Mirantis Neutron team regarding OpenStack Neutron being production ready for large-scale deployments and they did really cool research around how many VMs can pretty modest hardware installation of OpenStack with Neutron handle and how do the workloads behave on those type of environments and how the data plan and control plan behave. On Thursday, we'll have our OpenStack performance team session regarding what was done during Neutron cycle and the Cata planning because this is only one of the experiments we run. We have a bunch of stuff to share and if you would like to know more, just join us on Thursday so we can jump to Q&A session. Any questions? Just please use the mic if you want to ask something because it's recorded. You have fake NOVA and Neutron agents which reported instantly, isn't it? No, actually no. You mean about the Inria or Mirantis experiment? I mean about the compute part of the... No, that was the actual agent. So they take time to respond for output of... Yes, yes, yes. Okay, thank you. That was the main point using non-original fake driver and modified one, yeah. Have you dragged the scale of Neutron servers? Basically, we initially we placed a lot of the servers combined across these 15 nodes in the containers and later we figured out just using kind of not the full scale like kind of 20,000 of nodes when we would like kind of work a lot with kind of 200,000 of VMs we take a look on which services consumption resource and after that we just moved all this kind of service like it was first we moved conductor then we moved the Neutron server and NOVA PI initially we separated MySQL and RebitMQ but MySQL later we returned it to the common pool because we've seen that the resources consumption not so big. Yes, as the container also. We just dedicate all the resources. We tried to use two of them but in our case we didn't use the load balancing I mean in terms of the HA proxy we used just DNS round robin so in this case it would be understand API request would be splitted but RPC would be consuming with both of them. So the load is kind of going just split it out to it's stable to both of them so that's also fine, yeah. What level would you like? Hello, I've got two questions using NOVA cells help with the scalability of the scheduler and the second one is did you look at how changing the frequency of periodic tasks would help with performance? Okay, so maybe I can. So in these experiments we haven't looked at the cells it's planned for the next cycle actually so here we only use the 1000 computers in the same pool so maybe in six months we will have some results regarding the cells. For your second question, yes we've changed the timing of the periodic tasks and it helps a little bit because of course the periodic tasks are not so doesn't load I think the periodic task was running like every 10 seconds for some part so if you increase that you can decrease the load on the database and all the load on the RabbitMQ as well so you can with 1000 load you can find the load of 100 clusters if you decrease the periodic task timing it helps. I could also add a little bit so about the periodic tasks that's the pros and cons of decreasing the time so if you increase in the time you decrease in the load in the state of the current cluster you wouldn't be able to pull it accordingly so for the 10 minutes tasks that should be something changed in another computer conductor to make it shuffled because as you've seen we spawn all the nodes and because of that we have all the spikes because if these nodes would be theoretically randomly launched they would have a different time and it would be ideally a static graph but that's not possible there's no such random stuff usually everything is spin up close to the same time so it's maybe worth a try to add the random time to this 10 minutes periodic because it's pretty huge on load for the one minute periodic of the conductor you could decrease it but the problem is that based on this particular task schedule the site which computer nodes have amount of research so if you decrease it to the 2 minutes then you will have the information only once the 2 minutes so scheduler may do the incorrect decision based on that so that's basically that time usually decrease it up to 10 seconds so to have more or less immediate information about the computer nodes you usually decrease it but it depends it could be pretty flexible on that so it depends on how many and how often you're placing your workloads on your cloud because you have kind of pretty monolithic cloud there's no changes but if you have some kind of CICD processes in it and it's kind of spawning a lot of VMs removing them in this case you probably would like to go with the pretty small periodic timings just a question about the like an architectural question so you didn't have in your setup NHA whatsoever no it's not you have just one rabbit ok so no mirroring of the kills which killing the performance of the rabbit we tried it's kind of it's not just killing it it adds a little bit overhead on the performance somebody else from Grantis was telling us on the session before your session that it's killing like with 3 members with the full mirroring of the kills it's killing well performance by half yeah that's true if all kills are mirrored if all kills are mirrored that's basically not the case that's the first case and as you probably seen we have 20 cores what even not top-notch kind of mediocre servers have about kind of well let's say 40 cores in this case it's still one main moment we used rebutting Q for all the services in case if you're thinking that well it's maybe not enough performance of some particular if you grab the new cluster situation with containers allow you to spawn a different cluster for the different servers just multiple rabbit mqs makes sense have you noticed or have you looked and seen their volume service this is actually like the problem is we seen their volume because from my experience there's a lot of customers of Grantis using completely different drivers for the sender and there's a lot of users who use each of them which storage did you use for this test we didn't use sender volume at all alright so that's in the room most of this no no no any more questions? just please use mine because it's recorded yes sorry so I had a special configuration of the database of the red band we used in this test by default configuration we did a little bit we have this small slide with the kind of settings that you probably would like to apply it's not mandatory but it's highly recommended to go with because you may kind of not fit into the kind of numbers that you already have so for database it's mostly about the connections we did nothing beside the connections for the nova api of course max pool size because there's a lot of connections to the database so you have to have pretty big pool size for the workers so for conductors we already mentioned that's usually kind of only number of workers for scheduler it's that's a general recommendation so to have a pretty well quite fast provisioning of the workloads you have to go with at least one scheduler per 100 components nodes and you will have this cons it will be close to 100% load so if this saturation is kind of pretty high you will probably see sometimes the rescheduling happening because of some of the schedulers will try to put different VMs on one compute nodes and one of them will be killed and rescheduled so that's probably it and about the linux you probably will have to go and tune up some of these settings on the older servers because there's a lot of connections going on so we changed the queue size we changed the max connection parameter that's things was mandatory but for we done that initially because we have everything on 15 nodes so it's kind of it's pretty huge amount of load but it's still true for the end that's probably it and we have run out of time sorry we just okay thank you thank you guys


This is Think Tech Hawaii, Raising Public Awareness. Aloha, I'm Prince Dax, the volunteer host of the Prince of Investing. Think Tech is important to me because it brings Hawaii's number one financial literacy show around the globe. For the first time, Think Tech Hawaii is participating in an online-based fundraising campaign to raise $40,000. Your thanks to Think Tech will run only during the month of November and you can help. Please donate what you can so that Think Tech Hawaii can continue to raise public awareness to promote civic engagement through free programming like mine. I've already made my donation and look forward to yours. Please send your tax-deductible contribution by going to the website ThanksForThinkTek.CostVox.com. On behalf of the community enriched by Think Tech Hawaii, 30-plus weekly shows, thank you Mahalo for your generosity.


In the Middle Ages, we see an important development in theology and that's where the term transubstantiation does become explicit. Because there were certain people, Beringer of Tours is one of the greatest examples of this, of people saying, well, you know, the Eucharist is just symbolic. And so the church had to say, no, there is a substantial presence. What does that mean? Well, substance is what a thing truly is. And so to say that the Eucharist is not just a symbol of the Lord, it is His true presence. His body and blood are substantially made present within the Eucharist. And Thomas Aquinas becomes the great theologian that lays out even how that happens, right through the sacramental grace of the church.