Hey guys, this is Matt Beck from FreeSalonEducation.com and I'm here to talk about client retention. This is something that's so important to the salon business. If you want to have a successful business, whether you're one stylist or 25 stylist in a salon, you have to focus on your retention. When I was away at the Millennium Experience in Fort Lauderdale just a few weeks ago, John Harms showed this report that was focused on retention and what it did was it showed the percentage of guests that are still with you after a certain number of visits. That report was called the MA210 report. So if you use Millennium Software now, check it out. It's such a cool report, but I just pulled it up for my salon. I about lost my mind because it shows you after six visits how what the percentage of clients you've retained is and we're at 12%, which industry average is actually 6%. So I'm not happy with 12%. If you're only keeping 12% of your clients after six visits, then that's that's a challenge in the industry and that's something that we have to work on. The reason I'm making this video is because it's something I want to start working on. So I think it's something that you guys need to take a look at. If you have salon software, look at the report. There's got to be a report that focuses on retention and look at, okay, so we have a certain amount of new guests, how many come in one visit and never come back, how many come in two visits and then never come back. The challenge in the industry is that we're so focused on the new guests and making them happy and then as we get comfortable with them over the fourth, fifth, sixth visit, we're just bringing them in, sitting them in the chair saying you want the same thing as last time. We're not focused on creating an experience for that guest so they stop coming and they go on to the next salon. You got to keep your business in your salon. So check out the report, focus on your client retention and I think you're going to see your business blossom and grow and that's my hope is to take that 12%, bring it to 20, bring it to 25. That's really the goal there is just to keep it growing, focus on your numbers.  Thank you guys for watching. We'll see you on the next video.


We good? Awesome. So if we can just maybe close the door there when you're finished in the bends, that'd be great. So hi, my name's Rob Clark. I'm going to first off start by apologizing for the clickbait.  What it really should have been titled was something along the lines of please, please, help with security. But the title I actually ended up with is OpenStack Security Project, securing the world's fastest, largest open source project. It is a non-trivial effort undertaken by many people for whom security might not actually be the primary function of their job. And I'm going to try and guide you through what it is that we do and how you might be able to get involved and be able to help us along. So quickly I should speak about myself. I'm the OpenStack Security PTL, and I have been for a couple of years. The security project actually started about four years ago myself. And Dr. Brian Payne created something called the OpenStack Security Group, and I'll tell you a little bit more about the genesis of that. I was one of the early VMT members. Those guys were all too smart for me, so they kicked me out at some point. I'm a security track chair, so the security track here at the conference and many others I've helped put together. I'm a cloud security architect and an IBM Distinguished Technologist. I've been at IBM now for eight days, so I have literally no idea what that means. But there you go. So I'm going to go through a few bits and pieces with you. I'm going to talk about the state of OpenStack Security, the different initiatives that we have ongoing and how you can contribute. So one of the things I think was interesting was to see some repeats of this slide at this conference. And there we have sort of 85% or so of surveyed enterprises saying that security is the number one blocker for adopting production-based OpenStack clouds, the number one blocker. This should not be a surprise, but it seems to be a bit of a challenge for OpenStack in general. We have these two security organizations, the VMT and the security project. The VMT is retroactive. The security project is very much proactive. And we complement each other quite well. But one of the areas we struggle with is bringing new people in. There aren't many people who are paid full time to do security for OpenStack. And it would be great if we could bring more people in if some of the major contributing organizations could throw a little bit more resource our way. Yeah, this entire presentation is just me begging for security resource. So there you go. So OpenStack security is a little bit like herding angry cats that just want to write code. This is how product people see security, see OpenStack, as you say, sorry. So we have various services. We can build little walled gardens around them. Those are the blue dotted lines. We can have a gated secure interconnects between them. Those are the bright green arrows. And then we map data paths. And that is generally how product people like to say that OpenStack works. If you ask a developer, it kind of looks a little bit more like this, keeping in mind, of course, we're missing all of the big tent niceness that's come in recently. So this tends to be how developers see OpenStack. And this is kind of how security people see OpenStack. And we're running around fighting fires. And we're doing our best. And I'm going to go through a little bit of the things we're doing to try and fight some of these fires. So I've done a lot of number crunching and some very big data analytics. And basically, we've come up with a graph like this that shows, it's very scientific, that shows the accumulative effect of vulnerabilities being detected in OpenStack. This is just a very general guide. But basically what we're going through here is the fact that when OpenStack started, there really wasn't much of an effort in terms of security. And when you look at some of the early vulnerabilities that were found, they were pretty bad. Security started to ramp up. We started to peak a little bit at some of their problems. And we saw a much bigger uptick in reported vulnerabilities. And now we're at a point where there are a couple of people dedicated to just looking at vulnerabilities in OpenStack and finding things. So we are finding more vulnerabilities in OpenStack as we continue to go forward. And this graph will never level off. But the things we find tend to be a little bit deeper. A lot of the low-hanging fruit has been picked from the OpenStack tree. So there are plenty of vulnerabilities to pick from in OpenStack. But this just shows you how vulnerabilities have evolved over time. So back in 2012, we were talking about really basic direct reach-a-versals and arbitrary file creation vulnerabilities within OpenStack. A lot of the time, it was unauthenticated users could just bring things into existence on your cloud, which is bad. Improper sanitization in instance names leading to cross-site scripting. 2014 had a whole rash of SSL and TLS failures right across OpenStack. And a lot of that traced back to Python implementations. And some of it drove the move across to using the Request Library, for example. Then we move on to 2014, 2015. And the vulnerabilities we find are a little bit more complex. And if you want a list of every single security vulnerability that has an advisory, where there's a link at the bottom of this slide, and I'll share the slides later. Also, if you Google OpenStack OSSA, you'll find a list. So what we're finding now is that the number of reported vulnerabilities is going down. The number of issued advisories is going down. And I expect to see similar numbers for 2016 and 2017. So the OpenStack security project started as the OpenStack security group. It was an informal group that myself and Brian Payne started about four years ago. The idea was that we would consult and try and work with teams to improve the state of OpenStack security as opposed to just reacting to vulnerabilities. It's made up of security engineers, architects, and consultants. We also have a number of OpenStack developers now who are just involved in the project. Michael McKeown, amongst others, people who've just kind of dropped in and said, hey, I want to do security things. And we're more than happy to buy the handoff and work them till they die. Consultants on security. So we consult on security advisories with the vulnerability management team. The VMT itself only consists of a few people, and they can't be expert in everything. We have a much wider group of people that are running public clouds, private hybrid, so we can provide extra perspectives and things on vulnerabilities. We have four to five calls at any one time. Around about 25 active participants. We have a couple of mid-cycles each year, and they're normally very well attended. And there's a lot of different security initiatives that we take on. So the vulnerability management team I'm going to tell you about in a little while, they exist organizationally as part of the security project. Previously, I think they were release management. They are largely autonomous in and of themselves. This just makes more sense from an organizational point of view. So I don't get to see any of the confidential vulnerabilities that the VMT manages, unless they bring me into consult on a specific issue. So we have developer guidance and security notes, the security guide threat analysis, lots of documentation efforts. We also have active code efforts like Bandit, Anchor, and Cintra Boss. I'll tell you a little bit more about those. And we still have an aspiration to do cryptographic audit and tracking. That is, being able to say to, if you're a cloud deployer, being able to say to your customer exactly which algorithms are being used at any one time with any set of configurations. Sounds easy. It actually gets slightly messy. And one of the things we're hoping to do is leverage Bandit for some of that. So around the false and time frame, we came together. And this is a bit of a timeline since then. So security notes started after Grizzly. The security guide followed threat analysis. An agreement to do VMT work. And then our code projects have followed on in the last sort of year and a half, two years. So vulnerability management is really important in OpenStack. And the vulnerability management team actually has a very well-established process for managing vulnerabilities that come in. So they receive, investigate, triage, document, and publish. Their actual official documentation looks like this, but I wasn't sure how well it would project. It's a very well-documented process. You could just Google the VMT, and you'll find it easily enough. And if you do find vulnerabilities, please follow that process. Opening a public launchpad bug makes everybody's life a lot harder, because we have to run around much faster to try and patch things in the open. And if it's a particularly bad vulnerability, it can be difficult for deployers to fix in short amounts of time. So we ask that people, if you find a vulnerability, that's brilliant. But if you could submit it as a private bug, that would make everyone's lives a bit easier. There were plenty of OSSAs to pick from. And this is just an example of one that's gone through the system. So this was in 2013, the 36th OSSA to be issued. And it was a cross-site reported by a Cisco employee. It was fixed, publicly disclosed, bug with marked public, which isn't great. Backports were completed. Got a CVE in the advisory, got published. Backports completed is a really important point. So vulnerability management team only release security advisories for things that can be fixed in supported versions, which leaves a really big gap for things that are just horribly broken, but we haven't managed to fix, like, Nova RPC. And I'm going to talk a little bit more about that with security notes. So security notes document common mistakes, bad configurations, and the mitigations for some of them. They're up on the wiki, much like most things a security project does. We have, in the order of 67 or so, security notes at the moment with a few more in the pipeline waiting to be taken on. Security note 64 actually went live, or is complete as of a review that was done earlier today. And the vulnerabilities themselves can be different things, but they're normally things that won't or can't be fixed in an existing release, or where it can't be fixed with a patch. So we create this documentation and goes through them. So it's written and managed by the security project. They very much are intended to complement security advisories. And that's a link to where all the issued notes are. The idea here is that we create a one-stop shop for the deployer. So although the security guide exists, if you're deploying an OpenStack Cloud, you should go and take a look at the set of OpenStack security notes and see which apply to your deployment. Because there's a lot of gotchas, there's a lot of considerations that need to be made things. Good examples are database caching and flushing with Keystone and things that you're just not necessarily going to think about. But you need them to deploy OpenStack securely. So they're linked into a couple of places. They're linked into the security guide, and they're online for review. And we do actually have deployers coming to us now and asking us questions about the OSSNs, which is great, because it means that people are actually going looking them, consuming them, and overall helping us improve them. So OSSN 64 is going to be published in the next day or two. It's in public review. This was a great vulnerability in Keystone, whereby deployers were leaving the admin token auth method enabled. That's an auth method that's generally used for bootstrapping the rest interface on Keystone when you're bringing things up. And it basically meant that if you set that in your header and gave the passphrase admin, because it just uses a default one, you would get admin and Keystone, which is bad, which is very bad. And it's not something that you could easily fix with a patch without breaking a lot of people's systems. So we created a security note for it that discusses the issue and different ways to fix it, like removing that piece of middleware after you start your cloud running, which is what you're supposed to do, but it turned out people weren't doing it. And that's a good example of the sorts of things that we try to protect against with security notes. So I mentioned before that this talk is about trying to get people involved with what we're doing in the security project. Security notes are a good way of doing that. They involve taking a bug from Launchpad, working through a reasonably well-documented process that is here. It's up on the wiki in far more of the both terms. The security project cause will absolutely support anyone who wants to take on writing a security note. There's always a back list of like three or four that need doing. Sometimes it gets a little bit longer. And we're definitely very keen to have more people come on, even if they're just coming on as reviewers to start off with. That's always very useful contribution. So like I said, security advisories and security notes complement each other. So security advisories are generally, this is a bug, there is a patch, it's bad, you need to go fix it. Whereas the security notes are more along the lines of, if you configure it like this, you're gonna have a bad time. So I mentioned the security guide a few times. I'm not sure if we've got Nathaniel in the room or Michael McCune, but those guys have been doing an excellent job on updating and pushing forward the security guide. So the security guide we created in 2013, it's available, or it will be when they fix the weird RST issue, be available again in PDF form. It's all online, it's all in the security docs, which is part of the OpenStack manuals. It reflects the current state of security within the OpenStack community. It's basically inspired by the Ops guide, which was released before it. And it's designed for deployers, auditors and even customers who are looking at different clouds who want a good set of questions to go and ask a provider, have you configured X or Y in a secure way as per the security guide? Again, this is another place people can get involved. It's great, if you would just wanna go and pick holes in the security guide, because something changed in Keystone or Nova works differently now or something else, then that's absolutely encouraged. Spelling errors, everything else, all fine. You'll get your ATC tag, you'll be very happy, you'll get an extra little thing on your budget in the next conference. We really, really do encourage people to get involved with these documentation type efforts. They're very good because they do reach quite far and wide. The security guide has been consumed a lot and we've had a lot of good feedback on it. A couple of example topics there. We go through a very wide swath of thoughts and choices you have to make when deploying an open stack cloud and the live links to the guide are there. We also have a set of security guidelines. The tag line there is don't write sucky code. I shouldn't possibly say please don't write sucky code. These are a list of things that happened a lot that we saw coming up in multiple vulnerabilities again and again. This actually was generated out of one of our mid cycles and inspired by something at the time I was working at HP and we'd created a set of coding guidelines specifically for Python, specifically for cloud type operations. We kind of forced it on our developers and we thought it was unfair not to force it on the rest of the community. So we put these together and made them available. The idea is that Bandit, our static analyzer, can actually tie back to these findings as well when you find people making the same mistakes again. So the guidelines are pretty simple. They're up on the open stack security page and it's a reasonably accessible list of things and they're written in a very conversational format. So it's example-based and unvalidated redirects. Here's an example of bad code that we see from time to time when people write things using Flask. And here's the example of how you might do it in a more appropriate way. They are like most things open stack community driven so if there are multiple ways of doing it, the author will probably write it in whichever way they're most comfortable. In this case, it was someone who's more comfortable with Flask as opposed to say Pekan or something else. Again, this is somewhere if you wanna be involved with what we're doing, if you work with a different framework, you could come in and tell us all about that. So that was the documentation efforts that we're taking on. I wanna talk to you a little bit about some of the code projects we have as well. I'm gonna talk to you about Anchor. A few people here will know that Anchor's one of my pet projects. And PKI has a really big problem that we can help you with. I can do about an hour and a half talking about either the problem or how we can help with it. So I'm gonna try and give you a really condensed version and at the end there's a link to a far more elaborate deck that goes through things about Anchor. So Adam Langley wrote this recently in his blog. Adam Langley at the time I think was a Google Chrome engineer.  It doesn't stop attacks. He's not wrong. But I'm gonna try and take you through very quickly why he's not wrong. And then some of the things we can do to try and improve it. So TLS, SSL, it's all the security that you have in a certificate, if it's on a certificate being revocable. That is to say that I no longer trust this certificate and you shouldn't either. There are a number of ways of doing this. The first one was certificate revocation lists. A bit essentially a signed list of certificate fingerprints that should no longer be trusted. Turned out it didn't scale. So the online certificate status protocol was created. Nice lightweight UDP protocol. You reach out to some machine on the internet and say is the certificate still valid? And you get a yes, no, maybe message back. Unfortunately, that didn't scale. So then we have OCSP stapling, which is when a server can fetch its own OCSP attestation, bundle it together and pass it to you when you do a TLS handshake. Because they're individually signed, it means that you can be sure that the certificate is still valid. There are a couple of problems. Firstly, if a man in the middle can remove that. The second is that it doesn't work with intermediate certificate hierarchies because there's only space in the protocol for one. Then we get onto something called OCSP must staple, which is, I think it's still an RFC, but it's a change to the certificate specification that means that when you get a certificate, it says that it has to have an OCSP attestation with it and if it doesn't, then you should reject it. Which gives you kind of short life certificates because OCSP assertions are only valid for a short period of time. Most browsers don't actually support any of that, at least not for general certificates. So especially Chrome. Chrome has CRL sets, so Google creates an audited list of a very small number of certificates that they believe you need to care about as being compromised, and those are all checked. And this is easy to check on your machine. You can go to, if you set up an open SSL, CA, create a little website, give it a certificate, and then revoke it with a CRL or with OCSP, and then go to it with Chrome. Chrome will just say it's perfectly valid. It just doesn't check, which is a little bit difficult to swallow sometimes, and I bring this up only to encourage people to go and actually have a look at what TLS is doing because I don't think it's doing what you think it's doing. So once CRL is Mozilla's approach to CRL sets, although they still do some OCSP, depending on the type of certificate they get. So, PKI is pretty broken inside the browser, and it's completely broken outside of the browser. We don't support any of those more advanced technologies in any of the Python libraries we use in OpenStack. And in fact, most of the C libraries that you can do crypto suite stuff with don't support them either. So, this is kind of a metric of what is supported outside of the browser. You can just about get OCSP stapling with some C libraries, but basically, the only thing that works at all across the board, and the only thing that works at all for Python is CRLs, which, as we discussed before, don't scale. So we created Anchor. Anchor create gives you very short life certificates. I'm not gonna go very far into this because I'm wary of the amount of time we have to go through everything. Basically, it solves a lot of these problems by only giving you a very short lifetime certificate.  And this gets rid of revocation problems because the one control that works universally across all the crypto seats you tested is the checking the expiry date on a certificate. So by giving you a very short life one, we guarantee that we know when things will be revoked. It gives us a lot of nice properties, like by looking at the audit logs from Anchor, we know exactly how many machines in our infrastructure are using certificate at any given time, which is actually quite a nice quality and a bunch of other stuff as well. So it's an ephemeral PKI system, super lightweight stack. The idea of Anchor was that it could be deployed, say designate needs certificate services internally, it can just deploy Anchor internally. It doesn't require an environment where you have a big corporate style PKI that it needs to be configured to plug into. It's designed to just be spun up by individual projects as they require it. This is not our actual logo. It is some public domain image I found. I wish our logo was that cool. So short life certificates, rule-based issuing. A key point is that the rule-based issuing allows you to give it some understanding of how OpenStack works so it can understand naming schemas for different bits of OpenStack, that sort of thing. Yes, so there is an Anchor presentation. This is a wonderful screenshot from that presentation. If you're at all interested in PKI and the problems it can solve and the way Anchor can help you with that, then there's a Google Shorten link at the bottom that will take you to that deck. Cintra Bus is an awesome new project that we've just come brought into the security project. It is a further, which I know a few people here enjoy, for OpenStack APIs and services. If finding security issues in the RESTful API, it allows you to create all sorts of interesting first payloads. It logs all the requests and responses. Nicely, it supports keystone authentication so it can start working its way fairly deeply through branching logic within OpenStack rather than just banging on the front door. It detects common defects and it's generating some useful output. It's going through a whole bunch of changes at the moment, but if you're interested in a more up-and-coming project where you can have a bit more impact and get really involved, then Cintra Bus is a really interesting one to take a closer look at. So here's an example payload where we've just got a fairly simple descriptor below about how something's going to be interacted with and it generates reasonably parsable output at the moment. There is work going on, or there was work going on to give it multiple formats for output as well, which is similar to what Bandit has. The idea being that you can start collating output from all these different tools with a sort of a similar look and feel. So it's open source, it's easy to extend. It's very much, it's open stack-centric, but much like Anchor, it's not too tightly coupled to open stack. So if you want to apply this to some of the tooling that you've built to work with open stack, which actually seems like a fairly smart thing to do, then you have the ability to do that as well. So Bandit, Bandit is probably our most successful technical security project. It is a dynamic static analysis, which doesn't make sense. It is a dynamic static analysis engine for Python, what it actually is, is a static analysis engine for Python. Bandit finds issues in Python code, it can find a whole bunch of issues, I'm not gonna read them all off the slide there. It does it by first taking the Python code, putting it into the Python AST, de-offuscating certain things, and then looking for certain patterns. It's found a whole bunch of stuff.  It's had a whole bunch of development done. It's, like I say, it's probably our most successful project we have in the order of four or five big open stack projects now running it in their gate. And our ambition is to have every open stack project that uses Python have it running inside their gate. So this is an example of output from the Bandit test suite. And here you can see it uses pretty colors to tell you which things you think you need to care about. It gives you severity and confidence metrics. This is something I ran a day or two ago against Kola. These are some of the issues that it brought up. In fact, these were the only two issues it brought up. And the good thing about this is when this is in the gate, you'll get these results. You can then go and look. And if it's not a vulnerability, if it's a false positive, then you can mark it. And Bandit will know to skip it next time. But more likely than not, when you have things coming back with a high confidence, unlike some other static analysis engines, it's pretty good at finding vulnerabilities. So I'm gonna show you something now. So I can't really see any of you, so I won't really bother asking. But if you've used any static analysis engines before, you'll know they're incredibly difficult to install. So I'm gonna go ahead and hopefully we won't run out of time and show you how to install Bandit. Okay, so Bandit's now installed. Now, obviously that's just pipped. One of the big issues that you have with any static analysis engine is actually working on configuring it and tuning it and making sure you don't get too many false positives because they tend to be incredibly noisy out of the box. Sometimes that's because they rely on heuristics over time and sometimes that's just because they need a lot of tuning. They're very delicate machines. And obviously Bandit's no different. So as this video goes on, I'm gonna try and show you some of the more complicated ways you can configure Bandit before you can get useful output. The dash R there is very important for getting useful output. And there you go. That's static analysis with Bandit running over a thousand lines of code in a fraction of a second, finding some useful output. The point there is that it is not static analysis in the way that you might have experienced before. It is incredibly quick. It has good defaults and it is finding vulnerabilities and stopping vulnerabilities going into OpenStack code base today. So it's open source. It's plugin based. It's super easy to write new plugins. You can read a whole bunch more about it on the wiki or go straight to have a look at the Git repo. And as I demonstrated a second ago, it runs very, very quickly with just a standard configuration. So I wanna talk to you a little bit about where we are. We have a security blog. It's up on GitHub because that was the path of least resistance at the time. We're always looking for new people to contribute and the blog can be somewhere that's quite useful to get involved as well. We've only started blogging about things since the Mitaka mid-cycle and we have a significant number of posts on there now. If you're at all interested in any of the threat analysis work that's going on, then I would love to talk to you more about that. There's a lot of good information about that on the blog. We actually don't have threat analysis in this deck, which is possibly an omission. But there's a lot of good information up on the blog and we're very, very actively chasing the threat analysis stuff this week. So if you wanna understand more about that, then talk to myself or talk to Doug after this. So security project in summary. We have around about 250 listed sort of lurking people on Launchpad. Around 20, 30 people active at any one time. There's really opportunities to get involved at any level. If you just wanna help us out improving our documentation, if you wanna help us out by working on these interesting projects. I mean, Bandit's getting picked up all over the place. Ironically, two of the biggest companies I know that run it outside of OpenStack are Uber and Lyft. I'm not sure why it's sort of taxi companies that like it, but they do. And other people are contributing all the time. Anker helps solve a really big problem, not just for OpenStack, for other people as well. It's such a different way of approaching PKI that sometimes it's difficult to get traction on. So if anyone's interested in working on or adopting Anker, then please get in touch. And Cintra Boss, which is the new upcoming project we have very much on the coattails of Bandit, really looking at trying to actively improve security in OpenStack. There's scope for writing your own tools. The OpenStack Ansible Security Project is a great example of some works going on outside of Security Project as a whole because it should sit more with Ansible. But it's definitely something that we dip in and out of and help with. This talk is really, like I said at the start, really about trying to get more people interested in contributing to OpenStack security. We have, again and again, analysts, enterprises, everyone telling us, yes, your cloud is wonderful and I really like the way Nova does this and all the rest of it, but it needs to be secure. We need it to have certain security properties before we can take it forward. We see that time and time again. And there are a few companies like IBM, HPE, Rackspace who have thrown some money at making this work. I was fortunate enough to be pretty much dedicated to OpenStack for the last couple of years. But it requires more effort from more contributors in order to really get it to the point where we can make everyone really happy that we can deliver on security. So thank you for coming to my talk. Thank you to Travis, Michael and others that contributed to this. This is a collaborative deck, the intention being that anyone who wants to go out and spread the world about OpenStack security and try and recruit people can. So thank you and I'll take questions if there are any. Okay, well thank you for coming along and please contribute.


My name is Brian Borkowski. I am the CTO of Aerospike. Here is our... Here's our picture of an Aerospike. An Aerospike is part of a rocket. It's an aerodynamic feature. It's a small disk that goes at the top, very top of a rocket. It creates this compression shockwave around the rocket itself and allows it to get past the sound barrier. With an Aerospike disk, you have to tune it exactly for the size, shape, speed of the rocket so that you have enhanced stability. This particular rocket also has an Aerospike motor, which is a little curved device that allows the flame to stick together as well. So that's why we named the company. It's a small piece of technology that makes a big difference. And what I'm going to talk to you about here today is the role of flash and SSD technology within NoSQL and real-time databases. So just a quick summary or an overview of where we see flash playing. In a lot of our installations, we see a big data landscape that looks about like this. There's an analytics tier. That's sort of our old school big data. That's Hadoop. That's various different analytics, query engines, Green Plum, Vertica, a lot of Vertica, even other systems. And you have a set of data scientists out there creating queries and creating new insights that come out of these analytics systems. The app database, however, I mean, these are systems that take, if not minutes, hours, sometimes even days, run individual queries. So you need something on the front end of this system that can actually store those real-time patterns and get involved with the patterns that you need. So I'll go through some examples of that in a minute. Within this new ecosystem, we see a couple different new pieces of technology developing. So besides, you know, the new analytics tiers like your Hadoop and Vertica as well. Within the app server domain, we also see a new set of app servers becoming more interesting. Most importantly, Node and Ingenix are part of the event-oriented systems that you can use, app developers use, to drive more scale within these kinds of big data systems. And at the tier where big data apps are required, you start seeing the new NoSQL players. Now, I've also included Memcache there. I think that's a lot of the father of and is still involved in many of the ecosystem because of the in-memory performance, as well as a lot of the new companies that are at this conference. So what does this mean in terms of a typical deployment for big data in real-time? We've got models that are usually in these different analytics tiers. Because different databases are optimized for different query patterns, we'll see, you know, let's say a volt doing a lot of in-memory where you've got very small things since they can't use flash. You'll see a lot of Hadoop for the large systems, for column-oriented. So you'll end up seeing a lot of these different databases here. But what you have on the front side of your system is usually has to be just one database because that one database, you can make one query to look at your user, look at your interaction, and make a single decision. As these systems become more richer, you will actually see multiple database queries because now an application developer has more power on this front side. Typically, there are writes that go into all of these systems in parallel as well as interactions with a data store that's doing things like pattern matching. So let's think of an example like fraud detection where you're doing a velocity calculation and one of those cases we talk about commonly within fraud detection where if you have a credit card that is used too quickly within multiple geographies, right, you have to write all of those geographies all the time and read them back in real time as well as putting them back for your modelers. What do we typically see at Aerospike in terms of deployments of flash databases for that frontier of the application? There's really been a radical change in flash over the last year. So last year, this time, we still saw a lot of that configuration of machine. We saw a lot of eight core Xeons with about, you know, 24 to 48 gig of RAM and we would see parallel 100 gig, 200 gig machine, SATA drives and the kind of throughputs with Aerospike that you do that and larger cluster sizes. This year there's really been a change in what has happened in the data center. We saw a new generation of Xeons and we saw price of RAM dropping dramatically giving machines that are tend to be about five times denser in terms of that and also flash that is nearly 10 times denser on a machine by machine basis. The increase in throughput is also fairly remarkable with these systems. So now instead of having larger clusters, we see smaller clusters where each node is a bit bigger. Sure. So the question is whether this is... SSD, a solid state disk, often uses SATA as its transport. So you can use either SAS or SATA in an SSD. There are always flash drives and the physical interconnect will either be SATA or SAS. The protocol that the disk is speaking. Let's move on. We can talk about it later. Talk to him about it. He knows the answer. So where we are at Aerospike is we really saw that the in-memory techniques of random access could be applied in a big data context if you use and optimize for flash. What are the kinds of different problems that you will satisfy using flash technology? Advertising optimization, great case because you often want to cover all of the cookies and all of the users that you see, billions of users constantly. Fraud detection, we've found fraud detection is a great use case. As I say, fraud is everywhere. It's true in-game. You want to look in-game at different user behaviors and histories. You want to look at it in financial transactions. You want to look at it in advertising. In advertising you can present and say, hey, I'm a website and someone clicked on me when no one in fact clicked there. Everywhere where money is transferred, there's a little bit of fraud and there's people looking at transaction patterns and use patterns to see them. We also see financial positions and T-zero, which is to say one to four second financial calculation cases. In specific, what application patterns are in use? Well, first of all, we're starting to see map reduce in real time, map reduce in milliseconds as opposed to using a Hadoop system, which is more of a batch-oriented transaction system, and cases where you want to suddenly track not just every cookie, every IP address, every search term that's been offered recently. Great cases of these are in advertising, where you really want to not just see what has happened recently, but try to predict for giving users where they're going to be in the future. How do we know that Flash is already large? I didn't get a chance to update this since last week. Anyone at the Flash memory summit last week? Yeah. You saw the Facebook keynote, right? We all look down to Flash. It wasn't full of Flash. It wasn't stuff. They were in Flash. Flash everywhere? Everywhere. So Facebook? Yes. I have to leave in a couple of minutes. When you say Flash, we talk about Flash everywhere. I saw Flash in the beginning. I saw Flash in memory. Server memory, storage cache, server cache, arrays, front-ending, HDD. When you're talking about the kind of Flash that you're discussing here, what are you specifically talking about? In-memory, in-night-out type stuff? We're talking about a generalized system to optimize databases for all of those. So putting Flash in your servers, whether that's SATA, SAS, SSD, whether that's PCIe-based, like a Fusion IO card or some of the Micron cards, and the new Flash PCIe interconnects, all of that stuff. What is a generalized way to deal with Flash in databases? Not in advance, if I'm serious. Not yet. Not as heavy as you see.  Yes, I did. So what we found at the Flash memory summit there's a very interesting keynote. I think it was Wednesday morning, it was Thursday, where this number is old. So what we know from Fusion IO is that Facebook and Apple spent about $200 million just in Flash hardware. Now they run their own software on top of that. They're using their own stuff. They announced that their architecture, they were actually making $1.2 billion worth of Flash buying this year. So why are they doing this? For all the reasons I was just saying, because of random access and application design, to build a rich application, you really want to be using Flash. Those guys figured that out. They had to build their own software in order to, and their own distribution mechanisms, in order to manage that much Flash. So, you know, some of the common preconceptions, misconceptions about SSDs is, first of all, I can just take SSDs in Flash and put it in my database, it goes faster. So how many people are using SSDs in Flash in their databases today? And it goes faster, right? Yeah, it does. Two or three times faster? Maybe a little more. But SSDs are so much faster than that. Instead of having 200, 300 IOPS per disk, you've now got 20,000, 50,000, 100,000 IOPS. Even more than that. So why is it only going two or three times faster? Well, that was one of the simply different than any device we have seen so far today in storage history. The way that they're different is what I call asymmetric use pattern. So reads are very fast, and you can do a lot of them, very small reads as well, but writes are still very slow, and you have to use log-based file system style approaches in order to be able to write efficiently. So where RAM, it's fast to read, fast to write, it doesn't matter what the size is. And with disk, reads and writes are both slow, and you always want to do streaming. Rids and writes are very different with SSDs, which is why you have to specifically optimize your database for Flash. The other point about writes with SSDs is if you keep your writes to large blocks, you get less wary amplification created by the device itself. The device will have to go through and pick out all those little pieces and bring them together and push them back and forth, which is similar in software to having garbage collection and defragmentation. So if you do this at the database level and do it knowing the application's use pattern, you can do it more efficiently than the device can. Now the good news at the device layer is the device knows its chip layout. Sometimes it's better to let the device do that, and I'll talk a little bit later about systems for doing that. Another important factor within Flash and SSDs is the parallelization that you have to get out of the application layer. Now in sort of the old-school database way, what does this mean? It means async I.O., and it means a lot of reactor models that allow you to process I.Os very, very efficiently. And as opposed to using a true-in-memory, you know, just blast your thread through no locks kind of strategy. So you have to code a little differently in order to make sure you get enough parallelism. A lot of these devices are happiest at 256 to 512 simultaneous I.Os in flight. So you have to write your code with threads and I.O. patterns that are capable of this. If you can, you'll end up seeing factors of performance that's much higher than simply the three to four times X faster. Another preconception about SSDs aren't they expensive? Again, this is an old slide. The last time I looked up at Dell's site, I found prices for RAM at about $30 a gigabyte. It's come down a little since then. And what we were seeing back in January 12 was about $10 a gigabyte for flash storage, which has come down now to about $1 per gigabyte. Well, with a micron M500, we're now down at 50 cents. And the rumors that I've heard about what Apple is buying internal to their infrastructure is with their pricing power, they've also been able to get down to about 50 cents per gigabyte. So that's really starting to encroach on where you would even start thinking about rotational drives. And I'll show you some results about very high performance flash, the highest performance flash that I currently know about at $8 a gig. So compared to in memory, it's really a lot cheaper compared to the same characteristics. Compared to rotational disks, there are also choices that are within the same price band. Aren't SSDs slower than in memory? Well, yeah, they are. I'll show you later some of the drives we've tested and where we see different drives fitting in in the system. But it is so much faster than rotational disks. So you can put... One of the nice configurations these days is SATA drives and you can get 20 of them in a chassis. Once you get 20 of these things all lined up, assuming you're doing your IOs correctly and you've got your RAID card configured correctly, you can get some pretty awesome performance. Performance to a point where the bottleneck starts shifting from flash and from storage, where it's been traditionally for nearly 30 years, the bottleneck starts shifting to the network, which is a very interesting result. So sort of taking some of those and pointing out what you can now do with SSDs is rotational disks, I just sort of took something from 3PAR's price list. 5K TPS, that's considered a pretty hot system. With SSD, you get 4K of storage and let's see, the TPS, 500K database TPS, right? Whereas RAM, it is faster, but look at the difference in price. Instead of 16K for 4 terabytes, you get 20K for half a terabyte. Another example, this is the kind of thing that I show customers when I'm trying to express to them the difference between in-memory with flash and in-memory with RAM, is a project budget. So I don't know about you guys, but I suddenly have to talk my boss into a new project and I often have trouble starting a project at $4 million in the initial hardware bot. If we had a high availability 10 terabyte system, this is what I got out of Dell when I was trying to build a reasonable 10 terabyte high availability system. Whereas with SSDs, this is with the Micron P320H, which is actually a very expensive drive, it's an $8 a gigabyte. I came out to only $200,000. And you know, I could probably talk a boss into $200,000. That's within my power persuasion. Now some customers I say, well, you know, I don't really need 10 terabytes. This is the kind of price savings. I was on a whiteboard at some guys in Asia and they said, well, we're just starting out with a 200 gig system. What is a 200 gig in Redis with RAM versus a flash deployment? Well, 10 servers, the particular kind of servers they had versus three servers and still hitting the same performance goals that they wanted to hit. So I'm going to do $20,000 right off the bat in our initial buy. So let's talk for a second about what it means for a database to optimize for SSDs. So I was talking a little bit before about the kinds of IO patterns that you need to use. The primary one is using those large write patterns and also being able to keep indexes and RAM. Indexes and RAM has two different positive benefits. One is that's usually the choke point of parallelism in the database is access to the index. That's where you've got multiple applications coming in. They're all trying to find exactly what they're looking for. And RAM works better for parallelism. There's a lot of great lock free algorithms within RAM. And so you can get the parallelism that you need at the device level if you keep your indexes and RAM. The other important point about keeping your indexes and RAM is there's a lot of small writes that occur when you're writing to your index. And like I was saying, writes to the index, you don't want to do those small writes. So you do your large writes using log-based file systems and if you keep those indexes in RAM, you're going to get the high performance that the device can actually give you. Another problem that you have to deal with is background defragmentation. So when you have a log-based file system, that means you have to defragment. Instead of relying on your file system to do this, we found that that's a real impediment in Linux especially to high performance. So what we've done is we use a native device layout similar to what does Oracle do? Oracle uses native device layouts also. This is the common way to build databases. But then you have to constantly defragment. If you keep your defragmentation up until a later period and run for a while without it, then you'll slow down and you'll have unpredictable performance. So having that consistent, constant defragmentation as part of your database is key. We've also found that RAID hardware creates a lot of unpredictability because most of them are not yet built for Flash. They're trying to make these big, long stripe sizes to make sequential access efficient. Well, guess what? That's not how SSDs work. So instead, what you have to do is route your IOs to individual devices knowing the device structure and get the RAID card out of the way. Those things are a huge bottleneck right now. And we always have to be very careful, especially with the LSI configuration tools to make sure we can really get enough throughput through those devices. The application, from their perspective, we're the application, the database, manages those devices directly. We're going to get fast enough storage. So another interesting optimization there's a lot of talk in SSD about trim support. We have found in general that trim tends to slow down and create a lot of pauses within devices. There's no good reason for that. That's simply an implementation issue that device manufacturers will eventually overcome. So we typically don't use that. We have a configuration that allows it. Maybe someday we'll find a device that does use it. There are next generation interfaces available for Flash. And at Aerospike, we were the first database to use the open NVM KV interface. What the KV interface does is it supports using a primary key index, a key value hash table style interface, directly from the Flash card. So instead of supporting block level interfaces where you have to do all that refragmentation and extra space, you can use the KV interface within the device itself. It saves you on RAM, and it saves you on this refragmentation, allowing you to use less storage and pay less to the Flash cards. So that's a very interesting thing. Fusion I.O. and Dustco developed it originally. Fusion I.O. opened source to that interface. We expect to see more device developers pull that interface into their system. We're quite excited about it. Our DMA is another interesting optimization. It allows you to deal with some network issues. So one thing we've really learned about Flash is it's simply a new technology. Optimizing for Flash means knowing which Flash devices really work at any given time. A guy comes out with a... a micron comes out with a new device. Is it really fast? Does it have predictable performance? Of course the datasheet says it does, but really it doesn't. We thought that we had to create our own certification tool for SSDs and promote that to the Flash makers. Because we would have deployment situations where we would measure out that the Flash subsystem and storage subsystem was unpredictable and slow, and then our customers would ask us to go prove it and get the Flash guys in line and do a firmware fix. So we found that the only way we could talk them into the fact that it was their problem would be to give them a bunch of source code that they could analyze and then actually see the problem themselves. So the workload that we tend to configure this for this tool is one again where there's large block rights and small reads, and you measure the latency of those reads. Because latency of reads is really what slows down a lot of the database access. We tend to see a lot of use cases at the 1.5 kilobytes per object basis. That would be very common within behavioral cases. I expect that will go up a lot as the cost of Flash drops. So let me show you the kind of results we can get out of these tools. This is a particular device. It's a PCIe device called the Micron P320H. And it's not cheap. On the other hand, it's still less expensive than Fusion IO. And the way to read this table across a 6-hour period is more than a millisecond. At the level of 150,000 read IOPS and with 200 megabytes of rights simultaneously. Compare that to some of the other players and devices that are available right now. The Intel 3S 3700 is a great drive if you can get them. They're priced a bit lower. They're $3 a gigabyte. You see a real difference in the latency that they give. And these are much lower throughput rates. Now you can stack a lot more of these in a chassis. So we all end up going through with our customer what kind of chassis they use, how much floor space they're dealing with, whether PCIe is a better solution for them than SATA, etc., etc., etc. There's a lot of ins and outs to it. And the Intel can be a very competitive device compared to the Micron, depending on the exact latency characteristics you want. And the Fusion IO, IO Drive 2 that's a 700 gig MLC part priced at $8 a gigabyte doesn't really seem to fare too well in the same test scenario. One thing that we see is people trying to use true consumer grade devices within these chassis. And you know there's firmware bugs in some of them. Especially when you're dealing with higher throughput. And I don't really mean to put OCC on the spot but this was an email that we got from a customer who didn't take our recommendations to heart and was seeing what was his phrase. We're currently in the process of replacing. I think he actually said something about dropping like flies. So this is one of the things that gives SSDs a bad wrap within the big data and high performance community because someone has an experience like this and suddenly SSDs and flash are unreliable. Well, more importantly, the vertex 4 and possibly the batch that he got was like that. OCC puts a lot of different flash chips because they're buying from whoever they can find at the time. Compared to people like Intel who have their own chip lines and have different levels of quality control at a higher price. So these are things you'd have to watch out for. To put some other perspective on it, these are some of the devices we saw probably more than a year ago starting nearly three years ago now with the original Intel X25. We saw a fair number of 64 millisecond requests. That's a very different world from where we are now to where we came from. That Samsung device was really a very solid device at the time. Intel had a pretty good device with its 320. We recommended a lot of those. And you can just get an idea for some of the trade-offs we try to help our customers through. So let's talk a little bit about how Aerospike actually approaches the HA problem. Because once you're dealing with high uptime high availability as well as SSDs one of the great benefits is being able to have a distributed system. I'm going to skip over a little bit here. So our architecture allows us to uniquely do the kind of SSD optimizations I was just talking about. A particular independent company called Thumbtack did a set of benchmarks and you can read through the details of what they did online. They wanted their system integrator in New York that deploys a lot of these different NoSQL solutions. They wanted to see what different databases work better in which scenarios. We were pretty happy with what they found with us for their SSD based systems as well as the latency that they found. One of the benefits of our system and our design that I'll mention in a minute is the ability to continue operating even in the face of failure. So this is a graph from their system that shows what happens when one node drops out and when a node arrives again. So the system that we use in order to get those benefits is one where it's a shared nothing architecture. There's not another server. There's not a name node map. There's not a Mongo S in the middle that directs traffic.  The way that we do this is that when new nodes are detected we run the distributed consensus algorithms in order to figure out which nodes should be in the cluster and are added to the cluster. This allows us to create a cluster master during that period that allows that one machine to determine what data is moving where and what node has what responsibility. So from an application perspective it looks all like the same on the outside. But internally there's a master and there are specific responsibilities within the system. That allows us to have a very high level of reliability in our data rights. So what happens in our system is for any given row in the database there is a master and there is a replica. Now that's all randomly distributed. Every node is a certain percentage master and a certain percentage replica. You don't know from outside the cluster what is master and what is replica and if there's a failure within the system, if there's a hardware failure or if you add more capacity then you will these responsibilities may shift depending on whether you want to automatically advance or pause and wait for the system to come back up. Having masters and replicas for individual data rights is the only way to get a high level of consistency within your system. Yes, as many as you want. Of course that slows down your rights. So what we've also found is that having a client which knows about the data layout within the cluster at any given moment is also of critical importance. So we've seen more of the NoSQL systems move to this where for example Hector within Cassandra is capable of tracking the cluster and we've had that from the beginning. It's really very important to being able to tolerate failure. So within a client library within the process itself there is tracking of where data is and who the master is on a moment basis. You don't have to reconfigure you don't have to specify a list of IP addresses that you might have to do with a manual sharding system. You simply have the client running and it is automatically learning cluster reconfiguration in process. What we've also found is that being able to replicate your data between multiple data centers is critical for high availability. If you're only in one data center in some sense you're not serious about availability right because data centers go offline. It happens. So what we have come up with at Aerospike is both the capability of doing a single master star topology and also using multi master rings. This particular customer who deployed has one ring across three data centers, another ring across four and a very small ring with just two data centers. So every time they do a write the replica that write copies itself throughout the data center set. With different data pools that's always very important. Monitoring is crucial so we ship for usually for people who are getting started with their system they want to see that their cluster is performing well. What we recommend for our higher scale production customers is that they integrate with their own monitoring system. Because you want to do a full stack analysis usually as you're having performance problems is my problem in my app, is my problem in the data layer is it in the network. However having a simple tool that allows you to see what's going on is often a first step for diagnosis. So one thing we're doing here at NoSQL now is our new Aerospike 3 capability that allows us to do queries and gives extensibility to the database. So where we've started at Aerospike in this release is the ability to do queries on index values or columns just like in a standard relational system. Columns of course can be defined on the fly there's no need to pre-create your columns. However if you want an index column you do have to tell the cluster that you want an index on it. We support background recomputation as well as allowing the index to be created on the fly synchronously and with consistency. So the scheme that we have for doing this is that the index is on the same machine as the data itself. Now there's pluses and minuses to this there's another way of doing indexes that we will also get to probably in the next 6 months. The benefit of this kind of scheme is the random distribution that we get across the cluster is maintained also within the index. When you do an insert you're touching the machines that hold that data and you're updating the indexes that are in RAM on that same machine. Remember all the benefits I was saying about having indexes in RAM. The minus of this kind of architecture is that every query has to touch all machines. Now as long as you are touching say hundreds of rows and your cluster is tens or hundreds in size it is actually efficient to do that and removes hotspots from the system. So in a system where you are focusing the data that is similar together it's very easy to generate hotspots. A classic case of that is a time oriented index where all the data from one period of time all the inserts end up hitting one server. You tend to see that hotspot roll amongst your cluster on a second by second basis. You're not actually getting parallelism and you're not getting the scale out capabilities. And then similarly if you are only querying say the last 2 or 3 seconds you can see those queries hitting one server after another you're not actually getting scale out. So the benefit of this system is the queries go to all machines okay that's not necessarily the best. On the other hand each one is doing less work and you've got no hotspots. So we think this is a good design for this system. User defined functions a classic database technique now in Aerospike. So if you take for example the work that we've seen out of VoltDB and a lot of stone breakers ideas this level of database optimization we're very excited about because so many applications have their own very specific needs in terms of the data sources that they have the data structures they have and the optimal data patterns. By writing a small user defined function you can often get a much higher level of optimization in your database. It's like having your own query language being able to say hey I'm looking I've got this document and I'm looking for all of the H1s within my document. You can write a user defined function to pull those out for you instead of having to actually build those into the database. So we look forward to using this kind of extensibility to cover things like all of the Redis operations and some of the various operations. Once you have indexes and you have user defined functions you have the benefit of doing map reduce within the the new Aerospike 3 system one of the benefits is that we use the secondary index to touch less data. So imagine that you only want to map reduce over the last minute or say even seconds worth of your data. Imagine a query that goes into a Hadoop system where you often need to touch all of the data in these big table scans by only touching the few things that have changed or updated or let's say in advertising or let's say file detection you only want to touch things for a particular class they address. You can do a range scan hit those things reduce over that particular portion of it instead of having to touch all the data. Touching data really is one of the primary bottlenecks right now within map reduce. So you know some of the benefits of Aerospike, the tunable consistency we're very excited about our flash support the fact that we've been used by so many great customers over the last few years with very high availability. Aerospike is available in Community Edition today including the Aerospike 3 functionality in APIs as well as an Enterprise Edition which has a full set of functionality. The Community Edition is only limited by the cluster size and amount of data other than that all of the APIs are available. So thanks very much for listening and any questions? Great so as I mentioned at the top of the talk you know I know that a lot of people are here from other NoSQL companies and this is exciting to you. We are always hiring and we have a great engineering team both in here and in Bangalore so feel free to drop by and talk with me. Thank you very much.