Hello, everyone. So today I'm excited to share with you an intriguing success story, the migration of the web arrangement platform to Envoy. So this is a story of how we had Spotify revamped or web proxy layer to use Envoy, obviously. We'll dive into some technical details, share the lessons that we learned, and a bigger picture of how this looked for us. But I want to mention that we are initiating this endeavor with a purpose to present to our peers the practical application of Envoy. Our intention is to share some experiences, the lessons that we learned, as I said, and challenges that we've encountered while adopting Envoy. And we hope that by doing so, we can initiate some fruitful discussions about the future and potential improvements and future that we can all make together. So yeah, today we have a concise yet impactful agenda. We're going to start by presenting the Spotify parameter. We're going to show an overview of our operational scope. Then we're going to zoom in into the web management platform. And we want to tell you about how this is a key component that enriches our ecosystem. Then we're going to explore the motivation behind our migration and also share some technical insights into the actual migration work. To conclude, we're going to wrap up and look at the migration process and look ahead at our vision and evolving infrastructure. So with no further ado, let's embark on this journey together exploring technological evolution of Spotify. But yeah, first and foremost, allow me to introduce ourselves. My name is Sabrina Zotti. I am a software engineer at Spotify. I'm based in Milan, Italy. And today I have the privilege to present alongside Oliver Sol, staff engineer at Spotify, based in Stockholm, Sweden. Oliver and I both work within the computer networking product area at Spotify. And specifically, we are part of the ATC team with a primary focus on the design and management of Spotify's infrastructure perimeter. I will now leave the spotlight to Oliver to introduce our work. Thanks, Sabrina. So we'll talk today about our web enrichment platform. But before we do that, let's zoom out and talk a little bit about Spotify's overall perimeter, which can provide some context for our presentation. Spotify runs back in services in Google's cloud platform.  Just behind that load balancer, we have our Envoy-based edge proxy layer in all five of our GCP regions. We presented our edge proxy migration story here at EnvoyCon in 2019, and we'd be happy to take questions about that later or any time during the day. Just a side note, many of you may be curious. As an audio service, of course, our clients need the audio files themselves. Those are streamed from a CDN, and we're not going to talk about those today. Edge proxy, however, takes between 6 and 10 million RPS every day. Here's a recent graph of a week's traffic by GCP region. However, the vast majority of that traffic is our service traffic, like when your Spotify client needs to load a new playlist or you add a track to your liked songs. Only about 1% of our traffic is web traffic, and you can imagine that that traffic has much different patterns and more specific needs in the proxy layer than our regular service traffic. We route the web traffic through edge proxy and then to our web enrichment platform, which has many web-specific capabilities that our web service owners rely on. That's what we'll dive into today, and Sabrina's going to tell you a little more about that. Thank you, Oliver. So before zooming in to the web enrichment platform, I'd like to start by mentioning an interesting fact. So three years ago, when the web enrichment platform was created, back then it was Team Nibbler. The primary goal of this project is to dismantle this monolithic system supporting.botspotify.com into separate web services developed by a different team, obviously. To make a long story short, this project resulted in the development of the platform that we are not going to present that allows the segregation of client-facing applications from common business logic concerns. But not only that, as well as also breaking up the monolith in multiple leaner simpler applications. This aspect of the platform caught a lot of attention and became quite popular within Spotify. And fast forward to today, the web enrichment platform is now standard for the web traffic. So about the web enrichment platform. So you might wonder what this is exactly. It's an infrastructure designed to handle the routing and the enrichment of the web requests. From an architectural perspective, it consists of three layers. So a configuration layer, a routing layer, obviously, and the enrichment layer. Take away until now is that all requests directed to Spotify will pass through the web enrichment platform. But let's backtrack for a moment. So Oliver mentioned the concept of edge proxy, right? So if you tuned in in our previous presentation in 2019, edge proxy serves as the initial point of entry for all the requests that hit Spotify. And that still stands. However, in the case of web services, edge proxy directs the request to utilize the web enrichment platform. So within the routing layer, the request can be configured to either directly access the web service, meaning that it will only use the routing component of the platform, or it can be enhanced with enrichments. More on that on the next slide. But currently, the routing layer is based on Envoy and directs the request to the appropriate web service. The enrichment layer is a tool, a feature, that allows developers to incorporate business logic into their requests, such as cookie handling, market analyses, or even compliance to information standards like CSRF or CSP. But what does the enrichment layer actually look like in practice? Once a request reaches the web enrichment platform during the routing phase, the platform will verify whether the request should be enhanced with some business logic or not. If yes, the router will initiate a sub-request to Concierge. Concierge is another component of the platform and serves as the enrichment orchestrator. So to ensure a flexible infrastructure, the enrichment layer operates based on a plugin strategy, so Concierge. Any member of our organization can develop an Enricher, which is essentially a library that performs a specific function. This approach allows developers to extract business logic that maybe it's not really related to their application scope. And this helps their work more efficient, their application leaner, and while also providing the rest of the organization with reusable functionalities. We have an actual directory of Enrichers, kind of like a marketplace where people can go and browse and search for an Enricher that fits the requirements. If such an Enricher doesn't exist, then individuals can develop and enable it with the assistance of Concierge in the Web Enrichment platform. This sounds something like a marketplace of Enrichers, I can say. I think it sounds pretty convenient, right? So if you think about it, if we developers at Spotify are already using the platform for routing, adding this kind of logic here is not a bad deal. Currently, we support four types of enrichments within our system. The first one is content by Enricher. In this case, the response will include a custom status code and optionally a body content. So in this case, the request will not pass through any web service. The second one is content by redirect. With this option, the response will be a redirect instead of hitting the web service. Then we have content by proxy. In this, this feature applies the instructions at the routing layer and directs the request to the appropriate web service. But the web services will process the request and will return the response further on to the client. And then we have the final one, client response. If Concierge responds with client response, then these instructions will be implemented. And the instructions will only be visible to the client and not to the web service. One important thing to note is that this process involves a mutual exchange of information. For instance, when using a content by redirect, the request will pass through the web enrichment platform multiple times. Spoiler, if this sounds familiar to you, you might be onto something. Oliver will now tell you more about the behind the scenes of the configuration layer. Thanks, Sabrina. OK, let's talk about the configuration layer. And this is where we have to let you in on a little secret. The web enrichment platform was originally built around Nginx. So this configuration layer may not smell very much like an Envoy control plane, but bear with us. So how does it work? Logically, the configuration layer has two responsibilities, transforming our configuration API into Nginx configuration and performing service discovery of the upstream web services. But architecturally, we need to manage a fleet of proxies that autoscale. And since we don't have a control plane, we need that configuration logic to be running alongside each proxy replica. The configuration layer then is co-located next to each proxy replica as another process in the Nginx container and runs service discovery every 30 seconds, dynamically updating the Nginx configuration. On the other hand, when there's an update to the static configuration, like when a new web service is added, we make a new deployment of the router layer and the configuration is bundled along with it. As we move to Envoy, we use the same approach, supported by Envoy's dynamic configuration mechanism. The configuration layer simply produces Envoy YAML rather than Nginx configuration. And during the migration, the configuration layer played a key role in abstracting away the proxy implementation from our users. Ideally, they would have no idea that we're changing out the engine, so to speak. OK, so we just covered the overall structure of the Web Envision Platform. And now I'd like to cover some of our rationale for rebasing the platform to sit on top of Envoy. There are many reasons we chose to migrate to Envoy, but the primary reason was organizational. The team who originally created the Web Enrichment Platform, as Sabrina mentioned, they were called Nibbler, had seen that Envoy was gaining a foothold at Spotify and that Edge Proxy had benefited greatly from using Envoy. They also saw the potential of merging the Web Enrichment Platform directly into Edge Proxy, which of course would never happen unless it moved to Envoy. Now, from the technical perspective, Envoy's feature set was roughly aligned with the features we needed to support in the Web Enrichment Platform. The team had done some discovery into Envoy and hadn't found any major feature blockers. And additionally, they'd noted that Envoy had a vibrant open-source community, all of you, which made the team feel confident that we could contribute to Envoy to fill any gaps that we might find. The Envoy feature set that was most important to the Web Enrichment Platform was the set of features supporting extensibility, XSALT-Z, XPROC, Support for Lua, Wasm, the native plugin model, and as we heard earlier, Go as well. However, it's important to note that although Envoy supports high performance, performance wasn't terribly important to us for this application. Extensibility was really the key feature. Anyway, the team decided to go ahead with their decision to use Envoy and started engineering work early last year. Although we don't currently have concrete plans to merge the Web Enrichment Platform with Envoy, we have joined the teams. Nibbler merged with ATC earlier this year. And now, Sabrina's gonna tell you a little bit more about how the engineering work actually went. Thank you, Oliver. Let's take now a look at how we re-engineered Web on Envoy. Luckily for us, the transition was quite intuitive and something that was very helpful in the process was obviously Envoy flexibility. Many of the changes were primarily concerning the configuration and the routing layer with varying degrees of difficulty. We were lucky to have a lot of one-on-one matching between the features, but I want to mention a few of the things that we need to change. Starting from the upstream handling, first we need to slightly change just how the service discovery works. Envoy played a crucial role in this process, as our services need to find each other through engine room, the configuration layer. Then we had the health checks. For now, we rely on the active health checks through the outlier detection to ensure that the services are healthy. Another thing was timeouts. We can now configure timeouts request per upstream, and this ensures responsiveness. Regex says, so Regex says this feature is very important for us, because we use them for request processing, so we had to work a bit around limitations with the Regex engine in order to keep Regex patterns efficient. But thanks to Envoy being open source, we were able to improve this feature to fit all requirements, and this resulted in a PR that we opened, allowing us and potentially others to expand the capabilities. These changes addressed an issue and countered with the Regex rewrite as part of the redirect when the mesh path does not match the entire path. The problem was that only the mesh section of the original path, for instance, the prefix, was substituted with the Regex rewrite, whereas the expected behavior is to replace the entirety of the path. Another thing is that now customers have the ability to determine the appropriate course of action when errors are returned by a vertical. This is a feature that we offered in the platform. This can involve intercepting the error and invoking another vertical that displays the error page. As a result, all the verticals within a website can use the same error page without needing to independently implement it. It appears that such a functionality is still not ready for production in Envoy, so we had to depend on our custom law filter to fulfill this requirement further. Moving on, speaking of integration tasks, we're incorporating Envoy to validate the configuration and prevent unsupported or complex Regex patterns. We also managed to simplify our system by using built-in Envoy features, reducing the amount of custom code we had to maintain. Features like service auth filter, dynamic forward proxy, or external authorization filter played a significant role in reducing our custom Lua plugins. Additionally, we benefited from some new features as well. For instance, we may greatly use of the Envoy admin endpoint to support our operational tasks. On another note, as Oliver mentioned before, throughout this process, we were so pleasantly surprised by the supportive community and assistance that we received. While this project is still in progress, we are very pleased with the progress that we made and so enthusiastic about our upcoming plans. Sabrina just covered many of the successes we had as we implemented the Envoy layer, but of course it wasn't all perfect. There were a few places Envoy wasn't quite so aligned with the features we already offered in the platform. Let's talk about a couple of specific issues we had. Rate limiting is the first example. The web and version platform supports rate limiting and quite a few of our users had enabled the feature. But the core issue here is that rate limiting in Nginx and rate limiting in Envoy differ in some key ways. Specifically, Nginx supports using the client IP address as the rate limiting key, whereas Envoy does not unless we use the global rate limiting filter and the separate distributed rate limiting infrastructure. We didn't wanna build out that separate rate limiting infrastructure, so we decided to look at the problem a bit more holistically. We looked at our rate limiting users and tried to understand their various use cases. And it turned out that most of them use rate limiting to protect their services from unwanted traffic spikes. But supporting that use case has a few problems. Users don't have a good way to determine a specific rate over which they wanted to limit their traffic, especially when their traffic varies over the course of a day. And from an operational perspective, we would have to do capacity planning of a rate limiting infrastructure, which is especially hard when we don't really know what traffic to expect. So we'd been down this path before with Edge Proxy and we didn't feel like blindly using the same approach. So the team decided to pause and look at more long-term solutions. And unfortunately, I must say we haven't settled on a solution quite yet, but rather we're investing in our broader traffic protection strategy. The other example I wanna talk about is file serving. And I should start by saying that this was not really an Nginx versus Envoy issue. We had a feature in our web and Russian platform that allowed users to define raw Nginx config. Yeah, yes, you heard that right. It's the definition of a leaky abstraction. Whatever good reasons the team had for including this feature in the first place, clearly we couldn't port that forward, port that functionality forward to be supported under Envoy. So again, we took a look at what our users were doing with this mechanism and luckily we found that everyone was using it simply to serve small files such as a robots.txt file. I think we really dodged a bullet here. This could have been abused much more widely with many more use cases. But given our findings, we settled on building a proper file serving feature into the API and currently the implementation simply just proxies a request back to a GCS bucket. But in talking to our users about this feature, we also heard that they want the ability to use a downstream cache. And so we're considering if and how we might support HTTP caching in the platform. It's not commonly used for backend services but it would be useful in the web space. Now Sabrina's gonna tell us a bit about the actual migration. Thank you. So with this slide, I just want to provide a quick snapshot of how the migration process looked. The good news is that everything went smooth and we currently are somewhere near 90% completed in terms of the migration. Until now, we have made the switch with zero downtime and pretty much zero issues almost. However, we did take some precautions and systematically moved our traffic following this procedure. So we set up some migration cohorts dividing the user base in order to facilitate a control transition. Then we contacted the owners of the web services and maintained a clear communication and collaboration with them. When it was time to schedule the migration window meant being an optimal timeframe for minimal disruption. One thing that was very helpful for us was that we were able to temporarily operate traffic splitting with the help of envoy proxy. This helped us oversee potential issues in time and also avoid disruptions. Monitoring was key. After all, we prepared a set of dashboard with overview of traffic and what was happening with the requests and the enrichers as well as monitoring resources. When we assessed that everything was fine then we did a full switch on traffic. So 100% of traffic going to the web management platform with envoy. And then some more monitoring to ensure peak performance as well as collaborating with the service owners for the long term. Next, Oliver will share a snippet of our future plans. So now that we're almost done with the migration where do we go from here? Well, we're not sure exactly but a main focus area in the next planning cycle is to evolve the web enrichment platform. The first couple items that we have here in this list are almost certain, a control plane. Let's face it, the current configuration layer is a bit of a hot mess and we'd love to reuse our expertise with the edge proxy control plane and apply that to the web enrichment platform. Also, the tap filter. We already have a few use cases for the tap filter or at least use cases where we think we might start with the tap filter and iterate towards custom filters if and when we think that makes sense. One specific effort we're actually currently working on is creating a granular data set of traffic going through edge proxy. We've already run a prototype that uses the tap filter and some sampling and I think it's pretty likely that we'll use the same approach specifically targeting our web traffic in the web enrichment platform. And the other items here are a bit more exploratory. As Sabrina hinted at before, careful viewers may have noticed that the API that we offer in Concierge and the enrichment layer looks very similar to Envoy's XPROC API. So perhaps we rebuild Concierge as an Envoy filter and call external admirers with XPROC. Another idea is to have some coordination between the configuration APIs of edge proxy and the web enrichment platform. Maybe we could automatically provision endpoints in edge proxy if we know their web enrichment platform users. And lastly, the mythical idea of actually merging the web enrichment platform into edge proxy itself. There are so many ways we could actually do this and I'm sure there'll be plenty of ideas by the team over the coming planning cycles. So of course, we'll have to wait and see what the future will bring, but I can speak for the team and say that the future is bright with Envoy. So lastly, we'd like to say a heartfelt thank you to the current and past members of both teams, Nibbler and ATC. Thank you to the conference organizers for inviting us to speak and the countless others who helped this migration to success. I extend the thank you as well. And with that, we'd love to take any questions that you might have or give you a short break otherwise. And also please use the QR code here to submit any feedback that you might have for us. Thank you. Thanks for the presentation. I noticed one of the things you listed as a challenge during the migration was something to do with redirect policy for customer response. Also dealing with some interesting coax of internal redirects in a migration at the moment, so I'm curious to hear more about that if you can elaborate. That's something we might have to look up. I don't remember the details. That was something that I believe one of our past team members submitted an upstream patch in Envoy where I think there was, I think there was something where you could redirect but it didn't maintain the host header. It was something like that, do you recall? Yeah, I'm not sure. I'll tell you what, we'll chat later and look it up and get you a better answer. Curious about your experience with Envoy filters? I mean, I've heard typically that it's preferable to use C++ but what about some of the other languages? What was your experience and the pros and cons? I can take a stab at this one. This is an interesting question. I think specifically for the web enrichment platform, we already kind of had a choice made for us because we had so much custom code in Lua already to support Nginx. We basically just moved everything over and we didn't try and redesign things at that point. From a more strategic standpoint, taking into consideration both the web enrichment platform and Edge Proxy that our team also maintains, there are differing opinions in the team, I think. But maybe we can say that for Edge Proxy where we're really running all of Spotify's traffic, we would prefer to have something which is very maintainable and very high performance. But perhaps there are cases where we might want to prototype something in perhaps a language that team members are more practiced with. We don't have that many people who are actually good at C++. So I think we're interested in looking at some of the other interfaces. I'm also personally interested in Wasim as well because I think some of the logic that we might want to see in these various proxy layers, we might wanna run them outside of Envoy as well. And so then having a portable format would be useful. Thank you. I have two questions for you. I guess first, are you using Envoy for API also? Is that your team, a different team? I'm just curious what that looks like, web versus API? So it used to be multiple teams but those teams have joined. So now it's all one happy family. Gotcha. Okay, but are you using Envoy for your APIs as well? Yes, edge proxy that we talked about before. That's where all the APIs go. Got it. And then you mentioned control plane. Have you started your journey into that yet or it's just like an idea right now? I'm just curious, like have you started looking at, I mean, there's lots of control planes for Envoy out there, right? You know, I'm curious if you've found anything that looks interesting for web specifically or not. Yeah, that's interesting. Do you want to answer or fill in? Yes, well, as Oliver mentioned, many of the decisions that we took are coming from the previous architecture of the WebRange platform, but control planes definitely something that we are looking into and plan to invest in this of Copic cycle. And I can say as well, I mean, part of your question is interesting. Anything specifically about web, we haven't really looked at anything specific about web, but edge proxy does have a custom control plane written in Java. And I believe we're also the maintainers of the Java control plane in the Envoy community. So that's pretty much the no brainer direction will go. How we actually support specific web features, I think that'll just be an engineering exercise on the team. Got it. Okay, thanks. Sure. Great talk, thank you. A bunch of questions come to mind, but I'll just narrow down to two quick ones. One of them is for your, in general, this seems like a pretty high throughput application for Envoy. I'm wondering if you've looked at your usage of features related to the performance and scalability of your proxy, in particular, your reliance on regexes for doing redirects. We've been shy about scaling, about having lots of regex in your URL, I'm not thinking they'll be a drag on performance. And if you've done flame graphs, et cetera, to look at that. I'll just also shoot out my other question, which is that you talked about writing some custom filters and I'm wondering if you're following along what Alyssa referenced about XPROC in a way to offload some of that to maybe other services that you might have running to service those content modifications. Thanks. Yes, okay, so those questions. Around performance, as I mentioned before, for this application performance, performance isn't very high on our list, but I imagine we will get to performance at some point, but in my mind, it very much depends on how we consider merging the Web Enversion Platform with Edge Proxy. In Edge Proxy, that is very performance sensitive and we're doing a lot of engineering work to increase performance of Web Proxy itself. Around performance for regexes, yes, that could conceivably be a problem in the Web Enversion Platform, but we're not so worried about it. We can always just throw more hardware at it for the time being, so I think. And around XPROC, yes, we're definitely looking into XPROC and I think XPROC can really help us evolve the enrichment part of this platform. Thank you for the presentation. In my company, UF, I was actually going through a similar migration from other reverse proxy to Envoy. Actually, the first question, were you able to achieve 10 miles in the retries to upstream within Envoy or just via Envoy filters? That is a question I don't think I can answer. Do you have an answer? I think we don't extensively use retries. I think we set them to one only, so. I see, okay. Cool, yeah. The second question you mentioned about the global control point, right? In your scale, I assume that you probably have lots of Edge proxies, right? Is the control point, how are you guys handling, let's say, updates to Envoy, updates to the filters, the plug-ins, source code, how are you achieving that with this kind of distributed Edge proxies? Do you want to come answer? Nina is a part of ATC with us. Yeah, so the scalability of control point for the Edge proxy layer has been a known problem for us, so first of all, we switched to Delta XDS. And recently we had a problem with Java control plane when it got hammered by Envoy, like in a single region. Let's say we had some issue with the DDoS attack and then we, okay, let's scale up the Edge layer and then after that, when the control plane restarts and all the Envoy's tried to connect roughly simultaneously and to request the first state of the world snapshot, then our control plane just went crazy and it just stopped serving requests and Envoy's basically got the stale EDS assignments. And for that, what we have implemented as a short term roughly low cost remediation was what we call XDS rate limiting and we also reached out to the community and asked if other, like rest of the community would be interested in that, basically how that works. Every time there is a new XDS client connected to the control plane, we keep internal logic in the Java control plane that checks, okay, this is a new client connected and I'll have seen that client before. If it's a new client, it has a configuration of how many concurrent XDS streams it allows per second and if that one reaches, it sends, I think, some five or code and like, resource exhausted, try later and we rely on the mechanism in Envoy to do retries and like say, let's say in the largest region, it takes us with 10 concurrent XDS per second, streams per second, it takes us roughly 30 seconds for all the Envoy's to get the first snapshot but in that way we stabilize the control plane and we can scale up. We tried a very large, like 500 Envoy machines in a single region with a single control plane and we did control plane restart and XDS rate limiting has been proven like a battle proof mechanism for that. Yeah. That's great. Just one kind of ball up question.  Let's say that you are rolling out a new configuration. How do you make sure that you know to control the blast radius to start with? This is a feature that we are very much looking forward and currently we are running on GCE and this is not supported and we are on the way to moving to Kubernetes and this is the feature that we are looking for. Basically every Envoy upgrade for us is a stress event. We like it's create the risk events calendar. We notifying their company and then we are having our hands shaking and like looking at the graphs for early detection but we have cost bunch of bad incidents because we don't have that capability of reaching our rollouts. But we do like canary deployments and then rolling out globally. Cool, yeah. Thank you. This is a good input for our deployments team. We're definitely looking for globally progressive rollouts but we don't have that at the moment. Hi, I'm wondering what you've considered on the sort of simpler side of rate limiting so that you don't wanna roll out the whole infrastructure that Envoy can provide and that your users are used to something else that's simpler and different.  Yes, this is complicated. We can talk more about this afterwards but I mean we're looking from a technology perspective, we're looking into RLQS. That doesn't necessarily address the product side of the problem really. One idea we had is because we're interested in per second rate limits. Durability of the rate limiting infrastructure isn't so important. And so we were considering using Maglev on the rate limiting cluster and then basically you're distributing the IP addresses along the cluster hash. And if one of those rate limiting instances might go down, it doesn't matter so much, it doesn't take down the whole infrastructure. And that way you could also auto scale the rate limiting infrastructure taking care of some of the capacity planning concerns. But that's just an idea. I have an open tab in my code editor that has something kind of working in Envoy but that's not supported at the moment. I think we're about out of time but I think we're about out of time but if there's any more questions we'd be happy to take them offline. Thanks guys. Fascinating talk.


One of those days For those of you who don't know me I'm actually one of the original founders of the open stack project I had the great pleasure while I was the chief technology officer of NASA to pull an amazing team of engineers together They built Nova which is the compute engine in open. We open sourced that and I had the opportunity of getting up with Lou Mormon and announcing open stack on stage at Osconn back in 2010 But as I listened to surrender talk about his with the alpha Hello guys All right, it reminded me of how open stack was born in a crucible of fire our first customer actually That was powered by what we built at NASA was actually the White House and we had a pretty high-profile website things that President had when he was a senator in Illinois kind of gotten a bunch of Support to build this thing that would provide visibility into four trillion dollars of us spending and it's called USA spending dot gov and We worked for about six months on what you might call open stack alpha alpha 0.001 to try to make this thing work and you know one thing I want to point out a lot of the questions here about How does open stack move from? being used by the early adopters to the enterprise and I think the answer is Innovators like the panel you just saw here on the stage folks like surrender at Xerox Park I mean these these guys have the courage to take a new technology and define some use cases They might not be their ERP systems. They're not the mission critical applications But they're big data applications. They're mobile applications their applications where you can run them maybe in multiple Computing environments perhaps on multiple clouds where if you're private cloud or if your public cloud for that matter It goes down you've built your application to be resilient And I think that's one of the things that we're starting to see as we see more adoption of open stack in the enterprise You know is this transition of getting more and more workloads more and more applications Moving into environments where it doesn't matter if a server fails or even if a rack fails or if an entire cloud fails and You know thought leaders like Adrian Cockroft at Netflix Have open-source projects like Osgard which are now starting to get integrated into open stack and enable this transition and enable Enterprises frankly to use open stack for more and more stuff. So anyway I am reminded of the crucible of fire of surrender ready in you know the alpha version of our product almost a year and a half ago You know and you know the White House back when open stack was first formed I think that is the story of open stack. You know it was it was forged in a crucible of fire It was really put to a test. I think the thing that enterprises need to realize is We wouldn't all be here. There wouldn't be thousands of people trying to solve this problem if it wasn't a problem I think companies are realizing that there's a lot to be gained by using open stack and They're willing to make investments in making the technology work into figuring out ways to move all of the new workloads to open stack and not leverage Existing commercial technologies to solve these these new problems So take a minute to tell us a little bit about nubbila because you've got a somewhat unusual approach to no development stack Yeah, I don't think it's an unusual approach Frankly of taking a technology like this to market You know we take open stack and a whole bunch of other technologies and we put it into a controller You can plug off the shelf industry standard servers HP's a partner of ours Dell IBM Cisco and you turn the thing on and you know it boots up as a cloud And so our goal is to make that as easy as possible and we routinely install open stack clouds in a couple of hours and they work and they scale and they're reliable and That's really hard to do. I think that's been highlighted here And it's really hard to do even when every one of our customers has an identically configured system Where we have spent, you know, thousands of hours running tests 24 hours a day on the exact same servers So when an enterprise brings an open stack, you know, Nova alone has 750 config options when you run it so for any Organization to bring in a technology with that kind of complexity and then try to make it run on any arbitrary network fabric on any arbitrary server in any arbitrary configuration with any arbitrary storage vendor the combinatorial Impact of that means you'll keep a consulting team busy forever Trying to run that thing. So you have to make a choice, you know, do you need to run a cloud eBay does But most enterprises do not and they will buy a product like Nebula So surrender Mentioned that the sort of the difficulties and say the upgrade cycle something that fairly commonly raised Objects about open staff.  I will do that personally and it does work But I think that's a great. That's a great example backups. We do backups. You can back up and restore Nebula cloud You know, there's a lot of features security features open stack doesn't solve any of these problems Open stack doesn't solve the problem of how you orchestrate your network or your physical servers. It doesn't solve Security problems like how you use TPM secure boot Doesn't, you know, it's unencrypted traffic It's you know, you have thousands of things you need to do to make open stack, which is kind of a reference implementation An operational cloud environment, you know, and I think that's what a public cloud open stack Isn't a cloud service or a cloud product Nebula is a cloud product HP cloud is a cloud service rack space cloud is a cloud service So it needs to be product-atized or service-atized in order to be used period Now a number of people who have been up on the stage Previously have talked about how they want to avoid the lock-in of choosing specific vendors So you're choosing a strategy that is very much actually oriented towards lock-in for convenience Tell us a little bit about that. Well, I mean, what what are we locking you into you can choose your favorite server brand So we're not locking you to servers. It's not like we're putting our logo on a super micro server and trying to sell you the servers At a huge markup though companies can do that successfully. We're not trying to do that We're not trying to convince you to lock your applications into proprietary management framework. We're open stack So as long as you're Focusing your energy as a company on running on app open stack you can run on Nebula You can run on HP cloud you can run on rack space cloud, you know You're not locking yourself into anything You did buy a box and that box saved you thousands of hours of pain And to what degree does Nebula contain any proprietary extensions and how do you feel up private extensions to open stack? Today and actually our customers have asked for Nebula APIs. We don't have a single Nebula API everything our product does it does with open stack APIs and I think there is a case to be made for things that go beyond the scope of open stack like infrastructure management There's a case to be made for APIs that go beyond open stack But we're a hundred percent API compatible with open stack and for that matter We're API compatible with Amazon EC2 and Amazon S3 as well And do you intend to maintain that compatibility, especially as the foundation? Kind of pushes more towards breaking that compatibility. Absolutely. Yeah, I think you know the most important thing people ask What open stack is it's an open source Reference implementation for cloud and the most prolific cloud today is the Amazon web services cloud And I think as Google Compute Engine Continues to gain traction. We'll look at putting API support in there for that. It's frankly. It's what our customers want and API compatibility and interoperability is is frankly a customer driven thing So what are the use cases that you think are suited for the Nebula approach versus let's call it You know one of the software only approaches Well, I mean, I think if you're trying to build your own Amazon and be a service provider I'd invest heavily in knowing how to do that If you've just got some folks that want to move something off of Amazon and you want to run that locally We've got a great product if you Frankly are an enterprise CIO that is is used to things just working. We've got a great product You know if you've got if you've got a bunch of existing investments in storage You can plug a solid fire appliance into your Nebula and it seamlessly mounts like a thumb drive on a laptop But it's hundreds of terabytes you can we're working with other major tier one storage vendors to have that same experience with the storage You know the the ecosystems of storage that you already have in your data center and you simply can't do that with software You know if you can configure a software distribution in thousands of ways what are you gonna do when you call support and You know that you're gonna send a consulting team in there to figure out how to make your storage work And that is not you know That is not a viable solution for an enterprise CIO They want to be able to resolve that issue now So open stacks a lot of vendors involved in the community and many of them are very very large For an enterprise who's looking at you know who should be my provider of open-stack products and services What's the compelling use case to go with the startup as opposed to an established vendor? They have a relationship well I mean I heard the phrase punch above their weight I mean for the small number of developers we have we contribute a lot to open stack so if you look at the Whole history of open stack You know Nebula is one of the top five contributors of code of all time across all the projects in open stack And it's because we have incredibly productive developers that have been involved in the project for a long time And we have a lot of influence beyond just the code we write as well So I think we're also Switzerland.  We really don't care what storage you buy HP built some great servers So does Dell you need to make that decision and I don't want to lock you in based on proprietary management extensions and try to use these These techniques that have been used for the past couple of decades to keep people buying the same switches the same servers the same storage on Nebula is Switzerland and it truly Ironically gives you the the flexibility of not getting locked in and I think a lot of other companies are gonna gonna find it too easy to To slip down that path again So what do you think is most important for vendors to do in order to drive more mainstream adoption of open stack? Well, I mean, I think the thing that we need to do is ensure that there's true interoperability Why you ask why choose Nebula if Nebula just happens to be the fastest easiest Most secure way to get a cloud running the only reason you wouldn't choose it if we were incompatible with other clouds And so what Jonathan and the foundation are doing to ensure that there's interoperability between clouds If you know that Nebula can install a cloud in three hours say you want to go and do your big open stack deployment And you're gonna go spend, you know millions of dollars on an army of consultants You can go do that and we can have your cloud up next Monday And then while those consultants are toiling away you can essentially head your bets, you know And but the key thing is there you don't want Nebula to be incompatible with that open stack cloud You're building so interoperability is key and I hope this is just making it easier to run applications So we're excited about heat. We're excited about the project that essentially provides cloud formation style application orchestration So that there can be libraries of applications that you can just run Making it easier, especially in the enterprise for people who don't know how to use a cloud to run applications on a cloud is going to be critical to accelerating adoption and The community is was working hard on that. We're gonna make it really easy to run applications on Nebula And we're gonna do it in an open way. So we're not locking you into a proprietary pass layer If you use our product, thanks. Thank you. Thanks. I guess HP William We're not on stage together. Are you? I'm gonna play off Something that that Chris was talking about which Chris has just mentioned the importance of the compatibility with say AWS API's and the future say GCE API's HP is actually deliberately moving away from that as the foundation as a whole You want to you want to give a viewpoint on that? Since it's the HP of your point is contrary to that Nothing like starting with a hard question So I'll play it somewhat somewhat slightly contrarian I've been dealing with with open stack and other open source projects for real long time And there you know, there's a big debate inside the community There's a set of open stack people who want to be really tight to AWS and there's a set of other people inside the community that don't Most of them are arguing from either sides and very few HP people are actually involved in either side of that argument We're sort of sitting back in the background but If you think about it the Amazon API's are controlled by another company that isn't but isn't a both an open source company And isn't also part of the open stack consortium. So sooner or later and I think Sitting on one side of the argument you could see Open stack becoming very very large and Very strong with a massive degree of adoption and if you went one way support trying to support AWS and compatibility Amazon the authors of those API's could go off in a totally different direction and make it really hard for the community To maintain anything because they don't we don't control any of those API's There's no there's no open way just is I think Jonathan's over there Just as Jonathan was talking about the open stack design summit We all come together every six months and talk about how we're advancing stuff If the API's are going to change in Nova, they're going to change because the community makes a decision about that changing If Amazon wants to change the EC to API's the EC to API's change and this community Since most of you I think are gonna I'll call you all open stackers None of the people in this room would have a great effect on being able to choose that change so, you know, I can argue the other side which is You know, there's a very large community. There's a lot of people who are using Amazon and that's sort of that big de facto piece That's kind of the same line of reasoning that I think was Bill Gates used about of course Microsoft Windows is standard everybody's using it so I kind of fall in either way and I think HP in a number of these areas is kind of taking a wait-and-see position, right? We're certainly doing some things in the public cloud, but you know, we don't want to stifle the debate that happened started to happen It wasn't in Hong Kong, I think it was actually important wasn't it? Around where we started to have this it advanced more in Hong Kong and I certainly expect At the ice house summit in Atlanta for that dialogue to continue. I think once the dialogue is finished Then you'll see multiple vendors sort of immediately moving into that because we're all interested in tracking as Chris said tracking the interoperability of So how does OpenStack fit into HP strategy? It's very very crucial to HP strategy HP Believes and what's kind of core to our strategy is something one or two of the other people have spoken about sort of a hybrid cloud Big belief that public cloud is really crucial for a lot of what people are doing with cloud HP runs OpenStack based public cloud, but we also sell technology Software hardware storage networking to service providers around the world. They're standing up public clouds So we want to enable that but we also recognize for I think it was Ken Peppel said for compliance reasons for security reasons I also sometimes even say for ego reasons people sometimes want to run their own private cloud and You know in order to be able to burst move workloads from a private cloud to maybe a managed cloud up to a public cloud and back and forth we wanted a common architecture a common set of tools that we could use across all of this and HP as a very long history and open source. So we looked at the different options and Really chose OpenStack. We were involved with the folks at NASA and the folks at Rackspace helping to create the foundation and That technology is used in products from at HP from both Software to services to the public cloud and you see us moving more and more into trying to fill that vision of where we think hybrid is going And so why should we taking a particular sort of strategy as a vendor? You're not selling an open stack distribution for instance You've generally chosen to open stack as a as such an open source core to commercial products Any thoughts around around that strategy versus let's say, you know the type of work that never was doing that's more core open stack so I think What we're trying to do at HP is is deliver cloud solutions to customers and in certain parts of open stack We're fairly core. I mean we have Reasonable number of core contributors on most of the projects including what I'll call a lot of the core projects that PTL for Glantz actually works for HP Sorry horizon and But we've also done I'll say some of the other projects that we think address Both the scale aspects and some of the enterprise aspects. So things like a triple O Heat and bare metal provisioning or big areas that we've been making a lot of investments in the PTL for two of those three projects Are also HP employees We were co-authors of the security book along with you know rack space and Nebula and the NSA and Now we know this is being recorded Variety of other people so I would say that you know the areas that that we're Spending a lot of our time in are in what I kind of consider the big install upgrade update space That's sort of intersection of triple O heat and bare metal provisioning Security and then we've been doing a lot of work in the networking space So we're trying to augment some of some of both core But also what we would consider the things that are needed for enterprises to successfully pick this up step up and run with it Now of course that a price is deciding to get open stack of software really has two choices They can choose to get open stack as a distribution or they can choose to get an open stack embedded in some other product like HP Cloud System Enterprise or IBM smart cloud orchestrator or like So why choose something like HP Cloud System Enterprise rather than choosing an open stack distribution? What's the advantage or disadvantage to doing so so I think and I Think there's there's two. There's I think there's a couple of different pieces for it first different customers have different requirements and different wants and Even from some of the panelists who've been up here, right? They are bringing it in on their own I think it was the gentleman from eBay the gentleman from eBay and PayPal. They're bringing in their building and on their own There are other customers that really are looking for a base level solution And and I very much harken this back to the early days of Linux. I've been doing this for for a long time You know Even in the early days of Linux, right? There were a few people who would go off to colonel.org and they'd grab a colonel and they'd go off to various open-source projects and put something together and then there were really Early, I mean Ian Murdock was a little late later to it But you know Ian went off and built Debian and there were people who would just go and grab Debian And those were both, you know, open-source ways of doing it, right? I haven't mentioned any of the Unix vendors that that are out there But if you look today, I don't think there's a single person in the audience that goes to colonel.org to get their Linux Colonel today, right? But some people pick it up through a distro like Red Hat others pick it up through a package Pre-packaged almost appliance type system. So HP is trying to give customers choice Some of them are gonna build it on their own Some of them are gonna consume cloud products either through a managed service or through a public cloud So what do you normally see as the typical use case for let's say HP Cloud System Enterprise Versus, you know, someone who might be picking up a distro from something like Red Hat? It's a good question. So We talked to obviously HP is a huge company and we talked to a lot of customers and I spend a significant portion of my day ergo I'm in a black suit in our executive briefing center and and I have talked to customers about OpenStack specifically and the use of OpenStack Ranging from small teeny enterprises to huge global fortune tens government in sectors ranging from, you know, retailing Manufacturing people who build massive turbines airlines. It runs the whole gamut And I would say consistently that it really comes down to a set of business choices that they need to make and As I said earlier part of our strategy was trying to use this to build customer solutions So in in many instances if I talked to the CIO of a fortune 500 she's trying to solve a specific business problem and She thinks about it in terms of the problem She's trying to solve in the solution space to solve that problem and that drives Her decision of do I build something based on a private system or or a public system? And if I'm rolling it on my own, you know, am I using a distribution getting it from somewhere else? I guess they're coming to take me away hearing So I think you see it more that way I think also HP's perspective on this has been to carefully look at where we wind up with the distribution space as well because History is a great sort of predictor of the future right in the early days of Linux we had hundreds and hundreds of distributions when I started my career in the really early days of Unix I think it was 1981 or 82 there were more than 150 Unix licensees By the time you get to 1992 there were less than 20 So I think you know we could if we aren't careful we could wind up getting into the distribution wars with 50 different spins of this and From HP's perspective, we've tried to focus more on not that side of it But building solutions that customers can actually solve to real real world ways of doing it Which I think is not too dissimilar from what Chris has been trying to do over at Nebula now because of HP's position as an IT operations management tool vendor and the existence of you know Sort of a full-featured cloud management platform and things like cloud system enterprise You see not just the use case I think of these sort of net native applications, but also people looking at moving existing legacy workloads. Oh, yeah So what do you think OpenStack needs to do to enable that use case to make it easier for customers to choose that? so I I believe and HP is very heavily on this we're in what we call the new style of IT We've gone through the mainframe revolution the client server revolution and the cloud revolution and If you think about the mainframe revolution for a second in the early 80s There were lots of CIOs that said we're gonna move everything We're gonna be on this new on this new world in three year are in three months And there were massive amounts of transformation projects that just went up in smoke And I think people have learned from that and they're a lot more cautious about it So we envision a world over the next maybe say five to ten years where you have people and We still have customers that still have small stuff on mainframes, right? So you've got people with a mainframe world. I'll call it a legacy or traditional client server world and this new cloud world. I talked to Companies that are doing greenfield work on the cloud. They're making business decisions You may have an application that Keeps you in compliance with the FDA, but the FDA regs are changing in two years So they make a again a business decision about do we leave this where it is or in two years Do we re-implement it for a cloud world or do we look at trying to migrate it right now in many ways? We drive it more towards a business decision that what we're trying to do in the products We build the talk to open stack and other things when you look at some of the orchestration things that come out of HP allow those orchestration Products to manage both the client server world and the cloud world to manage the traditional aspect of doing development Staging production move through there, which works really well for socks compliance and other things But also supporting the DevOps model So you wind up with you know tools that need to live in both of these worlds and the unfortunate thing is if you talk to a CIO of a global fortune 50 they have to live in those two worlds for the next five years, right? It's going to take a while for legislation to change. They have to live with that degree of compliance So simply saying oh everybody's going to be over here. That's not what the world We may all wish as technologists and engineers that everything could be over here But in the actual fact most of you get paychecks Part of your payroll processing may still be running on a mainframe. So it you know we live in that world It's the reality and with net native applications and programmatic infrastructure and so forth How is that affecting HP outsourcing business? I people look at a new model of infrastructure and infrastructure management. I think the cool thing is I would say I Think that side of our business runs runs really well because as companies more and more want to look at the Technology today, they want to look at that greenfield piece. They still need to find somebody to manage their vax 1170 running VMS There's a very nice vax 1170 downstairs when you walk in the history and a PDP seven, but you know all kidding aside, there's that degree of Managing that long-term history and what you find more and more corporations doing is they're attempting to outsource that part of it Because frankly finding somebody who knows how to manage a VMS machine today is pretty hard So if you can outside outsource that to somebody like an HP where we have all that expertise You can then take your your in-house talent, which is usually very expensive and focus them on where you want to go Thanks, sir. Thank you. Good question and Dave from solid fire It's shift to the left We'll all bit shift over one So tell us how open stack is relevant to solid fire and how it and what your role is. Yes. I'm Dave right I'm the founder and CEO of solid fire and You know solid fire so we're a storage company So we build scale out high-performance storage specifically for public and private clouds And so you can already see the intersection there there with open stack And we've as Jay mentioned in the beginning been a longtime fan as well as contributor to open stack and we're Very heavily involved in the initial creation of the cinder block storage project and the contributions in that area and the reason that We found it such a strategic place to invest and even as a smaller startup Wanted to put resources into it is that we we did see this potential to become a Uniform universal cloud operating system and one that was an open platform that anyone can plug into And when some of the customers in the last session talked about You know wanting this vendor you try the wanting the ability to remove vendor lock-in and have the ability to switch in Different solutions and choose the best kind of solutions to their problems That's a great fit to a company like us who you know is an innovative storage company and and we want to be able to Remove the barriers to vendor lock-in that exist in other ecosystems and when the predominant virtualization vendor today is 80% owned by a storage company You know there's there's clear incentive to make that successful And so we want obviously very much to see open stack and other open cloud projects be successful The storage architecture has always been an interesting question with open stack And even as you go from open stack distribution to distribution you find that the choice of basically the backing store is different Right. There's the question of do I use stuff? Do I use sender? Folks who are still on old Nova volume How would you cut how should customers think about their storage choices? So, you know, obviously every app every seven application every use case is you're going to have different needs Generally, I break it down into and I think this is really where the storage world is moving to you're going to have a capacity Optimized storage platform you're gonna have a platform that you're really looking at lowest cost per gigabyte something that you're gonna store massive amounts of data Either for backup archive batch processing Content storage, whatever and that's more and more becoming object storage and obviously, you know open stack has the Swift project It has an open you know open source object storage project That's part of that and there are plenty of other object storage vendors that are now integrating with open stack to provide alternatives to that Seth is another open source project But there are other commercial object storage vendors that want to plug in and offer that kind of capacity oriented layer But then the other side of it is performance optimized storage and in any Environment you're gonna need the primary block storage for the virtual machines the images the databases and that type of things And that side of it is at least on the open source side of things a lot less mature There's there's no volumes. There's kind of basic Linux ice guzzly But when you get to high-scale high performance highly available block storage You tend to fall into the realm of proprietary vendors But one of the great things about the open stack architecture and cinder specifically is it has a model that allows many different vendors to Plug into that and solid fire as well as others have done that and there's a richness of choice there That doesn't exist as well in other platforms and so Based on the requirements for the performance storage the scale of it The price quality of service and other things that may be important to customers They can decide which of those solutions is gonna work so We've we've hit the point earlier on this panel around Being locked into potentially your hardware vendors To what degree is it important initially for customers to pick the exact right things versus switching architectures midstream? Yeah, it's a good question and You know obviously open stack gives you that ability That being said nobody wants to change the tires on a car while it's driving down the highway Even if you know it was possible to do that And so we do think part of that upfront assessment process of what the use cases are for this cloud What you're gonna be putting on it near term and where you might want to go longer term It is very important because you could actually in some cases with the wrong choice Destroy the user experience in that environment and storage is one of those things that is very centric to having a good user experience for your users of this cloud Where you might turn the organization off of it if you have a bad experience there So, you know, there's a lot of you know important choices to go into it up front But we think making a good assessment and good choice on the storage side of things is absolutely central to having that good Successful first experience with whatever that use case is that then you can build on for other use cases And so when do you think it's important for customers to make those decisions? You know, how far do they have to be beyond the proof of concept stage? Yeah, I I think I certainly think at the point that you're going to do a proof of concept You need to at least understand what the options are out there and what the pros and cons and the trade-offs are And whether you're doing a POC with the kind of final solution or whether you're just trying to get a handle on open Stack itself certainly by the time you've moved through the open stack evaluation process and made the commitment You know what I'm definitely moving forward with open stack You need to get pretty serious about understanding what storage and what platforms you want to deploy that on because If you just take something that may have been okay for an open stack evaluation and try to roll it into production There's a good chance that you are going to hit stumbling blocks there and it's going to it's going to hurt the user experience So to what degree do you find that your products are the products of competing storage vendors or other people in sort of the hardware chain Get pushed by professional services organizations helping users deploy open stack versus independent user evaluation without the aid of Yeah, and we see both of those we deal a lot in the service provider space We help a lot of cloud providers both open stack and non open stack cloud providers With the storage for their clouds in a lot of cases that is very self-directed again They have the expertise in their organizations to go out and make the evaluations on the infrastructure side of things But as we move into the enterprise and particularly as you move into open stack specifically there is You know more of a drift towards consultants system integrators And other vendors that can help them put together a solution And so we really work through both of those both direct as well as through channel partners So cooperation and competition in the open stack vendor ecosystem can complicate Some things especially as the number of vendors who want to plug into open stack proliferate As we look at more and more curated distributions of open stack We see that you know the vendors whether they're canonical or red hat over essentially choose Which are the which particular vendors they're going to support within their ecosystem. How does that impact smaller vendors? And how does that impact enterprise choice and what enterprise you should be thinking about when they're looking at picking a distro And picking the hardware that goes with it especially as a bunch of people have already brought up the interoperability issues between different solutions That should work together, but don't yeah and the approach for solid fire has always been to contribute to the core first and put our focus on the Contributions that we're making the integration and testing that we're doing with the core of Open stack because that is the base that all these distributions are picking up And if we can work with those vendors to prove that hey We are fully compatible with the entire core of open stack And yes, we may need to jump through some hoops of a certification process with that vendor And we've done that with companies like nebula with rack space private clouds distribution as well as red hat It makes that process a lot easier Whereas if you just try to kind of do an end run around the community and just go to a vendor and say You know hey, we're gonna buddy up and we're gonna take this product to market And you're gonna stamp a seal of certification in here, but I actually don't support that core platform That's where you get those lock-in effects Because that is where if the customer wants to then move to a different distribution or a different platform And that solution wasn't actually in the core distribution then they're gonna run into trouble So who do you find your typical open stack using customer to be? Yeah, we see a good mix of service provider and enterprise these days in the last You know kind of year and a half since we've been selling the product early on it was it was almost entirely service provider That was adopting open stack You know the move to public cloud and that model in the service provider space is rapidly underway Everybody's trying to compete with Amazon and differentiate there And and so we saw strong adoption of open stack in the service provider community But we are now seeing this shift to more balance between service provider and enterprise and on the enterprise side There's a range of companies. There's internet companies that are using open stack with solid fire But we also have traditional retailers. We have other technology companies as well as non-technology companies And you know the use cases have been discussed and it's really the same for us Dev test big data Infrastructure as a service platform as a service internally as well as SAS applications and vertical Applications, you know again on open stack not a lot of the traditional enterprise Legacy applications being deployed today, but I do think that in all of these cases They look this as a platform that can be a broad infrastructure management platform including those applications in the future So a question for all three of you How should customers think about assembling an open stack based solution? I would I would plug a solid fire appliance into some HP servers into a nebula box and turn turn it on I esteemed my esteemed colleague from Switzerland I can make other suggestions, but you know, so I think You know, there's there's a couple of options that are out there So certainly you can work with a consultant like Salini up front to really understand what you're trying to solve Somebody that knows the ecosystem and can point out the vendors and partners that can make the kind of best solution there And in a lot of enterprise cases that is the best option because unless they want to spend a lot of time getting into the community doing the Research sorting through the bud and the you know vendor hyperbole out there They're not going to know what works and what doesn't and so that's you know Really a good way for a lot of enterprises to start in a lot of cases though I think we're going to see the same thing with Open stack that we saw with Linux in the enterprise Which is it didn't start as a is a big kind of tops-down project It started because somebody had a need and they didn't have the budget to go out and buy You know a big system and so they just they downloaded some open-source software They put it on some servers and they started using it and before you know it that thing Which wasn't actually that important has some really important stuff on it and it gets elevated up And so I think we're going to see both a tops-down approach as well as a bottoms-up approach To open how open stack ends up pervading the enterprise. I mean, I think giving an honest one other than the switch for one comment You know it starts from two sides as he said but I Think the customer is generally trying to trying to achieve a business outcome, right? so and They're trying to try to achieve this business outcome in an area where they have less budget they needed faster and others and a You know a rapid way of doing it is you you know, you've got Younger more recent graduates. They're more familiar with this technology They understand how open-source works they get that they're trying to do that But in the same instance the person who's trying to get the business outcome is trying to balance this So I think we wind up seeing at least at HP We're starting to reach a point of and I think it was actually gardener that called it out But I'll I think you guys call it strategic adoption. So you're starting to see Accounts for which cloud is sort of their strategic direction They're going and then there's a small team of people who apply it by grabbing some of the source code grabbing what else is open source and really trying to do it as quickly as possible and in many ways I think that drives and frames how they think about where they're trying to get to So what can the developers do and what's the business outcome? I'm trying to achieve Questions from the audience for this group. I think in our in our industry You know biggest problem is you know a lot of businessmen actually they are not technically serving Well, on the other hand a lot of technologists typically like You know business sense my question is you know What are innovations driving open-stack development? By that means not only adoption I mean what are innovation driving open-stack progress by that means the progress not only adoption I'm most interested in Development open-stack So, you know, I'll answer that from from my perspective in the storage, you know part specifically Which is that you know the last kind of? Two and a half years that we've been contributing to open stack We initially were focused on you know We need to get this to parody with Amazon and we talked about earlier That's where a lot of people started with open stack is we've got to get the basics filled out And now in terms of where storage is going, you know I don't think really the question is really about Amazon anymore The question is about what are the users need to run a wider range of applications on open stack And what are the questions you have to answer about storage to make that successful and there are a Number I mean storage is one of the most complex part of the infrastructure and to be able to run very complex very Performant applications. There are questions that need to be answered questions like the backup question And how do you you know back up an open stack cloud questions around quality of service and performance? And how do you get the guaranteed performance to each of these applications in a cloud things like remote replication disaster recovery? live workload migrations and other things that are just very instrumental to the way that people think about enterprise architecture today Because we are looking to expand the use cases for open stack beyond just cloud native applications where the data doesn't matter and you Can assume the performance is horrible Because that's not the majority of the world we live in today So that's really where the innovation is going in the storage side and where we're contributing On the storage side to open stack to broaden the number of use cases that can use open stack on storage Yeah This question is for Chris Hi Chris I used to see bright lights Okay, yeah The labula was started from a trailer in NASA I Know NASA has enough data center. Why you started the labula in a trailer. Is this to show for the stop spirit of Silicon Valley, I heard security, but I wasn't clear on the question. Oh, sorry that any problem and like be a lot was built in the trailer of NASA or Container like a trailer I Think originally it was easier for me to use semi trucks than build new facilities So we were using big shipping containers to put some of the servers And I think that's been an approach some big service providers have used but that is somewhat more of a historical fact You know, then it is something that is really I think the other thing is is that we didn't want people to manage servers And people don't like to work in shipping containers in the middle of the field So it was a way to kind of solve both of those problems at once Yeah, a couple question The first question that as the panel already mentioned that Open stack is not our box ready product is a reference implementation model So in your own experience that how much work you have to put it into alpha product and finally into a product Really for enterprise use and on the enterprise use that the ready product that how much The open stack cost is I mean that in the how much portion the open-stack cost they occupy and the second question is that with so many Company jump on the open-stack bandwagon How do you see it down the road that you see a lot of MMA or you see a lot of bloodshed? Thank you So I'll touch on the first question, which you know I think was about the amount of effort you know required to kind of you know Turn open stack into a product or actually kind of use it as as a product And you know, I don't I don't want to sell short the advances that the community has made to not necessarily completely turn Open stack into a product But make it actually something that is downloadable and usable make it something that actually has decent documentation make it something that people actually can play around with and To go to the point of actually putting it in production There there probably is a need to bring in some outside expertise to bring in some vendors and to really think through what you're trying to do with it But the reality is it has come a long way from where it used to be and there are some great guides out there great tutorials great Documentation that allow people to actually get it up and running and get started with it And yes, if you're gonna run it at scale you probably need to work with some people that have experience and can guide you On the right way to make that be successful But I don't want people to be scared about this. This is not alpha level code, right? We're version 8 going on version 9, right? Windows is only on version 8. So we're way ahead of them Linux I think One of the interesting things about open stack is We have a lot of people focusing on Greenfield deployments and standing it up and you know, you can install it and you can go ahead and use it What really happens in an enterprise is, you know, you install it and for the next 15 years you use it So it's it's not simply about just standing it up. There's a whole life cycle management surrounding us and You know as I spoke a little bit about where we're investing our sort of you know engineering work and others It it's helping to build things like Triple-O and in heat and some of the things that help both manage the life cycle of owning this But also help orchestrate and manage the infrastructure side as a service Folks who don't know can you explain triple-o and heat? Sure. So Heat is Chris mentioned it a couple of others. So he eats the project inside open stack Around orchestration, but it's not necessarily orchestration for the applications running in the application layer It's orchestration around the infrastructure So it's how do you orchestrate all the different services that are inside of the cloud that you're running that are that the applications Are then running on top of it triple-o stands for open stack on open stack and and the idea behind that is actually Installing and bringing up and managing open stack with effectively open stack So you have an under cloud and an over cloud and I can get into all of the technology of it It started as a project at I want to say it was San Diego Where a set of us got together to talk about it? but Where a lot of this came from is if you look at open stack in the really early days sort of cactus Diablo and A cloud was really a rack or a set of racks and as you get customers that really are trying to deploy in the large Right people want to deploy this in not one data center But two or three and the whole installation paradigm when you're trying to install in let's say a data center that has 30 or 40,000 servers that's drastically different than if you're trying to install on two or three or four racks Right or even if you're trying to install in a semi And the nice thing about a container. It's the same problem as a data center Right what you really want to do is be able to power that sucker up and within you know Three to four hours be able to provision storage and compute and other services and ideally you'd like one person to throw the switch and that's all so For open stack in the enterprise that's sort of the stuff that we're working on and that's where some of the challenges come in Right, that's where you wind up getting some consultants to come in others It's not necessarily to first stand it up. It's made huge advances in the last three releases of getting Open stack to stand up if you tried to do it with cactus or Diablo You really needed to work on the project understand that to do it if you do it with grizzly or you know and then it's actually pretty easy now But where the challenge comes is if you run it for a while, how do you upgrade it the gentleman at park, right? He's trying to figure out how he gets the latest version. That's where a lot of the challenge comes in One thing I'll also say is we're talking about product features here when you talk about upgrading something or installing something am I installing it on a Large-scale data center infrastructure hyper scale infrastructure. Am I installing it on a laptop, right? I think that the whole idea of open stack is that companies like nebula can productize it as an appliance that plugs into HP servers and Solid fire storage HP can turn it into a public cloud Rackspace can turn it into a public cloud and enterprise can go turn it into something that powers eBay or PayPal You know you have the flexibility of doing whatever you want with it And in some cases as an enterprise you should buy it as a product and not worry about how it works That's what Amazon is you pull out your credit card and you start consuming services And that's easy and you put all of your energy into making your application run on Amazon That if you if you want that exact same experience, but you want it in a you know in a data center next to your existing data That's what nebula is for if you want to go build a massive service and go compete with Amazon That's where you want to go bring in an army of consultants and go differentiate your service offering and tailor it and and leverage your assets That that are unique to your organization. It's all about abstractions, right? I think what open stack provides is an abstraction for the entire Computer industry right whether and the thing that we have to do to make this thing innovative and successful in the future Is make sure there's a place for all these companies to plug into it and that the community Welcomes and is constantly bringing in these new innovative features. Who knows what storage is going to look like in five years or ten years? I don't but I know as new products come out It will change the abstractions that exist between things running in memory in a CPU and the things that are in storage Object store is just the most recent iteration of this, but I think that's where we have to you know Open stack brings the computer industry back together. It's it's not all happening in a secret room in Redmond or in Los Altos, right? You know we are all coming together, and we're trying to figure out how to build the next generation of computing Together and and there's a place for all these vendors to plug their products in in a way that they work together And I would just add one thing which is if there's something you specifically want to see coming out of open stack We'll all be in Atlanta the second week of May come you know put put code in effect how affect how it works get a voice and Surprisingly, I think it was Jonathan said there are multiple Enterprise class customers now that are actually putting code back in and participating in defining where this goes. It's very vibrant as an open source Yeah, you had a question about the number of companies that are in the ecosystem in M&A And I think there's two things realized one is that a lot of the companies that are in this ecosystem Are established companies and are not exclusively open-stack companies solid fire is is you know a great example of that We do a portion of our business in open-stack.  In fact, I would argue there's probably far more opportunity in the open-stack ecosystem Then then there are companies to realize it at this point And so I think there are going to be some very innovative start-up spawned in different areas of the open-stack ecosystem We've already seen that with companies like like NYSERA and Nebula and others And so I think there's great opportunity for startups in this ecosystem. It's not overly crowded Really quick two questions one question for William for former HP one for Chris So first question for HP is you mentioned a triple O. We are very interested on Triple to the quick question is the home mature triple E. It is and the any live massive production system actually initiate by triple That's first question second question for Chris is Nebula, you know, is a like kind of like a triple O type of appliance So basically you are the seed cloud or the cloud initiator What technology behind using triple O or using some like a Racer format or something you know technology like that or Can I not only doing the cloud initiator for the open-stack property? I have my own private cloud can use your appliance to time some customized the image to initiate as my private cloud so real short on triple O triple O is a project that's I'm still Effectively in incubation so it hasn't it's it's core technology hasn't released in Side and if I remember correctly, I mean If I remember correctly, I think their first instance of it is supposed to be an ice house I'm looking for Jonathan. He can check me and keep me honest. Is it ice out? so the In in terms of its degree of maturity you can go in and get the code now and and take a look at it There's a significant number of developers from both HP Red Hat and one or two other vendors that are working on it So the intention I think in ice houses for them to be able to stand up not only you know Racks of servers, but rows of racks of servers in a data center The long-term goal is for it to be able to basically stand up a complete data center, but it's an incubation Yeah, it's an incubated project. I mean our product today you plug servers into it and it works It's a shipping product. It's been shipping since April of last year. We have you know customers using it every day So I think that's the difference. I think is these technologies and open-stack evolve We're closely watching them because the last thing we want to do is put a lot of our Limited resources as a as a startup on something the community solved So we have to pick you know very careful pick the technologies. We invest in very carefully We use a lot of the core open-stack technology in our product And that's why we effectively have you know thousands of developers, but we don't I have a comment and a question for you Chris Comment everyone is welcome to comment on I feel open-stack what it really is enabling is Defining API's for managing a set of disparate resources and maybe over the next one year We're going to start seeing more and more vendors supporting those API's out of the box allowing Customers to kind of create custom solutions for their use cases regardless of vendor because almost everyone is supporting these API's Yeah, that's the comment. I yeah, I would just just comment on that I think you're absolutely right and that's why when I when I refer to open-stack as a cloud operating system to me It's like the Windows API or the Unix API This is the infrastructure API and it is going to provide a centralized point for all these tool vendors Who used to have to do this proprietary integration with every piece of hardware with every stack that came out? Gives them a central point to build that and it's going to create much more powerful robust Management orchestration tools as well as development platforms and we're seeing the layers now start to appear in both open source as well as Proprietary products on top of this core open-stack API and you know the question about Amazon capability Look, I think in five years the question is gonna be turning the other way around everybody's gonna be asking Amazon Where's the open-stack capability? Because it is going to be the far more universal platform Amazon will always be a one-vendor platform Yeah, I keep saying that people keep telling me I'm crazy, but one day Amazon will support the open-stack API now. I think that mark my words That's gonna get retweeted so I think seriously though. I think that what we have to do is be open to new Ways of introducing a look at Seagate putting ethernet ports on hard drives that support Swift object store directly I don't know what the right answer is, but I do know that open-stack provides a way for all the stuff to plug together To actually build systems and that's very different than taking software written in 1995 and Putting it in a holodeck and it'll run like it's 1995 in your 2025 You know, that's a very different if you look at what people in the open-stack community you're trying to do They're trying to innovate. They're trying to power next generation applications They're trying to take next generation hardware and put it to use in hyperscale data centers to power the applications of tomorrow This isn't about yesterday's software and it's not about tricking yesterday's software into thinking that nothing's changing It's about enabling the next generation of software to use the power of infrastructure to change the world Which was around the Nebula controller. I was curious to know What's the starting size? What's the maximum size that Nebula can support and how does this kind of scale? I don't want to turn this into sales pitch. We have we have a booth back there You know, it's a controller. It powers a rack, you know, though It can power multiple racks You don't want to buy it for just one server. So it's kind of a mid-range solution right now. Thank you Okay, thank you very much panel fantastic. Thank you So Lydia's gonna give us some closing comments and then I'll get up We'll wrap up this section and get started with the next So of all of you who came here thinking, you know is open-stack ready for my organization How many of you changed your mind today based on what you heard Yeah, I was about to ask that based on what you've heard about maturity and everybody else's challenges And for how many of you think it's more likely that the that there's more maturity than you thought there was And I Think that to sort of contextualize the experiences of a lot of the people who Did testimonials here? One of the things you're seeing is you're seeing folks who deployed in an earlier time frame, right? If you're gonna go out and you're gonna deploy using Havana today on a current distribution You're gonna have a different experience than these folks who've been using up a stack for a year or two But you're still probably going to encounter challenges I think you know the folks like Selenia and so forth were seeing current deployments are still seeing those challenges The question for you is probably going to be right Is what do what I want to get done for my business? worth the hassle and Is anything else gonna be any less hassle than working with open-stack, right? But in the end you're really gonna make a business decision of is this the right thing for me, right? What is the level of readiness in my organization to do this myself? Do I have the resources? Can I hire the resources? Am I gonna need to work with professional services? And I'm gonna need to work with a managed services provider on an ongoing basis How does that change my cost equation when I was looking at let's say, you know standing up a private cloud in-house versus going to Amazon for instance? Right the additional personnel cost is going to influence my overall total cost of the solution And what is my total cost of ownership over the years? And then what are all the components that I need to build get together integrate by? That are going to make a solution possible All right, so you're still looking at what is functionally a non-trivial deployment But the question is at your sort of you know technology company Relatively early adopter state is open-stack ready enough for you Even if they're not necessarily ready enough for your average, you know non-technology company with non-net native use cases So I hope this has been a good day. I hope that this has been thought-provoking And that the panelists have been helpful and thanks for having me here, too Thank you very much


Fireworks are a very important part of human celebrations. The bright colors and patterns of the displays never cease to amaze and entertain across generations and societies. In this lesson, we will learn how to test for positive ions, using the idea of fireworks as our learning base. Let's say you are given six files of solids. You are told that they are ionic compounds and you have been asked to determine the identity of the cation or positive ion of each one. To do so, you would conduct a flame test. Clean nichrome wire in concentrated hydrochloric acid and place in a roaring Bunsen burner flame. Repeat until no color is produced, thereby confirming that the nichrome wire is thoroughly clean. Place the nichrome wire in concentrated hydrochloric acid again and scoop some of the unknown solid onto the loop. Place at the edge of the roaring flame and you will find that each sample produces a characteristic color. Ensure that the nichrome wire is adequately cleaned between each test or contamination will occur and the flame color produced may not be strong. Sodium ions produce a yellow-orange flame. Potassium ions produce a lilac flame. Calcium ions produce a brick-red flame. Strontium ions produce a red flame. Copper ions produce a blue flame. And barium ions produce a light green flame. Notice that with the exception of copper, all samples are either group 1 or group 2 metal ions. Why is concentrated hydrochloric acid used in a flame test? Please pause the lesson to think about this and resume when you are done. Concentrated hydrochloric acid is used because the resulting metal chloride formed is more easily vaporized, hence making the flame test easier to conduct. These flame colors arise due to the fact that electrons become excited when heated and jump to a higher energy level. When they fall back down to their original energy levels, they release the energy absorbed in the form of a wavelength of light. The wavelength of light released is unique and we see this as a certain color. This chemistry has made fireworks shows possible around the world. In conclusion, a flame test can be used to identify certain positive ions as they emit a characteristic flame color.