Well, welcome. Guess I'm on here. Wow, I actually didn't expect such a full room for the last section. So thanks for sticking around. My name's Jack McCann. I'm the tech lead for Neutron, an HP's public cloud. And I'm going to tell you a little bit of tale of how we got to where we're at today with Neutron. Before I get started, I just wanted to ask, how many people have deployed OpenStack Neutron? Wow, OK, wow. How about Nova Network before that? OK, I don't know why I'm here. You're all experts, right? So I want to start with a little background on HP public cloud and what we've been doing over the last few years. We started about three years ago. We made a strategic decision to go on to OpenStack for a public cloud deployment. At that time, we didn't have any options. We went with Nova Network for our first gen computer service. This was in the Diablo time frame. And around, maybe it was August, the multi-host feature showed up in Nova. And we thought, that's what we want. So we adopted the multi-host feature. And there were a couple of things with multi-host that we thought we'd like to do it a little differently. We wound up with a few extensions to that model. Were we able to run the Gateway IP address, the same Gateway IP across all our compute servers, the same Gateway Mac, and a couple other modifications? So that's what I mean with HP extensions. And that model worked very well for us. We got that deployed in 2011. In early 2012, we took a look forward to see what we were going to do going forward. And there were some features that we didn't get with Nova. We had a flat shared network for our tenants. And we really wanted to get on to pertinent private networks, bring your own IP addressing type of things. And so we started looking at quantum at that time. So over the last two years, we've been working with quantum, standing it up in our test environments. And about a year ago, we went into public beta with quantum neutron, switched over to neutron. Forgive me if I say quantum. So we got into public beta with neutron about a year ago. And really started to hammer it with load. And we ran into a lot of issues. It was a bumpy road, the latter half of 2013. And what I want to talk about today is that those bumps that we hit in the road and what we did about them and what we've contributed back to the community to try to make neutron better. So a little background, the model we run in our public cloud, this is our tenant facing network model. So it's pertinent routers with private networks. And some folks have asked along the way, why this model? And it was basically this was because you can deploy neutron in multiple ways. And this was really the only model that gave us all the features that we needed, the isolated private networks, overlapping IPs. And then we had inherited other functional requirements from our Nova stand up around security groups, Nova metadata, floating IPs and DHCP. So this was our tenant facing deployment model.  So our deployment model, I'll get a little bit into now how we actually deploy the service. And we use a standard server building lock. We've got a few flavors of servers that we run over time. But they're HP servers. The basic server building block, we run x86, dual socket eight core hyperthreaded, so 32 CPUs presented to the operating system. Most of them have 128 gig of RAM, so we generally don't have any memory issues. Local storage varies. Two terabytes is pretty standard for the non-storage oriented servers. We run a pair of 10 gigabit NICs in a bond. And what we do with these NICs is each server in the rack, one of the NICs runs up to one top rack switch, one of the NICs runs up to another. And we use HP networking gear in our network fabric. 5,900 series switches. And one of the nice features in those switches is we can run them in this intelligent, resilient framework. Basically, it lets us cluster the switches so they present as one down to the server. So the server sees one switch at the other end of that link aggregate, so we can lose a tour and still have connectivity to our servers. So that's the basic building block. Now, what do we do with it? So the heart of the deployment is a neutron server. In our deployment, you guys see this OK? Font's a little small. The heart of the deployment is the API server. So the neutron server, we use a pair of those server building blocks behind a load balancer. And we have a dedicated API access network for our tenants to come in onto the neutron endpoint. We take another pair of those servers plus a quorum node clustered to run our database. We run MySQL. Another pair of servers for RabbitMQ, again with a quorum node clustered for high availability. And if you missed it, there was a talk on Tuesday by a couple of my colleagues that went into some detail around how we run our HA clusters. So you could look back and take in that talk if you managed to miss it. Compute servers, so many, many compute servers, I think our largest deployments now are about 1,000 compute servers in a single neutron control plane. And finally, because we didn't really have any other choice, the dreaded network node. So we run several servers to host network service functions. The DSP agent, the L3 agent metadata. We do run these in a clustered configuration for high availability. We've got some failover scripts that run. If one of the nodes fails, those networks and routers get failed over to the remaining network nodes for HA. Those servers are connected out to the internet for the external network for floating IPs, and the router static SNAT addresses, pretty standard stuff. Underneath these servers, we've got a management network where the RPC database and other management traffic runs. And finally, we do use an overlay network. I'll say a little bit about that. And we break that traffic out onto its own VLAN. So this is essentially our basic deployment model using those building block servers. Now, this particular slide, you'll notice there's a couple of colors on here. There's some green, and there's some blue. And in the interest of full disclosure, we are running our own plugin. And I'm going to ask you to kind of set that aside for a little bit, maybe suspend disbelief for a minute. Because the green here represents code that we run straight from upstream neutron. Basically, 85% of the code we run is straight from upstream. And if you've looked at the open source plugins, open vSwitch Linux bridge, you know that in the plugins, it's basically usually a pretty thin wrapper around a chunk of common neutron code. Our plugin is similar. It's a thin wrapper around some common stuff. So the blue is kind of meant to represent the HP specific code, and it's a small layer. And we also have some HP stuff in the L2 agent. And you could think of that stuff as analogous to, say, the OBS or Linux bridge plugins with VXLAN, L2POP, and an R-Presponder function. That's basically the functionality that's living down there. Why aren't we using OBS or Linux bridge with all that stuff, basically because it didn't exist two years ago when we started down this path? I would use it today, though, if I was going again. And we are actually moving in that direction with our new HP Healing on OpenStack release. But I'm not here to talk about the plugin. I'm here to talk about the other 85% of the code. And the ratio of problems we ran into was roughly in that proportion, 85% in the common stuff, 15% in the proprietary code. So the neutron server, the heart of your deployment, you've got to get this right. It's got to be provisioned right and tuned properly. Or if this is not running right, if it's clogged up, the rest of the system starts to behave in really interesting ways, starts to fail in interesting ways. I mentioned that we run a pair of these neutron servers behind a load balancer. So we've got two of those very capable servers I described earlier behind a load balancer. Handling API requests from users, RPC traffic from the agents down to compute nodes and the network nodes, updates to the database servers, calling out to Keystone. And now handling API requests from Nova as well. What we found, one of the very first things we ran into as we started to load up this system, was a single process neutron server as a bottleneck. So we've got 32 CPUs presented up to the operating system. One of them's pegged and the others are twiddling their thumbs. What are we going to do about this? We kind of expected this problem because we'd run into it two years earlier with Nova when we first stood Nova up. And we did the same answer that we did for Nova. We added support for multiple worker processes. These links here, these are hyperlinks to the upstream patches that we've contributed back in Icehouse. So now in Icehouse, you can go in your neutron.conf file and you can tune the number API worker processes and the number of RPC worker processes. When you do that, you should be able to see on your server that load will spread out and it eliminates that single process bottleneck. And when we did this, it really, we had a rash of issues when we had only a single process API server. And as soon as we deployed this in our test environments a year ago, that whole rash of problems went away. And a whole new rash of problems cropped up because the problems moved. I will caveat. There is a caveat on RPC workers. It's currently tagged as experimental. It does not work well with QPID and 0MQ, I think. It has issues with, but with Rabin MQ it works pretty well. And you really want to not be bottlenecked on your RPC processing in your Neutron server. So Neutron server, heart of the system pumping strong, going well. Now we're ready to fire up some VMs. Our tenant Sally here is going to help us today through our presentation as our tenant Sally. And the first thing Sally wants is an IP address. We'd like to make our tenants happy, but Sally doesn't look happy. So we debugged all sorts of these throughout the year last year. I can't ping my VM. How many debug? I can't ping my VM. OK, you feel my pain. I feel yours. So symptoms, VM console log, the dreaded route info failed message in our VM console log. Pretty typical indication that the VM just didn't get its IP address from the DHCP server. So we had encountered various issues with DHCP server over the course of last year. And this is a series of the fixes that we had to make to the DHCP server to make it run better. And these are the actual merges that we've pushed upstream. So some of the problems that we ran into. Well, the first one is just having a default route. So if you're using the default DNS service from DNS Mask, it can actually go talk out to the internet and be a proper caching server. And then several that are fixed in Icehouse here. We ran into interesting cases. Well, the first one was DHCP agent sits in a loop trying to synchronize its state with the server. And we found some cases where it would try to configure up a network. And something would go wrong. And there was probably some lower level bug that was blocking that network configuration. But that would fail. And it would kick out the sync state loop. And the agent would stop processing the network. So it was a case of one bad apple spoiling a bunch. And it would come around again and try to sync up again and hit the bad network. So this was causing us issues with new networks and updates. So we fixed that. We ran into a couple issues with DHCP agent cache getting out of sync. Some interesting corner cases. There was this merge to keep DNS Mask. DNS Mask thought it was still holding onto the lease. And the DHCP agent thought the address was freed up. So we made some changes to help synchronize that. We had some cases where that sync state loop was taking so long to complete a new sync state would start up. And that caused some issues as well. So we addressed that. This was a good one. Noticed in the log files and the API server log files that were thousands and thousands of these calls from the DHCP agent to get DHCP port every day. Just filling up our log files. And it turned out that just the way the agent was coded, it was doing to get DHCP port call when it didn't really need to. It just needed to do that once and remember it. And all of these combined to fix up some corner cases, make it a little bit more efficient. DHCP agents getting a little healthier. This last one is one we found recently when we want to move a network from one DHCP agent to another either in a failover situation or if we're evacuating the node for maintenance. What we found was when we remove it from one DHCP agent, the old port gets freed. And when it comes onto the new agent, that new agent grabs a new port with a new IP address. So all your VMs that are sitting back there using that default DNS mask as a DNS server on the old IP address suddenly have no DNS until they refresh their lease. So we've got to change this one's in review. If there's any cores in the audience, nobody wants to raise their hand. We've got to change up in review. Hopefully this will land shortly in Juneau where if you do move a network from one agent to another, it'll keep the same port or attempt to keep the same port to address that problem. So DHCP agent getting healthier. VMs got an IP address. Now it wants to get some metadata. Let's see if Sally's happy. Sally's not happy yet. Anybody debug? I can't SSH? Yeah, OK. Again, the dreaded giving up on metadata message in the VM console log, that's the typical symptom here. And we would see this just clouding it trying to connect out to the metadata server and it would time out and give up, in this case, after about three minutes. So we turned our attention to the metadata agent. And for those of you that aren't familiar with how that flows, basically a VM, when it wants to get its metadata from Nova, it'll connect to a metadata proxy process that runs in the router namespace. That proxy process will in turn connect via Unix socket to the metadata agent. And that metadata agent will then go and talk to the Nova metadata server to get the VMs metadata. And you can have, be launching multiple VMs in parallel, like some of our QA team likes to do, like 100 VMs, 300 VMs. And you can have other tenants that are launching their VMs and trying to get their metadata connecting through. Anybody seeing a bottleneck here? So what we ran into, again, we ran into actually two things. The first thing was there was a single process metadata agent, a lot like the API server. It's just one process, too much work. The other interesting thing we saw was the default listen size on the agent's listen socket was too small. So we were seeing connections getting dropped. So we pushed a change, a pair of changes. Again, I think these landed in Ice House to be able to spawn multiple metadata workers and to configure that listen socket to have a higher backlog. So you can change in your metadata agent INI file. There's now these two configurable parameters to address that problem. Now, the other potential bottleneck here is your Nova metadata server. But I'm going to punt that one off to the Nova guys. You'll have to go Nova talk about that. That does need to be properly tuned to make this work. So unclog that bottleneck. It's got our IP, got our metadata. Is Sally happy? No, she's still not happy. Yes. So this graph here, what this graph shows is the blue bar shows. This is from some of our internal monitoring software. What the blue bar shows is the time that a VM takes to become active. The green shows the additional time it takes to become SSH-able. And the yellow indicates when the VM failed to become SSH-able within a specific period of time. So you can see here that things were running along fine. And all of a sudden, we had a gap where the VMs didn't become SSH-able. And then they picked up again. And we saw this happen. These intermittent reachability issues. And we eventually traced this back to a restart of our L3 agent. So we take a look at the L3 agent. And this was an interesting one as well. So some interesting times with the L3 agent. First one, a restart. Every time we restarted the L3 agent, it would go and tear down all around our namespaces, all the floating IPs, and plumb them all back up. Yellow lines on our graph all over the place. So that got fixed in Icehouse. After that, we were still seeing some outages on our restart. You know, your restart, you want to upgrade the software, fix the bug, tune something. And we found a case where during the restart, the agent's trying to go make sure everything's plumbed up the way it's supposed to be. And when it got to the floating IPs, it would actually delete them and then add them back in. So there was outages, temporary outages there, and we got past that. One of the other factors contributing to the size of that gap was when we do configure a floating IP, add a floating IP onto a router. We do configure to send gratuitous ARPs out. And we've got our set to three, so we'll send out three gratuitous ARPs. So it's an ARPing with a count of three. That takes three seconds. And if you're running in the L3 agent and you come along and you want to add a floating IP and do an ARPing, all of a sudden, okay, I'm going to stick around for three seconds while the ARPing runs, and then I'll get along to the next floating IP. So it was slowing things down, particularly when it had a lot of work to do, a lot of floating IPs to add. As might be the case in a failover or a migration. And so we made a change to put that, spawn that into a thread, and that landed in an ice house and unclogged that bit of bottleneck. And then this last one, this was a pretty good one too. The agent, these are some things that we've done during ice house. There's been a lot of work in the community to improve these things as well. I should mention that. One of the improvements was the ability for the L3 agent when it's restarting, it goes out and it wants to know, okay, what are all my routers? It's got to re-synchronize its state. And there was a change done to do that in a loop and do that in parallel so you could process multiple routers in parallel. The thing we were in into with this was as you're processing these routers in parallel, they're adding floating IPs into the namespaces, doing IP tables commands to set up the NAT rules, and there's an IP tables lock to serialize access around the IP tables update because it's not atomic. And what we saw was all these updates, so we're trying to happen in parallel, got serialized behind the IP table lock. So one of our guys said, well, why don't we just use a separate IP tables lock per namespace because they are actually independent and you can update this namespace IP tables and this one at the same time. So we've got to change upstream and ice house to fix that by using independent IP tables locks. And the last one here, and this took a while, was during that restart when it's trying to sync up its state, there's new stuff, new work coming along, new routers, new floating IPs, maybe some going away, but it wasn't processing that new work during the restart until it finished synchronizing. So one of our guys, Carl somewhere, I don't know if Carl's here, but has a change in review, any course? Course, change in review to try to fix this so that while it's processing, resynchronizing, if there's new work that comes along, that'll get some priority because probably most of the existing stuff is already there and it's just checking to make sure it's still there. So a lot of improvements in the L3 agent, very nice house, these and others from the community. Let's check back in with Sally. Sally happy? It's always the network. You notice the problems are getting harder too? So Sally's not happy, the network's slow. So kernel version matters. We ran into several issues along the way, particularly as we started to scale up the load on those network nodes. We had a situation where if you try to delete a namespace, kernel would panic. And you really, by the way, if you don't have namespace deletion turned on, you really want to delete namespaces. You don't want those things just kind of piling up and it's not a good thing to have those laying around as we found. V8 pairs, if you got V8 pairs in your configuration, we found those to be a throughput bottleneck in earlier versions of the kernel. We're trying to figure out, I think we couldn't push more than three gigabits through one. And if we took it out, we could push the full wire rate of nine or 10 gigabits. Issues with network namespaces the number of network namespaces on the servers increased. The overall system performance slowed down. And finally, another one we ran into was the contract table in the queue router namespace. We found occasions where it would fill up with these unreplied entries. Now the good news is most of the problems we ran into had already been fixed in later kernels. So the namespace delete, panic, null pointer, dereference, I forget the details of that one. The V8 performance fix, I think that was around using per CPU statistics. Network namespace performance, you really want that one by the way. If you've got a lot of namespaces. It was a problem coming out of each time the agent issued a command into the network namespace as a process into the namespace. And on exit, a process coming out of the namespace. And the kernel would hold a VFS mount lock as it was coming out. And that was a global lock. And if you're running a lot of these with a lot of namespaces, it took a while to clear out of the namespace. So the whole system would slow down. And finally, the contract table full, what we found was by default, a contract will install entries for inflate connections with a default time out of five days. One straight packet, five days filling up your contract table. This fix reduces that to five seconds timeout. That's a good one to have too. I talked a little bit about that namespace issue. So commands, commands can help. This particular issue we saw as the number of interfaces on the system and large number of network namespaces, SUDU would slow down. And almost everything those agents are doing, every command they run, they run SUDU. And on a smaller system, you don't really notice it, but when you get say a thousand network interfaces and hundreds of network namespaces, the slowdown was considerable. And what we found, one of our guys found was that SUDU when it starts up, it goes in a numerates old network interfaces on the system. Doesn't do anything with it, just. So there's actually an option in version 1.8.10 of SUDU to disable that. In SUDU.com, few set this probe interfaces to false. And you can see the change, the difference it made here. You know, without the fix, just to time SUDU sleep once, that should take one second. It was taken one and a third seconds to sleep for one second. Most of that being system time. Afterwards, not so much. That's a good one to have. So, check in with Sally. Is Sally happy, anybody wanna guess? Oh, Sally, Sally, Sally. So those of you that might have attended the sessions earlier this week, IPv6 is not quite there. Hoping to make a lot of progress on that during Juneau, the guys at Comcast are really driving that forward. We've got some guys at HP helping out with that. Sally's gonna have to wait a little longer on the IPv6 addressing though. How many folks trying to run IPv6 and their deployments? Well, obviously it doesn't work very well, but. Okay. If it worked well, how many people wouldn't wanna be running it? Okay, sort of like Neutron, right? Oh, did I really say that? Okay. So, Sally, Sally hopefully is about as happy as we can make her right now. I do wanna say a little bit, looking forward to Juneau and one of the gaps from Nova Network to Neutron that I'm sure a lot of people in this room are aware of is it wasn't really an alternative or a analog to the multi-host model. So, when we started down, this just looks stepping back a little bit. We started down the path to public cloud and HP. We looked around internally in HP at some different technology and one of the pieces of technology that we found out at HP Labs was something called Diverter Distributed Virtual Router. So that's an interesting thing. By the way, that's a published paper. That's a hyperlink to the paper if anybody's interested. So that's something we might wanna use in cloud. Fast forward a couple of years. I talked about our first-generation compute service on Nova Network. Multi-host was a long way, along a big step towards DVR. We had some extensions we did to get it to use the same IP and Mac across all the compute servers. So in our V1 cloud, no matter what compute server your VM landed on, it always saw the same gateway IP and Mac. Fast forward a couple of more years and we've got now, there's been a couple of starts on multi-host for Neutron over the last couple of years. What we did last fall was we took a team of HP engineers and dedicated them to making that feature available on Neutron. So there was some talk of that in the design summit session yesterday. That looks pretty well on track for Neutron. That's the blueprint link right there. We really hope that's gonna land in Neutron, sorry, land in Juneau. For those of you not familiar with DVR, a quick word about that. Essentially, each compute node provides the routing services for its local VMs. In our subnet traffic, east-west will flow instead of having to go to the network node and back down, it'll flow directly between the relevant compute nodes. Floating IP traffic from the external network instead of going through a network node, we'll go straight to the computer server. There'll still probably be a network node out there for default SNAT, perhaps DHCP, some other services that can't be distributed, but this will go a long way towards getting us back towards a sort of model we had with Nova Network Multi-Host, which had some really nice fault isolation and scaling properties. You lose a compute node, you lose those floating IPs, but you've lost the VMs. So you can see here, in the virtual world, you've got Router A with a couple of VMs, and in the physical world, Router A follows VM1 down to the first compute node, and it follows the VM2 off to the third compute node, and the same thing happens with Router B. So we're really looking forward to that in Juneau, and I think that'll help out a lot. Oh, Sally's happy. Thank God. So summary and conclusions. Upgrade Neutron, if you haven't tried it lately, Ice House is better than Havana, is better than Grizzly. Neutron Server, the heart of your deployment, make sure it's properly provisioned and tuned. If you're using metadata, you might want to take a look at tuning the metadata agent, upgrade your kernel, grab that pseudo. There's probably other commands too that I forgot to mention. I think there's an IP route version that you need to delete namespaces, and I think we should see improved performance scalability in Juneau. So I'm gonna stop there and take questions. I live for it. No questions. Hey, well listen, thank you all for staying. Oh, is he leaving or has he got a question? I wonder, with all the problems you had with metadata of service and DHCP, why you didn't consider using config drive with CloudInnet? So a good question. Why use DHCP and metadata at all? A little bit of that's buried in a little bit of the history. We stood up our first cloud offering with a metadata service. We wanted that to be available to our second gen VMs. There is the option to do config drive. File injection we're not so fond of. We didn't necessarily wanna be mucking around with our tenants VMs and our cloud. Yeah, we went from file injection into using config drive. Config drive working well. CloudInnet, yeah, it's been trouble free so far. We may wind up having to go there for IPv6. Thank you. Thank you, good question. Anybody else? Hi, according to HP experience in the public clouds, how many VMs is suitable for a single neutral server to handle? So the question is how many VMs can a single neutron server handle? Yes, exactly. So that's a good question. I actually don't have the answer to that. I can tell you that we're currently running on the order of multiple thousands of VMs, multiple thousands of networks and routers on that configuration. We're not at capacity. Database servers not breaking a sweat, rabid MQs, twirling in his thumbs. There's plenty of capacity left. Okay, thank you. Thank you. Okay, thanks everybody. Thank you for staying. Thank you.


Okay, this is one of the hard daddy vintage bowling shirts they sent for me to check out and review What do you think? I think I like it. This is the medium, you know, I'm five six About a hundred and sixty pounds, right I got a little bit of a belly, but it's loose fitting for a medium It's not tight at all. It's not tight at all pretty nice for a medium, you know vintage bowling shirt by our daddy There you go


Okay, we're back here live in Silicon Valley, San Jose Convention Center, this is theCUBE, this is Hadoop Summit, this is Hortonworks, Yahoo, the ecosystem of big data developers in the Hadoop ecosystem talking tech. We're theCUBE, we're here in the ground zero for two days of live coverage. I'm John Furrier, the founder of SiliconANGLE, and I'm joined by my co-host. I'm Dave Vellante at wikibon.org. Merv Adrian is here at Merv. Merv gave a keynote talk today. He just completed a big survey. Merv's with Gartner, the lead big data analyst down there, Merv, welcome to theCUBE. Thanks, great to be here, and I can't take credit for the survey. Couple of my colleagues did it, that's the nice thing about being in a big research firm. You get to take advantage of other people's work. So how'd that work? You sort of gave them input to help design it, and then they go off and execute it, and then you get the data dump, right? Yeah, the big research firm process is very collaborative, we share content early, we share content late, we design survey instruments, we analyze the information together, and we write the research together. So I got to help shape some of the questions, but I can't take any of the credit. They did a great job. It's our second year. It's a thing we call the Gartner Research Circle. It's a set of people who are enterprises. The mean for this particular survey was about 3.2 billion in size, 5,100 employees. Global, every geography represented. And it's the second year we've done the survey. So you get some longitudinal comparability, which is very useful when you're looking at an emerging market like this. I'm really excited that you're on theCUBE because, one, we've been following your career obviously in big data since the beginning, since it started, actually. And we do a lot of the events, I'll see all the Duke summits and Duke worlds and strata and whatnot, and you're active on Twitter, so we see you out there every day, you're doing some great work, really one of the best analysts on the planet in terms of your data. You go deep, you poke at things that people aren't seeing, and so just kudos to you. Thanks for coming on theCUBE and sharing your awesome insight. So let's get right to it. So what are you seeing right now? Let's go in and talk about some of the things that you're seeing here today. Obviously, don't mention security in your keynote. You got Storm, you have Knox, you have... A lot of different stuff going on. What are you seeing? What are you looking at right now? What's emerging out of this summit that you could share with the folks that wasn't mentioned in the keynote? Sure. Well, the theme there was maturity, the fact that a lot of things are actually getting close to being used by real people for real things. Obviously, there's real deployment going on now. Serious use of the technologies is underway. And I have to tell you, I was reminded during my prep for the speech, looking at a little bit of past coverage and I remember seeing something from a year ago about yarn and how cool 2.0 was gonna be and hear all the great things. And I heard almost exactly the same speech today and that's an artifact of the open source process. Things are visible in the open source community long before they're ready for mainstream consumption. And that's a good thing, but it also is a challenge because people come up and remind you, you said this last year and it's still not shipping. What's the deal? Commercial software companies get to cover that. They don't talk about it until they're reasonably ready to put it in beta. So what we talked about last year is real now. And some things that we hadn't talked about last year are real now. So search is really showing up. The folks at Lucidworks are really excited about everybody coming back to Lucene. We're seeing the deal they did with MapR. We're seeing the work that Cloudera is doing to leverage this and other people are doing search as well. So once you bifurcate the data resource into the pre-curated optimized data warehouse side and what everybody seems to be increasingly calling the data lake, the undifferentiated pile that we're gonna go and pull stuff out of from time to time, search becomes the absolutely impossible to avoid first step to do anything useful there. So that I think is gonna be huge. The rise of interactive analytics on Hadoop is probably the second biggest top of mind theme right now. Even though there's a lot of, I think, misstatement about it being real time, there's a continuum here. Batch is at one end and real time is at the other. And what's in the middle is something called interactive and that's what we're doing now. There's a little activity in real time, but not much. So I got to ask you, obviously, Dave and I were talking about this on the intro, is that Hadoop has run, it's still so much demand for it, but there's been an effort to almost force commercialization a little bit too early, whether it's making it more of a business-oriented event or trying to push for monetization. We saw just two years ago the big data fund from Excel and Cloudera saying it's going to be a tsunami of apps, but yet only analytics kind of hits the stage. So I got to ask you, do you see applications coming next? Do you see platform stability and community coherency or our stability as a job one to the bigger picture of app developers? Because MongoDB has got a lot of traction on the developer side because of how easy it is with the LAMP stack. So where are we with Hadoop? I mean, are we yet to hit that wave of apps coming in or are we still stuck on the analytics and data warehousing piece and platform stability? How do you see that? I think we're in the early moments of the platform becoming accepted as a vehicle for moving forward with a variety of different applications. There are a couple of necessary preconditions here and really the gaps that I talked about at the end of the presentation are those preconditions. We have to have confidence in the security of the platform. That's the number one problem. We just heard Intel talking about it. There are vendors out there who currently are making their play on it, Zettiset, is talking about their distribution as the secure Hadoop distribution because the market wants that before they rely on it. We need a governed platform and we heard a little bit today about Falcon. We need security and we heard about Knox. For the open source community to catch up with what commercial software platforms already provide, they've got to fill those gaps. Once those gaps are filled, we'll see the kind of explosion you're talking about. I call it the weakest link in your talk today and it clearly is you put your data in Hadoop. The biggest problem, what I said was the biggest issue we see year after year for at least five years in Gartner surveys of CEOs, when we ask them about IT, what's the number one thing? It's trust. Can I trust what these guys are handing me? And I spent a decade building governance for data in my organization and income these hoodies with their new Hadoop platform with a bunch of data. Where did the data come from? We downloaded it from the internet. Well, what did you use on it? We got these open source tools. Okay, who's the owner? What's the provenance? How private is it? I don't know the answers to any of those questions. Okay, so you want me to rely upon the results you've derived from working with that data? I don't think so. My customers, Gartner's customers, are mainstream organizations for whom that trust in the data remains the number one issue and until they can have confidence in it, that barrier is still going to be there. So one of the big findings of your survey was that 31, and it was consistent from year to year, last year to this year, last year, there's 470 people this year, over 650, 31% said they have no plans over the next two years to initiate big data investments. Yeah. So on the one hand, that sounds pretty big. On the other hand, I'm not that surprised depending on who you were talking about. Now were you predominantly talking to IT people or was it sort of a mix? Were you going after the line of business folks and would you expect that response to be different if you were going after those? This was very skewed to IT and it was across all industries and all geographies and sizes as I said. But at the end of the day, it's IT that the CEO and CFO rely on if they're going to make business decisions. And certainly a lot of experimentation happens outside and I talk with the guy I call Chuck with the cluster in the closet who's off experimenting outside the IT organization and coming up with exciting new things. But at some point if there's value, it comes back to IT and the suits are right. They are the guys who get called in to clean it up and make sure that it's always available and that can always be relied on. And for that to happen to your earlier question we've got to fill those gaps. And I don't know that the third of people who aren't ready to invest in big data are not doing so because they don't think the platform is ready. I think they're not doing so because they don't have a real understanding of the value. I would say I... I'm confident maybe too. Yeah, confidence is an issue but what I'm really going after here is that I get as many questions still at Gardner about what is this thing? Is it really just hyper? Is there real value there? We still have some more selling to do to get some of those guys to believe that this isn't just the latest marketing campaign from the IT organization. Yeah, because business value oftentimes trumps some of those security concerns and the confidence concerns. Quite often. Because of business value they'll oftentimes take the risk but we've seen this movie before, Meriv.  We started with distributed computing. We're certainly seeing it now with the cloud. Do you think that the dissonance between shadow IT, lines of business and whatever you want to call it in a core IT will, that gap will close faster than it has historically? Are we going to make the same mistakes that we have over and over and over again and create a mess and then ask IT to come in and clean it up? What's your thoughts on that? Well, if I could restate your question I think you're asking me do even people who do know history find themselves doomed to repeat it? I think the answer to that is yes. The truth is we do keep doing this and as we look at wave after wave of technology innovation, when its value is proven it becomes hardened, it gets put into place and it becomes the new religion until the new profit appears in the wilderness and we're here at the Hadoop Summit but there's already people running around here talking about forget that and let's start talking about SPARC, okay? So there's always innovation at the edge and consolidation at the center. That's a good thing, it's a very healthy thing and one of the most interesting metaphors I heard today was the Peloton metaphor and if you heard that at the end if you know bike racing the Peloton is all those cyclists that were together and every once in a while a few guys break away and if you're watching on TV or if you're one of those riders that lasts for a while until the Peloton catches up. It's a wonderful metaphor for the open source community that people will come up with extraordinary innovations and folks out here are going to say you know what, I can write that, I can reproduce that code and I can do it out here in the open source community. Well that's democratization, that's competition, that's a free market. And you talked about alternative architectures to Hadoop. Are you suggesting that open source ultimately will be the Peloton that catches up to those alternative architectures or can they add enough value fast enough? This is an entirely new market dynamic. We've seen it develop with increasing rapidity since I guess Linux was the first really big poster child, right? This time around the cycle is a lot quicker. Yeah, sure is. I mean nine years ago they were adding the distributed file system to Nutch, right? And here we are with 2,500 plus people, 70 sponsors selling products and an enormous number of clients I talk to every day who are calling saying, okay, I need to get involved in this thing. How's it going to help me and how do I get started? So we talk about those guys all the time. So I talked to a CIO, he was talking about compliance issues, that's an inhibitor, but he's still doing a huge POC, billion dollar operating budget they have that do a huge POC on Hadoop. Big issue is compliance. So I was a little skeptic there in terms of confidence, but still value, just looking at the value. But I got to ask you about alternatives because we're living in an API world where there's layers of abstraction where value can be created at any different layer. Think OSI stack, whatever you want to analogy you want to use. Amazon has certainly shown that, hey, I can produce a kick-ass cloud stack and do big data. Dave and I were also commenting throughout the past month that all the different CUBE events we've done that the industry standard bodies now are open source communities. As almost as a matter of fact, I'm pretty confident on that. But I got to ask you about a role of Amazon as an agitator, as an innovator, as a commoditizer and also open stack. So now these are market forces that are somewhat adjacent to Hadoop pure play. So is that another angle that people have to catch up to or what's your take on that? How does that all shake out for you? I tend to divide things into twos a lot and I look at the web-native organizations out there, businesses that started on the web and I think that it's unlikely that they'll ever go to an on-premise world. They may have a tiny fraction of their infrastructure at some point in the future on-prem, but if they started in the cloud, they're probably going to stay in the cloud. For the rest of the world, tracking what's going on at Amazon is a little tricky because they're quite opaque. They report large aggregate numbers and they tell us stories. The stories are useful, but it's not always clear how representative they are. So we know that they started five and a half million elastic map-reduced clusters between March of last year and March of this year. That's a pretty amazing number. Even if half of them were Hello World, that's a lot of elastic map-reduced in the cloud that nobody bought CapEx for. Nobody went out and bought servers and storage and ramped up, they just used an interface. Great place to experiment and we know of people who are doing useful things in the cloud. What we don't know until we really do some serious formal research and I hope we can get to it is even those poster children, how much of their portfolio is inside Amazon? Lots of people have some stuff out there and it's working very well for them, no knock.  And some production apps. Right now, Amazon's stated vision and goal is to be the high volume, low margin provider. And that's a very, very interesting differentiation from all the other commercial IT providers. Most of us like to think about extracting margin. Certainly the Wall Street guys who follow those companies want to think about margin. Amazon is determined to drive margin down and to dominate the market by making themselves unassailable. It's really hard to compete with them. And bless you. And at the same time, preserving the value proposition that you would normally expect is to be associated with high margins. Remember, I want to come back to this notion you were talking about, about the open source cycle. Sure. And the pace of innovation and how much faster it is now than it was, say, during the Linux days. We asked this of all our smart guests but we want to hear your opinion. Are you going to ask me too? Yeah, we are. Will there be a red hat of Hadoop? Not in the next three years. Well, it took Red Hat a long time, right? Because the world kind of left him alone. But John Furrier's latest answer was, well, the Red Hat of Hadoop might just be Red Hat. Not yet. They haven't moved in that direction. But look, I think it's clear, as I said in my speech, that at every layer, I've got this sixth layer model, pick your own number of layers. But at every layer of the stack, there are multiple alternatives. Even the things that Apache call core canonical Hadoop, like HDFS itself or MapReduce, are substitutable by other pieces already, some of which are open source, some of which are not. Every distribution is a composition. And there's lots of room for more compositions. And the idea that anytime in the near future, the set of possible technologies and functional capabilities is going to become so uniform that somebody can really own it, seems unlikely to me. I don't think we're anywhere near the end of the innovation cycle where new things are going to continue to pop up in the stack. I love the way you laid out the horses on the track. You had leading pure plays. You had others, specialists. You had the mega vendors and the mega partners. Like a NetApp or a Dell, for example. And a good way to look at sort of the ecosystem. Well, my customers, Gartner's customers, tend to skew to the somewhat conservative mainstream adopters. And as interested as they might be in a pure play and as much as they might see value in something truly new and innovative. They don't want to get fired. They also have a large investment in the rest of their portfolio. And if they're, pick one, pick an IBM shop because there's lots of them. And they've got Tivoli and they've got Guardian and they've got DataStage. And they say, if I'm going to drop this new rock in the middle of this lake, how bad do I want the ripples to be? If this thing can enter the water smoothly and not disturb the surface, that makes a lot of sense. That gets back down to your value question, which is understanding the consequences is a way to assess the value. I was talking to one bank in Boston, a financial institution, they said, hey, I got a lot of NetApp filers. I got some EMC drives running email. I have Red Hat and I got IBM services integrating all this stuff. So Amazon, what are you, high? I need OpenStack. That's a direct quote. So I'm like, okay, so why OpenStack? It was, honestly, OpenStack gives me a warm blanket of comfort because I know that I might be able to tool it and develop with it. So do you hear that same feedback? I mean, is that something consistent that you're hearing? Because again, that's legacy. Is he saying OpenStack or is he saying HP OpenStack, for example, or Raxby? No, he's just saying OpenStack in general, OpenStack Summit, we just came back into OpenStack Summit. So that was essentially saying, OpenStack's the dream scenario for him because he now can look at the ripple effect and saying, okay, I know what I'm dealing with with OpenStack. It's modular, I can code, I can take the best practices of this and plug it into there. At the first point, they want safety too, right? Amazon was a little bit risky, it was a little bit of a black box for them, they were concerned, so that's what we were referring to. People don't want black boxes unless they are completely confident that the black box is going to operate seamlessly without errors and they have confidence in the person providing the black box. So you have to separate the question of which architecture do I want from the question of who am I getting it from? And the mega-vendors on that chart are people who are perceived as being strategic partners to their customers. And so they've already made that mental leap that I can rely on these guys. And so if I'm going to get this technology from them, I have a level of comfort. And if I'm an alpha and I want to play with every piece of technology myself, that's a different kind of company. There are plenty of them, there always will be, but they represent a small slice of the market. So if the Gartner customers are ultimately controlling the chessboard and the spending, which remains to be seen. Yeah, I wouldn't suggest that. No, I'm just saying, if in fact that's the case, well they have historically, and certainly in the enterprise with mobile that's changing, but to the extent that they are able to control that, that would say the cartel ends up winning. Oracle will make it that position. The rich get richer. When has it ever been any different? The oligopoly. Okay, so we're getting the time crunched here. Murph, we'd love to have you on theCUBE as long as we can because you're a great friend to us and on Twitter.  So there's nobody in front of us right now going like this. We're high-five. No, no, we have us, the airplanes are lining up as Mark Hoppe has always said. They're all wanting to land and they're running out of fuel. So I got to ask you the final question though is, is that what are you looking at now? So you, 27 days ago, I saw your tweet, I got a lot of briefings, my presentations in flux. You had a lot of changes going on. You had a great keynote here, but what's changed since your keynote in your mind from the conversations, what you were expecting, not expecting, and what are you looking at now? What are you poking at? Since this morning? What are you poking out on the horizon from the conversations, from the feedback? I might have missed that. I like that trend. What are you eyeing in terms of, what rocks are you picking up and looking under? What areas are important to you? Well, I start with where I perceive the gaps and I've kind of already identified a few of those, security and governance and administration. And certainly we heard a lot about that here. I'd say we heard more about that this morning than at any other Hadoop event. I've ever gone to. Suddenly we've got Knox to talk about. We've got Falcon to talk about. Didn't have those things before. As an analyst, that means I've got some work to do. I haven't dug very deep into those things yet. And I have a little bit of a safety net. My clients, of course, are, to some degree, lagging the leading edge. They'd like me to come back and report from it. They're not ready to jump there yet themselves. So I have a little time to go do some digging and understand and try to put it in context. That's my next job. They want to understand where their blind spots are, as John often says. They rely on us to say, you've got some perspective. You've taken a look at this. Is it time for us to think about it yet? And I have to give them a reasonable answer. No, or maybe, or you're late. Well, Murib is doing some great work out there. Gartner, Murib is the analyst, top analyst at Gartner in big data space between Gartner and Wikibon. We've got the world covered. Obviously some of the firms have to do some great work. Congratulations on the survey. An excellent keynote here at Hadoop Summit 2013. This is where all the developers are. This is where the communities are galvanizing, making the decisions, hardening the platforms, trying to onboard developers, get the white spaces, gaps filled, security, governance, administration. All this stuff is killer. This is the cloud, big data story. Thanks for coming on theCUBE. We'll be right back with our next guest after this short break.