Hi everyone. Welcome to this talk. Today I'm going to be talking about how to unleash ArgoCD observability superpowers. My name is Leonardo. Everybody knows me by Leo. I'm one of the Argo maintainers. I help driving the contributors meeting. Maybe you already met me in that meeting and I'm going to be presenting you today. Unfortunately, Deng was supposed to join me today. He wasn't able to come but he provided me the recording and I'm going to be playing it during his part. So let's get started. And I wanted to start just providing a really brief story of what the problem this is trying to solve. So imagine that you are passionate about technology and you work for this company and developers are having issues deploying components in ArgoCD, their applications in ArgoCD. And you have this, sorry, to deploy applications in Kubernetes and then you have this great idea to install ArgoCD in your company's infrastructure and suddenly it's a success. Developers start using ArgoCD and deploying their application. And one day one of the developers come to you saying they can't deploy their application. There's a problem going on in production and they ask for your help and you check the UI. There's no much of information there. So your nature is to inspect the logs and you open ArgoCD logs and you have that feeling. So you have to find which one is the faulty wire, right? And you don't know much about ArgoCD. It's a little bit intimidating to understand the logs. And this talk is about how you can use some observability functionality that is built in inside ArgoCD to help you whenever this day arrives. So basically I'm going to be talking about metrics and when I say metrics is about Prometheus metrics. And I'm going to be talking as well about distributed tracing, how ArgoCD leverages distributed tracing and enables you to have a much better overview of what's going on in your ArgoCD instance. And Deng is going to be talking in its recording about how they run continuous profiling to find ArgoCD bottlenecks at a large scale. So Deng works for ByteDance, which is by the way the company behind TikTok applications. So it's a very large infrastructure they have there. So speaking a little bit about the numbers when I say about running ArgoCD in large scale environments. So my team operates ArgoCD at Intuit. Just really brief numbers here. Intuit has more than 14,000 employees, around 20 locations. We have more than 2,000 production services. To support that we have 44 ArgoCD instances, which sums up more than 16,000 ArgoCD applications. So you probably already know, an ArgoCD application can sync several resources. So it is quite a bit heavy load environment that we have and that we have to support ArgoCD at Intuit. So let's start talking about metrics and how we leverage Prometheus metrics and Intuit MIT. So basically if you check the documentation basically, you're going to be able to see that ArgoCD has built in several Prometheus metrics that you can use to really inspect how applications are updating. So for example, the first one here in the list, I'm going to provide you an example. So you can write queries, for example, to check for the health status of your application. And whenever this health status comes from healthy to degraded, you can configure to send alerts to your team so you can act on these issues. So there are several metrics that you can use. And one thing I wanted to mention is that there are on the top here different components that you can see. And just talking a little bit about the ArgoCD architecture because things are going to be much better to understand. If you understand a little bit about internals on how ArgoCD is designed. So basically users interact with API server. And so this is the same component that serves the UI. So most of the operations are provided by this component, which is pretty much a back-end service. We also have repo server and sometimes depending on your configuration, the CMP server. So the boxes on the top represented with orange color are basically back-end services. And the ones at the bottom in green are controllers. So just showing you what I was displaying in the previous slide, those metrics are exposed in specific ports that are already configured in ArgoCD. There's no specific configuration that you need to do to have those metrics available. The only configuration is really pointing Prometheus to scrape those metrics from those ports. And you're going to be able to interact with those metrics. All right. But what we do actually inside into it to leverage those metrics, right? So we leverage Grafana dashboards. We have a collection of... So we have our dedicated alert dashboard. That's how we call it. And so basically you wrote a collection of Prometheus queries leveraging those metrics. And whenever an error happens that we know we need to act, we send a PagerDuty alert. And the thing we like, I think it works really well for us, is if there is a problem that needs human interaction to help solving the problem, right in the PagerDuty alert, we provide a link. So we can just click that link. We go straight to the dashboard and we have an overview of what happened and we have some history. So I think it's something we found really useful to have alerts linked with dashboards. So it provides some visual whenever something goes wrong. So this is what I'm showing here to you. It's a default dashboard that is provided by ArgoCD repo. You can start using this dashboard today if you're not already using in your infrastructure pointing to it. So this is basically a general purpose dashboard. But as I said, the biggest advantage you have is by the time you start tweaking those metrics towards your needs and sending alerts whenever you need some action item. Okay. So what if you're running ArgoCD in some sort of a multi-tenant environment? So what if you're sharing the same infrastructure across different teams? So if you're just querying one specific metric, what happens is that you can easily route alerts specifically to a dedicated team, for example. So this is one feature that I wanted to quickly show to you that ArgoCD provides a similar feature that is available in cube state metrics. So if you're familiar with that, it allows you to define some labels in your application resource. So here you can see that I'm specifying two hypothetical labels. So app team, platform, app, BU, business unit, infra. And if you configure ArgoCD with this command parameters, what ArgoCD is going to do is basically start emitting a new metric called ArgoCD app labels with the values that you define in your application resource, which allows you to join this metric with all the other metrics that I was showing before. So you can basically do a much better routing on those metrics and send specific alerts to specific teams. So this is available today. Yeah. And moving forward, distributed tracing. This is a new feature we introduced in ArgoCD 2.4. So ArgoCD 2.4 embraced open telemetry. So yeah, there was a little bit of work to get things updated, especially on the GRPC side, not getting into that. But the truth is, yeah, it is available. There's not much you need to do to enable ArgoCD to start emitting traces to any open telemetry collector that you have in your infrastructure. But the only thing you basically need to do is to edit this config map. So if you're not familiar, this is something I wanted to highlight as well. I'm not sure if there are too many developers aware, but ArgoCD provides this config map called ArgoCD CMD params. So before we used to ask developers to patch ArgoCD containers to customize how they want to configure specific settings in the tool. But this is not always great because, for example, for the open telemetry configuration, you would have to patch at least four different containers to add this additional configuration, which is basically the same configuration for all of them. So ArgoCD provides this config map called ArgoCD CMD params CM. You just define the attribute. We have a complete documentation about all the possible configurations it can add in this config map, and it automatically injects whatever you define it in the appropriate containers. So this is just a tip. If you're not using it yet and you're still patching ArgoCD containers, this is maybe a good thing you can start using. So to have a complete list of attributes, you can configure it by using this config map. You don't have to memorize this URL. Just remember that we have a section in ArgoCD documentation called operator manual, and inside you have a page called declarative set up, and inside this page you're going to have a link for all the config maps we have. So let's talk a little bit about distributed tracing. This is not ArgoCD UI. This screenshot here I extracted from Jager UI that I used to configure. So I configure ArgoCD locally, running locally in Jager, which is an open source tool to collect open telemetry traces. So here you can see that immediately there is something that is made available in the UI, which is most of the distributed tracing UIs, the ones that I used to, will provide a similar diagram that shows you the interaction with the different components that you have in your application. So as I showed you before in the architecture diagram, ArgoCD is not a single pod that you just install and run, so there are several components involved, and there's an ArgoCD to work properly, those components that they need to collaborate together. So by leveraging distributed tracing, you can see those components, the interaction with those components, and whenever some of those components fail, usually the UI will provide you a way to navigate directly to the point where this fails. So for example, ArgoCD communicates with GitHub to get the desired state, and then it reconciles this desired state with the live state and provides you the diff that you're already familiar in ArgoCD UI. So just with this very simple example, you understand that ArgoCD depends on the connectivity between your ArgoCD installation and where you run your Git repository in order to calculate this diff. So if something goes wrong, if your ArgoCD is communicating with GitHub, for example, and GitHub is down, ArgoCD isn't able to calculate this diff for you at that point, and relying on tools like distributed tracing, things are going to be much easier to realize just by looking in logs or even looking at the UI. All right, so I recorded a really quick navigation in Yeager UI, mainly to show you how it looks like and to have a feeling about what are the type of information you can get from it. So you can see here on the left that you can filter by the different services, and those services mean the different components that I was showing you in the ArgoCD architectural diagram, and you can also see the different operations. So for example, ArgoCD needs to list applications to show you all the applications that are deployed in a specific ArgoCD instance, and that is served by this GRPC call that needs the collaboration of two components. The one represented in orange are the API server, and the one represented in green is the API that is called, it's a component inside ArgoCD that we call repo server. So this is the GRPC call, and it shows, here I'm showing you the operation, you can see all the details and inspect, and you can expand the tags and see the details of it. Okay, so this is a pretty simple and quick navigation on the type of information you can extract by enabling traces, but these are all happy scenarios. So let me quickly show you what happens if something is not right. So in this example, what I did, I manually killed repo server, and I called the same, navigating in the UI, I invoke this service to list the application. So as this operation requires the connectivity between API server and repo server, obviously repo server is down, in the distributed tracing view, it's very easy to spot where the error is. So if you click in exactly the point where you see this red circle, you can inspect the traces tags, and you can see that there's something going on between the connectivity between API server and repo server. So this is the type of visibility that enabling distributed tracing in your infrastructure, enabling ArgoCD distributed tracing in your infrastructure is going to provide you. Okay, so moving forward in a presentation, so this is the time where I'm going to play the Dengs recorded presentation, and again, yeah, he's going to be, let me see if they're recording. I'm Deng Zhou from the Biden's Edge Platform Team. Today, I'm going to present how we do continuous profiling in our Edge Platform. This is an agenda for this talk. We will start from the definition of the continuous profiling. Then I will give an introduction about the high level architecture of the Biden's Edge Platform. For that, the deployment and the setup of the continuous profiling on Biden's platform will be discussed. Last but not least, two case study about ArgoCD performance troubleshooting will be presented. So what is the continuous profiling? Profiling is a dynamic method of analyzing the complexity of a program, such as CPU utilization, or the frequency and the duration of the function calls. With profiling, you can locate exactly which part of the application are consuming the most resources. Continuous profiling is a more powerful version of profiling that adds the dimension of time. They are two major type of profiling tools, instrumenting profiler and sampling profiler. The instrumenting profiling inserts code into function calls to collect function execution details, while the sampling profiling periodically collecting the function call stack to represent the estimate function calculation. This is the high level picture of our Edge Platform architecture. In Biden's, we have hundreds of seeding pops. Each pop is converted into one or more connectors and are managed by the Edge Platform. So the Edge Platform is a control plan of our Edge cluster federation of aggregations. The platform is equipped with info service like storage, monitoring, logging, tracing, and so on. It provides unified console to expose service like Github, metrics, billing, IM, and other product level features. They are actually many Edge workloads already running on the platform by sharing the resource and infrastructures like CDN, Proxy, RTC, API and game and other services. So why do we want to continue providing on our Edge Platform? There are a few motivations. First of all, we are seeing scalability challenge on our system. We have hundreds of Edge cluster all over the world, and more than five applications running on it. All the applications are managed by a single ROCD instance. So we want to understand the performance issue and the root cause. Second, traditional performance test environment is hard to set up for us. Our Edge cluster connection is not reliable due to the networking instability, which is very hard to mimic in the testing environment. Second, our application of this scale is very hard to mimic in terms of pattern. Even we can have similar number of applications, we are not able to mimic the unproduced activity and the remote health status. And then our Edge Platform use continuous delivery model, and we shape new features and fixes every week. As not all features enablement need corresponding age pop change or update. It's very high cost to simulate all the changes. Last but not least, even if we can do it, we have no intention to set up a very cost problem reproduced environment in terms of both money and engineering. Because of those, using continuous profiling tools to collect information from the production environment for the troubleshooting is a front-line solution for us. So this is a set up for continuous profiling on our Edge Platform. Every piece of the software about Kubernetes is deployed via GitHub's manner and managed by the ROCD. We have two profiling tools deployed, the Paca and the Periscope. Currently, only the control plan cluster enables continuous profiling. All our targeted services are running on the control plan cluster, including signals, premises, ROCD, and other self build services like resource manager, billing, ticket system, and so on. The first step I want to share today is the ROCD application list performance. As our internal SIE becomes multiple project owners, we are seeing that his application listing waiting time is several times longer than our admin. This is a finger captured and provided by the Paca, one of the tools we are using, when the performance degradation lasts. From the S-frame figure, we can find out that application service list handler takes most of the waiting time. Inside it, we realize that our back enforcement takes significant amount of time. After checking with the source code, it turns out that ROCD will go through the owner role list to check the permission of each application for the user when they do the application list. 5,000 application times the length of the owner list is a major source of latency. As a quick walk around, we give the SIE the admin instead of owner list. The list latency has reduced from 40 seconds to 7 seconds. Beside the workaround, we have redesigned the permission management by using of the project role instead of global role to avoid the checking. In addition, we have figured out that ROCD does not have a good caching about permission interpretation results. We have further ideas caching on that. With a fixed application listing can reduce to 2 seconds instead of 7 seconds. Another case we want to discuss here is about the CPU efficiency improvement for ROCD with the continuous profiling help. Our SIE observed that ROCD application controller has a very high CPU organization in most of time, even though there's no user activities. So we took a look at the profiling data collected by a periscope, which is another tool we used in our system. We observed that the function secret cluster, which sounds like a quick lookup, consume significant amount of CPU. So we want to know what happened inside of this function by checking another profiling data point as shown in the in the slides. It turns out that the unmask of the cluster secret is a major CPU cost. Actually, the cluster secret is a very large data object, which contains all the information regarding the cluster include the certificate and the connection credentials. But the code showed that the controller just need to get the get the cluster name but unmask of the whole secret. After reporting this, ROCD upstream has fixed it by just adding an index of that and the CPU utilization become much better since then. For anyone who are interested in learning more about continuous profiling, here is the reference. The first one is about what is the continuous profiling and how you can use it. The second thing is about another explanation of what is continuous profiling and the different type of continuous profiling. And the third one and the first one are two different tools we have, I have used and demonstrate in our case. So these are the two findings we want to share today among not all cases we encounter internally. Besides ROCD, we actually find the problems, performance problems in Sanos and our self-building service as well. Continual profiling is indeed an excellent tool which can help address the most unreproducible problems that show with very little overhead. Thank you. Okay, and mainly to wrap up this talk, I have this last slide, mainly to provide a conclusion. So in this talk, we spoke about where we are and into it and how we leverage Prometheus metrics, having a dedicated dashboard that we use internally. I also spoke where we're heading. So distributed tracing was enabled in ROCD 2.4. This is not still configured in our infrastructure. This is a working progress, but we're looking into going in that direction and improving the instrumentation that we have inside ROCD. And Deng showed how they do continuous profiling by dense to find problems that are really hard to reproduce because it requires a really huge environment to happen. Okay, so this is what I had for you today. I don't know if you have time for questions. Any questions, anyone? So thank you.


Hello and welcome to the Grand Line Review, your source for everything One Piece. Today we are going to be incredibly subjective for this here Top 5 list as we are going to discuss my favorite characters in the series. And I thought I'd do this because some of the most interesting comments I get on the channel are when I talk about a particular character and then all of these people just spring up and say oh they're my favorite character and I'm usually just like what really them? But hey given how expansive the One Piece world is there is almost certainly a character out there for every individual. However that can also make it very difficult to choose one's favorite characters but I think I've managed to do it for now anyway. And so the criteria for this list is as follows. I need to like the character enough to put them here that's pretty much it. So this will more than likely be the most subjective list I've made on this channel and hopefully therefore be immune from complaining. And furthermore all characters on this list will be canon because well I don't really need a reason this time but I guarantee you that they definitely all will be. But with that out of the way let's begin. Welcome to my top five favorite characters in One Piece. Number five. Dolflamingo. From the moment the Heavenly Demon stepped into the series all the way back in the giant arc he captured and held my attention for the next uh how long have I been reading the series again 12, 13, 14 years? Somewhere in there. But a lot of Dolflamingo's initial appeal to me was his aesthetic. There is something very intriguing about someone who permanently smiles especially one with such clear ill intent. But the other main appeal of Dolflamingo was the sheer amount of mystery surrounding him. Now I get that in 2019 this character has been explored pretty much to completion and for new One Piece fans this feeling probably doesn't exist. But prior to the whole punk hazard dress-roser business Dolflamingo was a looming and threatening presence in the world. His motives and goals entirely unknown and yet this damn Charlie Roger just kept appearing and I needed to know more. So when he did finally get his chance to become the main antagonist in the New World era I was all for it. And personally while the dress-roser arc does have issues Dolflamingo did not disappoint me whatsoever.   Number four. Crocodile. Here's another form of villain who made a big impression on me in the classic era of One Piece. In fact just for some context when I started reading weekly we just begun the anti-slobby arc. So at the time Crocodile was kind of seen as the definitive villain of One Piece. Far more powerful than his predecessors of East Blue, far more intelligent than Anel and we had yet to see what Rob Lucci would bring to the table. So Crocodile reigned supreme as the face of antagonists in One Piece. His design was amazing the sort of suave mafioso look with the massive golden hook and his devil fruit abilities were serious business. I mean sure his powers may feel a bit lacking these days but the way he wielded the Sooner Sooner No Mi was creative, fearsome and an aesthetic treat to watch in action. Plus I would also say that one of my favorite parts of Crocodile is his voice in the anime. It's just so gruff, unique and suits the character perfectly. And look the dude's just cool. And I could not be more thrilled whenever he re-enters the story which is nowhere near often enough, especially in the new world era. Number three, Charlotte Katakuri. Here we have a very recent addition to this ball in the grand scheme of things and I feel like it's a huge tribute to Oda that two decades after beginning a series he can still consistently come up with characters who are able to completely blow me away. I mean this guy really has no right to be as cool as he is. He rocked up halfway into Holcake Island and just completely stole the show through a combination of fantastic design and incredible abilities and as the arc went on an ever unfolding personality. The journey that Katakuri's character went on is incredible and I won't restate it here because I have a 15 minute or so video detailing it all, but by the end of the arc there was no way that I could really call Katakuri an antagonist, at least not in the same vein as previous characters on this list. I found myself rooting for him and Luffy equally, satisfied by the idea of either one of them emerging victorious in their fight. It does also greatly help that his fight with Luffy, which come to think of it makes up the large majority of his involvement in the series as far, was some of the most superbly choreographed action I have ever seen crafted on a manga page. And as much as I often felt like the climax of Holcake Island was going on a bit too long, there was never a dull moment when this character came into focus and as a result, in my subjective eyes, Katakuri has more than earned his place here today. 2. Monkey D. Luffy Pretty difficult to go past this one, as Luffy has carried the series to glory on his shoulder for over two decades now and for very good reason. Despite being a seemingly tropey shounen main character, Luffy always finds a way to surprise me, be it with his stupidity, directness, or on the odd occasion, brilliant left field thinking. And of course Luffy is also a visual feast to watch in combat. I don't think there's ever been a Luffy fight that's conjured a shred of disappointment to me and a lot of that credit is due to the way that Oda has thought about Luffy over the course of the series. I mean it's crazy to think that a man with such a simple ability can evolve himself so consistently and maintain long term interest. But the bulk of the charm in regards to Luffy comes from his sheer determination and lack of concern for what anybody else thinks of him. It's very difficult not to like a character who relentlessly pursues their dreams no matter what obstacles stand in their way, as we've seen time after time in the grand line. Luffy consistently conquers the impossible through sheer force of will and his character really does serve as a personal inspiration for me and as a result how could he possibly not be on this list. And yet with all of that said there's still one character left who far and away claims the title of number one, Charlotte Pudding. Just kidding it's uh it's obviously Zora. If you've been watching this channel for any stretch of time this will probably not come as any surprise to you whatsoever but for me I credit Zora as the entire reason why I began reading and watching the series to begin with. Basically the old story goes that one day I was flipping through channels and I caught sight of this dude bruh wielding three swords and as a result I began watching the four kids dub of one piece. That's right Zora was so awesome to me that for quite some time I put up with that truly horrendous dub because even that could not ruin him. But beyond amazingly cool design I'd have to say that the main factor in enjoying his character is his simplicity. Kind of like Luffy he's a man relentlessly driven towards one goal and he gives no shits whatsoever about what anyone has to say about it. But he does also have a couple of great character quirks like always getting lost or being a raging alcoholic that helps balance out the more often than not serious demeanor that Zora sports. And I don't really know what else I can say really. Any time Zora is involved in the story it piques my attention and he very constantly delivers on fantastic action, comedy, and general amazingness. And while that probably makes me sound like a Zora fanboy I can't really deny that either. It's that simple and it is for all of my subjective reasoning that Zora finds his way to the top of this list. But that pretty much does it for my top five favorite characters in One Piece. If you enjoyed this video and the content this channel produces in general then please do consider donating to the Grand Line Review Patreon because the support of all of your amazing people is what continues to make this channel possible. Also do check out my Teespring store if you're interested in shirts, hoodies, and other miscellaneous items with the proceeds going directly to support the channel as well. And if you'd like to join the fun at any time then please do head over to my Discord server where a wide array of shenanigans takes place on a daily basis. And finally please do comment with your own favorite characters in the series. This has been the Grand Line Review and I'll see you next time. So wait, what exactly was Zora even looking at here? Oh, oh.


Live from Orlando, Florida, extracting the signal from the noise, it's theCUBE. Covering Enterprise Connect 2016. Brought to you by Oracle ZDLRA. Vonage and Cafe X. Now your hosts, John Furrier and Peter Zantesonio. Hello everyone, welcome to theCUBE. We are on the ground and we are sponsored here at Enterprise Connect by Oracle ZDLRA. Take the worry out of data recovery with Oracle's recovery appliance. It's a DVR for the Enterprise that records everything. It's bulletproof. Thanks to Oracle's ZDLRA. We are here on the ground. I'm John Furrier, Peter Zantesonio. Peter, talk about the BC Strategies Group.  We're partnering with you guys with this new space. We're going into, you guys have been covering it for generations. What's happening in the UC space? Well it's interesting, 10 years ago when we started dealing with this, it used to be called unified communications. And it was a term that nobody really understood. And Jim Burton, who was my partner, actually coined the term.  So today, we've evolved our name to BC Strategies to reflect that. And I think the really important thing is, this is not a discrete technology or solution or whatever, it's something that weaves into everything. You guys have the epic wine party because Jim's obviously from Napa Valley and he has a little, it's a great networking event. But also the who's who in the industry is there. And it's great networking. And we were talking after a few glasses of wine last night about the future of unified communications. And there I was getting all animated. It's totally in transition. There's really two camps of people, old school and new school. And they're all coming together. And if you're not moving to the new school, you will be dead on arrival. This is the big shift. What is your perspective as an expert in the field with your analysts? Are you guys seeing the same thing? What is this new transformation? Why is this area so hot? It's so hot because basically it's the integration of the new way of communicating with the old school business processes. I mean, the world today revolves around mobile devices. The world revolves around multi-dimensional communications days of email and our classic kind of communications are gone. And so even voice to some extent in the classic form is evolving. So what's exciting here is there's two dimensions. One dimension is all of the classic silo type communications modalities are merging and some are going away. And the second thing is communication as a discrete process is going away. It's becoming part of everything you do. Touching a customer, doing a process, doing manufacturing, it's seamless. And so as all this IT stuff for many years went on and on and on and building silos, it's all getting flattened and communications is the glue that's pulling it together. And Enterprise Connect is the show that where all the industry revolves around. This is the big industry show, Enterprise Connect 2016, where all the action's happening. And you've got the big whales out there. You've got Cisco, you've got Microsoft. Now with Skype for Business under pressure with the whole Slack conversation, you've got companies like CafeX and Vonage just absolutely coming out of the woodwork. Vonage Trends, they were disruptive on the voice over IP. They basically eliminated the telcos, the I-Lex, the last mile around voice down there doing it for the Enterprise. CafeX, a 30 person startup doing, basically telepresence at a low cost. That's disrupting Cisco. So the whales are on their heels. The new disruptors are coming in. How do you make sense of all this? What is the folks out there see all this noise? They see all this data. How are they going to get the signal from the noise? Well, the problem is there's so much noise out there.  And that creates a lot of noise. These new guys are very, very focused. The other thing that's happening out there is how people buy things is changing, right? Historically in the telco world, we had a multi-layer channel, right? You had a company that sold to a distributor, that sold to a VAR, that sold to an end user. Today, with cloud services, people are buying direct. And so this thing that you build, this use channel thing, and it's the barrier to entry is gone. And so there's multiple things going on with respect to the distribution channels, as well as the endpoints being mobile, and the fact that the classic legacy silos are getting flagged. Jim Lundy from Aragon Research tweeted yesterday and wrote a blog post on his research site, UC is dead as UCC for unified communications and collaboration with the emphasis on collaboration. It's a perfect storm. The cloud in memory, mobile technologies, DevOps, and this time-to-value acceleration is happening right now. It really is the perfect storm. Do you see it that way? And what do you see happening this year? You're going to see some dead bodies, you're going to see some new growth. What's going to happen? Yeah, there's definitely dead bodies that are going to come along here. One of the challenges that people are not really that aware of though, is the fact that when you take large companies, you know, Fortune 100-type companies, they can't change their infrastructure on a dime, right? So you're seeing this stuff take hold at the SMB, the small to medium-sized business thing, and work its way up.  And you got these big guys like Microsoft and Cisco with these big platforms, and they're trying to figure out how to come down because they used to be up here. So, you know, it's a question of trying to understand who is going to learn how to communicate direct and sell direct and deliver that value proposition. Yeah, the middle man goes away. This is the internet. It's been a business model for the internet for years, but now it's accelerated because unified communication are buzzwords and siloed stacks and software stacks of the past that have chat, video, communications, voice. Now, that's being jammed into one blurred, integrated solution, unified, aka unified. But now with cloud, the users want that consumer experience. They don't want to have an experience at work different from what they do at home. Exactly, and one of the real problems here, John, is that, you know, classically the legacy companies, every application, whether it's voice or video or whatever, had its own interface, right? Its own stack. And now people want to basically get on and they want to say, hey, I'm on the phone. Hey, let's do this. Let's share this document. Let's have a video chat. Let's bring this guy in. And the ability for all those things to seamlessly connect across many different vendors, that's where the people that know how to integrate cross-platform in an open environment are the people that are going to win long term. I can't wait for Snapchat for business to be hitting the scene. Can you imagine the work environment where you have Snapchat for business where it disappears? That's good stuff, we're going to see. But this is the future. The future of things like the Snapchat. Okay, you're in the Cube, we're on the ground. This is the Cube, brought to you by Oracle, ZDLRA, Vonage and CafeX. We would not be here. It wasn't for the support of our sponsors. Go check out those companies, get the appliance from Oracle. Go check out CafeX and Vonage because without their support, we would not be here. This is the Cube on the ground. Thanks for watching.