Welcome. This is the afternoon track of App Developer Con. My name is Carlos Santana. I'm a senior specialist essay with AWS. And we have here my co-speaker. What are you doing? Metal Molic. And we're here to talk about mainly orchestration versus calligraphy patterns on event-driven applications. And this is in Kubernetes. We'll talk a little bit about resiliency and scalability. So I'll give it to you, me, so to get started. Go ahead. Hello, everyone. So let me start with the whole context around why you want to build event-driven architectures. What is the whole reason why we are here? And what problem does it solve? So some of the benefits of building event-driven architectures. Traditionally, we have build apps which use APIs, microservices. One of the service, the front-end service, calling a middle tier. It, in turn, calls another microservice using an API. And it fulfills a certain business functionality. But what it leads to is a lot of interconnectivity, a lot of dependencies, and a lot of challenges with respect to maintenance and building, making more agile systems. So the goal of event-driven architecture is to make it more fault tolerant so that if one of the systems or microservices were to go down, you're still able to operate. It gives you that scalability since you're now trying to build microservices which are connected with events. You're able to scale them independently without having to change everything that's involved in it. You're able to have small systems and which are using asynchronous communication to connect each other. It is extensible because you're now able to bring in new functionality as you need because you'll be using what we call as an event broker. And that event broker is the channel through which all these microservices are able to communicate using what is called as a public subscribe type of a pattern. And it makes it more extensible because as your businesses evolve, you need to bring in new functionality. You simply connect them to that event broker without impacting any of the other pieces in your architecture. It gives you that agility because you're not dependent. So your teams can think more quickly. They can bring in new functionality and they can build faster. So as I briefly mentioned, in an event-driven architecture systems and development teams communicate via events. So unlike the conventional API-driven way where you're making an arrest API call, for example, here you basically use a middleware or an event broker to act as that intermediary through which the systems or microservices are able to communicate. As you can see here in the picture, you have the producer or the application which generates that event. An event is nothing but a state change. It basically indicates, for example, an order has been placed, order has been canceled, a transaction has been placed, and so on. So anything that's happened in the past that you cannot change. You can have another compensating event, but as such you cannot go and change it. So that event is then transmitted onto an event broker and based on rules, you have subscribers which basically consume that event and communicate with each other. As you can see here, the benefit here is that these consumers are able to connect with that event broker independently and there is no direct dependency with the producer. The only thing that it needs to be aware of is the schema for that event. So going further around choreography and orchestration, the whole idea of this talk was to take you through some of these patterns. So what does choreography and orchestration solve? So as most of you would know that in order to build something which is customer facing, you need to have more than one microservice talking to each other. And this is where you need to have something like an orchestration or a choreography coming into play. As you can see here in this picture, we have an order that got created but as part of that order creation, the backend needs to do multiple things. It may need to do a pre-authorization on a credit card just to make sure that the customer has that amount through his credit card or his credit card and then we get a pre-authorization. Once that pre-authorization is done, then you actually make a call to the payment gateway to apply the order, that kind of an amount. And finally, you go and apply the same transaction onto an accounting system for that particular business who's the merchant for this. So all these typically would be in different microservices. Now, given that we're talking about event-driven architecture and you want to make systems as loosely coupled as possible, how do you achieve this?  You may have a situation where all these microservices may have a purpose-built database, right? One of them may be using a relational database. The other one may be using, for example, a NoSQL database and so on. So how do you achieve such a business functionality? You may also need to have ordering maintenance that needs to be handled in a sequence. And these business requirements may evolve. So you don't want to be bringing in coupling between these microservices. This is exactly where the solution with a saga pattern comes into picture. So as you can see here, what the saga pattern does is it basically is a sequence of local transactions. So in other words, the first microservice, for example, may be doing a certain transaction, applying it onto a relational database, and then generating an event that we passed on to the second microservice. And the event of a failure in those cases, you need to be able to have something called as a compensating transaction. So the picture, as you can see here, is showing that each local transaction updates the database, publish the message, or you want to trigger the next local transaction, and then for error handling, you need to have similarly like a reversing transaction if you are compensating. So the two options for implementing saga. The first one is called the choreography-based saga. In the case of a choreography-based saga, what happens is you have an event broker or a message broker, as we know more commonly, and the microservice is basically just independently publish and subscribe events as they need to. What it means is that when an event occurs, the first microservice may be putting it onto the bus or the event broker or the event bus, and then the other microservices may simply be subscribing to that event bus based on certain rules. For example, when an order is created, you may have multiple microservices which may need to act on that same event in different ways. One of them may be related to a financial aspect of it, meaning that it may go and update the customer's account. There may be one which may be just simply taking that event and putting it into a data lake. There will be one which may be more from an analytics point of view, and identifying if there is some kind of a trend or some kind of an alert that needs to be generated because of some kind of fraud kind of an event. So all these may just go and subscribe to that same event broker based on certain filters or certain subscription criteria of that event, based on the event attribute. Orchestration-based saga, on the other hand, is the one where you have a central orchestrator. So imagine the use cases where you basically need to execute them in a certain sequence and you want to control the order in which this happens. It also helps you with visualizing that what are the different microservices that I'm calling, what happens if there's an exception, how many times do I need to retry, and how do I log these errors? So if your requirements are more around that, then you typically would want to use what is called as an orchestration-based saga. So diving deeper, in the case of event choreography, as I briefly mentioned, that you have an event broker on which you have different microservices, basically publishing and subscribing the events. There is no centralized logic as such, meaning that you can have new microservice or new publisher subscribers coming in and subscribing to these events as they need to. The only thing that they need to do is basically have that selection criteria for that event. This is mostly used for publishing events across business domains. So if you're coming from a domain-driven design point of view and you have gone through some of those concepts, you're likely to have seen how you have different business domains and how you create these context maps and the dependencies between them. Now, as you start building microservices that are aligned to those domains, you'll realize that you need to have certain patterns on how you exchange events that are across these business domains and events that are within the same business domain. So the event choreography typically is used when your one business domain is wants to generate an event that needs to be consumed by another business event. And in those cases, an event bus has the most scalable and the most flexible way to publish those events and have an event choreography. Event orchestration, on the other hand, involves a centralized event broker. And what the centralized event broker, as I said, is you use another service or another part, a component in your architecture, whose sole responsibility is to control the order in which these microservices are being executed. Most commonly, you will see that this pattern is used when you have microservices which are within the same domain. As I showed you in the other example, a previous example for payment processing in case of an orchestration, and we will see it in a demo as well, where once you get a certain event, you may have to go through multiple systems and look at multiple sort of components in your workflow before you generate a final event. And that is what orchestration allows you to do. It is easier to visualize and you're able to control the order. And it is typically used for orchestrating events within the business domain. So as I said, that across domains, you would typically want to use choreography, but if you're trying to coordinate or rather orchestrate microservices within the same business domain, orchestration gives you that flexibility. If you're using an orchestrator, you can not only use asynchronous communication, but also you can use synchronous API call as well. You have that flexibility. In AWS, we have a service called AWS Step Functions, which basically gives you that capability to have a cloud-native service for orchestrating microservices. It works across over 200 services integrations that you have. You can invoke, for example, a Kubernetes job. You can have a Lambda function being invoked and you can orchestrate them together. So we have a demo for a managed care plan. So this basically shows how an EDA is in practice or how it is used in a real-world scenario. So for example, in this case, I'm showing one domain, which is like a provider management. So if folks are coming from healthcare background, this may resonate much better. Let's say you have a doctor or a nurse who's publishing their availability in terms of the services they provide. So you have this domain provider management, where they're saying that they're published their availability, that I'm available at this time, and then you have something called as a patient care plan. So think of it as like an app that you may have on your phone, for example, and you simply go there and say that, hey, I want to see a doctor. This is the specialty I have. This is the language that I want my provider to speak. And these are the times that I have. I would like to schedule it. As you can imagine, this can be in two different subsystems altogether, and it gives you a classic way of building a system which is event-driven or how events may be exchanged between the two to achieve a certain business functionality. So the architecture that we've used for that is all based on Kubernetes, and it uses most of the components here are open source. We use Spring Boot as the framework on which the core functionality, business functionality is built for creating the APIs. And then we use Kafka to act as that event bus through which these events are exchanged. And once we publish those events and we need to create a workflow out of it, we use Argo workflow for that. So as you see, as I dive deeper into this architecture, there are different parts that we have around provider schedule, the care plan, and then the part where I show the event choreography, this is where you have the, as the bookings are made, for the same booking, there may be different states. For example, when a provider says that okay, I'm available at this time, you have the care plan which says I have booked this particular slot for my appointment, now the patient may decide that I want to cancel it because something else has come up. Once a patient does the cancellation, the state of that event now changes, but it's published on that same topic. So you see here what is called the care plan bookings topic. And based on this attribute of that particular event, be it confirmed or canceled, you have different handlers. So this is how you basically implement what is called as like a choreography, where the events are being published on the same topic or the same event bus, but based on the attributes on that particular event, you have different consumers which are consuming it, processing it, and maybe submitting it back or sending it back on the same event bus. And then if you dive deeper, the other use case where orchestration makes the most sense is basically when an event or a booking has been confirmed, you may want certain resources for that booking. In other words, let's say this is at a clinic, you will want to schedule personnel for it. For example, the nurses, you may want to have a room booked for it. So, and those may be different steps in a workflow. So this is where an event orchestrator comes in and we use Argo workflow for that to execute the different steps in that particular business processing. So for the demo, I'll just open up, I hope it's visible. So this demo will walk you through some of the pieces of this particular solution that we have built. So this is, as I said, we used data on EKS. So if you are familiar with data on EKS, this is basically our EKS blueprints where we have taken a lot of data platforms and show you how to run them on EKS. So this is the Terraform scripts that you, templates that you get with it. And as you can see here, the Kafka and the Argo workflows are the add-ons that we have used here to deploy that both Kafka as well as Argo workflow on EKS. So this shows you the Kafka cluster. It uses that Strimzy operator to deploy that. And then we will, I'll show you next to the, the Argo workflow piece of it. Yeah. So as you can see here, these are, this is the event bus that's there. We use the default event bus. And then use Kafka event source for it. And then the workflow is that, is the sensor that we have, which basically gets triggered. And going through the code, this is basically the Spring Boot application that I was talking about, which shows that business functionality we have around booking an appointment. It uses a Postgres database to store these bookings and the availability of the providers. And these are the, the deployment artifacts that you have, as you can see here, it's using the same Docker images that we have for these different components. So next, I actually show you how it exactly works. So as I mentioned here, this is where we are invoking the API to submit like an availability. So as you can see, the provider says that he or she is available from this time to this time, then counter type is on site. What it means is that whether you're doing a virtual visit or in person. So this has now created the availability for that provider. Now, if you go to the get all provider availability, you see the host URL has changed. What that means is that I'm invoking it on the care plan site to see the availability that was just posted in the previous API call. Mind you, these are two different microservices that I showed you earlier. Now, this is where I'm creating my availability. Sorry, my booking. So I take that availability ID and I say that I want to book an appointment. I'm waiting for confirmation. If you remember from the previous picture, it sends an event to get the confirmation from the provider side. Now, if I go and check the booking that I just made, whether it has been confirmed or not, you'll see that this is now showing as confirmed and the payment mode is showing that it's an insurance pay or versus a self pay. Similarly, as soon as the confirmation has happened, this is where our Argo workflow will get invoked, which basically shows the different steps, post confirmation for handling the resources in an actual provider setting. So once I take the token, go into my Argo workflow console, and this is where you see that there is a workflow that just got executed the one 50 seconds ago. Just come to it. There you go. So this is the one. And it then shows you the different steps. So for example, this is the input. This is the event that I just received with respect to confirmation. It's making a room reservation that is needed and then sending a confirmation to the patient saying that, hey, your booking has been confirmed and this is the clinic where you need to show up and so on. So with that, let's get into some of the deployment pieces of this and I will hand it over to Carlos. Thank you. So switching hats in here, literally. So I'm going to switch this hat from developer. I have one here that says platform engineer. And is there any platform engineers here? Yeah, app developer can rock. So two components that we talked in here, we mentioned DataNICS, but DataNICS is a space, we work with the community on data on Kubernetes to say working group, people from our team, working there, how to run best, how to run the best way to run the stateful workloads in Kubernetes. In AWS, for example, we have managed MSK, Amazon. MSK, we say manage Apache Kafka where you say give me a Kafka service and it give you just the broker URLs. If you're familiar with Kafka, basically that's the only thing that you worry about. When you're deploying Kafka on Kubernetes, there's different aspects of resiliency and a scalability you have to take into account. And one of them is when you deploy Kubernetes, you want to deploy in different failure domains. In the cloud, you will have two availability zones and these are like, for example, different buildings if it's on premises and things like that. And the brokers and zookeeper, they need to be close to each other because there'll be a lot of communication between them and you want to have them in the same availability zone and not cross over. The other aspect is, Missou mentioned the operator. So using an operator to deploy makes this setup more scalable in terms of scaling the configuration of them. Using StreamZ is very recommended to run Kafka we use in our patterns. And the other aspect is having the consumers, right? The microservices that will consume from these stateful workloads, right? And this will be your producers and consumers. They could be in the same cluster, but most likely you want them in a different cluster, but close to each other. So you don't have, you have low latency between them. Remember that these are like TCP connections when you're talking about Kafka where you have producer producing events into Kafka topics and then consumers taking out those messages that are sitting in partitions in the brokers. So that's basically, we have different patterns there if you're running on EKS or data on Kubernetes, there's different patterns but at least for us, we have a project called the EKS Blueprints where you can deploy a Kafka and then get it up and running and we have helped a lot of users and users running Kafka efficiently on Kubernetes. So let's come up a few of the tips of running Kafka in production. If you have run Kafka in production, it takes more than that, but at the high level, this is a good starting point of what will be the aspects of when you get started deploying Kafka and Kubernetes. The second one is Argo workflows, but like we in the demo, we mentioned Argo events and usually they go together because that's the event source for Argo workflows and Argo events has a sensor for Argo workflows. Basically it instantiates a workflow. So in terms of scalability and scaling these Argo events, talking about Argo events, there's two aspect on it. It's the event source which in our demo was Kafka but it can be a GitHub pull request event like some type of events and there's a couple of them there. They scale by deployment so there's not that much problems of scaling them. It's just to have a lot of deployments or pods handling. The one that will be your bottleneck or what you need to be more concerned about is using the event bus and in Argo workflow, in Argo events, there's one that is the one that we use in the demo which is using NATs. It's the name, we call it the native. There's also JetStreams which is the next best one from that and then you have, not saying that it's the best one but then we have folks using Kafka because they are using Kafka anyway so they want to reuse Kafka for the event bus and that will be one way of gaining that scalability resiliency from Kafka that you already deployed with StreamZ. When it comes to Argo workflow, Argo workflow where you have to watch out for scalability is the Argo workflow controller. The Argo workflow controller, the way it works, it creates, when you create an Argo workflow, that's an instance of a running job, you can say. And then any status that is on that workflow or inputs or outputs are safe on the CR. So you're running back to the HCD and as you get a thousand, 2000, like that many CRs, the Argo workflow controller has scalability problems. So you need to start sharding manually on which workflows match to each controller and you can have multiple controllers. But we started a SIG in Argo called the SIG Scalability between, we have a SIG Scalability where we are looking into helping writing the control or scaling Argo CD and Argo workflows. And to finish, as Argo workflows can run jobs in parallels on DIACs, the number of workflows that you run in parallel at the same time basically is how many posts you can run on an EKS cluster and it has a garbage collection into it. In terms of scaling Argo workflows, you have to take into account the different type of storage. So if you have a thermal kind of storage, you go at the bottom like using SSD, the local storage of the node, where you have more resiliency and more better data for resiliency will be S3. And then between them, there's different aspects of what storage to use. That will be also affecting your scalability of Argo workflows. And with that, we want to say, give it a try, this is the URL for the example out of the box.  You can use things like the GitHub spreadsheet if you want to deploy the hem charts and all these and all the patterns are under the data on EKS where we have patterns for Jupyter Hub, Spark, Ray, all these stateful workloads to run in Kubernetes. They're kind of like complex to run but we have a patterns in there that people are using and sharing and open pull request is an open source project, the data on EKS. So with that, I think we're done. Thank you so much. Thank you.


theCUBE presents UiPath, Forward Five, brought to you by UiPath. Welcome back to UiPath, Forward Five, you're watching theCUBE's wall-to-wall coverage. This is day one, Dave Vellante, with my co-host, Dave Nicholson. We're taking RPA to intelligence automation. We're going from point tools to platforms. Mirage Matar is here, he's the director of intelligent automation at VMware. Yes, VMware, we're not going to talk about vSphere, Aria, well maybe we are, but he's joined by Thomas Stoker, who's the principal product manager at UiPath. And we're going to talk about testing automation, automating the testing process. It's a new sort of big vector in the whole RPA, automation space, gentlemen, welcome to theCUBE, good to see you. Thank you very much. Thank you. So, Neeraj, as we're saying, Dave and I, you know, really like VMware was half our lives for a long time, but we're going to flip it a little bit, talk about sort of some of the inside baseball, talk about your role and how you're applying automation at VMware. Absolutely, so as part of us really running the intelligent automation program at VMware, we have a quite matured COE for last four to five years, we've been doing this automation across the enterprise. So, what we have really done is, you know, over 45 different business functions, we've really automated quite a lot different processes and tasks on that. So, as part of my role, I'm really responsible for making sure that we are bringing in the best practices, making sure that we are ready to scale across the enterprise, but at the same time, how quickly we are able to deliver the value of this automation to our business users as well.  What's your area of, main area of focus? The main area of the UIPAV test suite. For your role? For my role is the RPA testing. So, meaning testing RPA workflows themselves. And the reason is, RPA is matured over the last three years, we see that and it has adopted a lot of best practices from the software development area. So, what we see is RPA now becomes business critical. It's part of the main core business process and operation and testing it just makes sense. You have to continuously monitor and continuously test your automation to make sure it does not break in production. Okay, so and you have a specific product for this, is it a feature, is it a module? So, RPA testing over the UIPAV test suite, as the name suggests, it's a suite of products. It's actually part of the existing platform. So, we use Orchestrator, which is the distribution engine. We use Studio, which is our IDE to create automation. On top of that, we build a new component, which is called the UIPAV test manager. And this is a kind of analytics and management platform where you have an oversight on what happened, what went wrong, and what is the reason for automation to break. Okay, and so Niraj, you're testing your robot code. Correct. Right? Correct. You're looking for what? Governance, security, quality, efficiency, what are the things you're looking for? It's actually all of those, but our main goal to really start this was two front, right, so we were really looking at how do we deliver at a speed with the quality which we can really maintain and sustain for a longer period, right? So, to improve our quality of delivery at a speed of delivery which we can do it. So, the way we look at testing automation is not just as an independent entity. We look at this as a pipeline of a continuous improvement for us, right? So, how it is called in industry as a CI CD pipeline. So, testing automation is one of the key component of that, but the way we were able to deliver on the speed is to really have that end to an automation done for us to also from developers to production in using that pipeline and our testing is one piece of that. And the way we were able to also improve on the quality of delivery is to really have automated way of doing the code reviews, automated way of doing the testing using this platform as well, and then, you know, how you go through end to end for that purpose. Thomas, when I hear testing robots, I don't care if it's code or actual robots, it's terrifying, it's terrifying. Okay, great. You have some test suite that says, whoa, yeah, we've looked at the code. Why is that terrifying? It's terrifying because if you have to let it interact with actual live systems in some way, the only way to know if it's going to break something is either you let it loose or you have some sort of sandbox where, I mean, what do you do? Are you taking clones of environments and running actual tests against them? I mean, it's like testing disaster recovery in the old days. Imagine, imagine. So, we are actually not running any testing in the production live environment. The way we build this actually to do a testing in the separate test environment on that as well by using very specific test data from business, which we call that as a golden copy of that test data because we want to use that data for months and years to come. So, not touching any production environments or things. All right, because you can imagine. Yeah, absolutely. Yeah, it's like, oh yeah, we've created a robot. It changes baby diapers. Let's go ahead and test it on these babies. Yeah, that's right. I don't think so. What's the, does it matter if there's a delta between the test data and the production data? How big is that delta? How do you manage that? It does matter. And that's where actually that whole angle of how much you can in real life can test, right? So, there are cases where you would have, even in our cases where the production data might be slightly different than the test data itself. So, the whole effort goes into making sure that the test data which we are preparing here is as close to the production data itself, right? It may not be 100% close, but that's the sort of boundary or risk you may have to take. Okay, so you're snapshotting that, moving it over, a little v-motion. Yeah. Yeah. Okay, so, do you do this for citizen developers as well or is it you guys pretty much center of excellence writing all the bots? No, right now we are doing only for the unattended, the COE-driven bots only at this point of time. What are your thoughts on the future? Because I can see some really sloppy citizen coders. Yeah. Yeah, so as part of our governance, which we are trying to build for our citizen developers as well, there is a really similar consideration for that as well. But for us we have really not gone that far to build that sort of automation right now. Narrowly, just if we talk about testing, what's the business impact been on the testing? And I'm interested in overall, but the overall platform, but specifically for the testing, when did you start implementing that and what has been the business benefit? So the benefit is really on the speed of the delivery, which means that we are able to actually deliver more projects and more automation as well. So since we adopted that, we have seen our improvement or speed is around 15%, right? So 15% better speed than previously. What we have also seen is that our success rate of our transactions in production environment has gone to 96% success rate, which is, again, there is a direct implication on business on that point of view that there's no more manual exception or manual interaction is required for those failure scenarios. So 15% better speed at what? At implementing the bots and actually writing code? And to end, yes. So from building the code to test that code, able to approve that and then deploy that into the production environment after testing it, this really has improved by 15%. Okay, and what business processes outside of sort of testing have you sort of attacked with the platform? Can you talk to that? The business process is outside of testing. You mean the one which we are not testing ourselves? Yeah, so just the UI path platform, is it exclusively for testing? This testing is exclusively for the UI path bots which we have built, right? So we have some 400 plus automations of UI path bots, so it's meant exclusively for that. But are you using UI path in any other ways? No, not at all. Okay, okay, interesting. So you started with testing? No, we started by building the bots. So we already had roughly 400 bots in production. When we came with the testing automation, that's when we started looking at it and now building that whole testing platform. What are those other bots doing? Let me ask it that way. There's quite a lot. We have many bots in finance, in auto management, HR, legal, IT, there's a lot of automations which are out there. As I'm saying, there's more than 400 automations out there. So it's across the enterprise on that. Thomas, both of you have a view on this, but Thomas's views are probably wider across other instances. What are the most common things that are revealed in tests that indicate something needs to be fixed? So think of a test failure, an error. What are the most common things that happen? So when we started with building our product, we conducted a survey among our customers and without a surprise, the main reason why automation breaks is change. And the problem here is RPA is a controlled process, a controlled workflow, but it runs in an uncontrollable environment. So typically RPA is developed by a COE. Those are business and automation experts, but they operate in an environment that's driven by new patches, new application changes rolled out by IT. And that's the main challenge here. You cannot control that. And so far, if you do not proactively test what happens is you catch an issue in production when it already breaks, right? That's reactive, that leads to maintenance, to unplanned maintenance actually. And that was the goal right from the start from the test suite to support our customers here and go over to proactive maintenance, meaning testing before and finding those issues before the heat production. So I'm, oh good, please. Yeah, yeah, yeah. So I'm still not clear on, so you just gave a perfect example, changes in the environment. Yeah. Those changes are happening in the production environment. The robot that was happily doing its automation stuff before, everyone was happy with it. Change happens, robot breaks. Okay. You're saying you test before changes are implemented to see if those changes will break the robot. Yeah. Okay. How do you expose those changes that are going to be in a production environment to the robot? You must have a, is that part of the test environment? Does that mean that you have to have what? Fully running instances of like an ERP system? You know, a clone of an environment? How do you test that without having the live robot against the production environment? I think there's no big difference to stand that software testing. Okay. The interesting thing is the change actually happens earlier. You are affected on production side with it, but the change happens on IT side or on death side. So you typically would test in a test environment that's similar to your production environment or probably in IT in a pre-proud environment. And the test itself is simply running your workflow that you want to test, but mock away any dependencies you don't want to invoke. You don't want to send a letter to a customer in a test environment, right? And then you verify that the result is what you actually expect. And as soon as this is not the case, you will be notified, you will have a result, a failed result, and you can act before it breaks. So you can fix it, redeploy to production, and you should be good now. The main emphasis at VMware is testing your bots, correct? Testing your bots, yes. Can I apply this to testing other software code? Yeah, you can technically actually, and Thomas can speak better than me on that, to any software for that matter, but we have really not explored that aspect of it. You guys have pretty good coders. But good engineers at VMware. But no, seriously, Thomas, what's that market looking like? Is that taking off? Are you applying this capability or are customers applying it for just more broadly testing software? Absolutely. So our goal was we want to test RPA and the application it relies on. So that includes RPA testing as well as application testing. The main difference is typical functional application testing is a black box testing. So you don't know the inner implementation of that application. And it works out pretty well. The big opportunity that we have is not isolated, not isolated testing, isolated RPA, but we talk about convergence of automation. So what we offer our customers is one automation platform. You create automation not redundantly in different departments, but created once, probably for testing then you reuse it for RPA. So that suddenly helps your test engineers to move from a pure cost center to a value center. How unique is this capability in the industry relative to your competition? And what capabilities do you have that are differentiators from the folks that we all know you're competing with? So the big advantage is the power of the entire platform that we have with your iPath. So we didn't start from scratch. We have that great automation layer. We have that great distribution layer. We have all that AI capabilities that so far were used for RPA. We can reuse them, repurpose them for testing and that really differentiates us from the competition. Thomas, I detect a hint of an accent. Is it German or? It's actually Austrian. It's Austrian, well, so. You know, don't compare us with Germans. I understand. Hi, German, is that the proper, is that what Spoken in Austria, is that? Yes, it is. So, point being? Point being exactly as I drift off. Point being generally German is considered to be a very, very precise language with very specific words. It's very easy to be confused about between the difference between two things. Automation testing and automating testing. Yes. Because in this case, what you are testing are automations. That's what you're talking about. You're not talking about the automation of testing, correct? Well, we talk about... And that's got to be confusing when you go to translate that into... But isn't it both? 50 other languages. Is it both? It actually is both. And there's something we're exploring right now which is even the next step, the next layer, which is autonomous testing. So, so far, you had an expert, an automation expert, creating the automation once, and it would be rerun over and over again. What we are now exploring is together with the university to autonomously test, meaning a bot explores your application and the test and finds issues completely autonomously. So, autonomous testing of automation. It's getting more and more complicated. It's getting more clear by the minute. Sorry about that. Alright, Niraj, last question is, where do you want to take this? What's your vision for VMware in the context of automation? Sure, so I think the first and the foremost thing for us is to really make it more mainstream for our automation developer itself. What I mean by that is, is to really... So there is a shift now, how we engage with our business users and SMEs. And as I said previously, they used to actually test it manually. Now the conversation changes that, hey, can you tell us what test cases you want? What you want us to test in an automated measure? Can you give us the test data for that so that we can keep on testing in a continuous manner for the months and years to come down, right? The other part of the test changes is that, hey, it used to take eight weeks for us to build, but now it's going to take nine weeks because we're going to spend an extra week just to automate that as well. But it's going to help you in the long run and that's the conversation. So to really make it a much more mainstream and then say that out of all these kinds of automation and bots which we are building, so we're not looking to have a test automation for every single bot which we are building. So we need to have a way to choose where their value is. Is it the quarter end processing one? Is it the most business critical one? Or is it the one where we are expecting a frequent changes, right? That's where the value of the testing is. So really bring that as a part of our whole process and then, you know, go ahead. You're still fine doing that, great. Guys, thanks so much. This has been a really interesting conversation. I've been waiting to talk to a real-life customer about testing and automation testing. Appreciate your time. Thank you very much. Thanks for having us. All right, thank you for watching. Keep it right there. Dave Nicholson and I went back right to this short break. This is day one of theCUBE's coverage of UiPath Forward 5. Be right back after this short break.