I'll let you say your name unless you want me to introduce you to it. Go ahead. Okay. Okay. Yep. Got a few people coming in. All right, everybody. Let's get started. See a couple of people coming in. Now, I just want you to know you don't have to take pictures of these slides because they are all available on the internet right now for download. Or, of course, this session's being recorded so you can watch it later. I'm Doug Soltes. I work with SwissDAC. This is Clay Gerard. He's one of the core Swift developers. And today, we're going to advance you further into the Swift realm. So this is Swift 102 beyond CRUD. Now, we've got a quick agenda. And today, this conflicts a little bit with Summit.  There's certainly going to be Swift API in here. But we decided we'd thrown a little bit for operators. There's a lot of new stuff in the Mintaka release, 2.7. Clay is going to be talking to that. And in case you have not seen us talk before, and all this stuff is too new and too complex or whatever, well, guess what? All the OpenStack videos are available online. Clay and I have spoken before in Tokyo. We've spoken in Vancouver, Paris, and other summits, as well as other people in the community. So Christian has spoken. He's given you how to build Swift web apps, how to build Swift middleware. And so if you need any getting up to speed, this is the best way to do it. So without further ado, I'm going to let Clay take it over and talk about what's new for operators. There's been some new talks given this summit as well. So when we do this again in Barcelona, we'll have even more stuff. Yeah, so Doug and I were talking about what we want to go over in this cycle at this summit. And I was looking back at what I'd been working on, both upstream and Swift, and at Swift Stack trying to operationalize Swift for folks. And I really want to talk about some operator features. Yeah. Because we've done some stuff on the back end that has been some good improvements. And that's what I've been working on, so that's what I want to talk about. What happened there? I don't know how to use these things. I don't know. Clay, I told you it's just up and down. Up and down. It should be simple enough. Yeah, luckily they don't let me run slides in my day job. I just work on distributed source systems. There we go. See if they're up down. There we go. So one of the changes that we were working on, there was another core developer, Matt Oliver, and a few of us had been talking about this idea, was a way that we can improve some of the operations, the speeds of some operations under failure. Swift is a distributed system. It takes advantage of the hardware that you can give it, and it's going to be a fast storage system as you need it. But when things fail, there's always opportunities for timeouts and things are not going to go well. So as the storage system itself is degraded, you can, you know, that some of the clients will be observable. And it's our job working on the system to not only work around those failures and make sure that we can maintain availability, but also to optimize in those failure cases to make sure that we're still remaining as fast and performant as possible. So this is a new feature. It just came out in the last OpenStack release. It is currently not on by default in the default configuration. So as you're doing your review of the changelog and getting prepared to deploy your Swift clusters, you know, make sure that you are going and enabling that. In Swift Stack, we did some internal testing and on all of our new clusters going forward, we have some numbers based off work that Doug has done that we're going to be, you know, having this on going forward because it is an awesome optimization. But if you need to turn it on, it's just a couple of config options. And the goal is to improve first byte latencies under failure. So let's look at a little bit how that works. And again, I apologize. I'm going to get down into the weeds because I've been thinking about this stuff for the past release. Now, this is a nice, good failure condition in distributed systems, high available systems we love to think about when things are really broken. So here we have a triple replicated system. There's three primary nodes that are holding this object data. And in the first two attempts to reach that data, the node timed out. This is, you know, we'll think of it as some sort of error. It wasn't a quick response where the drive had been unmounted or some of the, one of the other operational monitoring procedures had sort of marked that offline. This has been a discovered failure that just sort of just happened. Network congestion, there's some sort of a disk IO that's hammering that particular device. And so from the proxy that's organizing all of these object servers and talking to them, when it goes to make a request to one of the nodes, it can't wait forever for that guy. It doesn't understand, you know, we don't know what's going on with that guy's disk, write, queue depth or whatever. So if that node doesn't respond, we're going to time that out and move on to the next one. So here we've got a real good failure. After that, the second node was also failed. So finally we fail over to the third and in his situation it was a very quick short response. He was able to respond and then we can return to the client. So we're working around all these failures, but the client can observe this. This is latency on the order of seconds potentially and it can be observable. So with concurrent gets equals true, we take a very similar sort of failure mode. We have a couple of nodes that are timing out, just we can't wait on them any longer. But what we do is we stagger when we start the request, this little waterfall of when we push out the request. So even though we haven't given up on that first request, it may yet still come back. We're going to sort of pre-seed the request queue a little bit and send out another request now after a little bit and see just sort of hedging our bet that maybe that second node is going to be coming back. So in the same failure condition, we've actually lowered the time to first byte that the client improves by 2x. So this is sort of a perfect storm that this particular situation is in some respect what it was built for. But when we were implementing this feature and thinking about how we can leverage concurrent gets, we started thinking about all the different kinds of failure modes that we might be seeing. So taking a very similar setting, you turn on concurrent gets true, you have a sort of few hundred millisecond sort of waterfall request pattern. If you have a little bit different failure mode, where the first node times out, the second node is slow. It's still going to, you know, there's still going to be some difference in time, but that second node is going to be slow. Eventually we get back to response, but we haven't gotten it back before the time whenever we wanted to, again, kind of hedging our bets, send out that next request. So, you know, the interesting thing that you see here is that we respond as soon as we get back that first successful request. The third request that actually gets sent out, that's going to end up, you know, piling into the disk queue and doing that read from the object server's perspective, it needs to service this request. It just doesn't happen as fast, you know, any faster than the other node that got started a little bit ahead of it. So in some ways you can think of this as useless IO. It's not perfect in efficiency, but not having the ability to see into the future. It is sort of perfect in implementation. Well, you know, as good as we can do. So here's a different failure mode. This is a great one. We have total primary storage failure. So this is something where, you know, you might see under a cluster, we're going to talk about these a little bit later, we've got disk getting full, we've had a massive partition in the network, and none of the primary storage devices for an object or down. These are typically going to be in separate zones. So this is some sort of systemic failure going on in the cluster. Very exciting stuff. We, you know, we're going to do our waterfall pattern and we're going to hedge those bets, getting out all of those requests. And then we're going to go to a handoff node. If this is a new write coming in and the previous write had also observed a similar sort of failure pattern, it wouldn't have been able to write to the primary nodes and it would have written that data into a handoff location, which the consistency engine would repair later. So there's still a good shot that, you know, we can find the data that we're looking for on one of the handoff nodes. So there's a number of requests up to two actual replica count that we'll sort of dig around into the cluster to find a viable disk that we can write to for writes or potentially based off this stable ordering, assuming that a write may have come in through that sort of degraded state, we can find it in the way that it goes. So we hold off on the last request. We never have more than replica count requests in the queue. So it's not a perfect waterfall. You don't see immediately after, you know, your concurrency timeout setting of that third request that the next one gets started immediately. We actually want to wait until one of those kind of falls back into the queue so that we never have too many outstanding requests because this is a pretty exceptional situation where you have a total primary failure. And we'll look at some of the other cases here and you see a little bit why this makes sense. We already talked about that extra request when you already end up serving the one that you started earlier can be a little bit of wasted IO. So this is concurrent. It gets true. The timeout's a little bit smaller. It's a tunable so you can kind of shape that waterfall pattern based off of your average time to first buy it, your 95th percentiles. Doug's going to look into a little bit of that. So we have some slow responses and we don't start that fourth request doesn't get started right there along that waterfall. It would not happen in this case because the second request that we started was able to get served. So there's no extra pending IO that most of the time is not going to be useful. And here's another similar situation where concurrency timeout is very low. We're firing out multiple concurrent requests. This can introduce some extra IO into your cluster. This is not the default. It's just meant to be an example. But if you shoot them all three basically out at the same time, the first one to respond is going to be the one that we come back with. But we never have, it doesn't just keep going out through all that 2x replica count into all the handoffs. We'll never have more than your replica count request pending at a time. And this is some of the work that we did. It was we were working on this object and working on this new feature and figuring out how to make it work best for folks. Yes, I'd be glad to talk about this.  So it doesn't have to be that you're using this for a node or a drive that's failed. This can actually accelerate your workloads. And so let's talk about the workload that we ran. We ran through SS Bench and we had a workload where we have small objects and we have large objects. And it's a very common workload that we see when we're speaking genomics or media and entertainment, right?  And what's happening is that when I'm writing those big files or when I'm reading those big files, my tiny requests, my small requests can get queued up behind that on a drive. And this is a perfect case to, in a healthy cluster, speed up your first time to bite with your... Quality of service. Yeah, with your searches. Our tiny objects are 4K to 32K in this example. Our big objects are 50 megs to 128 megs. We did 90% tiny objects, 10% large objects. And within tiny objects and large objects, we were doing 50-50 read-write. And so here's what this looks like. With concurrent time-offs off, the way that you'd have it in Swift 2.6, we're seeing that on a tiny object, on average, we're getting about 174 milliseconds. So again, this is a small cluster. It's three nodes, 12 drives per node. It's something you'd see kind of typically as a small Swift deploy. And so we're hammering it. I think I was doing 150. It was in my SS bench on the previous slide. 150 concurrent requests, trying to hit it as fast as we can. And we're getting an average time-out of 174 milliseconds. Our 95th percentile, right? So if you're looking at the worst results, we're getting, yeah, we're getting 698 milliseconds. And so once we turned this on, the first thing I did was I set it to 200 milliseconds. So again, if I'm queued up for more than 200 milliseconds, it does that concurrent firing. And you can see I got a 74% improvement in my average, and I got 134% improvement in my worst cases. And if you reduce this all the way down to 0 milliseconds, where it's firing all the requests simultaneously, which I think is a bad idea. We'll talk about that in a second. The benefit just gets better and better and better. The reason I'm going to say that asterisks here do not set this below, or I would not recommend you set this below 50 milliseconds, is because the IO, right? So I had 36 drives, and when I set that that low, I'm doing all these, as Clay called them, useless requests. And I actually started getting a couple of errors on the drives, where my request queue, these are SATA drives, are getting a little too long. I think during that 0 millisecond run, I got nine request time-outs, 503s, right? And that's not something you want to see in a healthy cluster. So if you're going to play with this in a cluster, you know, start high 200 milliseconds, go down to 100 milliseconds, work it that way. And the other interesting thing here is it even helped the first bytes for large objects. So when you think about a large object, there is a statistical case where a large object is going to get queued up behind another large object, and it helps with that, too. Now, if you ask, hey, Doug, why do you have in this chart the tiny object's last byte latency and the big object's first byte latency, it's because, honestly, there's about one millisecond difference with a small object between first and last byte latency, so you might as well leave it the same. Here's the same thing as a graph. And so if you look at that red and orange line, that's my 95th percentile for the large objects and for the tiny objects. The two lines at the bottom are my average latency for the small and large objects. And you can see you get a big benefit even just turning it on to 200 milliseconds. And then you really get diminishing returns up to zero milliseconds, which is why I'm really going to advocate that you don't blow up a production cluster and set it at zero right off the back. I really like this graph because it sort of validates when we were working on it and we were working through these situations and trying it out in various lab and dev environments. This is exactly what we wanted to see. We understood that there was a range here and that there was some trade-offs. And so that diminishing returns graph is exactly what we were expecting with it dealing with. Doug did all the hard work. Well, you programmed it, so. Okay. Well, so that was Concurrent Gits. Turn it on. It's faster. It's great. I want to talk about another operational thing that I've been working on. I don't know. Failure is still an interesting thing. And in this particular case, I want to talk about a specific failure, the 507. I've been involved over this last cycle in a number of engagements it feels like where their growth curves look like this. Everything is going great. We're growing. We're adding data to the cluster. They've got new use cases coming on from internal partners and stuff that want to transfer expensive storage onto the Swift storage, API-based commodity storage. And suddenly someone in some internal team finds, man, this is working really well. I want to go back, fill a bunch of old data or I want to start applying this to new applications and all of a sudden the growth curve. Because it's just API storage. It feels like it's infinite unless you're the guy operating the cluster and then you actually have to plan for these capacities. So everyone can sort of imagine exactly what happens in the next few ticks on this graph. But what you may not necessarily want to ever have first-hand experience with is what happens if that growth curve does outpace your ability to manage your cluster and bring in new nodes and plan for that capacity and get them online. So I'm going to talk a little bit sort of through the process what happens when Swift runs out of storage. So first of all, the object server in attempting to write down a object to the disk if there's not enough space to write that disk to write that data to the disk it returns immediately in 507 HTTP status response to the proxy which the proxy will observe and it will error limit that particular device and it won't try to reform future rights to it. So back when we were talking about that primary storage failure, one way that you could get into that case where new objects are being written entirely on handoff nodes instead of on their primary nodes is if all of the primary disks are full. But Swift will dig through up to two extra replica count all these handoffs and it works its way around the ring in every place that the partition could be placed in this stable pattern until it finds somewhere to store this disk. It will dig around for every last byte that it can possibly find so that it can write down that data and service that request. Swift is a system of turning disk space into HTTP 201 created status calls. It won't give up until it absolutely can't go on any further. But one of the interesting artifacts that comes out not in the right path but once you start observing these 507s which can get returned all the way out to the client. The client it could observe a 507 storage space not available because the cluster, the proxy despite its best effort cannot find anywhere else that it can fit these bytes. So then you're sort of left in this situation what do I do? What do I do about these 507s? So one of the first thing a lot of people want to think is okay I want to send a delete. I've got some old data, some test data, and I want to delete that. But people have talked about distributed systems doing deletes is sometimes harder than doing writes and the way that Swift has to implement that is really kind of a transaction. We have to write down that you wanted to delete this object. So particularly the place where things are under failure mode, we've got devices in different places, we still have to record that delete operation and in Swift that's implemented in Tombstone, a 0 byte file marker that will get written down to indicate the object was deleted and sometime after your consistency engine runs that as well can get reaped. But these Tombstone objects can get written on to handoff nodes not on the primary locations and generally in you know you think about down on the data structure first we have to write down that it needs to be deleted and then we can unlink the fat data file that actually has hundreds of megabytes of objects in it. So you have to write first and then you can unlink the primary data holder is over here but he can't be written to because he's out of space the Tombstones could actually end up getting recorded onto another device which will have to be replicated over, again it's a write, before it can get deleted. So a delete request may just create 0 byte files and not actually remove anything from the cluster. And of course while all this is going on you know replication is actively trying to repair this and any handoffs, any data that's been written out of place is going to be trying to get moved back to the primary modes, the primary devices where it's supposed to be stored. So Swift is doing what it's supposed to be doing but as an operator when you're in this situation you really are trying to figure out okay I need to come up with a better plan I gotta do something to actually deal with this situation and Swift provides a number of tools to operators so that they can make a clusterful situation, hope you never get into it, be essentially a non-issue so I talked about those clients getting those 507s, when you call the operator and say hey I got a 507 that is either a very calm conversation where he says it's no problem I'll have the new drives on later this afternoon and I'll enable merge to your replication mode or you can delete some old data or it's you know their hair is on fire and they're freaking out because maybe they are not aware of all the things you kind of gotta put together to make the system work in a healthy mode. So here's some stuff that you gotta think about. First tool that we have available is F Allocate Reserve and basically this just says that during a write I talked about how if there's not enough space to store that object it will return 507 to the proxy. F Allocate Reserve allows you to say I want to hold on to some additional space. Give me a few hundred megs in addition to the space of the object and if I don't have that plus go ahead and return the 507 find a different disk to put this on this one is near full holding a little bit back means that you will have room when a delete request comes in to write those tombstones directly over the data files on the primary locations where they're supposed to go to get things via the API to reduce some of your hold back storage space. Another thing that is a new feature F Allocate Reserve has kind of been in there for a while but the replication engine we talked about and we've learned and observed is you know it can fight against you even if the object server allows you to write something else over here the consistency engine is going to be wanting to put it back over here trying to repair that failure as it observed it. So with our sync modules per disk first of all it's just a great tuning it's a new way for people to kind of balance their I O so you can have per device concurrency count so you can sort of tune an individual drive level rather than server level you can tune it right to the drive so you can marry that closer to your your Q depth good default probably to be started for a lot of people have been running around 25 or so for a node but that means you could potentially have 25 connections going to a specific disk with our sync module per disk and the our sync module template that you can put in your object server config you can tune that such that if a single device already has four other nodes in the cluster trying to push a partition on to that device you just have to wait and come back around to it whenever slots available for you so it's better load balancing you know just to your disk I O and the advantage of having this means that you can also shut down those replicators at the per disk level and we'll look at that in the next slide the next thing that you need to think about is emergency replication mode there's a couple of tunables I kind of group them together because you're going to be using them together a lot when you're in a situation where you really need the consistency engine to be focusing on moving new data to fill up your new nodes there's a couple of tunables that you can use to sort of indicate that this is the primary work that I want you to be doing it's just a couple of seconds in the object replicator setting it's not it's not the default though because it does sort of it focuses work on one particular thing and the consistency engine normally has to take a larger view not only do I need to be moving data I also need to be repairing durability and the consistency of the processes so disabling our sync modules you know Swift stack implemented this as part of our node agent that already is sort of monitoring devices and alerting you if you have devices where they're getting too full and that sort of thing so it was easy for us to add this here everybody's got different ways of doing it there's been a number of Swift operators and we've all been sort of sharing okay how are you guys doing it how are you guys doing it and you know it's hard to be just prescriptive because everybody kind of has some of their own stuff built up but it's also nice to have examples so go to the just check it out I mean it's just a very short Python script you can see it sort of goes over all of your devices it does a stat if they don't have enough space that you want to have on them you disable our sync otherwise you make sure that our sync is enabled to disable our sync you just write down this little template file that takes that devices our sync module sets its mask connections to negative one which means that nobody can talk to that disk via our sync and of course if the disk is healthy or once it becomes healthy because even though it won't be receiving our sync connections it can still push data out be the our sync client you want to make sure that that configuration file has just been removed you just unlink it just get rid of it and if it's not there you know that's fine move on to the next guy so not necessarily that you would be running this script I'm sure you can come up with something more eloquent but that's the general idea mark the our sync disk module as map connections never want a negative one when it's full and pull it back off whenever it's ready to go now this allows sort of you see how it all plays together so the first step that you got to think about is a new rights coming in we're going to reserve some space on this full device here so the first three are our main primaries user rights to the proxy the proxy rights to the nodes the one that's full or getting full is going to return 507 and the proxy is going to write that data down somewhere else the next step that's going to be going on in the cluster is we're going to be trying to repair that right the primaries want to fix the fact that the other primary that's not holding the data should be and we are not going to let that happen we're going to disable those discs so that we can not like end up backfilling and working against ourselves because we know that what we've got coming the real solution for Swift is to add new capacity and when this new node comes online rebalance partitions are going to get moved around the emergency replication mode ensures that the primary work that the consistency engine is focusing on is pushing the data off of our full discs onto our filling discs and that is how you can make a full cluster kind of a non-issue awesome so I'll just speed through we got a couple of cool new things to talk about also so those are definitely the best ones the high ticket items that's where we like Clay talk about them so one thing that Swift has always been great at is having multi region right so you have a single name space you've got these links but they're highly latent this is something that eventual consistency totally helps with and yet we've gotten a bunch of support calls at Swiss stack where people haven't configured it right and we want to make sure the whole community knows how to configure Swift correctly for multi region so you set up a region in London Australia and in the Americas and the next thing you know customers are calling because they're getting high latency on their puts their gets on everything so the first thing is really obvious everybody probably already knows this you need to set up read affinity read affinity says that when I come to a proxy node in my region I should look first for an object in my region and not go across the wire now of course if it doesn't find it there because it hasn't replicated so far it's going to go across the wire now the second thing you probably already know is we should use right affinity right affinity is a double edged sword because it is saying that when I write data all three copies are going to go to the same region and then the replicators going to actually move them to the other regions again read affinity would still be able to access that if you're accessing it from another region where it hasn't replicated to yet however with right affinity you do have to worry because right now for this small moment in time I have all three copies in one region and if that region you know fell off the face the earth I don't have the same durability I normally have in Swift but what you probably don't know are the next two settings and some of these came out in the Tokyo release and we didn't have time to talk about them so you set up read affinity you set up right affinity and users are working and they're much happier now but every once in a while you get a user coming and they're going hey my connections are still really late and why on every put get etc well when you think about it the way we authenticate in OpenStack is with tokens and tokens are stored in Swift to cache and speed things up and that's in a memcache pool and if you have your memcache pool across all these regions there's a chance that I'm sitting in London and my token is being cached in Australia and that's not a very good thing and so what you want to do is you want to start taking your proxy servers you want to alter your proxy server dot com and you want to set up the proxy server to only use memcache servers within its own region and so now your customers are much happier but then every once in a while you get a customer coming to you and they go hey you know I've set up storage policies I'm writing to data to write to London and it's only writing to London my objects are just in London I'm not using the full you know multi region settings and yet I'm still highly latent what's going on and here's what you have to remember the objects system is able to use storage policies and storage policies if you don't know we'll talk about later but they allow you to control the durability of your data the geolocation of your data and the tiering of your data and so if you've picked the geolocation of your data to only be London well the account container which is our database layer does not abide by those storage policies it always replicates across the entire region and so there's a setting out there for async container updates where essentially what happens is the database in your region gets updated and the databases in the other regions get asynchronously updated by the object server and so your customer is always getting that fast 200 on their puts it really just affect you on puts and updates or deletes now so there's our hey this is how do you you should make multi region really work for you another quick thing that you want to talk about for yeah so this is something that we've been working on this is another long when the post operation something that Swift provides to perform metadata updates to objects that you have in the system the original implementation notated the transformation near the object down in a metafile next to the data file which will allow you to override or change some of the metadata that's associated with that object it can be useful if you're doing some sort of indexing you need to keep some metadata with the object but it didn't update the container listings which one of the things one of the pieces of metadata that they include is the content type and since there was no update to the database layer the container listing layer some pieces of metadata that you couldn't change on an object so later a new implementation that wanted to sort of make that a little bit more consistent was to transform the post operation into a server side copy which is exactly how S3 does metadata updates but it's cost intensive on the server itself it has to actually read all the bytes do any transformations that it wants to do the metadata in there and then store it back down so we were never really happy with this but post operations weren't a common thing in most workloads but in workloads where post was an important thing there was still a tunable the object post is copy which you could change it to false and you'd fall back to the old behavior with all of its restrictions so one of the things that we were working on in this cycle was unifying those two implementations so you get the same behaviors and expectations from the client side you can update whatever metadata that you want on the object but removing that side distraction of it actually being transformed into a server side copy and therefore being slower particularly on a large object you might want to upload hundreds of megabytes and then just do a quick little post request to add some additional metadata to it now you could have sent that metadata with the object itself but if you're calculating different kinds of checksums besides the md5 that Swift already keeps for you you may not want to know all that information until after you've read it and so you want to just have a little quick way to notate some information on the object after you've uploaded it so now when you set object post as copy to false in Swift 2.7 you get fast post which has all of the behaviors and all of the good stuff that you get from post as copy but with none of the bad stuff where you have to transfer a lot of bytes I would say you should use it as the default use it every time so the Swift community is very theory about we want operators to have time to migrate to new settings we don't want to change or make any surprises for anyone but this one of the sessions that we're talking about now is okay what's our deprecation cycle how can we get rid of post as copy there is just no downside for post as copy false alright so let's move our talk on a little bit to talk to the developers in the room right we've definitely spoken to you guys before and we want to tell you about new updates to the Swift API so the first one is last time Clay and I talked about ranged SLOs and that was a whole new feature in the Swift API one of the new things in the Swift API as well is that now by default SLOs can allow for a one byte segment so previously we had this slide up here and we said hey why would you want to use a ranged SLO well I've got three different objects these could even be SLOs and I want to make them into a new object and so I can use these ranges in my new object but the caveat was none of these ranges could be under a megabyte well guess what now they can now I do have one warning to throw out there and that's if you are going to use ranges under a megabyte by default now in 2.7 Swift we are going to rate limit that rate limiting works like this it says ranges under a meg we will only deliver one segment per second now of course your admin can change that you can change that to deliver five a second or you can actually set that minimum size from a meg down to half a meg but I did want everybody to know that this is now yet another feature this is actually the third time in a row that we've talked about new improvements or new features in the API for SLOs it's really been turned out kind of an interesting feature and it creates a very small little blob data that really didn't exist by comprising it of other things right another new thing out there so everybody knows that Swift can list files we can do markers we can do things like that and now you can do reverse listings and so the use case for this is generally log files right you're putting log files in they're segmented by date you do a listing and it's coming up in what you would think of as reverse order and versioning objects and so it's really simple when you do a query parameter you do question mark reverse equals true and guess what you will get your objects back in reverse order and if you need to know more about the query parameters in Swift John Dickinson the PTL gave a great talk back in Vancouver and if you look this up you will learn about markers prefixes, delimiters and everything else you could want to do with query parameters now I have a wag of my finger at a couple of the developers in the audience we talked almost two years about Swift announcing storage policies I spoke in Vancouver about how to implement that and put that you know query them put them into your programming they're an extremely powerful thing they allow you to determine the data durability as a end user or an app developer the geolocation in the tier of drives and I have seen very few applications on the market giving the end user the customer the ability to pick their storage policy and so what's happening is it's just going to default so back in my example earlier when I talked about multiple regions you need to expose that to the operator to the end user so if this is you know your backup program you're using backup or convolta or whatever there needs to be that drop down I took a quick screenshot of an application where when you create your container you know it gives you an option of all the storage policies out there and again this must be done at container creation you cannot do this the audit is go through your code find a place where you're creating a container and think about if that could be pointed at a cluster where there's actually different reasons why they might want to use it for storage policy and figure out a way to give that option to the other thing that developers have hit me with a ton is been gee object storage is really slow like I don't like it I use it and it's terrible and I'm going well I'm using it right and so one of the things we want to make as a again a performance knowledge transfer is that object storage is highly latent and what you need to do is not treat it like a filer or a block system where I do request and then wait for that to come back in another request another request when I work with developers and their application runs slow it's generally because they're doing one request at a time and there's a huge amount of latency and so the thing you need to do with object storage is to fire off a ton of streams of connections and generally when I work with anything I always start with 10 streams or 100 streams object storage is perfectly great at handling that and that's not something you normally would do with a filer because that would be a terrible idea and so I can show you right here that on both puts and gets if I have a single connection and these are 128k objects so kind of small objects I'm getting pretty poor performance but doing the same thing with 100 connections is giving me a 3040 objects improvement in my speed and my transfer rate and then if you combine that with next slide with larger object sizes you get even more advantage and so this is the same graph right the same information I just show you but instead of just having 128k objects I'm showing deep performance with 2 mega objects and you can see on the right with a 100 connections transferring data I think I was using get put as my benchmarking tool 2 mega objects on my get side I was almost maxing out a 10 gig pipe and this was again against that small system I did the other benchmarking with 3 nodes, 36 drives and I was limited to 10 gigs that was all my proxy had going you know in and out of it the last thing I want to bring up is some best practicing best practice around sharding and so what sharding well I've seen a bunch of developers treat swift as if every container is infinitely deep and I'll tell you why even Amazon S3 the containers not infinitely deep they've got a knowledge based on there and they go here's how you need to name your objects if you want this bucket to be infinitely deep it doesn't matter if it's swift or seph or anybody doing object there's no such thing as an infinitely deep container and so it's up to you as the developer to create multiple containers in fact when you think about it swift can have and each account could have millions of containers and we need you to code your application so that it doesn't try to dump everything in just one container and so there are a couple different ways to do this a number of applications I work with so people that do document management systems that they're already creating I give you a document it's called cat dot you know Excel and it's about all my my favorite cats and you put into document management system but you you rename the file you put it into a container and you've gotten a database that cat dot Excel equals you know this this new name if you're doing that it's fine to do what's called a fill and spill so you decide that you're going to maybe max out at a million objects in a container and remember you can always head a container to find out how many objects are in it and once you hit a certain amount you create a new one and a lot of document management systems have this already because they're used to juke boxing with DVDs and they go well when's the point whatever it is seven gigs of storage is in this this directory I'm going to spin off a new directory do the same thing for object storage now if you want to do something where operators or end users would actually be able to look in the system now you need to do categorical and so categorical is make it intuitive so if you have a log base system maybe you create a new container each day with the date and you put all the log files for the day and that maybe if you have a video surveillance system you do the same thing but you also append the camera name to it because again we can have millions of objects and it's much easier to clean up a container like that you want to take a directory and put a hundred thousand entries into a directory because windows would not have a good day and just because we can put 50 million objects in a container doesn't mean you should file sync and share same thing certainly build you know accounts maybe one for each user and allow them to instead directories have many containers or if you have to have one account and have containers for each end user when you think about the data model and lay it out based on something that makes sense that's normally the best way but you can also use consistent hashing it's just really important for everybody that we make the scale here's a quick something we ran in the lab this wasn't on that same system I spun up a couple instances so millions of entries in a single container down there and I think this goes up to like 50 million entries and so what I was doing is I was just testing the AC layer and so I started off getting a rate of around 300 puts per second on the database side right and as I kept putting more and more objects in it you can see that the database for the single container started to slow down and remember that in Swift every container has its own unique database which means if you had 10 containers it would be 10 fold the amount of performance now I was using an SSD drive certainly there are environments out there that are running account container of the database on a spinning drive and the spinning drive is going to have much worse results here I mean we've certainly worked with developers who put 100 million objects into a single container on a spinning disk but what you're going to see is this graph keep trending down towards zero and so at some point in time you're only going to be able to do 3-4 puts per second do yourself a service in charge the biggest containers are normally the ones that are filling up the fastest so you're really trying to maintain a high request rate and it's going to drop off after a point well Clay we are right on time if there are any questions hit us right now and if not I think they're going to kick us off stage feel free to catch us afterwards we have a question amazing talk guys the concurrent gets is really exciting just wondering what can be done in terms of improving put performance I know that we only return once all the replicas have been written right so is there anything that can be done there to optimize that path there yes and actually some stuff is in we didn't get to talk about all the great stuff that we built in the talk in so two seven so one of the things you get we have a post quorum time out you don't have to actually have all three replicas return success you just have to have a quorum of replicas plus a small time out you can tune that in essentially once two of three has responded success the proxy is going to ultimately respond success it doesn't absolutely have to wait on that third one to be completed because it already knows it does have enough to indicate success you wait a little bit longer because it helps things appear like more consistent congealed but that's a tunable and so you can bring that guy in if the final get response on a put request is some latency that you're looking at another thing that changed into seven one of the component one of the jobs that the object server has to do while rights are coming in is maintain some data directory data structures that help optimize replication has to invalidate the portion of the namespace that is going to need to be most most quickly evaluated by the consistency engine that was in line in the request right along with the F sync and potentially the synchronous container updates so you can tune the container updates down the data structure is actually now a much quicker lockless append-only file we just tell it that hey the next time that you evaluate this data structure you also need to apply these changes which have accumulated so yeah performance is also something we're working on so I would say that the biggest thing is if you look back at and this was in the Liberty release but we didn't get to talk about it the a sync container update so right now when I write an object I need quorum so if I'm making three copies two of them have to land on disk and within a reasonable time two of them have to be updated in the database layer and so saying that the database the account container asynchronously can be updated is going to allow as soon as it lands on disk for it to come back now depending on your database like we were just talking about that could be your bottleneck if you've got a container of 100 million objects in it certainly decoupling it allows even for the database to work more efficiently in a batch mode so take those region settings that I gave you and apply you know at least that one even if you don't have multi region the post quorum thing that's off by default like it's not is that like it's only controlled by its timeout setting so by default it may be the case that you observe that generally speaking that third response happens before the timeout would have kicked it out I believe it's something on the order of seconds so if you want to bring that in you really need to return as quickly as possible the async container updates is maybe the first tunable that you look at the post quorum timeout in the proxy server is going to be the next one that you want to make sure that you're bringing that stuff in as tight as you need it and the post quorum timeout in the proxy we've got the settings on a number of these slides if you look at them afterwards and again they're all available for download right now last thing I've always wondered about Swift and too lazy to check myself is between the proxy and your storage nodes does it do any kind of HTTP keep a lifestyle connection pooling like keeping connections open no most of that we leverage the kernel's TCP stack there's a number of prescribed TCP tunings that we want to have those sockets open I mean it kind of depends on the cluster layout how many different guys that you want to be connected to to the ports but it doesn't we don't do it we just tune it down at the socket kernel level there's some stuff there's a paper out from Intel on those tunings especially when you're doing a racial coding the network tunings have sort of carried us this far and it ends up just being a prioritization game of all the other places where we can really make a difference in our application code versus sort of optimizing what the Linux already do but if you hadn't seen those before I mean does that answer your question? Yeah, it does. Great, well thank you guys very much. Thank you.


My name is Narayan Raman, I am the author of an open source tool called Sahee and this was released in 2005, it was a test automation and web automation. About 3-4 years down the line I started a company called Tito Software that helps organizations with their automation process, okay. So we are a product company, we are not a services company, we are a product company, it is a product that we have called Sahee Pro and that actually we have been selling it to organizations which have faced specific problems with automation. Now what I am going to talk about today is not about the product itself, it is about some of the principles that have helped our customers to have automation which keeps pace with their product. What I mean by that is products evolve pretty fast especially nowadays, right, like you have all these processes around it, development has become much easier and the expectation of the business is that people deliver very quickly. Now when that happens you also have these cycles of releases which are quite short and when that happens you need automation also to be squeezed into those cycles. Now in fact you have, you need testing squeezed into those cycles. Automation is helping you to actually squeeze them there, okay. Now I know that a lot of you are, this is an agile conference, you may be thinking of Scrum, you may be thinking of how do I get it into this Kanban cycle or whatever that is. Now we like, my talk is not going to address that at all. What we are going to talk about is this on the basic main pretext, software exists because it enables a business, okay. Now this I kind of stress upon because we get carried away software for software sake kind of thing. But that real thing is software enables a business, okay and you need better quality of software to minimize the risk to business.  In dynamic businesses where the business is changing fast, where both software and business change. So in the sense that there may be business changes which means that you know the newer kind of payments banking has started in India. So that was not there before. So that is a new kind of business. The other thing is you have the same kind of business let us say a bank but now you have a mobile interface through which you want everybody to actually interact. So these are different ways by which like businesses and their implementations are dynamic and software quality systems should ensure continuity of such business. Whenever something new comes in a business it should be ensured that doesn't have it doesn't add any risk. In this talk we are talking going to talk about automated tests which ensure quality of software, okay. So why am I talking about all this? Because when we talk about automation we become tool centric in our head, we become framework centric in our head to set the context that you know this is not specific to any of those things, okay. While it will have examples it is not specific to any of those. So now I will jump into the little bit more technical thing, right. So when do you use automation in your, in your let's say how many of you do web automation in your organizations right now? Okay just before I go to that slide let me ask you like what problems are you facing right now? Just to actually if you can just shout out what problem you face, what is, what do you think is the problem? Why are you unable to keep pace with evolution? How do you feel we are keeping pace with evolution? No problem at all. How many of you are fine with the pace of your automation right now? Okay so like what problems are you facing? Do you have any words that crop up in your mind quickly? Sure, okay. So most people feel that you know this is what like we've realized when talking to people. When people think about automation failure they think about flaky tests, unreliability of their giving you results that you can believe on all the time. So a lot of times the suspicion of a failure lands on hey maybe we need to add some weights there, maybe we need to tweak that particular thing there, right. So a lot of times like automation is thought of at that level, okay. So let me just take that and we'll go to the next step, okay. Can your automation help when migrating a jQuery application to AngularJS, these are JavaScript frameworks. You may be migrating from one framework to the other, okay. Can your automation scripts work when you're moving from Angular 1 to Angular 2, when I say like can automation help, it means this. To your business you had an application which was working fine with jQuery, okay. Now there are a few more requirements that people are asking for and the development team feels that it may help to actually migrate that particular user interface to let's say Angular, okay. Or they want to have parts of their overall application in Angular. Now your automation tests should ideally allow you to migrate from one to the other with minimal changes so that you can catch differences in the behavior between these two things, okay. The next thing is how about migrating from Angular 1 to Angular 2, similar problem, okay. This is a very real problem actually, Angular 1 to Angular 2, people struggled with it because there was a thing called protractor which came out with Angular 1 itself. Have you ever used protractor? So you start coding with that, you had your automation scripts with that. When Angular 2 came, it was not immediately compatible with protractor. So a lot of the automation scripts went for a toss at that time. But then afterwards, two, three versions down the line, they actually like added support for it. If you have worked with Salesforce applications, okay. There was a Salesforce classic user interface then it moved to Salesforce lightning interface that is what they are actually like trying to move people over to. Now, in this case also, there is changes in the user interface. There are changes to various things. Can your automation help? When migrating an applet or a flex application to web, this is totally different technology, right? Can your automation scripts help you in that? When extending a web application to mobile? Again, user interfaces are different, technologies are different. So this is normally the problem, rapid development is the need of the day. Automated tests, in these cases that I spoke about, it should ideally help. If it helped, think about it. Your automation scripts are working across these technologies. So how confidently can you move? It is much better, right? You have a very good harness system like a net that you can actually utilize when you are migrating these technologies. You can make changes, you can make sure that you verify that the automation scripts are passing, hey awesome, if it works, very nice, right? So it gives you a lot of confidence before the release. If whatever used to work before will continue to work within your own. But automation scripts break easily when applications will work. This is the main problem and fixing automation scripts steals time from quality testing. What I mean by this is, a lot of times people spend more time in fixing those automation scripts and maintaining them than actually finding new bugs in the system, okay? Automation efforts stagnates if you feel that suddenly the stakeholder will say, hey, you know what? I want the release now. Customers are waiting for it. You have to release it. What goes out of the, what do you throw out first thing? Hey, forget automation. Let's fix it later. Let's actually just manually test it and release it. It'll fix the automation later, right? So that is a real problem. You start doing that and then automation slowly is no longer in sync with your product. It starts stagnating. So you need to have a way by which it can, it is fast enough, okay? So what does it take to be in sync? Always like, in any task, time needed depends on the total amount of effort by the number of people who can actually solve that. So in automation tasks, okay? If you look at a normal automation thing that you do, you have all this code that you write to do things that the user interfaced it, okay? Or the user, sorry, not the user, what the user did? Now, okay, just to step back on this, okay? When we are talking about this, we are talking about UI automation test, okay?  There are these unit level test, there are these like service level test, there are different layers you write those test cases and the more number of tests you have from the bottom up, the easier it is for you to actually deliver your product. And UI automation is just a small portion of it, okay? Your dependence on this is absolutely wrong, okay? But ignoring this may also cause you business trouble because the end user is interacting with the user interface. So you should not ignore it, but you should not overdo UI automation either, okay? Now, here, when I'm talking about your automation task here, I'm talking about UI automation. So what are the different phases of UI automation? One is creation of scripts or automation code, execution, then reporting, analysis and fixing, okay? Now, these can, some of these themselves can be automated, okay? The creation of automation, people say that recorders are bad, right? How many of you believe recorders are bad, okay? So the thing is, some recorders are bad. There are like, there are, there are tools which have good recorders also and if your application lends itself to recording, you may be able to automate part of your creation task using those automated tools, okay? So the creation process can be, if not the recorder, there is definitely a use for what is called, you know, an element spy or an object spy, okay? Or what you do with, you know, right like inspect element kind of thing. These are all like elements which, these are programmatic ways of identifying the element and giving you a way by which you can, you know, interact with them. So these things are useful, okay? Now, you need to utilize them to actually make your creation process simpler. So creation can be manual and machine, manual because you know what your test cases, you know what your business is. Only then can you actually automate that scenario. Execution is of course machine, okay? So that is when you get automation results, automation benefits. Next thing is reporting. This also can be automated. Analysis is manual. The sense that when things fail, you need to look at it manually and figure out why it is failing, what is happening, etc. And the fixing is also manual and is also iterative, okay? Why are we talking about these things? Because different, each of these things take different amounts of time. So if you look at fixing a bug, okay, in the test automation script, you go, you play it, you look for, look at an error, you go and figure out on the application, fix it.  So this is an iterative process, okay? And the longer your scripts are, the more time you waste in them, right? So what we want to do is reduce the effort. We want to offload to the machine as much as possible, okay? And increase people contribution. Now, increasing people contribution. The first part is of course the reducing effort, offload to machine as much as possible. We can do it through the automation tool. Increase people contribution is also something that we can do by choosing the right kind of people and the tools, okay? We want to make sure that automation is not one person's per view. It is not just the DevTest guy or the developer or the testers whole. It's not owned by that person, those people alone. We want to maximize the ability to contribute to your automation script, okay? The first thing I would say is like this is very code centric. Use version controlling. Why do we say that? Version controlling solves a lot of problems that people are still struggling with, okay? People actually copy paste their code into a central folder, shared folder, everybody takes a copy of it, works on it. This is still happening, okay? So when people actually move to version control systems, they automatically solve a lot of these problems. And even non-binary files, we suggest that you use text-based files. For example, Excel, you store it as a CSV and put it into the version control system, okay? The next thing is how do you enable everybody to contribute to this? So you go into smaller role-specific files. So this is what we're talking about is who are the different people who actually are in a team? Broadly, business analysts, quality analysts, and developers, okay? Now, these people have slightly different kind of mindsets. They have different kinds of competencies, okay? Developers, of course, are like super thrilled about writing code, okay? BAs and QAs, depending on what kind of organization it is and what kind of team you have built, may or may not be so thrilled about it. Especially in an organization which is domain knowledge specific, for example, a banking domain. Now here, the business analysts don't necessarily want to write code. They can very well interact with an Excel-like interface. They can actually formulate all their use cases. They can give you manual test cases, return an Excel that you can actually execute. But they may not be thrilled to write code there, okay? Similarly with quality analysts, should they be spending all their time writing code? Or should they be doing like finding newer bugs, exploring what is happening in the overall product, automating what they don't want to repeat all the time? That is what their overall focus should be. Now, because this, and QAs again, like testers, ideally they should be like having a mindset of figuring out problems in the application. They shouldn't have to actually write code, if possible. Of course, they will have to, but if lesser, better. Now, what we say is, break down your automation code into ways that you can enable all three of these, okay? For business analysts, you may want to expose an Excel-like user interface. Interaction code that actually performs actions on the browser, can be done by somebody else and some other things that we can do, okay? But basically what we want to do is separate out so that everybody can get involved, okay? So, and the other thing we say is minimize programming, if possible. Programming seems to be like, hey, everybody should do it. Maybe, do it for the right, right reasons, if there is something that already works without programming, use that, okay? What are the types of changes in applications? Now, mostly you can break them down into three, okay? So let me just show you a small thing before that. So, this is a small application. I'm going to, you can see this here, okay? Now, follow what I'm doing here. I'm just going, I'm logging in into the system. This is showing me a user interface, like this is a banking user interface. If anybody gets, anybody can guess which bank this looks like, yes, yes, yes. So, this, this interface, right? Like, a simple case maybe, I log in, I see that I have 25,000 available here. I go to funds transfer and transfer some money to Rajesh, let's say, 1000 rupees for, let's say, books. I click on continue, confirm and then go to the accounts and check that the total is lesser now, right? It's 24,000, 24,000, okay? This is a simplistic test case, right? So, if you were to automate it, what comes to your mind when you're automating this? What is the first thing that you may be looking, thinking about? If I said that let's automate this, start from logging in, very nice, okay? Now, if you ask this question to some subsets of people, they'll say, okay, how do you log in? You actually put in the username into that field, put in the password into that field, click on logging, okay? Then you go into that, so there are two ways of talking about this problem. One, as you said, is start with log in. The other is to say that, hey, interact with the system by putting in the username, the password and clicking on log in button, okay? Now, these are different things, okay? Now, let me actually, you've seen this interface now, right? Can you see this? It's a little hazy, right? Okay, let's see, let me just do this, okay? Now, look at this, what is written here, okay? This is log in retail user with test and secret, verify that the balance is 25,000 rupees, transfer money to Rajesh 1000 for movie tickets, verify the balance is 24,000 rupees and log in, okay? Now, this test case, does it talk about what is your user interface? Does it say that it's a web interface? No, does it say that it's a mobile interface? No, right, like all it says is this is what you're doing here, okay? Now, is this what is important to the business? Absolutely, right? This is the business scenario, what are you doing?  Now, if you had actually, if this test cases can be given to your business analysts, they'll be happy to maintain this and they can write more of these test cases, right? Any thoughts on this? Like, do you think like this is something that the business analysts may not want to actually work with? This is definitely better than giving all the interaction code to that person, right? So, one layer is the business layer, okay? And here, we do not have any interaction code at all, okay? So, you must have heard about Cucumber and other tools and a lot of times, while the people who actually wrote Cucumber say that, you know, don't put any steps of the interaction in it, most Cucumber implementations we see have interactions in them. They'll talk about, you know, put the username as this, put the password as this, click on login, then on the next screen, do this. But the screen specific logic should not be in the outermost layer. Now, if you have a test scripts like this, it doesn't matter whether your actual code is implemented in Angular 1 or Angular 2, whether it is implemented as, you know, a desktop application or a mobile application. This will survive across your backend changes and your user interface changes. But how do you actually make this run, right? So, the next layer that we talk about is this layer, okay? So, from an interface which look like an Excel that can be modified by business analyst, we get into a little, here we get a little more intimate with the application itself, okay? And in this, what are we doing? We're actually saying that, okay, login as a retail user, okay? Here in, you do the actual interaction with the application. You're saying set value into the text box, set value into the password, click login, submit, right? So, this is what you have broken it down to. And what are these text box user, etc., these are the elements that you're interacting with, okay? Now, these elements are very specific to your web application, okay? Now, if your code changed, let's say if you are back in, if your UI framework changed, it is likely that some of these things will change, these elements. And what are these actually? You see that this actually defines the actual element on the browser, okay? This is where it is actually coming to the nitty gritty of what it is on the browser, okay? Now, if you were to execute this whole thing, so let's say that we played back on Firefox, okay? Just kidding. So, the start URL is wrong, I'm just changing that. So, this will launch a browser and execute the test cases on that, okay? Well, it's demo time, give it a minute. Yes? So, the thing is that these test cases run very fast. So, you don't really will not have time to actually use it at that point of time. What you normally do is like you offload it to other machines to actually execute, okay? So, this is, and this again, like you're getting into the nitty gritties of how this tool is executing, but the point is not that. The point is that you have these different layers in the automation and you want to make those things execute together, okay? Every part is part of your automation code, but can be owned by different stakeholders. For example, the Excel like user interface can be owned by your business analysts. The interaction can be owned by your QA and the element identification. There is an interesting play we do on that, which is actually to say, you may be able to have a smoke test which actually passes through all those user elements and that can be run with your developers build, okay? What that allows you to do is when you're this smoke test only goes through the elements that normally are exercised in about like 70, 80% of your application flow. For example, in a banking application, you will have to log in, you will have to like go through accounts like, you know, funds transfer, etc. Most of these things involve that. So, you just have like smoke test which actually go through these and assert on these various elements once, okay? Now, when you do these assertions and you actually hook in the script along with your developer's build, the developer is making UI changes, his build fails because of this particular thing, he goes back and fixes it directly. Now, it doesn't even come to the QA before like when there are just element changes. What that helps you do is, you know, see, there are different in your overall how you set up your continuous integration, etc. There are these smaller smoke test that you want to run with each build. And there are these like larger speeds which actually run for 8 hours, 10 hours, etc., which you want to execute like, you know, overnight, okay? Now, you don't want to trigger these large tests without ensuring that most of the code flow will work correctly, okay?  Now, this if it failed and the developer is fixing it, the developer build is not even passing over to the QA for actually executing all these nightly builds. So, what you do is make sure that they actually pass before you pass it on. And for that, if the developer owns those tests, very nice. Yes, so the thing is, so that is where, right? Like, what are the different kinds of changes, right? Like, so I'll just like, so here actually just laid back, let me do that again. Right, it's doing all of that and like, and doing, you know, finishing. And if you look at the reporting, it actually gives you something like this. So, these are the different steps that you perform. So, even when analyzing reports, okay? This should be accessible and, you know, like meaningful to all stakeholders. If a business analyst sees this, he can say that, okay, if it failed and verify balance, he knows that, okay, this is where it failed. So, he understands that, okay, is this a business logic change or is this a code level change? So, he can say that, you know, the business logic hasn't changed. You go ahead and fix your scripts accordingly. Then internally, the developer can go or the tester can go in and say, hey, this is the interaction that happened. I go in here and then check what is going to, what is happening here, okay? Then he goes on to actually like say, okay, this is where it is. So, I edit it and fix it, fix that particular script, okay? Now, see, this is all part of the tool called Sahipro, okay? But this is not specific to all what we are showing is not specific to it, actually. You could do this with cucumber, you could do it with robot. And any of these frameworks can do something similar to what we are talking about, this separation. This particular thing is a product called Sahipro and this comes, the editor comes with that, okay? Now, let's actually make a small change to our application, okay? I'm changing the application itself. Now, if I were to play this back, this is definitely going to fail, okay? Now, let me just show you the application, what has changed in it. So here, the first screen does not take the password, it just takes the user and then like you say continue and then you put in the password, okay? So there is a slight breakage in this. Now, let me ask you this, with this change, what in our layers should be affected? Do you think the elements that actually are involved in the automation, will they be affected? Yes, because there is one other element which has come in, okay? Second thing is that, which other layer will be affected? Or which other layers will be affected? Do you think it, this has changed? The logic of, so do you think anything has to be done here? No, right? So, why is it not like that? Because what has changed is a small user interface change.  But it does not change the business itself, okay? Now, if you actually go into that, here is where like we want to fix the particular thing, because here it says like set the value to the text box and value to the password and then submit. Now, this definitely has to change because in between we have a continue button which has come in, right? And the other thing is, you will see if we actually play this back, so now, if the person understood whoever was actually making this, they can very well actually work on this change as the product is being evolved. They can think that, hey, this is going to change, let me actually add the continue button here. This is not very tough to do. Just because you know where this overall thing is going to lie, you can easily pinpoint and do it. And there are not too many places, because you have one function which actually deals with the login of retail user. You do not have to change it in too many places. And if the element has changed, you just go and change the element itself. So, let's actually do that. I think it looks something like this, let's copy this from here, right? Now, if I play this back, okay, it's not confirmed, continue actually, right? So, if I do this, by the way, like you want to see what it looks like when it fails, so the log actually, if your reporting system is nice, it should actually show you like what is going on there. So, you see that, you know, the retail user failed, it actually went in and said, hey, I couldn't do this. It says, submit the confirm button was not available, and here it shows a screenshot because I was working on it showing me the screenshot, but it would have shown you the actual screenshot of the application at that point of time, okay?  If you are only to throw, let's say, a stack trace, and say that, you know, this is the problem. Now, that needs a lot of analysis as to like why the stack trace happened, what happened before that, and who is actually like really responsible for this whole thing. But if you have a full flow and say, this is the overall logic, this is the business part we wanted to do, then these are the steps that we did. It is much easier to say that, hey, you know, retail login user is failing. Why is it failing? It must be that the new button has been introduced. It's much easier to actually pinpoint there, okay? Now, yes, it is, and as I say again, it is not only this framework. There are a lot of these frameworks which actually do similar things, okay? It automatically comes with that. So, here we go again. Now, again it has failed. Let's look at that. It says verify balance failed, and here it says click accounts it failed in, okay? And it says that div accounts has changed. Now, the thing is, user element changes also cause failure, okay? Now, let's look at what it shows. You see that, you know, there is a div accounts here, there is an accounts here, but it failed to click there, okay? Now, what would you do? You can actually go to that application, and you are seeing that this is the accounts thing, right? So, you actually hover on this, okay? Now, this it shows that, you know, it is actually span accounts here, okay? This element has changed. So, what we are saying is, okay, let's look at what it is identified as in the system. If you go in there, so div accounts is actually sort of underscore div accounts. It's not span accounts, okay? So, this is what has changed. Now, what we are doing is we will, and what, you know, when this kind of a change happens, when the top navigation bar, you use a framework, let's say to actually create a navigation, okay? Now, when you create navigation using a third party tool, and you upgrade to a newer third party, you know, the newer version of that tool, what may happen is they may change how it is actually represented. So, all you care about is something that looks like this, okay? And your functionality is fine, but underlying the person who actually like developed it, you know, hey, styling is easier if I do it as a span rather than a div, so I will change it to span. Now, your automation scripts may start failing here, okay? Now, what you can do is, you just go to that central place where everything is located, okay? So, this is where all these elements are stored, okay? Now, here you can say, okay, this is where the div accounts is, let me actually change it to span, okay? And here again, I can change it here, and everywhere it is actually used, it will get changed. This is, again, a very standard concept, like if you are looking at Selenium tools, you may be looking at page object patterns, where they actually store all the elements together. If you are looking at QTP, they have their object repository, if you are looking at Sahi, it has its own access to the repository. This is not a new concept, this has just been there, only thing is that people forget to use it when they just create those scripts. Now, here we go, like we actually change this, these two changes done. Now, can this be owned by the developer himself? Yes, because he is the one who is actually making changes to the application. So, it is quite possible that if he had a small test, which actually ran through this overall thing and pointed out these errors with various elements, he may be able to fix it and before he even comes to the testers. So, what he says, element changes, one file, which can be handled by either the dev or the tester, flow changes actually comes to either the dev or the tester, depending on how your team is constructed, the actual logic changes, it can come from the business owner over at it, okay. Now, let's go back to the, so this is what we saw, there is something that we talk about in this. This expresses business intent and it is agnostic of the web application itself. In fact, it is agnostic of the application itself. It is agnostic of the testing tool. So, what you saw here, right? It doesn't have any assumption of what tool you are using. Right now, it is Sahi maybe, but if you wanted to actually get rid of Sahi and go to some other tool, look at this test case. It has nothing related to Sahi itself. These are just keywords and they make business sense. This, if you actually like sent it to your entire customer to verify something, you can very well tell you that, you know, this makes sense, it's fine, right? So, it is not tied to the tool itself. It should not be tied to the interaction code also. The third is the most commonly made mistake in automation, okay? People tied on everything in a single thing. They think that Cucumber should be actually doing all the interactions. They think that like even this Excel like user interface should view most frameworks that people build have an Excel like user interface where they say, hey, I can choose from click set value, etc. I can choose elements from here and put the value there. That actually doesn't add any value. That framework is actually useless. That layer is actually interacting with code. Let it have the full power of code. This layer which only talks about business and doesn't care about the code. Let it be devoid of code if possible. So, as I said, examples create user, approve user, log in user. These are all like business things. Well, change of business logic itself changes. By the way, can you tell me in the code that we were writing in this, in this user scenario, okay? Can you tell a business case which will lead to modification of this? Think of a business case. You do banking every day. Can you think of a business case which will ask you, which will make you modify this, generate an OTP, okay? Now, the thing is that is it the transfer money part, okay? Now, generate OTP itself may be just a detail of your transfer money, okay? The thing is it's an internal thing to your transferring of money that the OTPs. There will be other test cases associated with like OTP. So, another thing in automation, right? There are two kinds of functions you will normally be having. One is to test that functionality itself, in which case you will have lots of scenarios for it, okay? And the other is you're traversing this piece of this code because you want to get somewhere else, okay? For example, a login. A login can have a lot of use cases, test cases and assertion. But when you use it as part of a flow, do you actually do all of those? No, you just take that like login as a, as like login, go to the next step. It is just a precondition to actually going to the next step. So, in that case, right? Having the OTP may or may not affect this case, okay? It may affect it if you're testing the transfer process itself with the OTP. But if we are trying to verify that the totals are correct or the balances are correct, you may just encapsulate it inside transfer money itself, okay? So, I agree that that's a very valid case. But the thing is like, if you are in a flow, you may not actually want to make it outside. So everywhere where you do transfer money, do you want to also add a step of verify OTP? No, you may actually have that only when you're testing the OTP part. Otherwise, it just goes into this flow, okay? Any other, any other thing that you may actually come up with? No, that's okay. So, that is just a test. So, you are actually trying to, it does not change the business case. It changes who you are transferring to. Like, it doesn't change the business case really, okay? Right? So, if there is a minimum balance check, okay? Now, will this be part of this or will it be a different test case, right? So, okay, when you think about this, right, whatever you come up with, most probably will need to be added as another test case somewhere, okay?  The other thing is like the one where you said like, you know, minimum balance thing, okay? That is another test case. So, if you say that, there will be transfer money and if it is greater than, like, if it is lesser than the thing, maybe like disable the transaction, right? So, it will be another use case, okay? 25,000 to 24,000 may actually just work, yes please. Right? So, then like it may be, it may reduce the overall amount. That is the, is one of the only cases where this thing changes, okay? So, what he said was if the bank wanted to introduce a transaction charge because of which your overall balance will go down, okay? Only then does this change. The others may be add on test cases for like more of those, okay? Now, sorry? Yes. Yes, yes, they will also be get, so they will get executed. For example, right? This is a bank transfer related test case, okay? So, this will have all bank transfer related thing. This will not have login related checks. So, you won't check here that, you know, hey, if I put the wrong password, it actually like fails or like, whatever it could goes in the wrong password. So, you will have another which actually individually goes to that particular place and does the various operations there. So, each, as you rightly said, it's a journey. There are different stages in the journey and you may need to actually do extensive testing of each part of this journey, okay? But you do not want to do all of this in one go, okay? There are multiple reasons for this, okay? First reason is if you were to do all of that and let's say like you want to eventually, you are facing some error in the fifth stage, okay? When you rerun the script, you are going to do all these validations and reach the fifth stage, which is not useful to you, okay? And then you will be able to reproduce and fix it. So, the better way would be when you get to the stage in the easiest possible way without assuming that everything up to that stage is working correctly, okay? So, if you want to test on the fifth stage, assume that till the fourth stage everything is working correctly. You navigate in the simplest possible way to that place and then you do all the checks at five. When you want to go to six, again assume that five is working. Go in the simplest possible way and then do the six. So, in each one of these, if there were checks in verify balance, you would do it as a verify balance related test case. If you were like doing in the log in, then it will be a log in related test cases, okay? So, if right now it is about transfer money, assume that verify and the log in work correctly, then like verify the transfer related stuff, okay? Yes? No, you don't jump. You don't jump. So, the thing is, so let's say, okay, in this case, if let's say OTP is involved in this. So, here you will go in and say that here where you actually do transfer money, you may enter OTP. You will add it here, okay? This is what will take you to that place quickly. You are not testing OTP right now. You are not testing that, you know, the OTP functionality is not being tested. It is just part of your flow. So, you just add it here and let it flow, okay? There will be other test cases that specifically test the OTP, saying that, you know, if I put the wrong OTP, what happens? If the OTP did not reach on time, what happens? Those will be a separate set of test cases. Sure, okay? You assume happy till you reach that point where you want to be sad, okay? So, you reach that and then you, so you keep going stage by stage and then like put yourself through all the trouble of, you know, testing everything there, okay? So, okay? So, business layer agnostic of these things, I'm running out of time. Again, interactions, this I have already mentioned that, you know, it can be maintained by both QA and testers and developers. Here again, this changes when the element repository layer changes when particular elements are changed via HTML or JavaScript. So, I showed you a demo of like what happens there. So, what should change now when migrating a jQuery application to AngularJS, we see that the business layer doesn't change at all, okay? Most probably some parts of the interaction layer and a good part of the element repository layer will change. Migrating from Angular 1 to Angular 2, similarly, mostly it will have a lot of changes in the, in fact, like not a lot. A little bit in the element repository, a little bit in the interaction. Salesforce, Classic to Lightning, what they have changed is basically the navigation. So, this only affects the middle layer, which is the interaction layer. Applet or flex application to web. This, the business layer doesn't change if your overall functionality is the same. But if your interactions may change, the tool that you actually used to talk to these things may change, the element identification may change. Extending a web application to mobile. Here again, all these, all these layers may change because the business also changes to actually squeeze your overall interaction to a smaller one, right?  So, it's a little limited functionality. So, you'll have a subset of these things. So, it may actually affect all three layers. So, some guy on automation code is most useful when there is a lot of change planned in your application. This is actually converse to what people find nowadays. So, if you write it correctly, you can actually safeguard yourself. Its access is safety, night and the guideline. Automation code should not be thrown away when application technology changes. It should again act as a net. Building the right layers and strictly following them helps in minimal maintenance efforts and long-lived useful automation scripts. Thank you.