We need every single airman from every FSC, from every component, rather Air National Guard, Air Force Reserve, active duty to understand the role that they play in delivering air power on behalf of our nation. And we need every single airman to have a foundational understanding of why what they do matters. While we are living in rapidly evolving times where our world is really contested, it is because of our airman that we have the greatest competitive advantage over any enemy or against any competitor. It is truly because of our airman. I love that we have a playbook that is foundational to who we are and that we are focused on what is basic Air Force doctrine that highlights what we do, why we do, etc. But on the airman piece it's a reminder of how very much our values and our character matters and the airmen that we have serving in today's Air Force are going to be the airmen that are going to propel us into the Air Force that we need in 2030 and beyond and we've got to get that right and we can't train and develop our airmen in the same way that it was, especially considering joint all-domain operations like our airmen have to be smarter, more creative and more empowered to get after it. We're going to have the airmen that are ready and they're already going to be the digital natives, the very educated airmen that we need. The doctrine is, you know, it's our guiding principle on making sure that we have a foundational understanding of who we are, what we do, what we bring to the fight, but more importantly now the how we do it is really truly through empowering the airmen that are coming in. The global landscape that we're in today is so very different than when I came in and every year it continues to change and evolve and we actually have competitors and adversaries who do not wish us any goodwill, but the one thing for certain that we have in the United States Air Force, our most competitive advantage, are our airmen. So we've got to empower them and we've got to cut them loose so that they can be that creative and unique advantage that no other competitor has.


My name is Titus Fortner, this is my third career. I started out as a submarine officer doing nuclear engineering things on a submarine in the United States Navy. I do not recommend that if you enjoy Sunlight, which I do. It is, yeah, it wasn't the most fun experience. Software ended up being my third career. Day two of the, as a junior developer, day two manager got a group of us together and he's like, all right, we need someone to figure out this selenium thing. Looking around and no one else is raising their hand and I'm like, all right, I'll do it. I'll figure it out. And that was seven years ago. Since then I've worked at five different companies as a software development engineer and test, working with existing frameworks, writing new frameworks, mostly in Ruby, a little bit in Python. I've gotten to work with some of the really, some of the smartest people out there in this industry, I would say, and it's been really great. I'm currently at Sauce Labs as a senior solution architect, which means I get to go around to other companies and take a look at their frameworks, take a look at their code and give them feedback on what they're doing well and what they could do to improve the quality of what they are doing. Also, I'm really big in the open source community. I am a core contributor to this Selenium project, primarily doing the Ruby bindings and I'm also the lead developer on a project called Water. So many people have heard of Water. Yeah, so Water's been around longer than Selenium, but it re-implemented itself to use WebDriver when WebDriver came out. And so now Water is backed by Selenium and it's essentially a test library built on top of Selenium to provide a little bit more, kind of provide an interface that's a little bit more tester friendly than some of the things that Selenium does natively. And some of the stuff I'll talk about today is things, includes things that I've learned from using Water and tools in the Water ecosystem for making testing easier. Today is also my birthday. My gift is being here with you. All right, so this, yeah, thanks. So this, I'm sure I don't run out of electrons. So this is actually part of a much larger talk. I could do an all day thing with this. I have code for all of the stuff that I'm gonna be talking about in, I do releases with each workshop that I teach that I put all of the code and all of the slides. And so there's like six different versions up there right now. We're not gonna walk through, we don't have enough time to like actually go through all of this, we probably won't even have time to set everything up if we try to get everyone working on their laptop today. So I'm just gonna be walking through the code and talking about what I'm doing in each step of the code to try and get an idea of what things to be looking at here. And primarily, we're gonna be going over these eight things. And these are the things that essentially are components of putting a framework together and being able to write maintainable tests. So we're gonna start off with a framework overview. What is a framework? Everyone, this is a very overloaded term. Everyone uses this term very differently. So I like to define what I'm talking about. A lot of people will talk about the test runners, the J unit, the R spec, the N unit, Karma, PyTest, these as frameworks. And so oftentimes what framework are you using? This is what people are asking for or what especially at SauceLabs, what they're asking for. But I consider these test environments or test runners. These is one small component of what it takes to put together your tests, to execute your Selenium tests or your Appium tests. A lot of these projects will call themselves frameworks. Your WebDriver.io, CappyBorrow, Water, Selenite. These are the test libraries that provide the glue code, essentially, to your wrappers and your helpers to make it easier for you to do your testing. But I don't consider these frameworks. I kind of give an all-inclusive definition of a framework which is essentially all of the software that you need to test your website functionally is part of your framework. And the reason I like to give a very inclusive definition there is because everyone has a different framework. They're all unique to a certain extent because everyone implements them differently or creates their own glue code on top of what's out there. Even if you're using RSpec and Water, you're also probably using a page object gem. And maybe it's Cheezys or maybe you built your own or maybe you have a data management gem. Maybe you have something specific to how you configure your data. And it's important to look at it this way because you're making a lot of choices throughout putting this code together. And the idea is what choices can be improved on, what things can you maybe find a better way to do something. So I guarantee everyone's framework is really strong in some areas, hopefully. And also likely poor in others or just haven't thought through some of the ideas there. All right. So I'd like to start out. Record and Playbacks, Lending My D. I know there was some love for this in Simon's overview. He and I have a fundamental disagreement on this. If you're looking to build a maintainable test suite, which is what this talk is about, do not use the Selenium IDE. Do not use a record and playback device. If anyone has seen a large scale maintainable test suite that was written in Selenium IDE, please let me know. Because I've got drinks for those people or something to buy. Yeah, so I'm a little concerned about the Selenium IDE initiative. And I've been vocal on Twitter and things about it. I can see some positive things coming from it, some kind of code generation or make it easier to find selectors and locate elements and things like that. But as a solution for let's put something together with this, I feel a lot of people are like, oh, it's like training wheels. I'm like, well, that's great until you weld those on to your little bicycle. And once you start doing that, it makes it a lot harder to get to the point where you can actually excel and run a race that you need to be succeeding. All right, so the whole purpose of a framework is to reduce maintenance costs. Pretty much anyone that has ever maintained a test suite knows that the maintenance costs are more expensive than the combination of creation costs and execution costs. So every time someone complains to me, this is something like at Sauce Labs, right? So Sauce Labs, it's slower than if I run it locally. Yes. If that's what you're concerned about, maximizing, optimizing that execution time, you're looking at it entirely wrong. Your costs are coming in the maintenance time. So let us help you optimize the amount of time that you're spending on maintenance so that you have more time to create and focus on more of the business needs rather than constantly updating your code to make sure the tests aren't flaky, digging into poorly written exceptions to figure out where the actual legitimate problem is, and just spending a lot of extra time writing unnecessary tests. Dry, what does dry stand for? What is it? Thank you. Yeah, so don't repeat yourself. This is essentially when you're putting a framework together, everything should have a place and it should be obvious where that place is. And so some of the patterns I'm gonna be talking about here, I really appreciate because it makes it very obvious to see what things need to go where in order to be easier, like, ah, that got updated. I know exactly where to make that one change for everything to work now. So the components of a test framework. I've got seven of them listed here. We'll walk through each of these. All right, so as an open source proponent, I always bring the slide up here as far as there are a lot of options for test libraries that you can use in all different languages. Ruby has more than others for various reasons. Java, everyone kinda likes to do their own thing because Java, but there are a lot of different options for things so that you don't have to continue to reinvent the wheel. And if you're looking at an option that's out there and like, ah, this is close to what we want but not exactly, I would encourage you to not, like, oh, this only does 80%, so let's rewrite our own that does exactly what we need. Use the 80% and add on the additional 20% you need and if you solve a problem that other people might have, feed that back to the open source project and so that the next company that comes along and has a similar issue can have that solution solved for them already. All right, so actions on assertions. We start out listing the popular test runners in each of the different languages. When I talk about test runners, this is like the fundamental, this is what's providing the structure. This is what's keeping you from just running everything as from your main method or in just a Ruby file that's executing that either returns a zero or a one at the end. This is what allows you to structure your tests with before and after hooks and all of the assertions that you need. All right, so TDD versus BDD. How many people here would say they practice BDD? Actually, how many people here use a BDD tool? How many people that use a BDD tool think that they're doing behavior-driven development? One person confidently raising his hand over there. All right, so the idea behind test-driven development is that you write the test and then write the code and get it to pass. Accepted test-driven development is essentially just looking at it from a user point of view and behavior-driven development is supposed to be working together to do acceptance tests in a specific way, so it's supposed to be more collaborative. So the way I look at test-driven development when it comes to writing tests, write the test, make sure it's failing, make the code work, make sure it's passing, make the code better. So tell a story of a coworker I had at a company a few companies ago. He did not write the best code. I was being, as you get better at writing any programming, you get very particular about how things should look and getting really good about writing good code. And I was like, all right, I'm gonna abstract this out, I'm gonna create this other thing. So my boss tells me and my coworker to each automate some things and this guy knocked it out the first day. He got his stuff done and I looked at it and I'm just like, wow, that's kind of ugly, man. Look, two days later, three days later, my boss is like, how are you doing on that type? So I'm like, almost done, almost done with my stuff. I've got it, it's all pretty, it's gonna be great. And yeah, which of us did my boss like more? Was it me? Because he'd already done three times as much as I had. And this is where kind of the idea of gagny, who knows what gagny stands for? You ain't gonna need it. Don't over optimize if you don't know what you need to optimize for.   I can't tell you how many times I've seen page objects where people go through and identify every single element on a page and put it in their page object. Why go to that trouble, you're just polluting your class with stuff that you may never use.  And as long as you are not working for someone that's like, ah, you wrote it, you can't go back and touch it ever, because there are managers like that. As long as you're not working for that manager, just get it working and then worry about optimizing it once you see what the actual pain points are. All right, this is a video. Let me see if it, Internet Live is not a big thing here in India, but that's my favorite SNL clip. And I think it really accurately summarizes kind of how I think of agile just in general, of this idea of what's the problem, fix it. It doesn't have to be pretty. If the next biggest problem is making it pretty, then make it pretty, but if the next biggest problem is something else, work on that and just try to keep improving what you're doing until you have something that is optimal. All right, so BDD. I like to kind of just throw up an example here for people who aren't familiar. Essentially you're trying to write English sentences that get parsed through some kind of regular expression and then that translates into some kind of code that gets executed. How many people like working with regular expressions? Okay, if you say so. No, regular expressions are awesome for figuring out like how to parse things that are difficult and get exactly what you need from them, but they're very tricky to maintain. They're very tricky to get a lot of regular expressions that do what you need them to do every single time that you need them to do it. It's very easy when you're putting, I've worked at, I think three of the places I've worked have done cucumber and in two of them we had a lot of problems where the wrong regular expression was matching what I had just written to the wrong thing and it's really easy to forget like, oh wait, where did we put that one? How is this one different from that one? It's in a different library and a completely different file. It takes a lot of forethought and consistency to minimize those interactions. So I always like to put out the warning for BDD in general.  If you're not using it as a collaborative tool you're creating extra overhead of dealing with a bunch of regular expressions and trying to keep up with where you put all of the code that you're writing. I like to say that cucumber is for consultants because I think that's the right model to adopt cucumber is if you've got some vice president that's like, ah, we're not agile, we need to be agile. Let's hire someone to make us agile and that person comes in and says, okay, let's get all of the stakeholders in the room. This is the process we're gonna follow. We're gonna write these given one then statements that is gonna encompass all of the new features that we're writing and it's not for existing features on the existing sites, that's another thing. A lot of people will do BDD tools to write tests for features that already exist. And the whole point of BDD is that you're developing the features by agreeing on what the tests are and going from there. And the other reason that some people will say oh no, cucumber is great because that makes it really easy for one person to write the feature files because that's all they know is English and so they're just gonna write the English and they'll have someone else that deals with the regular expressions because that's a pain and we don't wanna do it and then we'll have someone else that implements the page objects and the fun stuff. That always frustrates because once you start trying to specialize within groups you're essentially saying you're never gonna be good enough to do this and we don't trust you to do that and I don't think that's a good way to build a team. You can have some people that are better at some things than others but everyone should be able to do everything on a team like this idea of cross-functional Agile teams that yes, someone might be better at back end and someone better at front end but you're on the same team and if that guy is out of town or she's on vacation you can take over what they're doing for them. Here's some code. It works. Is it a test? Why is it not a test? No assertions. I inherited a suite of tests once that didn't have any assertions and I was just like, okay. So I dutifully went through and was adding assertions. The funny part is that I found a bug and I submitted it to put it in the bug tracker and send it off and the next morning I came in to get an email back from product that it updated with priority five won't fix. All right, so why am I maintaining a test that no one cares about? Why am I putting all this time and energy and why are we running things and if this can't be the only test that we're running that you don't actually care about? So this idea of make sure that the tests that you're running are actionable. Things that if they pass, they give you some additional confidence that the system is working and if they fail you're gonna do something about it other than just say, oh yeah, that's nice. Like the three components here then of any given test is going to be an assert, arrange, arrange, act, assert. This is essentially the given when then of a Gherkin or BDD. There's a school of thought out there that says you should only ever have one assertion per test and I don't subscribe to that school of thought. I think the important part is the action that you're taking. That's the name of the test, the one thing that you're doing and so the point of the arrange is to do the minimum possible to create an exactly known state in order for you to take the focused action that you care about and then whatever things result from that action being successful, go on the assertion. So if you need to see a success message and you need to see the item added to the list, go ahead and put both of those assertions in there as far as I'm concerned because if either of them aren't working, the action is not doing the right thing and this also kind of gets into assertions versus exceptions. An assertion is the intent of the test. Is the intent met or not? Most test runners will make a distinction between whether or not an assertion failed or whether there was an exception. I see a lot of people putting assertions everywhere on their code. Navigate to this page, assert on that page. Click this link, assert link clicked. Navigate to this page, assert. And there's some value to that in order to get certain types of information but I like to make a clear distinction between have I tested what I intended to test and did it work, did it not work or could I not even get to the point where I could test that in the first place? And so if you feel the need to stop where you're at if some condition's not met, I would recommend raising an exception instead of just making an assertion in your page objects or wherever you have abstracted things. All right, and the last thing about tests here is their attributes. Tests should be atomic, autonomous and short. The goal here is to have very focused tests and this is again the action part. What is it, what action are you testing? What functionality, what chunk of usefulness in your web application? They should be autonomous so they can be run in any order or at the same time. No state should be transitioning between them. We'll talk about, the data is really where this gets a little tricky. We'll talk about that here in a little bit. And the other thing is for it to just be short. The longer your tests run, the more likely some weird problem's gonna come up that is gonna cause it to fail when it shouldn't and having a bunch of shorter tests is definitely preferable to fewer longer running tests when it comes to scaling. All right, so next part of our framework is the initializing and the cleanup. So usually this goes in some kind of a base test class of some kind where you do your setup and your tear down, start the driver, quit the driver, whatever other things need to go in here. Sometimes you put some kind of reporting thing in here if you're doing JUnit. How many people are JUnit here, static curiosity? How many people are testNG? Interesting. Okay, this is what I do in JUnit with essentially I've got to use SOS there for what's running locally if it's running on the SOS labs. But what are you doing if it's passing or failing? How are you reporting that? What this can and this can be into your Jenkins could get this information. You could be passing it from an API into any kind of like test trail or rally or anything like actually written APIs that will pass that kind of information back into all kinds of these third party tracking systems. This is where you set your options. This is an easy one to show because it's very obvious when the browser opens if the info bar is there or not. And it's just adding an argument for that. Chrome literally has hundreds of different things that you can toggle to do all kinds of things that you probably would never need to do but it's an interesting list. The other thing I'd like to point out if you're running things locally, one of the most common things on the Selenium forums, whenever someone comes in like my tests were working yesterday and now they're broken. First question, update your driver. Like always because especially in Chrome it's almost always, oh yeah, Chrome's got like a rolling three version or four versions that their driver supports. The browser automatically updates, the driver does not. So I like pointing out this library here, the web driver manager. I wrote something in Ruby that will automatically download and provide for the user the latest driver if they don't already have it on their system. I'm like, ah, it would be nice if we had this in all the languages and someone's like, Java already has it. Here it is. I'm like, yes, let's publicize this because if you are not using a third party service to execute your tests or some kind of Docker container or something that keeps everything together or don't limit your browser to not auto update, having something that will also auto update your driver is huge. And the other thing just to kind of walk through a quick SOS Labs, how it works. This is my account. I have not given you my access key but this is where it would be. And you just take that username and your access key and you pass that in as part of your SOS URL. So if you are already running on a grid somewhere, moving to a third party provider like SOS Labs is mostly just changing the URL and then updating the specific information. So we've got a little platform configurator here where you just use the UI to pick which things you want and then we'll give you the code in the different language down here that you copy and paste into your code to get that particular combination of browser and operating system. Also pointing out that W3C is gonna be changing. Like the biggest thing that the W3C is probably gonna be changing is the desired capabilities. Desired capabilities is going away. Java and C sharp are both using an options class and it says desired capabilities is deprecated. Please use the options class. The options class is implementing the desired capabilities underneath. I still have some stuff to do for Ruby to make that happier. It's on my list. But what it means for third parties like SOS Labs is that there is a defined list of things that are allowed to be sent in as a capability. And so right now SOS Labs, you just send that as part of the root level desired capabilities and it works. With W3C it won't. So we have to have a little SOS option. So like any third party, whether it's Google or Mozilla, so Google if they have special things, it'd be Google colon. Mozilla it's Moz colon and like that. So anything that doesn't fit what's specified in the W3C, everyone know that we got the W, yeah, Simon talked about it, W3C a couple of weeks ago. So we have our special key for that. Just something to keep in mind that when 4.0 comes out, this will probably be required for a lot of clients to be doing something like this if you want to use Selenium 4 with a provider like SOS Labs. And I don't want to take too much time talking about SOS Labs, but just wanted to show that when you execute your tests on SOS Labs, you get a screen that looks like this. You can run a bunch of tests at the same time that will give you your browser operating system combination, whether it passed or failed or through an error. And then each specific test is going to give you the ability to watch the video of it. You can see exactly which wire calls the test made and what the input and output of each call was and then screenshots if the DOM is likely to have changed with the action you took. We also have logs and made a data options. So those are all things they configured in the initialization. The next big thing is data. So this is probably one of the biggest things when it comes to scaling up your tests, like what needs to be fixed or addressed or improved in order to make your tests scale better. So there's three approaches to data. First one I call grab and hope. This is the one where you just take whatever's on the page and you hope it's in the state that it needs to be in in order for your test to work. Good example of this is an e-commerce site where you go to the page and you grab the first product on the page and you go to check out with it, which works great until the staging environment says it's out of stock and so now you're gonna get a failure for that product even though the system is working perfectly because you grabbed one thing and hoped that it was in the condition that it needed to be in and it was not. The next option is fixtures. So think of fixtures. This is like the spreadsheet model where you have a spreadsheet of these are the 10 users and each of these 10 users have these different properties and each test designates which of the hard codes in some fashion, which user it needs to use. This is difficult because whenever you have an additional property, you'll end up often doubling the number of things that you have. Like oh well we need half of them with this turned on and half of them with this turned off and so now we're gonna double the number of users we have so that we have the right conditions and still all of the combinations and permutations that we need, it gets really difficult to maintain. But this is the way a lot of companies, the way their systems are set up, it makes it difficult not to do something along those lines. The ideal is to use what I call a just in time system and this is every single test is responsible for creating all of the data that it needs for that test. And yes, that takes a little bit of extra time in the beginning relative to creating everything ahead of time and using that. But again, we're trying to optimize maintenance time, not optimize the execution time. And if increasing the execution time a little bit decreases the maintenance time significantly, that's a huge win. This is what makes it difficult for people to run at scale. This is where when you're running 50 tests at the same time, one test is gonna fail sometimes and you don't know why. And after hours and days of working with it you realize ah, it's failing because this other test just happens to be running at the same time and when they're both running at the same time there's a problem but if they're not running at the same time there's not. Very hard to track down but it makes your test sweet very unreliable and it's really hard to figure out why. And so if you can do everything that you can with a just in time approach, that's going to be the most useful. So we've got an email of useradexample.com and a password of password. What does this tell us about the test that we're running? Yeah, it could have something to do with password not being complex enough. It could have something to do with example.com not being a real address and so there's some failure. It could be that this user doesn't exist. It could be this user does exist. We really don't have any information that's useful that tells us why we're using this particular data. So I use what I call static data. I will create a valid user from a user object. This is just a plain old Java object here that's gonna have some properties and these properties in this case are just email and password. This will get a lot more complicated if you're doing an address book or credit card data or any other data that you're working with that can be a lot of fields here. But the goal here is just to identify what it is that you want and it's hard coded in one place here and it's gonna return an instance of that object with those values. Which is great until you need to sign up for instance. This will work for a sign in because that's a valid user but what about sign up? I need some kind of random user in my test for a sign up. So I'll do something like this. Faker, who here is familiar with Faker? All right, yeah. So I gave a talk earlier this year at the Automation Guild. It's the virtual conference that Joe Colantonio does and I briefly referenced Faker and everyone. The first three questions I got at the end of the talk had to do with talk about random data. Like how are you doing this? What is that class? How does that work? Essentially, I want a random email address and a random password and Faker.Internet will give me those things. You have a bunch of different options with Faker. Internet, you want an IP address, you want a port, you want a random URL, a name, first name, last name, suffix, title, address. You want something in Italy. It'll give you an Italy zip code that it's randomized. Loram is the text, finance for your credit cards, date and time for, again, all kinds of different localizations and Chuck Norris Fax. Who here does not know Chuck Norris Fax? Everyone knows Chuck Norris Fax? Oh, there's a few people. All right, okay, we're gonna, this is important. I didn't realize that people in the United States didn't know about Chuck Norris Fax. So this is Ruby, but it's easier to use for me. Anyone see this? All right, no statement can catch the Chuck Norris exception. Chuck Norris does not use exceptions when programming, has not been able to identify any of his code that is not exceptional. Chuck Norris rewrote the Google search engine from scratch. My favorite is Chuck Norris can recite Pi backwards. So yeah, so if you just need a random fact, that's a very important library. But there's literally dozens or hundreds of libraries that do all kinds of random things. And I see a lot of people, again, especially in the Java community, that will write their own randomization things from scratch when there's a perfectly good library that has all of that already available. All right, so we had a random user, but random user doesn't necessarily make sense either because we don't really care what username or the password is, what the email is, that data doesn't matter. And so it makes sense for us to set a default for these things. And so we shouldn't need random user because any time we initialize a user object, it should default to giving us random data. And so random user becomes redundant. Give me a user, give me a set of user data, might be a better way to say that, and use that, which is great until we need to do something like a blank password. So this is an implementation that shows how to override the default data with the data that is contextually important. In this case, password is an empty string. It might be we need to ship our product to Alaska and if it's going to Alaska, it can't have alcohol content because we can't fly alcohol content and so that can only go to the continental United States. That was a real test that I wrote a few years ago. And so you have to specify that I want an Alaska address, but you don't care what the first name of the last name or the street address is, you just care that it's going to Alaska with the right zip code. So the next thing to talk about is configuration data. These are like interrelated chunks of data. So this is an example of something you would send to SOS Labs, for instance, of I want Windows 10 platform, I want a browser Chrome, I want version 65. For the purpose of our test, this doesn't, it doesn't make any difference if platform by itself doesn't make sense. Browser name by itself doesn't make sense. These are things that have to be together in a chunk in order for them to make sense. And so I do not like to include this kind of data in my test logic. Most people put this data hard-coded in their before step. I'll see all sorts of configuration data, usually with really complicated conditionals for a lot of different things, just a lot of data interspersed in the logic. I much prefer to have some kind of key that says if I want Windows 8 IE, give me this chunk of things. And then my conditional becomes pulling something from an environment variable or a system property to give me the collection of data that I want rather than having to specify it in a long conditional inside my code. This is an example of a YAML format. I really like YAML just because it's both easier, it's serialized and easy to read. JSON I do not find as easy to read. You could do the same thing with like a Java properties file or something, but the idea is to chunk like things together and pull them outside of your test code. The other thing that I see a lot of is people putting a lot of variables and properties in their CI tools. As someone who has spent way more hours than I would care to admit to updating Jenkins configurations, I highly recommend that everything that is some kind of variable, that is some kind of toggle be put in your code. Version control either, like I wouldn't even use like string properties in Jenkins where you list environment variables. I don't even do that anymore. I've also seen in the build steps having all of these flags and parameters after each one. Create some kind of make file that lists out whatever that execution is and call it directly. So your build step is essentially just a one line thing saying this is the collection of stuff that this build is executing. It's a lot easier to maintain and it's easier to see when you change something than oh, we changed something, something's different, we're not sure what. It makes it a lot easier to maintain and see when there might be a problem, what might have changed. So the next thing is site modeling. So this is your page object model. One of the things I really enjoy talking about is this idea of declarative versus imperative. It's very common, it's one of the things I do when I'm looking at someone's, when I'm looking at a company's framework, if I'm doing a framework assessment for Sauce Labs, I'm taking a look. This is one of the big things that I work with clients about and that I've gotten a lot of good feedback on. This idea that the text of your test should be high level. It should not be including your implementation logic.  Imperative is the how, the implementation details, specifics, specifies all of the data. So this would be an example of something imperative. We're referencing a driver. We're making a specific call to navigate. We're finding something and clicking it. We're waiting for something. We are specifying non-contextually useful data. We're again, identifying an element and sending keys. We're clicking, that whole thing. Compare that to something like this. We're signing in, we're visiting, we're signing in, we're verifying. Or given when then. The only tricky thing here is we're initializing a new homepage instance. Everything else is very straightforward. Sign in page visit, sign in page sign in with data, a valid user, and assert that on the homepage we're signed in. I always write my tests first like this. One of the things I like to point out here is when I'm writing this test this way, the method that I'm choosing is sign in. I'm on the sign in page, I'm signing in with a valid user. If I was implementing the page object first, I probably would have implemented a fill form method. And that's just one example of something this is more contextually useful when you're reading the test. Sign in page, sign in data, rather than something that's a little bit more descriptive of fill form. One of the ways that I think about, well another thing that I think about with this is does this apply to both mobile and web? Your desktop browser versus your mobile browser versus the native application. Sign in is a function that works for all three. You want it to be true in all three of those instances. If you are specifying select dropdown, click this, click this other thing, click this link, those things are less likely to apply in a mobile device. And what if it changes from a dropdown to a check box? Does that fundamentally change the behavior of the system? It's an implementation detail. And so it should not, you should need to go into the text of your test in order to update those things. That should be done in your page object implementation. And then as far as the naming, like I said, I like thinking about what the names are from the test perspective, from the user perspective, rather than the implementation perspective. And this comes down to the idea of the two hard problems of computer science, passion validation, naming things, and off by one errors. All right, so yeah, so I always start with writing the test, then I write the methods for those tests, and then figure out exactly what elements I need, and then only write the locator for those elements. Like I mentioned earlier, instead of writing every single element on the page, it's like, oh, I'm already here, I might as well do all of these. It clutters up your test, when you make changes, you don't know what all changes need to be made. Like what happens if you have one that you're using and 10 that you're not, and you have to update the one that you're using, do you then have to update everything else? You don't know, you would have to go through, and it's not necessarily something that you need. All right, well, I have a whole talk on these. How many people use a page factory here? The people who wrote the page factory wish that they had not written the page factory. Page factory obfuscates some things and makes some things difficult in a way that's difficult to scale, but because it's in the project, people think it's the only way to do it. I just wanna put out there that it's not the only way to do it. If you're using it, I'm not saying change it, but if you're not using it and thinking this is the way I have to do it, it's not. And we still argue back and forth as to whether or not it's going to stay in the project or if it's gonna be rolled out into a separate library. I'm honestly not sure where we're at on that this week because it seems to change regularly, but there is a desire. I already talked about overpopulating the base page. Oh sorry, I didn't talk about the base page. Overpopulating page objects in general is a problem. Overpopulating the base page, I'll talk about that one a little bit more in a minute. Coupling, the original page object design pattern specified that every method in a page object should return an instance of another page object. In my experience, that causes more problems than it solves.  That's great. What happens if you're at the cart and you're about to sign out of the cart and it'll ask, do you wanna log in? Oh yes, I'll log in. So it takes you to the login page, you log in. Does it take you back to the landing page? No, it takes you to back to the cart. Do you need two page objects? Do you need two methods? Do you need, how do you handle that if you're coupling your page objects together in that way? So I do not find it overly challenging to just initialize a new page object at the places that I need a new page object. And I much prefer that than trying to have that navigational logic embedded everywhere in my pages. Multiple expectations from methods. This is an interesting one for the whole like fill form successfully, fill form unsuccessfully. This idea that both of them fill a form but one of them is gonna have a different outcome than the other based on what you intend. And as we're moving to be more declarative in the tests that we're writing, it doesn't necessarily make sense to call the same method if we're expecting one to have one result, expecting it to have one result one time and another result the other time. It could be a lot easier depending on the context to have separate ones that can have their own specific waiting methods or parameters as necessary. Conditionals based on state. This is one of the ones that really frustrates me. I've seen people stand up on stages like this one that people about the size sit in talking about how, yes, the way you have a page object for mobile and browser is to look for the mobile element and if you don't see it go down the browser path and if you do see it go down the mobile path. That really frustrates me because we are in an era of dynamic sites. So what happens if when you say is the mobile element there it just wasn't there in time and it's gonna show up point one seconds later. It's a bad pattern to get into. Your test already knows. Your test knows if it's testing a mobile application or a browser. It's either in a configuration file that figured out what it was gonna start up in the first place or you can ask the driver are you an Appium driver? Are you a Selenium driver? You should never have to query the state of the browser or an element in order to figure out what conditional. One of the companies I worked at had a really, really over engineered framework. It was really impressive actually. It would try and automatically figure out all sorts of things on the fly and so there would be these forms that would get filled out. There were like 10 pages long. And it would be like all right, so populate the, if you see this data then populate it and if you don't see this data then do this other thing and if you see a next button then click the next button then populate what you can see and then if there's a publish button hit that. If there's not then hit finish. And that would work great if the site stayed in a 2005 static presentation. But once anything started getting dynamic and once there was any latency lag Firefox and Chrome would behave slightly differently with these things. All of a sudden it's like hey click this if you see it and if not click this other thing that becomes a very bad pattern because again it wasn't there and so it went down the wrong path and we would get a failure when the system was working just fine. Specifying navigation this is I usually do a visit method on my page objects. So part of what I try to do when I'm doing my UI test is to try and have as minimal setup as possible to get the exact state that I need. So I don't necessarily care that I start on the homepage and I click on the sign in button to get to the sign in page and then click the sign up button to get the sign up page and then fell out of form. I just wanna be on the sign up page. So I have a visit method and I put the URL that I need to get to directly in there if it's a browser, if it's mobile you have to do something a little bit more. But I always have a visit method to say whatever is the easiest way for me to set the state of being on this page, make that happen. And so that is kind of coupling cause I will use other page objects sometimes. Reference on the homepage, click this button on the sign in page, click this button and now I'm on the sign up page. But I find that a lot more useful than coupling from the return statements of methods. Unhelpful error messages. This is another one that takes a long time to debug problems. You have, what normally happens, right? With the biggest error here, element not found. Okay, element not found. Why is it not found? I have no idea. You look at the stack trace. Well, the stack trace is usually two lines further down from the last action that you took. And so, and especially if it's any size test, like five minutes in we get an element exception, element not found exception. Let's, all right, we need to put a debug point here and then walk back through or take a look at the video that you have or any of the screenshots that you might have taken to see what's going on. Finding ways to put in useful error messages or finding ways to make assertion, raise exceptions in places that have more contextual meaning. So for instance, this is kind of where the successful sign in versus unsuccessful sign in idea comes up. If at the end of the method for successful sign in or unsuccessful sign in, you are waiting for a condition or you are expecting a specific condition, you can throw a more useful error message that will let you know, aha, the issue isn't, I couldn't click on the checkout button. The issue is that I couldn't submit the previous form because this field was missing something. So finding ways to increase the value of your test by getting useful error messages where you need it is very useful. I also try to avoid checking the absence of an element. It's a lot easier to explicitly say, is this element here than it is to say, is this element not here? Especially in a dynamic environment. So if you have a choice of is the sign, like are we signed in? Maybe check to see that the username is there and are we signed out? Check that the sign in link is there when it wouldn't be otherwise. Rather than just having a boolean that is, if you see it, then it's true and if you don't see it, it's false. Try to actively check for things to be true. Rappers and helpers, this is the fun part. Are we on break out there? This is like two sessions in a row here. So we're doing a little more here. Okay, so rappers and helpers. So this is the glue code. This is the stuff that water is really good at in Ruby. These are the things that help take problems that it would have in your code and make it easier to deal with from a testing standpoint.  These are the errors you're most likely getting. It's stale, not visible, can't click it. Almost all of these are addressed with some form of synchronization. There's five different ways to do synchronization. Three of them are local and two of them are global. So let's talk about the local ones. Hard coding sleeps. This is literally the first thing I do when I see someone's code for the first time. I do a search for sleeps and see just where they're at with it, like how good is what they have done to figure out how to make it work without a lot of band-aids? How mature is the framework? There are a couple places where I might use a sleep, but not for synchronizing a site specifically. So fluent weight, this is a specific Java implementation. The idea of returning self from each method so that you can update parameters. Usually I find this to be overkill, and so most of the time when I'm writing Java code, I'll just use an explicit weight because it's just this easy one line. So the issue, one of the other disagreements I have gotten into with some of the other core contributors in the Selenium project has to do with the idea of global weights. Traditionally, this is an implicit weight. An implicit weight, who here uses implicit weights? Who here used to use them and or knows please don't ever use implicit weights? Okay, who doesn't know what an implicit weight is? Okay, most of you are not raising your hand at all. All right, so implicit weight is a, so an explicit weight is the codes polling for the condition. Hey, element, are you there? No, are you there? No, are you there? Yes, okay, now continue. An implicit weight is a global setting to the driver that says go locate this and don't come back until you've found it and time out at this point. And so the code no longer has any control. There's no more, there's no test logic that happens. If you ever do a find element or a find elements and it's not there, it's gonna wait that time out length before it comes back and returns control to the code. I like the idea of implicit weights. I do not like the way that it's implemented in Selenium. There could be problems with if you do want to check to see whether or not an element is not there, it's you have to wait for the full time out in order to validate that. And the other issue is mixing implicit weights and explicit weights can have some really weird behaviors. You can get these weird exponential things that are really hard to debug and it's not fun. But there is another option other than using an implicit weight to get the same global benefits that an implicit weight provides. So how many people have base pages for their page object? Make heavy use of those. All right, I think base pages get overused a lot and we'll talk about that in a second. But one of the things that you can do in a base page is to provide a global weighting strategy or as I like to call it an abstracted weight. And that's essentially for every action. This isn't on location, this is on an action. If you're gonna click something, you should never say to click an element that you think might not be there. Your test should be declarative, it should be click this element, it's gonna be there and if it's not, then it's an exception. So go ahead and wait for that before you do the click. Make sure that it's present, it's displayed, it's enabled, whatever that logic needs to be for you to want to do that, make sure it's in that state before you take that action.  And what this does is it provides you global protection for the element not being in a state. One of the things in a previous version of, you know, it gets you into water history, but essentially when water was adopted web driver as its implementation, it adopted a very selenium view of taking action and if it doesn't work, throw an exception. And what people would end up doing is we had this method called when present. And so define element dot when present dot click. And just looking at it, it's like, well, this doesn't make any sense when would I ever want to click something that's not present? So why is when present sometimes there and sometimes not? So the first time anything would ever fail, we'd be like, oh, well, maybe it's a synchronization issue dot when present. All right, now just run it again and just wasting a whole bunch of time just seeing maybe that'll fix it, which is not the most efficient way to do things. And you would have when present scattered throughout your code. So what we ended up doing when we updated to water six two years ago is I put together logic that made it so that it was a universally globally applied system on the action. Every time an action was taking, we'd check to make sure that the text field was not read only. We'd check to make sure that the button is not disabled because you should never in a test try to click a disabled button because again, it would not do what the test would be testing for. So essentially it would look about like, well, so that's just showing how to extend the base page. Yeah, so the problem with that is performance.  We had someone from, it was at SauceCon last year and someone came up to me and was like, ah, I'm using water, I love it, it's great, it's super reliable, but it's just really slow. I'm like, ah, it's not that much slower. Running it locally, yeah, it's like 15% overhead. Well, you run it on an external service like SauceLabs and like, oh yeah, this is a lot slower. So we went through and I'm like, well, what is, it's super chatty, it's making all of these extra wire calls. Well, what if, instead of making all these wire calls, we move from going wait act to act wait act. From a computer science standpoint, this is not good design because this is flow control by exception, which is bad, but it's very performant. It only does the waits when you need to wait. And if you don't need to wait, it doesn't make any extra wire calls. So this is kind of the ask for forgiveness instead of ask for permission model. And if I'm working with anyone to speed up tests on SauceLabs, usually the issue with some kind of latency is just a lot of, there is some set amount of time for each wire call to get sent and received that vastly overwhelms all of the processing time on each side. Anything you could do to decrease wire calls is gonna have a huge improvement. So last summer, when we implemented a wait act wait, sorry, an act wait act strategy, Slack who uses water reported a 30% performance improvement. Great, excellent.  And we took another stab at it and got another 20% out of it by reducing some of our extra wire calls for some of the validations that we were doing. Things that were redundant. And so there's a lot of things like this that people design their tests locally where it doesn't matter how many wire calls are made because they're milliseconds. But if you have tests that are executing on a remote system, that's something to keep in mind as far as how many wire calls are you making and what can you do to decrease. So if you have any kind of global waiting strategy or even if you've got explicit waits in a lot of places, finding a way to take that explicit wait and doing the wait and then act and trying it and rescuing it. In water our logic for this is like 40 lines long. So there's a lot of conditionals and there's a lot of things to make sure that we're handling it exactly the way we want. We handle text fields differently than we handle buttons which we handle differently than divs and anchors. So there is a lot more logic that we put into it and a lot more logic that can go here but that's something to keep in mind. Another thing I wanna touch on is the idea of composition versus inheritance. One of the things I don't like about a lot of base pages that I see is it's like let's take all of the different methods that is common, anything that we might use on more than one page object and let's put it in the base page. Problem with this is you now have a whole bunch of things that don't apply to most pages or just pollute that namespace in a way that's not obvious. So anytime we can create objects that can be acted on in a more contextually meaningful way. So rather than just putting click as a method in the base page, what about doing a get element? We create an element object, we compose it with the selenium object and now we can query that object for taking actions or getting information or any of the things we're gonna do with it rather than just putting all of that stuff up into one huge base page implementation. So this is another thing to consider is minimizing how much extra stuff is in your base page by trying to abstract out into separate wrapper classes and maybe that means like having a browser class that you can ask the URL and the title and things of and having a Windows class where you can interact with Windows and alerts and things like that independently of just throwing everything that might be used in more than one page object in the base page. So this pretty much comes down to figure out the reason for the test, write your arrange act assert, figure out what data you need, figure out what pages are gonna be used, then I implement the data, implement the actions, implement the elements and then hopefully everything works. Hopefully. So getting this to work at scale. So this means parallel and parallel means autonomous. So I wrote a Rails app for this address book. It's at a.testadressbook.com. It's essentially the very basic component of any test automation, authentication and CRUD. So manual testers, when they go through a site like this, they're gonna try and find the most efficient flow that they can do in order to test all of the things that need to be tested. A journey for them might look something like this for the happy path, going through and doing the navigation, taking the actions, doing the verifications, all the way through. When we look at something like this though, there's really only six things on this list that are especially important. We care about sign up, sign out, sign in, new address, edit address, delete address. These are the functions that matter in this flow. And while for a user, doing them all one after another and doing whatever variations on those pages makes sense, from a computer standpoint, it doesn't make sense to do all of those in your automation together. That's the code for doing all of that. When I ran that on SOS Labs in water, this was before I optimized water, so it was slow. About three minutes to run it all in serial. So what if instead of running it all one after another, we break it up into the six separate components? You need to sign up. We'll create it, yes. Sign up, log out, log in, authenticate it, yes, addresses. In order to create an address, we just need to have a new user. New user could create an address and we check and see if the address is created. To edit address, we need to sign up a new user and create an address. There are steps that for some of these tests don't need to be taken to do the bare minimum to test the action that we care about. So I split these up, I actually split these up into eight tests last summer. And I got this running in a minute and a half. Essentially, we are limited by how long the, sorry, this is very difficult to read, but the edit at the bottom here is what took all of that time. Everything else completed in less time and so you're essentially, when you're running in parallel, you're limited to how long did the longest thing take to run. And then the issue is when you split it up, once you've split it up into something like this, then it's like, okay, now what can I do to make that even faster? And that's where APIs come in. We're talking about responsive websites and dynamic things. When we have your REACs and your angulars and your services that are taking the same information at the same endpoints from both mobile and web applications, you have some gateway. There's certain endpoints, there's certain JSON blobs it's expecting to see, certain JSON blobs it's gonna send back. When you have this kind of gateway, what that means is the same information that you're sending through the web application you could send through a REST client or some HTML library. Rather than creating an address by locating every single field on that page, locating it, sending text, locating it, sending text, locating, querying something about which is selected and selecting it or not selecting it down the list. Put all that JSON into one blob, send it to the one endpoint and you've now set state. You can now edit that address or you can delete that address without having to spend all of the time it takes to put that together. Another interesting thing here is authentication. This doesn't always work but it is possible for you to send a REST call to get a token for authenticating a user.  The only trick here is you have to be on the same domain as what the cookie is gonna be set in order to use that cookie.  You don't have to go to a page, fill out a form, it all happens in the background and you navigate directly to the site you need to be on. Sometimes this doesn't work in production for cross-site scripting issues, there's certain security things that could come into place. I've actually had success at companies that I've worked at to get special things set for staging environments. So at one place I had an admin password that you could automatically create credentials on a staging environment or something that allowed you to get a token that wouldn't work exactly the same way on production. So that's something to keep in mind that even if there's not something that is enabled in production, you might be able to work with your developers. Same kind of thing, creating a user, like sometimes they have to be provisioned or you have to verify an email address before you can start using that user in production. So that's why people will create all of that. We've got 10 users for all of our tests because it's gonna take us a while to provision all of those users. Well oftentimes you can work with your developers to automatically provision a user in some fashion, automatically set state for that user in a staging environment. And that might be, hey the message queue normally checks every 20 minutes for something. Well in a staging environment, let's set it down to 15 seconds, 10 seconds. And that way it's something that you can automate in a much easier fashion. And so I kind of refer to this usage of APIs as transitive testing. Because essentially if you can verify that what's going on in the UI is the same thing as what's going on in the API, you don't have to use the UI in order to set state. So you are creating a user in the API logging in and then you can just make sure that both the browser and the backend through the rest client are giving the same result. Now do that in the API. Create the user, authenticate the user. Now you can create an address. Same thing with deleting an address. Create the user, authenticate, create. All you're doing here in this test is making a bunch of API calls with the rest client, opening the browser to the base, since you have to open the browser to the base URL, set the cookie, navigate directly to the address that was created from the ID that you got back from the rest call, and then delete that, click the delete button and verify that it's been deleted. And once you can do all of this, you can run the same thing that took us a minute and a half. We've now got this down to 45 seconds. Because you're not going through the UI for everything. I throw this slide up here. Mark Brunner-Hamm has an API framework repository that has implementations in each of the different languages. I updated the Ruby ones with more syntactically correct Ruby stuff, and then I kind of took what he did and made something, a gem of my own, that kind of took these ideas and expanded them. But this is a good way to kind of take a look at how you could test, he's got an API for a web application for booking that he has a lot of examples for. So, all right, so that's my presentation for today. Does anyone have any questions for me? I think we've got a few minutes still. Yes, sir? The reporting? Yeah? When you're after hook or whatever. Yeah, yeah, yeah. So he's asking about reporting. I'm not sure I'm gathering the full context, but you're essentially like in the after hook, so either you have a watcher that's gonna pay attention to when hooks get sent off and execute at the end of the test, or you have an after hook in your test runner performance. It's more for specific APIs you want to measure how much time it takes. Well, the reporting I'm talking about is like the functionality. So it's essentially you've got an assertion, it passes or fails, and then what you do with that, you have complete control for where you're sending it, how you're storing it, how you display it, that you can create your own dashboard that'll do those things. There's all sorts of different options that you have for what you do with the results of the assertion that you make.


in this video are available on the opendatahub.io website in the Getting Started section. Specifically, we will be going over the quick installation steps and a few of the steps in the basic tutorial that will demonstrate how to access the Jupyter Hub application. So let's begin. Here I have an OpenShift 4.1 cluster that was deployed using code ready containers. I've configured this cluster to have six CPUs and 12 gigabytes of memory, which is enough to deploy a few of the components provided by Opendatahub. Feel free to increase these resources as your hardware allows. Let's get started. First thing we need to do is create an OpenShift project for the Opendatahub deployment. We can do that by clicking on Create Project, and we'll give it a name of ODH. In order to deploy an instance of Opendatahub, we first need to deploy the Opendatahub operator using the OpenShift operator hub. To access the OpenShift operator hub, navigate to Catalog and Operator Hub. Here you will see all of the community operators that are available for installation in the cluster. You can find Opendatahub in the list by typing a keyword Opendatahub and selecting it from the filtered list. Click Continue on the dialog. Before you install the operator, feel free to review the description of the currently available version for this release of the Opendatahub. For this video, we will be deploying Jupyter Hub and the RAD Analytics Spark operator. To begin the operator installation, click Install. On the operator subscription page, make sure the project that we created earlier is selected, and then accept the defaults for everything else. On the subscription overview page, wait for the status to show one installed. Now that the operator is installing, we can view the status by going to Install Operators and seeing Opendatahub operator. Now that the status has installed succeeded, we can begin to customize our deployment of Opendatahub. Click on Opendatahub operator, and then click Create New. This page, you're presented with a template of the Opendatahub custom resource. In this template, we can customize which components will be deployed and also how we want to configure those deployed components. For this, we want to make sure that Jupyter Hub is deployed and the Spark operator. You can control their installation by changing the ODH deploy property to true. This will make sure that these are deployed in the Opendatahub. We default to deploying Prometheus and Grafana by enabling monitoring, but for this video, we want to disable that. So we'll change its deployment property to false. Since we're running in a cluster with limited resources, we need to modify some of the properties for the Jupyter Hub deployment. Specifically, we want to change the CPU allocated to the Spark cluster nodes to one. We want to reduce the Spark memory to one gig. And we'll go ahead and disable the Spark node, worker node. All the other options will allow us to safely deploy Jupyter Hub in this deployment. If you want to customize any property further, feel free. Now, to deploy Opendatahub, go ahead and click create. On this page, you'll see the Opendatahub deployment custom resource that we just created. If you want to monitor the status of this Opendatahub deployment, click on workloads and then pods. Here you can see all the pods that are deploying as part of our Opendatahub custom resource creation. Specifically, we're deploying Jupyter Hub, a database for Jupyter Hub, and the Spark operator. Once each pod reports the status of running, we can proceed with accessing Jupyter Hub. So, to access a Jupyter Hub application, we're gonna navigate to the Jupyter Hub route that was created with this deployment. So we can click on networking and routes to view all of the routes that are available as part of this deployment. So, to access the application, we just click on the location field for Jupyter Hub. We can proceed by clicking sign in with OpenShift. And we'll go ahead and log in as the regular developer user that's available on this cluster. For the Opendatahub deployment of Jupyter Hub, we've added a few customizations to allow an individual user to customize the notebook deployment for their workflow. Specifically, they can select from a list of available notebooks, change the notebook pod size to fit their workflow, and if their GPU is available in the cluster, they can access those directly. Also, if they wanna add any specific environment variables to be available in that pod, they can do so now. Here, we'll go ahead and add a Hello World environment variable that will be available in our Jupyter Notebook. So, we'll go ahead and spawn the notebook pod. Here, you'll see our notebook pod is spawned and we can create a blank Jupyter Notebook. So, to create a blank Jupyter Notebook, we can click on New and then select the Python 3 notebook. In this Jupyter Notebook, we have an empty cell where we can start to insert our Python code. So, just to show you how to run this, we'll just create a very simple Hello World script and execute it. This is just a simple script to show how you can access the environment variable in the Jupyter Notebook pod and print it out. We can execute the cell by entering the information and just hitting Run. From here, you can start the basic tutorial that is available on the opendatahub.io website. The basic tutorial will demonstrate how to interact with a Spark cluster and read and write data to object storage from a Jupyter Notebook. For more information on the Open Datahub and its components, please visit our website at opendatahub.io and subscribe to this channel to be notified when new Open Datahub videos are available.