Wyyddo ni, gweld y gallwch ar bobiwch gynnwys, a fydd ni yn gefnogi gwaith gan peninsulaon gyfer ddigon o ddweud yn fanlif ... Is e'n gwellaeth ydym. ... ac mae'n gwneud yng nghymru sy'n gweld gan y peth... ... ac mae'n gweld â hynny. Cyn ymddangos yn y rhan. Dwi'n gwybod i'n wneud hynny. Felly... ... am fyddeinodd am ddifnwysgwyd... ... yn ymweld i chi'n ffordd ymddangos. Rydyn ni wedi allan fod o'r stori. Mae'r ffordd yw'r rhan. Felly mae'n gwybod i'r llynigol. Mae'n gwybod i chi. Mae'n gwybod i chi. Mae'n gwybod i chi'n gwybod i chi. Mae'n mynd i gynnwys i chi yw'r amllunol, y cynghwil sy'n gweld ar y w exact. Um yr un oedd eu gwaith yma. Joe yn yng Nghymru. Heidiwyd yn ei hunain sy'n gweithio'r gael. So mae e'n oed y gwasanaeth yn y gwelio ardisad. Ac mae'n gweithio mynd i gael yn roi, a yn ymarfer hyn maen nhw. Joe yn y magwmiad. Mae'n ffordd ffrindlu. Mae'n ymdwyf yn ei ddweud. Mae'n ffordd yng ngenief. Mae'r gwealth, maen nhw mae 28 honno. Now Joe is on his own. He just has some freelance writers. Joe went fully digital a few years ago and uses WordPress, and we commerce with subscriptions to run his site, and his site is well trafficked. It's also his primary source of income for him and his family. You're going to feel very sorry for Joe by the end of this. I should just warn you now. So get your heartstrings ready for Joe. Joe doesn't believe he has a budget for anything, but he doesn't have a budget for security. He doesn't have a budget for development. So he relies on plugins from the repo. He relies on occasionally buying a plugin. At the heart of his site is this popular page builder, which we're not going to mention. It gives him the flexibility to do whatever he wants. Joe is a really great guy. He's a pretty normal person too, and one day he was on Twitter, as it was called back then. Just browsing around where he saw a funny gif, but there was a link to a website where he could sign up to get more info, and so he was like, oh, I want more funny gifs. That sounds amazing. Honestly, I don't know why he signed up that day, but people do really strange things, and you'll sign up to any old rubbish. Over the next few weeks, Joe got several emails from the site. One called his eye. A build your own meme generator. That sounds great. It's like this really cool website. It allows you to go along and remix memes. One of the features was a browser extension to allow you to quickly insert your memes onto your own website. Joe installed the extension, and then logged on to his website to try it. He was disappointed. It didn't work. Of course, something had happened. You see, as his browser made the way logged in, and he went to the AP login, the extension made its own request. Back to a server far, far away, probably in a different galaxy. And it just sent the following. A URL, a username, and a password. The browser extension clearly malware. Malware designed specifically aimed at site owners of WordPress, Joomla, and Drupal sites. All the extension would ever do is harvest credentials. It never did the thing it claimed it was going to do. Weeks pass. Joe had forgotten he even had the extension installed. Actually, weeks passed, sorry. Weeks passed. The extension installed. Even though it was not working, he just left it there. Like so many people do, it didn't work. He'd forgotten about it. He moved on. And so it carried on, harvesting the credentials. He logged into several WordPress websites each time. I'll have that for these. Off it goes. The original meme site was long gone. The extension no longer available to install. Yet, one day on a forum, in the deepest, darkest parts of the web, a new post appeared, offering a dump of 50,000 legitimate WordPress admin credentials. The cost? $1 per account. Now that might seem really expensive. But that's actually $1 exclusive 24-hour access to the account. After that 24-hour period's over, that whole batch of 50,000 emails will be sold off in what 401 lump sum. So, as part of the public facing part of that, the dump contained the domain names, but obviously not the credentials. Hacking is a business. And like a business, it has supply chains. The group that created the extension and harvested the credentials sold the list onto a middleman. That middleman posted it onto the forum. He or she were then working with the buyers and arranging all of this through. The person at the other end, who bought the list, goes on to exploit the site. But then, they're more than likely going to come back around in this circle and post it back on the same forum. Basically, once you're on one of these lists, you're screwed. Joe, for Joe, life carried on. He went to his login page, though, one day, and a strange thing happened. When he went to login, his password didn't work. He didn't really know what happened, so he just reset it. New password arrived. Carried on with life. It wasn't for another two weeks before things started to go truly wrong. Joe was updating WordPress Core, and it just wouldn't update some sort of permission error. File permissions just wouldn't work. Joe would tell you he is an un-technical person. An un-technical person with knowledge of FTP knows about file permissions, has a deep knowledge of how he uses WordPress, and occasionally he even remembers to download the backup plug-in to his local machine. When someone tells you they're technical or un-technical, it's based on what they assume is their baseline, which is no way the same as what you assume is your baseline. Someone who says they're technical might be the most un-technical person in the world to you, but equally, there are many people, including people in this room, who will tell you that they're un-technical when in reality they know an awful lot. Joe does know an awful lot. You should still feel really sorry for him. He wasn't getting anywhere, though, that day, so he contacted his web host. An email a little while later said, fixed it. Brilliant. He logged in. It was fine. They'd fixed it. Let's leave Joe now, and briefly visit that web host. The technician who got Joe's ticket was incredibly busy that day. Things had been going wrong. The local coffee shop was closed. Someone was off sick. When he got into the office, several of his colleagues were off sick. The technician had logged on to the server and to Joe's files, and he used API CLI to update it. So, like Joe, he got the file permission error. Same error. So, what did he do? He does whatever a good person does. He uses Pseudo. Pseudo, do you want to update? He gets a horrible error message. Well, that's annoying. We'll just skip that and put our route at the end of it. Problem solved. The site is updated. Brilliant. The technician moves on to the next ticket in his list. Something to do with emails. The weekend passed. Joe went to the seaside with his partner, and so it wasn't until Monday morning when he got his cup of tea that he goes to log in. His emails have been quiet. Indeed, if Joe had been paying closer attention, his emails have been quiet since 2am on Saturday morning. When he went to his website, he wasn't presented with the login screen. Instead, he was presented nothing. His site wouldn't load. Panicking? He went to log in to his host control panel. It wasn't working. It was gone. He tried using his phone, different IPs, asking friends no one could get on to either of them. He went through the call log on his phone and found the host's phone number, and it just rang and rang. Nothing. Let's go back a couple of days to Friday night. The late shift at the hosting company were having a quiet evening. Really, the only ticket item for them was one of the shared hosts had triggered some sort of alert about the backups failing, which is odd. They weren't meant to run for a couple of hours. The admin logged into the server, found the SSH connection between the host and the backup server. Wasn't working for some reason. They tested with their own credentials. Absolutely fine. Some fiddling and they realised a background service wasn't enabled, so they just restarted. We'll go forward just ever so slightly to 1am on Saturday morning when things start to go very wrong. The backup server was running. The backup server is really actually a cluster of servers and it backs up all sorts of things. It has connections to absolutely everything, to every one of their hosts, to every one of their servers, to their email servers, even the office PCs. Suddenly things were going offline. It started with the server connections being lost, then the office desktop machines, and soon the entire host was offline. They had become a victim of ransomware.  In their place a ransom note for a very, very large amount of Bitcoin. They worked through the weekend to try and restore things. They had no way to reach out to customers. Even their Twitter account had been compromised. Do they pay? Would you? A week passed. The host had made the technical press not a good thing. Web hosting, victim of ransomware, all client data potentially lost. It was nearly 10 days later, when Joe was reached out to on Twitter by an account offering to restore his site for a fee. The ransom group had failed to reach an agreement with the host so they went to the individual sites that were affected and was offering to give them the site data. For Joe, the offer was tempting. But his host being the domain registrar and where he had his DNS, the data was almost less important. It wouldn't get his site back. It wouldn't get his email back three weeks later. The host has partially restored DNS facilities. Joe has got his email back and his DNS. He used an old backup from his machine from a few weeks before. He finally tried to log into his site. His password wouldn't work. Oh, that's right. He needed to use the old password as this was an old backup. But once in, everything looked fine. So let's, with the power of hindsight, go through and see what went wrong. Joe installed a browser extension, perhaps without really understanding how much power these things have. A browser extension with the right privileges can steal any data that is passed through your browser, much like a WordPress plugin can do anything on your site. Knowing and trusting your sources, whether it be a browser extension, a plugin, a friend, all of this is imperative that you understand and you trust. Passwords don't stop working. It's a huge red flag. In this case, our bad actor had deliberately changed it. But they deliberately changed it not to stop Joe. They changed it to stop other bad actors because they knew that in less than 24 hours, that list was going to be resold again. And so they needed to stop other people getting into his site. They helped him, sort of. This, as I said, should be a huge red flag. By the time that that happened, the site had already been compromised. They'd already changed the file permissions to lock to the specific version of WordPress. They'd already placed malware and that malware was specifically designed to both give them a secondary backdoor so that if the password was changed, they could still log in. They could still do anything they wanted, but also had us checked for whether it ever got enhanced privileges, i.e. got root user. Because once you have root user to a server, you can do anything with it. You are the most important thing on the machine. How could this have been prevented? Joe could have installed a two-factor authentication plugin. Done. That's all he needed to do. Even if his password had been compromised, if he had a two-factor enabled, he'd have a second factor. If you want a recommendation, two-factor is the plugin I would recommend to do this. With a plugin like this installed, there would have been no chain. This would have prevented the whole thing. There are other solutions you could use single sign-on. You could use magic links. But for most people, the simple solution is two-FA. The malware will never know the payload, but it was designed to provide a backdoor and a trigger if it was ever run as a privilege, which leads to that poor technician. I don't think I need to tell people never run things as root. You probably get the hint that it's a bad idea. But no, we put a big warning in. Never run as root. Also, don't iron things while wearing them. Similar levels of... Once the bad actor got that root privileges, they were able to start probing and investigating. From there, they could quickly work out, hey, I can get to this backup server because I can send stuff across to it because it's bidirectional. It's not just one way. Once they've gone to the backup server, they realized they could go anywhere they liked. So they were able to pivot to full control. Almost certainly this was never the original intention. In fact, the person who probably went in initially might not have even been the person who put the ransomware on. As we said at the beginning, hacking is about business. They simply sold went, oh, I have the potential for this. So they sold it on to the ransomware to crew. For Joe, he was screwed because all of his eggs were in one basket. Had he had his domain registered somewhere else, he could have then been able to move it to a different host. Had he had his DNS managed somewhere else, he could just point the DNS somewhere else. Had he had his email, at least with somebody like Gmail, he might have had a chance of having his email restored at some point. Gmail for business or whatever we're calling it. So he would have been significantly less screwed. By consciously spreading the risk to multiple service providers, you potentially increase costs and complication. But if one part is affected by ransomware or technical error, you can work around it. Two simple mistakes by two separate people resulted in a catastrophic chain. We can always put technical solutions for each of the problems. We can install two factor to prevent WPCLi, and we can prevent WPCLi from running as root. But security isn't about technical solutions. It isn't about that side of things at all. It's about mindsets and humans. Everyone is responsible for security. In your heads you were already putting blame on people as part of that process. But neither of those people were to blame. The whole process was to blame. So everyone is responsible for security and we all need to develop those tingly danger senses. We need to be given the space to do that and also not be put into positions where we have to cut corners.  And yes, that includes the MD who has an administrator account. Why? For those of you who don't know me, I'm Tim Nash. I'm a WordPress security consultant and professional doom speaker. I help organisations get better at their security, both the technical side and the human side. You'd be pleased to know that this tale was mostly fictitious, but was grounded in real events. There was a website name generator and a browser extension that grabbed WordPress credentials. More than once someone has accidentally run WPCLi as root at a hosting company. Warning signs normally have a history after all, and there has been more than one host who has become a victim of ransomware. I think we should let Joe get on with his day. It's been not a great few weeks for him. I guess we can just hope that he changes his password before one of the many buyers of that bulk buy list tries to log in again. Thank you very much. Wow, there we go. You don't have to ask questions. Yes? Yes. How I've seen it done at least once is that you can set a WPCLi.yaml file, which you can include a pre-execute file inside of that, where you say anytime you run this file, put it through. You could also put it in a pre-append file via php.ini. You could also just hide it in WP load, but you'd probably get spotted by someone doing some checksumming. You could put it in the WP config file. There's a load of ways you could do it. Probably the safest one that you wouldn't necessarily notice though is probably in the WPCLi.yaml file, because you'd not really think to ever look in there. Any other questions? No? You can always come and ask me your highly sensitive and scary questions afterwards.


Can I go? You're going to start, yeah. OK, so let's start the official part streaming now. So my name is Jelko Philippin. I work as a QA engineer for Wikimedia Foundation. And welcome to our browser automation workshop. So today, we will be talking about a lot of stuff. But we will have a few sections. So we will have a part where I'll show the feature that we want to work on. Then we'll show some code. And after I show some code, I'll run it. So you see how the test runs. And then I'll ask you to help me write another test. And then we'll implement the test. So that's like a quick overview of the workshop. So a little bit of housekeeping notes. So if you didn't already read the meetings page, so I've worked very hard to explain everything, all the setup needed for this workshop there. And if you didn't set up everything so far, please stop now and pay attention to the workshop. There will be time after the workshop where me or Chris or somebody else will be able to help you. I don't think you have any problems with installation. So let me show you the feature that we'll talk about. So I'll show my screen. OK, a bit recursion there. Can you see my screen? Everything OK? It's good. OK. So we will be talking about this WikiLog feature. And it's a nice and fun feature. You have to be logged in to see it. And you have to be on the user page of another user. So I'm logged in as my official account. And I'm on the page of the screen, Djelko. Share once more. OK, let me try it again. Can you see it now? Oh, wait, maybe I have to do something here. Maybe I should give the screen to Djelko. So can you see it now? So you don't see. You are not seeing now Djelko's screen? No, we have seen Djelko's photo. Now we see, Mike. That was it. That's it. Or was? I was. Kim, do you know what to do? Like no. Well, it's Djelko's screen when Djelko's talking now. Yeah, you just talk and the rest of us. OK, so if I talk, then you see my screen. Basically, yes. So I'll just keep on talking. Let me know if something's wrong, because I don't see the Hangout window while I'm showing this. So if anything's wrong, just let me know. So I hope you can see my screen now. So let's start from the beginning. So I'm talking about this. Sorry. So today, we will be talking about a big key log feature. It's a feature that allows you to send appreciation to another user. To see the feature, you have to be logged in. So I'm logged in as my official account. And I'm on the user page of this Selenium user that we use for testing. So as you can see, it's completely messed up. This is a test server, by the way. If you don't have an account here, please create one. You will need it. Like, if anything else fails, you will be able to use it like for manual testing. So when you are, and let me open, let me show you how it looks when I'm on my page. So when I'm on a page of another user, there's this little hard icon here. And that's way to assess the weekly log feature. When I'm on my page, there's no hard icon, because I can't send weekly log to myself. When you click the hard icon, a window opens, and there's several options of sending appreciation to another user. So for example, why would you send this weekly log thing? Can somebody mute? I hear some noise in the background. Is anybody not muted? Kim, can you take a look if somebody recently joined and mute them? I'll assume that Kim will take care of it. And please, if you just recently joined, please mute yourself. There's a button off the top right, so it doesn't make a big noise. So when you open the weekly log feature, there are several options. Bonds, barn stars, food and dream, kittens, and make your own. The reason you would send somebody this weekly log, it's like a gift when you're working on an article and somebody makes an ice edit or something like this and improves your article, then you want to send appreciation. It creates this good community feeling. You can send them a barn star, food or dream. My favorite one, of course, is beer. Or of course, there's kittens because internet. And you can make your own image and make up a header title and message. So this is the feature that we are going to test. So for example, let's try to click preview. So we picked barn star, just left this original one, entered some test message, there's a preview here, and we can send it. It didn't work yesterday when I was testing it, but let's see if it works now. It works fine. So when you open, so we are on Selenium user page, and we can see I sent beer to the user yesterday and barn star today. And this is from previous workshop, people sending kittens and everything. So this is the feature itself. Let me see my notes. So why did we pick this feature for these workshops? So first, because it's fun, this is probably one of the most entertaining features on media wiki software. Another one is because it's fairly complicated, but yet simple to automate. So the pop-up appears. There's a lot of options. And every option has sub-options. There's select boxes and links and text fields. And there's this flow of steps that you have to do. This couldn't be automated in a unit test, because you need to open a browser. You have to log in. You have to go to another user's page. You have to click this heart icon. You have to select a few things from here. So it involves JavaScript and everything else. So of course, JavaScript behaves differently in different browsers. So this is just a perfect feature for us to work on for this workshop. So for example, what could we test here? So let's turn this back to the honest. This is a workshop, not a lecture. So could anybody let me know on a test idea what should we test here? If we do something, what should happen? Like in a couple of examples. We can start with one, of course. And feel free to share your screen. I'm not sure how good that will work. Anybody? You can just talk, and I'll share my screen and then try to reproduce. Well, I know what I want to do. Don't we already have some tests? We do. I can show that yet. So let me show you what we have. So let me just see my notes. OK, so I can go forward and show you some code. So we already have some tests here. Since Chris suggested it, we can start there. So. Chris, funds, please. Oh, sure, sure, sure. So let me see here. So what I see is tests for the barn star, for food and drink, and for the kittens. And that suggests that we might need a test. Anybody see what's missing in that list? So I recommend everybody to select Zelko's screen on the Hangout so you don't see the rest of us when we talk. Yeah, so yeah, I forgot to say that. So if you click my screen on the bottom, you'll just see me. And when somebody else talks, you'll be able to hear them. But the screen won't change. Is this better, or should it be bigger? I think it should be bigger. I can't really read it very well. OK, let's try bigger. So this is my setup when I pair with Rachel. But just a second, you'll need to analyze funds. I'll increase to 20, and we'll see there. So 20. Is this better, bigger? Yeah, it is for me. I don't know about for other people, but it's better. Anybody else? Can you read this text? Or should I make it bigger? I can see it. I can see it. So this is the code. Let me just do one more thing real quick. There's debugging the ID that enables one of my custom settings. OK, so let me talk about this. So this is Cucumber, and this is the way we write tests. As you can see, it's pretty much English. It's structured, but it still reads like English. At the top, for now, you can ignore this. It just wants to say that we have to be logged in for this test to work, and that we want to run the test at this site. So we are testing Wikilev feature. That's explicit enough, I think. Background steps will run before every scenario. So if you've used another test framework, this is something like setup. So given I'm logged in, we have to be logged in. When I visit user page of this Selenium user 2, and I click Wikilev, this icon. So this will be executed every time before the rest of the steps executes, because we need some setup. Then we have barnstone options, food and drink options, and kitten options. So we'll go for one. When I click barnstars, I should see barnstar checkbox, and I should see message text field. So let me show you what it's all about. So I am logged in. I am on another user page. I click the Wikilev. I click barnstar, and there should be, let me just check what it says exactly, barnstar checkbox and message text fields. Where's the checkbox? But there's the text field. OK. So I would guess maybe this is what's my checkbox. I didn't write code, as probably you can notice. And we have some similar setup for other features. So we have food and drinks and kittens. So let's see what's here. So we have barnsarns, food and drinks, and kittens. Let me show you how the tests run. So while the tests are running, it'll take a while. Please think like, what could we test next? So this is the way we run the test. I'll just quickly check if I'm still in a hangout. Yes, OK, great. You should see a Firefox popping up at the bottom. Then it will appear on the screen. The browser will go to this test site that we call Labs. This usually works faster, but now I'm on a 3G connection. So it's working, but it's not as fast as usual. You can see the login screen logging in. We logged in as Selenium user. We will go to Selenium user 2, user page. The test will click this lovely heart icon. Click the barnsarn and check if these options are there and do the same for food and drink and kittens. So while the tests are running, anybody bold enough to propose another test we should automate? Like, I have plenty of ideas, and I'm sure Chris does. But I would like to hear from somebody else. And Kim can even cheat because he was at the previous workshop and we had really similar examples. Do I hear somebody proposing the next test? Can we now test for the barnsarn and for the kittens and for the food and drink? What's missing? Make it your own. OK, very good. That's one option. OK, so you saw the tests running. So we have three scenarios of them passed, 20 steps, two minutes, whatever. So the next step would be make your own. So if we go with the trial. So if you have barnsarns, food and drink, kittens, and make your own, I have another idea. So let's go even further. So I have a simpler test in mind. So we can pick one of those and automate it. So on this page, what could we test here? Like, my idea was, everybody has an idea? Do we have a WikiLav icon on your profile? No, that's what I thought we can test. But if you don't have, so you cannot send WikiLav to yourself. Yes, but that's an excellent test, right? So if something's broken and WikiLav appears here, there will be a bug, right? So we could write a test that would check if this hard icon appears on your user page. And it should not. And then we could check if it appears here on another user page, and it should be there. So that's a couple of tests, right? Yes. And then another test that, so we started from this screen and went the options. But can anybody think of an even simpler test? So I went way back. So when we are at the page and even checking if the feature is there or not. So when we check that the feature is there, one test that I would propose is just clicking it and seeing if this pop-up appears, right? That's another feature. So we can't test all these sub-options if the pop-up doesn't appear for some reason. For example, let's make a wild guess that when in I6 you click a hard icon, some JavaScript breaks, and this pop-up doesn't appear. Just a wild guess. I wouldn't expect it to happen, especially not in I6. But who knows? I'm not sure. I just want to mention that I got a question via IRC from someone that is watching the stream. So it is a pastry here in the chat. OK. Can you read it? Because if I go to the Hangout, then like, I got it. He wanted us to know about the page objects, and I told him we'll be covering page objects here in a little while. Yes, page objects coming soon. There will be page objects falling from the sky really, really soon. We are focusing on the feature right now, but there will be enough page objects, trust me, really soon. So I think we have enough ideas for now. And I would propose to write them down at the moment. So is this somebody suggesting something else? OK, it's not. So feel free to type with me and follow what I do. And I would even suggest you do so. So by the end of this workshop, you will have new code and you will be depending on how far along you went with your Git and Garret setup, you'll be able to submit it. You'll be able to have your changes either only in your machine, you could submit it to our GitHub repository or to our Garret repository. So let me close these two. So let's open my favorite IDE. And let's write some scenarios. I would, so should we have a vote, should I just like pick one and implement it? I think it'll be better just for me to pick one. So let's start with a really, really simple test. Let's move it way up here. So we are in this Wikilab feature file. Let's make a new scene. So Wikilab icon is visible at another user page. That doesn't sound really good. So please help me with English. So Wikilab icon is visible at this bar. Oh, so we have, yes, the background assumes that we already clicked Wikilab. So we'll have to move that a bit down. So let me rephrase. So given I'm logged in and I visit user page of Selenium user 2 and I click Wikilab, so this is what our background already did. So it's open Wikilab. So before we do some refactoring to this page, let's make it simpler. So when I click Wikilab, then we will just check that the Wikilab pop up up here. So Wikilab, given I'm logged in, when I visit user page of Selenium user 2, and I click Wikilab, then. So this is really, really simple, but it's an important test to make. So for example, when we click this, if this window didn't appear, none of the rest of the test will make any sense, right? Because we couldn't select anything if this test breaks. So I think it's important for us to have this test. If you have any questions, please let me know. If I'm not making any sense, if I'm going too fast, let's run this for now. So we have some cucumber test written right now. And let's just run it. I have a question. I have a question. So when you do it, it looks like very easy. It feels like the software will be able to digest just anything you type. But how do we know what the software understands, or doesn't understand? I'm going to show you that. So this is exactly so we were just finished the first part of the workshop where we talked about this Wikilab and what should be tested, and all this stuff. And now we are going into code, the second part of the workshop. And now I will show you how to get from this almost magic, like English, to something that will actually run some browsers. Is that the question? Maybe. So for me to understand, for this phase, is it good enough if someone else than you understands another human understands what you want to explain? That's the goal of this phase. Yes. Thank you for the reminder. So I'll just say a sentence or two about Cucumber. So Cucumber is this test automation tool that helps us run tests. But in my opinion, its biggest feature is communication. So it's a communication tool. It allows two people to communicate. So for example, me, I know how to make browsers cry and sing. And somebody else knows all about some software and how it should behave. So the two of us, for example, Kim knows all about Wikilab. And the two of us could just sit down or exchange a few emails with these scenarios and agree on how this feature should behave, or how this software should behave. And then it's really easy for me to make the code run. But the hard part is thinking about good scenarios and what should be tested and what should be automated. Does it make more sense now? Yes. In the meantime, Carol is asking another question. She says, how many steps are necessary in the test case for a scenario to pass? Do you have to have when, then, and words to trigger the right code? OK, excellent question. So you should have exactly one step, either a given, either a when, or a then. And steps are optional. So to have a test, you should have to like, every scenario should have at least one step, right? Because it's not a scenario, anyways. If it doesn't have any steps, it's not a scenario. So you don't have to have all three, given, when, and then, just one or two would be enough. And steps are completely optional. Does it answer the question? I will assume it does. Silence means approval. So for example. Thank you, yes, says Carol. So for example, in this case, let's make a story about it. So Kim and I talk about this Vicky Law feature, and something should be automated. And he says, oh, I think we should check if the window appears, because the rest of the steps will fail if this window doesn't appear at all, this Vicky Law window. And I say, OK, I write this test down. And I ask him, does it make sense? He says, yeah, that's pretty much what I wanted to say. And cucumber is good, because it will lead you the way. So you just run it. So unlexic cucumber features, you run one scenario. So if we provide line number, it will run just this scenario. So we can type either nine or 10, and it will run just this new scenario. So let's just make sure I save this. In case it's important, we cannot see what you are typing on the console. Let me make it bigger. Wrong shortcut. So it is important. And I'll make my text in the console way bigger. So this is good. This is for my computer, too. OK, can you read now? Yes. So bundle execucumber features, Vicky Law. Let me just make it like, say, half of the screen or something. Bundle execucumber features, Vicky Law feature, column nine, meaning let's execute just this one scenario. So let's run it. If it needs to be bigger, just let me know. I forget. Opens login page. It's a bit faster this time. Logs in as Selenium user. Goes to Selenium user two page. Clicks the Vicky Law icon.  So it says, we have one scenario and one undefined steps. And it even helps you. It says, you can implement the step definitions like this. And it gives you real code that you can copy paste. So let's just do that. So now we are going from Clucumber to Ruby. And we will soon get to page object. So Kim said for the last workshop, people were a bit confused with all the screens involved. So this is one screen. Cucumber feature file. Another screen would be a Ruby steps file. So let's open it. It lives in step definitions. We can have steps. As you can see from .rb, it's a Ruby file. We can just paste line to the bottom. We'll figure out later on if you need to move it. And that's it for now. So we just copy pasted this code to the steps file. Now we are getting into Ruby. Again, let's see what will Cucumber say now. We run the exactly same command, bundle exec. So not to get into a lot of details, this bundle exec Cucumber is just the way we run the tests and features. We can have feature is path to the feature file. And column nine says just run scenario in line nine. Firefox opens again. You'll see this Selenium user lining in all the time. Going to Selenium user page, clicking. OK, so now something else happened. So previously, Cucumber said, oh, you don't even have any code in the step. And now it says, oh, you have code, but it's pending. And I'll show you now what's pending. So we don't actually have any code in this step. So this is a step. So for example, this is another step. And this is page object code. And we will do something similar in this step. So for example, now it's pending. And let's just remove that code and do something else. So we can allow windows appear. So we can take inspiration from another step. So now we are going, now I'll say a few words about this page object pattern. It's just a fancy name for the way we organize test code. Page object pattern says that all the code about the page should be in one place, in this case in one file and in one class. I'll show you the implementation of the class later on. For now, let's just see how we use it. So we say, we are on the vehicle of page. And in this previous example, they say this vehicle of message element should be visible. So we'll do something similar now. We'll say page. So while Joko is typing, I'll mention that. One of the reasons that we chose this framework is because it offers prompts all along the way. Once you have a scenario and you run that scenario, the framework will tell you exactly what needs to go in the steps for that test. And then when you put that in place and you run the test, the framework will tell you exactly what steps are pending. And now Joko is writing a real step that he'll go through, but the framework will always tell you what your next move is going to be. It's a very, very convenient way to create these tests. Yes, and it usually doesn't take an hour to write a simple test. I'm going slowly now and talking a lot. But this is exactly the same workflow that I have. But when you get into the flow, you can do this in just a few minutes. So what we will do today in a couple of hours, it usually takes like five or 10 minutes. Let's talk about this code for a moment. I forgot to mention why do we use this page of the pattern at all. So it allows us to separate test execution from test implementation. So for example, we will define some elements on this page. And for example, we kill our window. We will define it. It's not defined yet. I can even show you. So let's go to the declaration. So this is a simple page object. So we have a class called wikilof page. We include some stuff at the top, but let's not spend time on that at the moment. This should be familiar. So it's user, selenium to user. It's part of the URL. The rest of the URL is towards somewhere else. And here, we have all the elements of this page in one place. And wherever else in the code, we need to do something with this page. It always calls this code. So for example, if this bar star select element, so if this bar star select element, this thing here, if it changes from select list to radio buttons to links to whatever, and we have thousands of tests using this element, we will have to change it in just one place. So here, we will change this select list to a link, a div, a text field, or whatever it is. The name will stay the same. George is asking wikilof window is missing. Yes, and that's where I'm getting. So really good observation. We'll get there. So for example, let me finish the explanation of this page object, and then we'll go to the implementation of the test. So anybody that's worked on software knows that software changes a lot. And everybody did the test software and did some test automation, knows how frequently tests break, because something changed. So for example, this select list, a designer goes to a meetup, and everybody says, oh no, select list. So yesterday, we have to use divs, and it changes all select lists to divs. And all of your tests break. If you have this code scattered around, if we hear, if we said, only kill off page, you find this select box with this ID, and then it should be visible. And if we had this code scattered all over our code, when the change happens, and it will happen, just a matter of time, we will have to go through all the code and make all the changes, and it's really prone to error. Because we always forget to change software. That's the reason we have another layer. So we have this cucumber text describing what we're testing. We have some code that actually runs browsers. And then we have page objects that are repositories of these objects that we deal with. So for example, this barn star select element at the moment is select list with an ID. If the ID changes to something else, or if it's no longer ID, but the name, or class, or whatever, we just need to change it in one place. And all the tests just call this barn star select. And they don't care if it's a select list, if it has an ID and what value it is. Like, it just calls this element. And as George nicely noticed, this wikilaw window is not defined here. So we are on wikilaw page. And there's no wikilaw window. So wikilaw message, wikilaw spinner. Let's see what happens if we run this code. So at the moment, we just made up this code. I think Chris cracked me if I'm wrong. But Cheesy, Jeff Morgan, he calls this wishful coding or something like this. Let's just make up the code and we'll make it work later on. So we just want this code to work. Like, we want to have an ICAPI on wikilaw page, wikilaw window element should be visible. That's what we want. And we'll take care of the implementation later on. So let's see how this works. Let's run the test again. Let's see what cucumber will tell us. So the same command again, bundle exec, cucumber features, wikilaw feature, line number nine. While the test is running, feel free to ask questions. And in any case, we're preparing questions, because in a few minutes, we'll have already passed the first hour. And I think we can have a little break, but not break. Prepare your questions for that break. Yes. OK. And now cucumber complaint. Let's see what it says. So it says, undefined method wikilaw window element for wikilaw page. And we already know that. But still, as you can see, you can just make up code. And cucumber will complain when something is not working. So let's implement the element. So we call it wikilaw window. Just copy, paste, and then type rows. It should go to the bottom. It's sort of vertically. Let's inspect the window. So now, we have to dig into the page and figure out what this window really is. So let's see. Any modern, any contemporary browser has an inspector. Firefox from recently has a built-in one. And the famous one is Firebug. And Chrome, recent IEs, oppressive fire. All browsers have built-in page inspectors. Some are not enabled by default, I think, on Safari you have to enable it. But it's just like a setting in options. So just right-click the element that you want to inspect and go to inspect element in the context menu. Let's see what it says. Oh, it's a deal, right? So it's a deal with a class. So there's a lot of things going on. Style, of course, there are no IDs. So let's see. Yeah, I think let's see what happens if we just use this huge class. So it's a div with this class element. Is there anybody not familiar with the document object model, the DOM, and how elements on a page are identified? John Fred, IDs are preferred. ID is the best way to identify an element in a page. And the reason for that is according to the W3C standard, IDs must be unique per page. So anytime you have an ID value, you can be guaranteed that you're addressing the correct element. However, they're not required. Not all elements will have IDs. We prefer that we have them. But we can get to these elements in any number of ways. You can identify these elements by class, by name, by title, any number of different ways. And in fact, the framework that we're using has a really lovely page in GitHub, which I'll find a link here. Well, let Chris find the link. So there's plenty of documentation about all this stuff. Document object model is a bit out of scope for this workshop. But we can still cover it like in one-to-one sessions or later on. So let's see what we did. So we inspected this div. We found that it has a class, that it is a div. Also, we inspected the element, found that it's a div, that it has a class. We already decided the name. We just checked that it's a div, and it has this class. And now this code should work. Let's see how it works. So let's collapse everything and just leave this one that we are working on. Let's run it and see what happens. There's a 50% chance that we did something wrong, especially with the live coding. In the meantime, there's one question from Karima. She says, in the future, the WikiLoft page object and WikiLoft's feature will be in the same folder of the extension, or all tests have to be in the same project of browser tests? That's an excellent question. So no, this browser test report, well, no, depending on what I answer. So browser test repository is like an example repository that we use when an extension or a feature from MediaWiki Core doesn't already have tests in their repository. So we are still selling this browser automation to MediaWiki developers. And just a couple of repositories have Selenium tests included in their repository. So to answer the question, Selenium tests could be either in the repository of an extension or like a core feature or whatever, or in this browser test repository, or whenever else. Like it doesn't really care. We can run the code from anywhere. OK, so the test, I hope I answered the question, so please rephrase it if I missed it. What's usually the best practice for that? The best practice would be to have the code, to have the test code next to the production code, in my opinion, because it helps with version controls. So if you wrote a feature and wrote a test and you know that this test works in this commit for this feature, then you can always check out this commit later on and know that the test will pass. And if you have test code in a separate repository or in a separate branch, then you have to set up some kind of tracking.  And that's really easy to mess up. So it's just like the simplest thing is to have the test code in the same repository as production code, in my opinion. And feel free to ask follow-up questions. Oh, so we had a timeout error. Let's try it again. So sometimes browsers misspeak. Let's see if it will be better this time. I hope I answered the question. So if you have more questions, let me know. Yes, you did. So WMF Labs is taking a lot of time to open, at least for me. Can anybody confirm that WMF Labs works for them? Is it just my machine? Yeah, it works for me. Fails for me. So it might be down. Yeah, it's not looking good in Chrome either. So let's try refreshing again. So this is the server. I got an error on it. OK, it works fine in Chrome now. Let's try it again. OK, it's working just slow. It's working at the moment. Let's see what happens. OK, so the test passed. This was unexpected for my part. So let's see what we changed so far. And should we commit this to the repository or not? So I like GitHub for Mac application for viewing diffs. So let's see what we changed. So it says we added a new scenario and we added a new line here. This could be ignored. This is just white space. The phones again are small. Oh, let me see. I'm not sure if I can make this. It's fine, I think. OK, can you see better now? Well, not really, but you can guess what is in there. It's a little bit small still, I think. Can you make it bigger or not? I see a lot. Oh, maybe the hangout. I made a zooming of Mac OS. Maybe the hangout is not picking up. OK, so let's forget this. Let's move back to here. So what we did, we added this scenario. So just one scenario, WikiLove window appears with one step. We added this step here on WikiLove page. WikiLove window element should be visible. And we added just one element to the WikiLove page. So this was pretty simple, right? And we made, I think, a pretty good and important test. So we checked. Go ahead. Be visible. Is that a reserved word or recognized by PageObject or is it defined somewhere else? Yes, let me go there. Excellent question. So this is a combination of a cucumber feature and PageObject selenium feature. So yes, this is a reserved word. So here, the way assertions are done in cucumber, you define an element, and then you define a should. Then you type that should. And then what should be true or false? So here, we say it should be visible. So it's a reserved word. For now, let's not go deeper. Since this test passed, and I have a saying, never trust a test that you didn't see fail, I want to see it fail first before I even consider committing it to the repository. The keyword is should not. And I hope I wrote it correctly, but we'll see if cucumber complains. Maybe there's... Cucumber is a very sophisticated assertion library. It allows you to match various assertions in a number of really powerful ways. In general, we only really use two matchers in cucumbers. We use the word should, and we use the word should not. You can get quite a bit more sophisticated, but should and should not gives us 99% of what we'll ever need. Correct. You can see the test running. And this one should fail, because we changed the assertion. We said that the wiki-love pop-up or window should not be there. We know it is there. So let's see what happens. And I will trust this test only and only if I see this fail. Okay, great. So let's see what happened. Expected visible to return false got true. So this is music to my ears. So let's go back to code and see what happened. So this visible, so how could I make it so you can see both? Okay, great. So the error message said we expected visible to return false and got true. So this be visible piece of code actually calls this visible. So this be visible keyword calls this visible question mark method on this element. And it expected to return false but got true. Meaning like in this case, meaning that the window, we expected window to be visible. And we said it shouldn't be visible, but it was visible. So let's fix this test again. So we know. So on wiki-love. So did I confuse anybody with now this making the test fail? Because I think it's really important, but it's a bit confusing. So yes, a couple of days ago I was pairing with Rachel and we created a simple test for something. And it looked like this. It looked good. Like it looked like something that will pass. And it really passed. And then I say, okay, let's make it fail first. And then it passed again. So this code really didn't check the thing that we wanted to check. Like we made a mistake and both of us didn't notice the mistake while we were working on the code. And then like we took a deeper look and figured out there was like, we made a simple mistake. But if I didn't insist on making the test fail before checking it in, we wouldn't know that the test really doesn't do what we wanted it to do. Like check this thing. So in this case, we checked that it fails. When we say that the window doesn't appear and it passes when we say that the window does appear. So we are going to commit this code. Let's just make a commit. So I used this GitHub for Mac. But yeah, I'm not sure how to make it bigger. It's not... No, okay. I'll zoom. I don't think it... Okay, never mind. So we have a new step. We have a new element in a feature file, in a page object. And we have a new feature and let's just commit. So I usually just copy paste the feature and the scenario. Okay. So in this case, we can just copy paste the scenario as a commit summary. Scenario, we can allow window appears. Comment, commit. I can... We are pretty much done. I'm open. Let me just check my notes if I wanted to check something else. Let me answer George's question while you... Yeah. Okay. So George, you asked about RSpec versus Cucumber. And they're really... They are two entirely separate tools that happen to just play very well together. And there are historical reasons that they play well together. Most important of which is when Cucumber got included by default in the Rails project. That became very important. So Cucumber is the tool that allows us to specify tests in plain English with a given, when, then statements. And that gives us all the pattern matching and the ability to write our tests in some very sophisticated ways in plain English. RSpec, on the other hand, is our assertion library. RSpec gives us the power to say that such a thing that we're testing for is true or not true, that it matches or it doesn't match, that it's visible or it's not visible. These are all steps in RSpec. And it's the page object framework that actually allows us to specify the elements in our pages that we are manipulating in the course of our tests. Cucumber can be used with assertion libraries other than RSpec, but it generally is not. As we mentioned, one of the reasons that we chose this particular tool set is that Cucumber plus RSpec is very much a standard in the Ruby community. It's everywhere you go, regardless of whether you're doing unit tests or whether you're doing API tests or whether you're doing integration tests or in our case, browser tests. Cucumber and RSpec are a pretty standard approach. This is going to be familiar to anyone, pretty much anywhere in the Ruby world. Yes, and page object pattern and page origin that we use are becoming more and more mainstream. I think that was a good pick, too. Another comment about RSpec and Cucumber. Cucumber was part of RSpec years ago, and then it split to another project. They share some history, but RSpec is more like a unit testing library and Cucumber is more of a communication tool, in my opinion.  Are there any more questions about that or anything else? Yes. RSpec is talking to Selenium, that's right. Sorry? Is that RSpec which is talking to Selenium? No, no, no. RSpec and Cucumber. Shall I take this and talk? Go ahead. Selenium, all Selenium is, is a way to drive a browser. Selenium is the tool that tells a browser what to do. Selenium is what knows about the document object model, and Selenium is what knows about what elements are on the page, whether those elements are accessible and how. So we are using the page object framework uses, is a wrapper for Selenium that understands a particular way of addressing the elements within pages. We then take the information that Selenium reports by way of the page object framework, and we check that against our assumptions using RSpec. Okay. But in a nutshell, Selenium, all Selenium does is drive the browser and nothing else. Yeah, that's right. That answers my question, thanks. If there are any more questions, let me know, or like in a minute or two, I would just like to wrap up and repeat what we did today, and then we can, we have like a half an hour left. Are we, like, do we, Kim, will there be a break, or what's the plan? Well, if we change then the activity, I think we can then go directly to the either path or start doing something different. So let me just, like, tell you what I did in the last couple of minutes before we start answering questions. So I made a commit. So we take a look at, it's a little small, but so I made a commit of the code that we changed. Then I pushed it together. So we use, so with this command git review, I pushed it to Garrett, and it's now visible here. I can make this. So this is Garrett. Let me go to home screen. We use, we use Garrett as a, instead of GitHub for hosting our repositories, we use it as a code review tool. I'll show you the code review real quick. We do have a GitHub mirror. So if you prefer to work, like in a GitHub way with forking repositories and sending pull requests, feel free to do so. We have tools that will communicate between GitHub and our Garrett install. So this is our GitHub mirror. All the links are in the meeting page. So if you go to the meeting page and to the code section, you will find links to the Garrett repository and GitHub repository. If you take a look at this commit, this is the code that I just pushed up there. So this is the commit message scenario. I just made a note that we wrote it during the workshop. I've added a few people that I know are in the Hangout. So Kim, Rachel, Chris and Tomislav as the reviewers. So all of them have a chance to go here, take a look at the code, for example. Let's see. So we do have some Jenkins jobs that will check if the syntax is fine and so on. So it helps reviewers. If Jenkins says, oh, you've committed something and like it completely, it's completely broken. Like it's even not valid Ruby code. Reviewers will have more information. So this is what we did. This is the diff that I wanted to show you. So the bottom is white space. So we just added a new scenario. We added a new step here and we added a new element to the page. And this is all we did. And as you can see, it really checked something. It broke when we said the window should not appear and it worked fine when we said the window should appear. And it was that easy and that's all there is to it. So when the reviewers, so the workflow that goes like when you create a code, when you commit it and push to Garrett or GitHub, when you push it to GitHub, it will end up in Garrett. Anyway, we have some communication between the two. You can either Chris and I both monitor the repository. So we will add ourselves as a reviewer if you don't. And reviewers now have a chance of commenting on the code. And some people have privileges of merging into master. So whenever, oh, I made a mistake and committed from master. Oh, so we should create a topic. I just fixed the problem. So I get to workflow is to always make new changes in a branch never into master. So I just made it here. I just made a change in Garrett tonight in a topic branch called workshop. We never merge the code only in the case of extreme emergency. You can self-merge the code into the master branch. The usual workflow is that somebody else will make a review and if everything is fine, we'll make the merge into the master. I wouldn't go too deep into the Git because that's even another part of the Git Garrett part. So I think we should go back to the topic of the workshop. I just wanted to mention. We have a question in the chat from George. So in Cucumber, any step, any given when or then step can optionally take a second argument which is and. That's when you have multiple conditions for a particular step. You can have multiple things that must be true at the start for a given. You can have multiple actions that you take under when. You can have multiple results that should be checked in a then statement. So any one of these ands can be, any given when or then can also have ands and it's simply our convention that each of these steps is indented to spaces. Correct. If I think, look, is there Chris, Kim, is there anything that I forgot to say? I think we went through everything now. Is there any questions or we can bear now with people? I do have a question. Your hard code is the fact that you are visiting the page of Cineum User 2. And it's sorting around the repo. I see it's someone you can pass around an argument for visiting a specific page. What's the best practice for that? I mean, that's regular to have a hard coded page. Okay. So I'll repeat the question just to make sure I understood what you're asking. So you're saying that we hard coded the username here and that we are always going to the same page. And is that the best practice or not? Yes. So I'm not a big fan of the term best practice. But yes, it's less than optimal. In my opinion, every test should make sure that everything it needs, so the test should create everything it needs and should clean up after itself. At the moment, we are doing that just in a portion of our test and just to make things simpler and make some automation and then worry about making it really, really good later on. We just hard coded some stuff. In this case, we just hard coded and sell any user. And we use that user page for this feature. But yes, in an ideal situation, you would create a random username, a couple of random usernames, a login with one, and send wiki live to the other one. And then after the test, you would delete the couple random users that you just created. So any other tests that do that at the same time or before or after this test, like nobody would step on anybody's feet. So that would be the best practice in my opinion. Every test should be available for itself and create everything it needs and clean up after itself. Okay. But we are practical. We do have a certain kind of constraint though. One of the original goals, when I first started with the foundation, we had no shared test environment. We had no usable shared test environment. Today, we have a couple of really useful shared test environments. We have what we call beta labs where we're working right now. We have what we call the test 2 wiki. And one of the goals of our browser test suite was to actually run against these institutional shared test environments. So in that context, and Karima has run up against this, in that context, it actually does make sense to have certain things hard coded. Certain users, certain pages. We want those pages always to exist in these shared test environments and we want them always to have a certain form. Because that is the main target of these tests. We are actually working on a generic, basically a smoke test suite. That would be generic for any environment and they'll probably run under phantom.js and not in a particular browser. That's the process of building one of those. But a lot of the tests that you see today are really intended to run in these shared test environments that are maintained by the foundation. I have a question. So, okay, I did this workshop and now I want to try out something. Realistically, what should I do? There's plenty of features in the wiki media software. So where is it more useful to contribute? Where do I find, where do I ask? You know, next steps, next practical steps to do a first contribution in automated browser testing. And excellent question, Kim. Should I go? I have a couple of answers for that. One is that, like every open source project, I mean, this is, it comes about because developers scratch their own itch. If you yourself have a feature of wiki media that you want to, that you have some investment in, you want to see this feature continue to work properly, then we welcome any test for that sort of feature. At the same time, we have certain priorities that are going on. If you don't have a particular feature that is important to you personally, you may want to know about some of our features that are under development. The visual editor is a very important feature being developed right now, and it's undergoing significant changes, Sorry to interrupt you, because what you say makes sense, but I also have heard you saying that visual editor is not the best product for a newcomer to all this, because for some reason... We also, GELCO has also identified a number of what we are calling easy bugs, and Tomaslaw has been doing a great job for us. Just doing some rearrangement and some refactoring and making the test run without deprecation warnings was the change from Tomaslaw that I just reviewed earlier today. We have a number of projects where that beginner should be able to tackle, and you always feel free to ask questions of GELCO or myself. Okay, so you had this list of bugs. How you got to this list of bugs that wasn't paying attention. Yeah, so I'll answer the question. I'll rephrase, Chris. In short, if you want to get involved, if what you saw here is interesting and you would like to get some training, feel free to join our QA mailing list.  There's a Get Involved section. There's a couple of links. One of them is to our QA mailing list. Feel free to join, click here, enter your email address and subscribe. There's no moderation, but you will get a confirmation email and the Gmail sometimes puts them into spam, so please check your spam folder. We had a few people telling us I was not allowed to mailing this. That didn't happen. We allow everybody, but please check your spam or junk folder and there will be confirmation email. Another thing, if you would like just to get your hands dirty and fix, just commit anything, just make the first contribution. As Chris said, Thomas is doing a great job. He already closed 3, 4, 5 of these easy bugs. My public to-do list is in this box. Whenever I find something in code that I think is easy to fix or something that somebody new to the project would have an easier time than something else, I will tag it with this keyword Easy. At the bottom of the meeting page, there's a link to Easy Bugs and it will open this box. Just read them, pick one. If you can't decide, just join the QA mailing list, and we'll take it from there. Just let us know what you're interested in and we have plenty of tasks for you. If it's not here, we have plenty more. Does that answer the question, Kim? To me, yes. I hope to the others as well. I have a follow-up question, actually. Let's say I want to test stuff. I think gadgets are probably interesting to test. Yes. I want to test gadgets on my favorite home wiki and I'm very lucky because there is a common wiki media beta on VMF labs. I'm very lucky, but is there a beta for every wiki around for wiki source or Japanese wiki news or whatever? If it's not, it could be fixed. Chris, Kim, do you know? Yes. I can say something to that. Our beta labs environment is intended to emulate the production cluster closely. In practice, some wikis are more close to production than others. Mostly, at this point, English wiki media is in pretty good shape. Some of the other ones are, I know we've had a request for an Arab to bring the beta Arab wiki media up, and there was another right-to-left wiki that we have an outstanding request for that needs some maintenance. The idea is that beta will emulate the production cluster. In practice, some of the wikis are more production-like than others. This is based simply on interest. If you are now interested in the Japanese wiki news, just push for it and you'll get it sorted out, I'm sure. The assistant administrator for the beta labs is a man named Antoine Mousseau. He lives in France. He's a very helpful gentleman and knows quite a lot about the wikis out there. If there are questions that we don't answer, or for the next steps for people that would like to get involved, please join the Q&A mailing list. This is the place where everything is going on regarding testing. More questions? I have another, but it's for everybody, actually. Thanks to Chris Angelco, I've been learning about the concept of pair programming and how good it is to learn and to just progress and get things done. So how could we organize slowly, but actually this pair programming among the people in this session and others we have in the mailing list and the Q&A mailing list, how do you think we should start to get organized with the pair programming? I would just say join the Q&A mailing list and let us know when is a good time for you. You can start with an hour a week and take it from there. Rachel and I, I'm not sure if Rachel joined. Rachel and I are pairing a few times a week. Chris and I at least once a week. Chris also pairs with Rachel. I think I can say that both Chris and I are depending on our other meetings and other stuff that we have to do, but we are open to at least a few pairing sessions a week. So just join the Q&A mailing list and let us know what your interests are and when would be a good time for you. So Chris is in the U.S., I'm in Europe, so we cover like half of the planet, so we could arrange a time that could be good for everybody. Okay, so the idea would be to start dancing with you and then try to find other dancers because I guess you don't scale much. Yes, so the idea would be that Chris and I would teach Rachel. So Rachel is our intern for this summer, so we would teach Rachel. Tomislav is doing good. I know a few other people from here. We would work with the people that are interested and then they would just take on from there, so they would teach somebody else and so on and so on. So Chris and I don't scale yet. That's a good point. But it's good to start with something. I think it's important. I don't think we have... So Chris, do you think... So we do have a few people that are more or less involved in browser testing, but I think the two of us are the only one available at the moment, right? Yeah, I think so right now. Rachel will be, Michelle will be, but right now it's Zyalka and myself out of primary contact. This is Rachel and I had a thought, maybe you could schedule a session like this once a month, a training session, and the same people came every month or most of them, then you maybe could get around and work with them individually or to explain different concepts or something. So a lot of people could be worked with at once. That's a good idea. I think we should have a workshop every month or two, depending on how we are able to arrange it. And every workshop could cover another topic, but I think we should also have parent sessions because in a workshop like this, it's more of like a lecture format. We can't really help anybody. And I think like when... Like you could probably tell Rachel, like when the two of us work, I think there's a lot of learning going on. And I know when I work with Chris or with somebody else, I learn a lot and I can transfer some of my knowledge. So I think there's a place for both this workshop type events and pairings. What might be really useful is getting a... or pairing up once someone thinks that they're mostly done and they need to get things through the Garrett step. Just as a software engineer, I've done this for a long time and if you've not used Garrett before, it took me a while to figure out and I've done this kind of thing for years and years. So for someone who isn't used to it, just helping them through the Garrett step would be really, really, really helpful, I think. That's why we didn't cover Garrett. And when I came to Garrett, Kim said, let's not go there this time because Garrett is not the most user-friendly tool that we have and I used to hate it. But now I got used to it and if anybody is able to figure out everything but how to send this code to Garrett, there are two options, let us know and we'll help you up with Garrett. And you can also use GitHub and if you prefer GitHub, just send code there and we'll take it from there. Zelko, are you seeing George's question here at the time? Yes, let me see. How do you manage cucumber steps? I know that the feature steps I've worked on can have slight variations, which needlessly creates more step definitions. Do you have a step library to see if a step definition has already been created? No, I do a review of the steps from time to time. I do some cleanup. If I see there's something really similar, I'll extract it. We try to organize the steps in files and then solve them alphabetically. So it's more obvious when something is duplicated but I don't think we have a good way at the moment. Chris? Yeah, I can speak to that in some detail. So George, a couple of things. One is that if you inadvertently duplicate a step that already exists, cucumber will give you a warning. Cucumber will actually inform you that you have in fact duplicated a step. One of the things that I encounter from time to time is that I will inadvertently write it down the name of the step that already exists but needs to be subtly different than the step that is in place already. That can be annoying to have to make the language of your new step different enough from the existing step to get it to do what you want. That helps to answer the question. I see there's more questions in the Hangout chat. So one that came posted from a stranger's name, DR0 or something. So somebody asked if I could go into some depth on the support printives in feature support pages and directory. Well, I think we are... I'm not sure... I'll do that. If you could show... Can you show something from a step file? Sure. As you're writing the step, as you're involved in writing the cucumber step, what's going to happen is you'll have... I always will say that on a particular page, and your page, in this case the Wiki Loft page, this is an object in Ruby. And then what follows the page that you're working with is an abstraction of the element. And this is free text. In this case, it doesn't have to be Wiki Loft window. It could be XYZ, ABC. This is an abstraction. It is a handle for the page element. And we can create that handle in our steps file on our page object using any text we want. And then for the rest of the entire sweep, whenever we use this element on this page, that is the handle by which we will address it. That's our abstraction for the element on this page. So this is... We always have the page with a dot followed by the abstraction of the page element, followed by the assertion and the argument for that assertion. What you'll often find in more naive implementations is rather than having an abstraction like Wiki Loft window, you'll have the actual address of the element on the page. And the problem with that is that anytime you have... What this gives you is duplicate code. You'll have Wiki Loft window mentioned several times in several tests. And then when that element on the page changes, you have a big, big, big maintenance problem because every time you define this in your tests, you have to change every single instance of that. So what we have instead is an abstraction. And then if we go to our pages file, if we move out of the steps file and we move to the pages file, what we'll find is we have a very specific way to identify these elements on these page. So in this case we have a div. And then our handle is the first argument inside the parenthesis. That is our free text abstraction for this element on this page. And so anytime we use the Wiki Loft window in a test, we'll just call it Wiki Loft window and we made up that text. That's how we get to this element. And it's only the final argument in this line, in this page file where we actually reach into the DOM itself and we find the actual identifier on the actual page and that's how we find that this thing called the Wiki Loft window. So this div may become a span someday. This class may become an ID someday. Regardless of how that happens, this is always going to be the Wiki Loft window to all of our tests forever and ever. That's a general page object. Instead of using an element on our page, you use an abstraction of that element in all of your tests. And this is just an institutional way to implement the page object design pattern. We have 15 more minutes. Are there any more questions? We have 10 more minutes. Oh, is there a George asks, is there a reason why page object doesn't support Capybara? That would be a question for Cheesy, I think. Kim, do you have a question? Do you know maybe? As far as I know, Capybara is another API on top of Selenium like water web drivers. So I think it's just that Cheesy decided to support Selenium web driver API and water web driver API and didn't have time to support another API. Is there a reason that you prefer Capybara? So George, that's another question. Yeah, I'll answer that. Capybara is not just on our radar. But let's talk about web drivers, though. Selenium, what we're using is Selenium 2.0, also known as web driver. It is radically different from Selenium 1.0. Selenium 1.0 worked by opening a browser inside of a frame and injecting JavaScript into that frame, and it was fragile in many ways. Web driver, but one thing to note, the original Selenium API was very large. The web driver API is very small. It's intended to be only a small set of tools, a very basic set of tools, a toolbox from which to generate your own API for your own application. Now, we want a more general API than that. So the water project, web application testing in Ruby, water itself actually predates Selenium. It was the original, useful open source test tool before Selenium ever existed. There's a guy named Yari Bakken for when web driver came out, when Selenium 2.0 web driver came out and had this very small API, Yari thought that it would be good to have the large water API as part of web driver. So he has actually, water is a very sophisticated API that wraps the very basic Selenium web driver API. Water gives you some very, very neat tools. In the Selenium API, for example, you cannot identify an element with more than one identifier. So water gives you this. We can identify an element with both class and name. We can identify an element with both text and index. Water gives us this. Water gives us a lot more power than we would have with just the pure raw Selenium API. And that's why Jeff put it into the page object, Gem and Ruby. George had another question. Do you guys use water red driver or Selenium? And he said he doesn't like a lot too much. So to answer his question, so we use both. So water red driver uses Selenium web driver. So water red driver Ruby Gem uses Selenium web driver. It's just a wrapper around Selenium web driver providing nicer API, a bit more high-level API. And Selenium web driver Ruby Gem is actually Selenium Ruby bindings. So we answered the question. We used both. Selenium has several Ruby bindings. I think the officially supported ones are Ruby, Python, C sharp, Java, and from recently JavaScript, but I'm not sure about for JavaScript. And there are bindings like unofficial ones in several other languages including PHP. We have 10 more minutes. So George asks. I do have a question. Let's say, I have a question about the chain of background scenario. In our case, the background is when I visit the user page of Selenium User 2 and I click Quick Love and this is the background for all our tests. But let's say that we want to make a test. When I visit, I don't know, a file page or an article page because we want to test that Quick Love is not present. That should be... We might want to do that, right? Shall we, like, refactor all of our testing to make it different and create a new feature? What is the best way to do that? As usual, it depends. There are several ways we could do that. One of them would be to refactor the existing file and remove the step that's not reused by others. Let me share my screen. I'll rephrase the question just to make sure I understood it. I hope you can see my screen. What you asked, I think, is what would happen if we wanted to check that this... If we wanted to check that this appears on the Selenium User page but doesn't appear on my page. This... Was that the question? Yes. And how we would refactor the feature file? So one way would be to... Since this, and I click Quick Love, this step would no longer be shared by all other scenarios, right? That was the question. What to do with this step? Now it's in background. What should we do? So one way would be to create another feature file and this one could be called WikiLove, I don't know, whatever. I have no idea at the moment. So we would create another feature file and just have these two steps in background. Or we could refactor this one and move this step to every scenario. Like this. And then all those scenarios that we type there. Does that answer the question? Yeah, you can start with the end... the end step scenario. Yes, you could... You could... Well, you could... I did a mistake. So given... You could, like technically, there's no point... There's no reason why you shouldn't, but I would complain, like I would give you a minus one review for this. So then I just did it myself. I would... I think that every scenario should read for itself. So if you don't care about the background, you know that the test has set up everything it needs. Then you should read the scenario as a story for itself. And when the story begins with an end, it doesn't really sound good. I would say, I would say when here. I would change these to when... Was that the question? So we would say, when I click Wickelow, the Wickelow window appears. I think that reads better than... and I click Wickelow, like it's... but I'm just starting the story and you're starting with an end, so that's not a good one. But yes, technically, this could be an end. Like, technically, you could make them all ends or all ends. It's just a convention. So given steps are doing the set up. So given steps, so I think this should be a given also. So just one thing. There's five minutes left. There's some questions now. But in any case, Gielco can stay a bit longer to answer or there's the mailing list for that. I can't stand right now. I thought I'll be able to stay longer. But I have to go. I have a family obligation. Then maybe you want to go fast through the questions you have in the chat. Yeah, I'll go quickly through the questions and please use the Q&A mailing list. I don't know how many times I've mentioned it today but it's really the best way to communicate because other people will probably have the same question as you and when you ask it there and I'll answer it once and everybody will see the answer. So Nero is also saying he has a meeting in five minutes so let's just... Let me see if what was... Should I go through real quickly through the questions? Or should we just end now and point people to the Q&A mailing list? I believe we're caught up on the questions. There are more. Headless browsers. Let's do something since apparently... It's also the morning in the US so people are more likely to have jobs to attend right now. Please look at these questions here and if you find that there's something that hasn't been answered please answer in the mailing list. We are assuming that anybody here is subscribed there. And in any case I want to thank everybody for these two hours of your time. I hope the experiment was satisfactory. Any ideas on how to improve this please share them with us. Any topics that you want us to focus please share this with us. We have run already like two, three sessions that are way introductory and I think now it starts to be the time to find sessions for the next steps. Please use the Q&A mailing list. Any questions that we didn't ask are answered just ask there. And see you on the next workshop or on the next pairing. Very good. Thank you very much everybody. Thank you so much. Bye everybody.