Test Automation Vision

When I was at the CAST2016 Test Conference in Vancouver, I also was visiting the keynote from Nicholas Carr who spoke about his vision for test automation in general. A lot of people think that it is a good thing to automate everything so we can reduce our manpower for the company. But Nicholass preached that it is the wrong mindset to have as an human being.

He told us how an hospital was reducing their manpower by automating their whole medicine order system, and  how an human mistake easily got passed through the system so that a young boy got 33 pills of antibiotica instead of 3. If there was an human involved somewhere in the process of the order system, he could already find the mistake and report it back to the doctors that they made a mistake with their input. But because of the lack of human interaction, the young boy got 33 pills which made him awfully sick.

This hospital is not the only one who is noticing these flaws in the automation vision. Another big company named Toyota is starting to hire human employees since 2014 again. The reason for this is that all the robots in the factory were good at their job, but because of the lack of human interaction, some flaws in the designs were discovered after the assembly fase. Imagine a batch of 300 Toyota corolla’s with the same flaw in their design. You can basically throw your 300 models away and start over again.

So we can conclude that the best practice for automation is changed. You don’t want to automate everything which is replacing employees. You want to automate something which is helping your employees with their tests: a human-centered automation.

To build an automation which is human-centered you need to adapt the following rules:

Automate after mastery

Back in the day when we were young we learned first to do math without a calculator. Just a pen, paper and our brains were the tools we were using. What I’m trying to say here is that we first have to master a certain skill before we can use (and trust) automated tools like the calculator. What if the calculator has a bug in it and tells us the answer for 1+1 = 3 instead of 2? Because we already mastered the skill calculation without a calculator we already can identify this bug.

Transfer control between computer and operator

In the aviation we know that a pilot basically only is involved in the start and the landing procedure of a commercial flight (but with today’s technology the auto pilot can also handle these scenario’s). The rest is handled by the auto pilot, so the pilot and co-pilot can relax during their flight etc. We all know that their lives have become easier with this awesome technology.

But history tells us that it is not a smart move to let the auto pilot take over. The list of airline crashes due to a malfunctioned auto pilot is long. Most of the time the auto pilot misinterpreted the data it obtains from the different sensors, like you are losing altitude but the sensor who is responsible for this is not working properly so the airplane is gaining altitude and at a certain point the aircraft will stall and crash.

The FAA came up with the idea to set a timer on the auto pilot. Basically the system just told the pilot to take over after using the auto pilot for a certain amount of hours. This method helps the pilot to stay focussed and in control during the flight, so he or she knows at all times what the status is from the plane.

So what we can learn from this, is that it is a good thing to transfer the control between the computer and the operator so the operator is always in control of the situation and not left in the dark because the computer does all the work.

Allow professionals to assess situation before providing algorithmic aid

Of course you are hiring a specialist which can automate your awesome app, but what kind of background does your specialist has? Is it a tester which had a crash course programming or is it a developer which only know that testing is clicking random buttons? A Testers mind is always focussed on “How can I break it?” while a developer is thinking in solutions “How can I fix it?”.

Test Automation can’t be done by one specialist or individual, you need a team which can aid your test automation. The best way to achieve this is to let developers and testers work together in small (scrum) teams. The tester takes the lead on which part needs to be automated and which part still needs to be tested manually. The developer can than build the desired test automation framework.

With this workflow you get a real good product which is accepted by both developer and tester, because every step in your test automation is assessed by a professional.

Don’t hide feedback

So congratulations on your test automation build street. everything is running smooth and you see that the test results are nice green. But what is the next step?

If you are working in a small company mostly testers and developers are integrated in one scrum team. So if there is an issue found with the automation a tester can easily see it and report it back to the developer. If you are working in a bigger company where you are working with feature-(scrum)teams, this reporting back to development is a totally different deal.

The best way to report your results back to different feature-teams is to make it visible in within the teams, by building a live dashboard which is showing the test results of the latest commit on the development branch or make it accessible within the company network so developers can access it on their own computers for example.

Because of this dynamic reporting strategy, different feature teams can already see what has been tested and if they broke an all ready existing feature. So an issue can already been discovered during a sprint for example. So don’t hide feedback, make it visible within the teams!

Allow friction for learning

As we all know, games are for fun and leisure. But what we often forget is that games are also ment to train our skills. For example with a RTS-game you need to make important decisions in a split second and adopt every time to the current situation.

For test automation it is the same. You first had a vision to automate everything. But later on in the process you discovered that some test scenario’s can’t be automated because they are too big or too important to let a computer run the happy flow. Maybe it is better to let a tester do an exploratory test to see if he or she can break the new feature.

So don’t be mad because the test automation build street is still not finished or didn’t automate everything. Because the situation can change at any time, a test automation development team needs to be agile and needs to learn to adapt to the current situation. But in the end your automation team learns a lot with these frictions and that can become handy when you decide to transfer these knowledge to other departments etc.