Automation testing for everything? Is it a good idea?

If you are in the field of QA testing, you are probably facing this question everyday, either by your peers or your managers. Should you automate all of the testing?

Let's start with a brief history of software testing.

Back in 1999, James Bach wrote the famous article "Automation Snake Oil". 

Even though the article was written nearly 20 years ago, we still see the same problems, which in many cases increased due to the principles of Agile and Continuous deployment. 


The aspect of saving time and cost in testing is so seducing that companies strive to think that test automation will replace the thinking people (a.k.a manual testers).

What IT professionals and IT managers tend to forget is why the automation tools were created in the first place! Automation testing is just an enabler, it is a tool, to help the testers in their work, and to reduce the risks of faulty software.

The automation test evangelists, as I call them, fail to understand the true meaning of testing. They often praise the scripts and that workflow, but they tend to forget that what they are proposing is just checking and not testing (Read more about checking vs testing Michael Bolton here)

Using automation, you check whether a feature is fulfilling the acceptance criteria, but it does not find things that have not been considered by the developer of the automation test. You can run that test 1000 times, and get a PASSED mark, but at the end, your support line will be filled with complaints of a non-functional software, and you will wonder what has happened?

In 2009, Rikard Edgren published an article called "In search of the potato" 

In short, his point is that the valuable and important aspects when creating the test design, are not just the requirements or the acceptance criteria, but as a tester you need to go beyond that. 

In one brilliant article by Martin Jansson, which was the inspiration for this article, called "Implication of emphasis on automation in CI"  everything is explained in one brilliant image.

Known Knowns 

Automation testing should focus on an aspect of the system - set of repeatable tests on the things you know (or you think you know). By testing the known knowns, all the changes (a.k.a evolution) of the system will reveal the unknown knowns. 

You specify what you expect, what is the correct result and what is the actual result. 

Known Unknowns

If you go beyond the specification, and you would like to learn new aspects of the software, how it functions, and evaluate any unforeseen risks, you cannot (in the most part) automate the Known Unknowns.

Unknown Unknowns

This investigation a.k.a exploring and testing, will reveal new things that no one has thought of. Through serendipity you find something surprisingly valuable. You rarely automate serendipity.

The Oracle problem

(The Oracle - Matrix)

A problem that is probably unsolvable, is that there are none (or at least very few) perfect or true oracles.

A “True oracle” faithfully reproduces all relevant results for a SUT using independent platform, algorithms, processes, compilers, code, etc. The same values are fed to the SUT and the Oracle for results comparison. The Oracle for an algorithm or subroutine can be straightforward enough for this type of oracle to be considered. The sin() function, for example, can be implemented separately using different algorithms and the results compared to exhaustively test the results (assuming the availability of sufficient machine cycles). For a given test case all values input to the SUT are verified to be “correct” using the Oracle’s separate algorithm. The less the SUT has in common with the Oracle, the more confidence in the correctness of the results (since common hardware, compilers, operating systems, algorithms, etc., may inject errors that effect both the SUT and Oracle the same way). Test cases employing a true oracle are usually limited by available machine time and system resources.Quote from Douglas Hoffman in A taxonomy of Test Oracles.

Here is a the traditional view of a system under test is like the figure 1 below.

 In reality, the situation is much more complex, see figure 2 below.

These images explain that, theoretically we can set up the initial state, the inputs, but in reality we do not have the full control.

The result we get in any form of testing is that something appears to be correct, but everything that might influence that input that we do not check, can be utterly wrong.

Next time you go with the idea that you would like to automate everything, you are implying that you trust your product in something that is lacking perfect oracles.

Final thoughts

Automation testing is a tool, that largely improves the workflow of a QA Engineer, and it can be used in many situations.

Please do not fall for the trap that you can rely heavily on on Automation testing. You can never replace the manual testers and their capabilities to find unforeseen issues.