What do you think?
Rate this book
The "A" Word
If you've read my blog for the last few years, you've probably read everything here already (although without some much needed editing). This is a collection of blog posts from my blog - rearranged, edited and polished (a bit) and put into an ebook.
The book is free if you want - or you can pay (all proceeds go to the American cancer society). I just wanted to gather my thoughts on test automation in one place and share somewhere.
The book is free if you want - or you can pay (all proceeds go to the American cancer society). I just wanted to gather my thoughts on test automation in one place and share somewhere.
- GenresSoftware
58 pages, ebook
First published January 1, 2013
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 3 of 3 reviews
July 30, 2018
Although the style of this small book is not as argumentative, with a strong anchorage in a scientific approach and having an accuracy for how notions are used, as is the case with James Bach and Gerald Weinberg, it does offer some revelatory insights about what is known as ‘test automation’ and attacks some big myths and folklore about test automation that is out there in the industry. Here are the most important insights which stuck in my mind:
- He starts by talking about Philip Amour’s Five Orders of Ignorance and relates them to testing. Testing has two dimensions: proving something that we know (checking if specifications are implemented correctly or not) and finding out what we don’t know– it’s the level of lack of awareness (when we don’t know what we know or do not know) and lack of process (when we don’t have yet a method of acquiring the above needed knowledge). Knowledge-acquiring testing is continuous exploration, experimenting and interaction with the product. With this idea in mind, Alan Page continues through his series of texts in the book.
- The biggest mistake of people who are initiated into automation is to think “how can I automate this”. If a (manual) test can easily be automated, then probably this is a shallow, superficial test and your automation strategy is of the same kind. You shouldn’t think how can I automate this, you should think how can I test this, bearing in mind that you have coding skills that will help you write tools and scripts which will make your life easier and open many possibilities in which you can exercise the program in interesting ways. Rather, automation (technical, coding skills) should be incorporated into the test design skills. Thus, you don’t write tests and then automate them, you design good, complex tests for which might include some automation steps (like generating or analyzing big data, inserting parameters from an Excel file etc). Alan Page says: “It pains me to hear that some testers design scripted test cases, and then automate those actions. This tells me two things: the manual test case sucks, and the automation (probably) sucks […]. Some teams even separate the test ‘writers’ from the ‘automators’ – which, to me, is a perfect recipe for crap automation.”
- Avoid automating through the GUI as much as possible – “For 95% of all software applications, directly automating the GUI is a waste of time. […] I’m not against automation – just write automation starting below the GUI and move down from there. I’m not saying that you shouldn’t test the GUI at all – I just don’t see why you wouldn’t want to test it manually and get people knowledgeable in user experience to help”. Don’t use machines to simulate human actions, but use them to perform actions beyond human possibilities (big data generation and analysis, stress tests).
Another problem with automating through the GUI is the need to babysit the automation scripts, to constantly adapt them in accordance with the changes in the UI/UX. In addition to this, you need to do lots and lots of tweaks to handle the delay of UI elements loading on the page.
A third issue with automating through the GUI is finding good oracles, means of telling if there is indeed a problem if a test failed. “If your oracle solution involves screen comparisons, I think you’re heading down a scary path – please consider this solution as a last, end-of-the-world type solution.”
In conclusion, I will quote one crucial idea of Alan Page which I also share, that: “test automation is just one tool from our tester toolbox that we can use to solve a specific set of problems. Like most tools, it works extremely well in some situation, and horribly in others.” And later he summarizes how this tool it got horrible:
“This mindset, although common, kills me:
* Testers who code write automation
* Automation is primarily made up of a suite of user tasks
* Automation is primarily done through the GUI.
All these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test teams in the wrong place and is, frankly, insulting to testers who know coding.”
The book can be found here: https://leanpub.com/TheAWord
- He starts by talking about Philip Amour’s Five Orders of Ignorance and relates them to testing. Testing has two dimensions: proving something that we know (checking if specifications are implemented correctly or not) and finding out what we don’t know– it’s the level of lack of awareness (when we don’t know what we know or do not know) and lack of process (when we don’t have yet a method of acquiring the above needed knowledge). Knowledge-acquiring testing is continuous exploration, experimenting and interaction with the product. With this idea in mind, Alan Page continues through his series of texts in the book.
- The biggest mistake of people who are initiated into automation is to think “how can I automate this”. If a (manual) test can easily be automated, then probably this is a shallow, superficial test and your automation strategy is of the same kind. You shouldn’t think how can I automate this, you should think how can I test this, bearing in mind that you have coding skills that will help you write tools and scripts which will make your life easier and open many possibilities in which you can exercise the program in interesting ways. Rather, automation (technical, coding skills) should be incorporated into the test design skills. Thus, you don’t write tests and then automate them, you design good, complex tests for which might include some automation steps (like generating or analyzing big data, inserting parameters from an Excel file etc). Alan Page says: “It pains me to hear that some testers design scripted test cases, and then automate those actions. This tells me two things: the manual test case sucks, and the automation (probably) sucks […]. Some teams even separate the test ‘writers’ from the ‘automators’ – which, to me, is a perfect recipe for crap automation.”
- Avoid automating through the GUI as much as possible – “For 95% of all software applications, directly automating the GUI is a waste of time. […] I’m not against automation – just write automation starting below the GUI and move down from there. I’m not saying that you shouldn’t test the GUI at all – I just don’t see why you wouldn’t want to test it manually and get people knowledgeable in user experience to help”. Don’t use machines to simulate human actions, but use them to perform actions beyond human possibilities (big data generation and analysis, stress tests).
Another problem with automating through the GUI is the need to babysit the automation scripts, to constantly adapt them in accordance with the changes in the UI/UX. In addition to this, you need to do lots and lots of tweaks to handle the delay of UI elements loading on the page.
A third issue with automating through the GUI is finding good oracles, means of telling if there is indeed a problem if a test failed. “If your oracle solution involves screen comparisons, I think you’re heading down a scary path – please consider this solution as a last, end-of-the-world type solution.”
In conclusion, I will quote one crucial idea of Alan Page which I also share, that: “test automation is just one tool from our tester toolbox that we can use to solve a specific set of problems. Like most tools, it works extremely well in some situation, and horribly in others.” And later he summarizes how this tool it got horrible:
“This mindset, although common, kills me:
* Testers who code write automation
* Automation is primarily made up of a suite of user tasks
* Automation is primarily done through the GUI.
All these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test teams in the wrong place and is, frankly, insulting to testers who know coding.”
The book can be found here: https://leanpub.com/TheAWord
August 14, 2022
It's essentially a collection of revised blog posts from the author - now in ebook format. It's a fairly short read - I read it in one day. He discusses the limitations of automation testing, how people's attitudes to automation is wrong (might not be suitable/efficient to automate, especially GUI testing). Part of his closing statement is as follows:
"I’m not against automating tasks for some testing. Automation is overused and overvalued in software – while the coding of diagnostic and analysis tools is extremely undervalued. In many cases; quick check of basics, some unit tests, monkey testing, etc.), automating a set of user tasks is beneficial and provides a lot of bang for the buck, but focusing entirely on automation of user tasks is a waste of any programmer's time."
"I’m not against automating tasks for some testing. Automation is overused and overvalued in software – while the coding of diagnostic and analysis tools is extremely undervalued. In many cases; quick check of basics, some unit tests, monkey testing, etc.), automating a set of user tasks is beneficial and provides a lot of bang for the buck, but focusing entirely on automation of user tasks is a waste of any programmer's time."
January 23, 2018
DAMN RIGHT
Displaying 1 - 3 of 3 reviews