When I create or maintain an automated test I use a “learning loop” so that I learn from the test results. This enables me to create stable test packs that return accurate results.
I automate tests with Playwright using TypeScript and with the low code tool Ghost Inspector. The “learning loop” is the same for both tools.
When I automate a test I use a plan-do-study-act learning loop:
- I plan the test. This includes planning what functions and modules I will need, and how I will assert that the test has passed.
- I then “do” the test automation. I automate the test using my plan.
- I then study the test. I run the test and study the test results. I run the test several times, both on my workstation and in CI. I study the results to see if the test fails. If the test fails I study the test and the test result to work out why it failed.
- I then act and take the findings of my study into a new plan-do-study-act learning loop and plan the fix for the failure.
If a test that is running in CI “flakes”, I use a plan-do-study-act learning loop to fix the test:
- I plan the fix. I consider why has the test flaked and plan how to make it stable
- Then I “do” the fix. I make the change to the test that I had planned
- I then study the test. I the test both locally and in CI several times to see if the test passes consistently. I study the results of the test runs to see if the test is now stable and returning accurate results.
- I then “act”. If the test still flakes I will go through the loop again. If the test now returns accurate results I will merge the fixed test so that it runs in CI.
Each time I go through the “learning loop” I learn about the application I am testing and the tool I am using to automate tests. I refer to the loop as a learning loop because I have learned so much by using it. A diagram of examples of this loop for both Playwright and Ghost Inspector is shown above.
This loop was created by Walter Shewhart to help improve quality in the manufacture of telephones and was popularised by W. Edwards Deming. It is usually known as the Deming Cycle and is widely used, for example in developing self-driving cars [1].
I am responsible for packs of hundreds of tests. If you use a “learning loop” when you create and maintain tests your tests will be stable and give accurate results. Automated tests do not have to be flaky.
References:
[1] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p137)
Learning resources:
- DemingNext – eLearning from the Deming Institute with a two-week free trial
- Out of the Crisis by W. Edwards Deming
TestAndAnalysis 