Go Ninja On The Test Automation Pyramid
Many companies struggle to build scalable automation suits and end up with hard maintenance, refactoring, endless debugging, and flaky tests which spend everyone time and money.
In this article we’ll I will explore different parts of the automation test pyramid. I will explain key guidelines and best practices
The Test Automation Pyramid
“Mike Cohn’s Test Automation Pyramid is a guide that is frequently cited as a good practice to follow. It suggests that:
1. the bulk of your automated tests should be unit tests
2. the next largest group of tests should be developed to target the service layer which includes the web services and business layer of your application
3. there should be a very limited amount of UI tests, as they are more brittle and. take the longest to execute” Angie Jones
A few years ago, when you heard the term “automation” all you could picture is Selenium/Appium end-to-end tests running on multiple devices and browsers navigating, clicking, verifying database and UI. You would also think about huge API flows executed by REST clients in a live environment.
We still want to run UI testing as it gives us the best user experience indication but we want to get a faster indication of our bugs, therefore we’ll execute more service tests.
Why less UI testing?
- hard to dive in and debug services issues
- long development time
- need need a lot of resources to execute (machines, devices)
- long run time
with that being said, we understand that we come to understand we need more service tests
More service testing
The pros are of course well known
- fast indication
- faster to debug
- fast execution and development time
The main disadvantage in long API flows testing is that in a microservices-based environment services are always deployed with bugs, break, and change contracts. You will find it hard to debug end to end API tests.
How to find more bugs in build time and reduce automation tests in a live environment?
Nowadays companies come to realize that automation should be part of the development process and first of all should be verified by developers with unit, component, and contract tests.
We all know about Unit tests. in this section article, I will talk about component, and contract tests using pact.io which gives a very creative solution for contract issues.
Contract Testing With Pact.io
Fast, easy, and reliable testing for integrating web apps, APIs, and microservices
Basic entities:
- Provider – a micro-service that provide the API
- Consumer – a microservice that consumes the API from the provider
- Pact files — JSON to describe the contract between provider and consumer
- Broker — third party the saves the pact files
Let’s create a contract test between a provider that exposes POST Phone data API to a consumer
Consumer example
Step 1 consumer expectation
Test it agains mock
Step 2 — publish to pact broker
step 3 — Now pact broker holds sharable contract between consumer-service and provider service
Implement the provider side
Step 4 — provider run tests, booting it server locally and execute /v1/phone to localhost and validate the contract using context.verifyInteractions() function
Step 5 — provider update the broker with the results
Isn’t it magic? it all in build time! which means no need for a live environment to validate contracts between services!
Next step: Component tests
We use component tests to verify our service business logic using mocked APIs and DB’s just like in contract tests but here we test one microservice with more deep dive into the business logic. component tests is also used for testing edge cases and error handling — service is unavailable, internal errors etc. There is wide documentation on this type of testing and this concept is well known. look it up =]
After DEV has created component and contract tests they are more confidant of the product, the environment should be somewhat stable and now automation developers can be focused on testing the product to the edge.
Testing 3rd Party services (or not…)
A lot of products count on 3rd party services to provide their data as part of the flow. For example — validating credit risks, retrieving flights info, syncing to calendars, and more.
Sometimes those services are not stable in a test environment and we want to isolate our development and testing cycles from other companies issues and just verify we are sending the request to the 3rd party as expected.
Solution:
Create a dedicated environment that contains all the relevant services for the test and mock the 3rd party
Writing all requests to DB with a key you can control from the automation test (for example requestId header)
Provide an API in mock service to get all requests triggered to the mock service from this identifier
After we verified we have no bugs within our own services we can now test end to end on the full environment.
This basic line will ensure more stable environments
Test E2E tests with Kafka events on the microservices environment
Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform. taken from confluent.io
First, let’s describe a simple Kafka based microservice environment
There are three services in the complete environment:
- Service A and Service B are owned by DEV TEAM 1
- Service C Owen by DEV TEAM2
- each service has an API to return the event data from its own DB
TEAM A just developed a new feature to send a new event from Service A all the way to C
The straightforward test would be :
Service A publishes an event -> use the service C API to GET and assert the data
with this approach, we will face multiple issues
- TEAM 2 have not develop yet a feature that consume this specific event
- Bug on service C will fail TEAM 1 automation tests
The solution
TEAM 1 should test they’re feature a soon and fast as possible. before the complete end to and we should add another layer — domain testing
TEAM 1 new test will look as the following:
Service A publish an event -> assert that service B published an event to topic C
In order to create such assertion we need our tests to consume events from topic C. To avoid loosing events to race conditions between test and service C the consumer must be from a different group-id.
What are our options?
- Each test case will create a new group id will open endless group id’s which is hard to manage.
- Each automation project will create a new group-id will still live us with race conditions when we want to test on parallel.
- Create an automation service with one group-id and provide an API to retrieve the data.
Summary
In this article, we learned to build our automation tests with different layers of the pyramid increasing build time, stability, and testing time.
If you enjoyed this article you can also visit my previous tutorial for Quality-oriented kotlin set up with allure and integration tests
Thanks
