Want to test microservices without standing up a full test environment? Want to do this before you push changes into a mainline branch? This is a challenge that we often encounter during project delivery with our clients. Recently, when we started a greenfield project to build a new aggregation platform for a client, we wanted to move as much testing and automation as we could to be executed pre-merge in our process. We knew that we needed a way to execute deep functional tests on our services and enable our teams to run independently of each other. This is what drove us to develop test scaffolding for microservice isolation and testing.

The Challenge

We need to test changes to microservices independently of other services before deploying to an integrated environment. This way, when we deploy changes into our upper environment, we can focus on testing integrations and transaction flow instead of the functional behaviors of a single service. While we could point our service under test at a test environment for its dependencies, this can cause two problems. First, an intermingling of data between users of the environment can lead to unexpected behavior. And second, latency issues and slow tests can occur due to chaining of calls between services. We quickly realized we needed the ability isolate services pre-merge to enable focused and fully isolated functional test automation.

The Approach

We decided to isolate services by introducing a concept called test scaffolding. Test scaffolding’s job is to mimic all external services our microservice might need to interact with. It behaves much the same way as scaffolding for a building, and as the microservice interactions grow we add those to the scaffolding. Each service had its own scaffolding, which was built and deployed along with the service pre-merge. Doing this provided our microservice the dependencies it was expecting and gave us fine-grained control of the inputs and outputs via our tests. This approach was also critical in building services independently and providing continuous early feedback to our engineering teams. In the next few sections we will dive deeper into how we defined a microservice for the purpose of this discussion, examples of the sorts of interactions, and some examples of test scenarios we executed using it.

Microservice basics

We will use AWS terminology in the examples below, but the concepts should be applicable across all platforms.

To get started, it’s important that we have a common understanding of how we defined the scope of functional tests that interacted with test scaffolding.

• Tests were confined to the boundaries of a specific microservice.
• Testing interactions of deployed services belonged to our journey tests.
• On the project we defined the boundaries of a microservice as the components defined by the services CloudFormation Template (CFT).

The most common set of components in our microservices are often a mix of API Gateway, Lambdas, Dynamo and SNS or SQS, which we can represent in a diagram like this:

For this example, let’s say we need to test Microservice A, and Microservice A communicates with Microservices B and C during a given data flow. Because the services were very focused in purpose they would have to reach out to other services to validate external information.

We don’t want to stand up instances of B and C to test A, so we need to isolate some interactions:

This is exactly what test scaffolding does. It takes the place of services B and C, providing a ton of control within the test about how we set up and manage inputs and outputs.

A few things to note:

• We only have one test scaffolding per service. We can add as many different types of interactions as needed into that single test scaffolding—even if they represent different endpoints across different services.
• The microservice and test scaffolding are both defined by their own CloudFormation Template (CFT) but stored in the same project repo.

Example use cases:

Now that we have a definition of what our microservice boundaries are, lets discuss a few use cases.

In our first example, we have a service that accepts an API request, processes the request, then sends output via an SNS topic. We use this pattern on our project to send information out to different aggregator services that would then map the data into the correct format for publishing to several third party services. We want to ensure that the input values get processed correctly and get output to the SNS topic. SNS is tricky to interact with because it requires a subscription to capture the response. This is tough to do on the fly in an automated test so we added a Lambda to handle the subscription to the SNS topic and DynamoDB to retain the response for validation to the test scaffolding.

This allows tests to send requests to the service, have the service process the request, then pull the results sent to the SNS topic from the scaffolding DynamoDB to determine if the result is what we expected.

Our second example is sort of the mirror of the first. Here, our microservice gets an event from an SNS topic and calls out to a service for an additional status validation, is the product active for instance. This status would then be published to the mapped external service. In this case our test scaffolding would contain an SNS topic, API gateway, Lambda, and Dynamo. The test populates Dynamo with the data we want the service to receive from the test scaffolding API call and triggers the flow from the SNS topic to start the tests. During the processing the microservice reaching out with its status call to the API Gateway in the Test Scaffolding and gets the response the test staged there. We validate the final output from the process in the DynamoDB that is part of the microservice.

As the example shows, using test scaffolding gives us a high degree of flexibility in matching the interaction requirements of the service, allowing us to test each service in total isolation.

Scaffolding setup, deployment, and execution

So how did we configure these services to ensure they were looking in the right areas? Parameterization for the win in our cloud formation templates. Test scaffolding, and the microservice under test, each have their own CFT to manage necessary resources. In the microservice CFT we leverage template parameterization and conditional statements to control the use of test scaffolding depending on how the service is being deployed. These parameters are passed in via the CI/CD pipeline. In our pre-merge deployments this lets the microservice look for the test scaffolding and in our integrated environments allows us to have that service looking for the appropriate deployed service.

Here is a snippet of a microservice CFT that conditionally uses test scaffolding based on input parameters:

Test scaffolding gets deployed as part of the branch deployment in parallel with the microservice. The diagram below shows how scaffolding code must adhere to the same standards for a deployment as the rest of our services. Once deployed we run our functional tests.

It’s only when the isolated service tests using scaffolding pass that it’s possible to merge into the mainline branch. Test scaffolding is NOT deployed in our integrated environments. Once the changes are in the mainline, we use a scheduled job to clean up any isolation environments (services and scaffolding) that might not have been torn down manually. This is important because there are caps on the number of available resources in a given account.

Testing, leveled up

Test scaffolding allows our teams to develop and test microservice functionality independently—before being merged to a mainline branch and deployed into an integrated environment—using the exact same technology as the service is developed in (CFT and native AWS resources). Controlling input and output via scaffolding lets us build a deeper set of tests around critical flows and efficiently ensure a very high level of quality.

Because test scaffolding allows us to create more than just basic create, read update and delete tests, we are able to build in-depth functional flows that tie back to our business use cases. These deep functional tests give us confidence in microservice changes before being merged to mainline and deployed into an environment. Additionally, keeping a bulk of the automation at this level (as well as unit tests) minimizes the use of Journey tests (i.e.: “end-to-end tests”) and supports a healthy automation suite across all levels of the test pyramid.

Some highlights:

• Tests are executed on every push into the repository
• The tests are FAST! The longest running set of tests took under 10 minutes for several hundred tests against one of our main services.
• The tests gate upcoming merges. If they don’t pass, your commits can’t be merged, and cannot impact other teams working on other services.
• Tests define and control all their own data, making them fully deterministic and hermetic.

Lessons learned

Test scaffolding required some trial and error as we learned the best ways to implement and leverage it. We had to define both how to incorporate it into our projects as well as the best way to build and deploy it as a part of our pipelines. A few key learnings:

KISS — (“Keep It Simple, Stupid”) There was a tendency to make behavior in test scaffolding more complex than it needed to be. You don’t need to duplicate dependency logic to generate an appropriate response.

Clean up — When changes were committed into the mainline, the test versions of the service and test scaffolding were left behind. Thus, we established cleanup jobs to remove artifacts from our AWS dev account that were no longer in active development or had gone several days without updates. The fact that everything was defined by cloud formation stacks made it very easy to redeploy changes as needed.

Contract evolution — It is critical that requests and responses used in test scaffolding align to the real service contract. Otherwise, you are not testing what you should be. Inter-team communication and contract testing is still critical.

Like any new process, test scaffolding took time and effort to develop and deploy, but eventually became a powerful tool and critical asset in our overall microservice development approach.