Contract Tests: A New Hope
A story about why and how we added ad-hoc contract testing
Previous Q&A before reading:
“Does it apply to my app?” We’ve no idea about your app, sorry. Read, try, learn, and repeat.
“Does it cover every single case?” Of course not; we’re still learning. We’re still iterating.
This is our approach. It worked for us. It gave us a lot of value. We hope the same for you.
A little bit of context
About us
v6 is the team in charge of providing a mobile application to Mercadona’s stores in order to manage the product stock.
If you’re a usual Mercadona’s Medium reader, this could ring a bell to you, and you’ll be totally right cause we explained what the team actually does in Our Bread and Butter. If you haven’t read it, it’ll give you more context about why we’re developing a new tool for our beloved store colleagues.
About the product
In terms of engineering, there are three main parts implied in making this possible:
- A mobile application written in Android (The beautiful part. It aims to make the workers’ lives easier.)
- A backend service in Django (The API, the logic, the magic. Also, it’s the connection between the app and the Mercadona services.)
- Multiple Mercadona services (The data. The single source of truth.)
About the scope
Let me show you some figures (Updated on 6th September 2023):
- 532 deployed centers
- 804 daily active users
- 779 app versions in not much more than two years
- 831 API versions in not much more than two years
- 10 different app versions running in production
- 2 production environments
As you’ve just seen, the magnitude of the product is enormous, and so is the team’s responsibility. That’s why we’re always thinking about how to improve the quality of our software, build the most helpful UI, or, in this case, avoid bugs before going to production.
The problem
Imagine we have a range of 10 app versions running in production (From 100 to 109).
Let’s say we have the following endpoint:
// GET /api/products/1234/
{
"id": "1234",
"name": "HUMMUS",
"description": "Hummus de garbanzos Hacendado receta clásica",
"image_url": "https://www..."
}This endpoint uses its own serializer:
class ProductSerializer(serializers.Serializer):
id = serializers.CharField()
name = serializers.CharField()
description = serializers.CharField()
image_url = serializers.CharField()Therefore, in the Android part, we have the corresponding model:
@kotlinx.serialization.Serializable
data class Product(
@SerialName("id") val id: String,
@SerialName("name") val name: String,
@SerialName("description") val description: String,
@SerialName("image_url") val image: String,
)Due to product requirements, we must remove the description from the screen so it won’t be used anymore. So, in version 110 we remove the UI, the implied logic, and finally, the description property in the model, and we deploy to production. Work is done in the client part! 💪🏽.
So far, so good; it’s the backend turn — easy peasy lemon squeezy. We have to remove the description property in the serializer, remove the affected code, and deploy it to production.
Well, actually, it isn’t so straightforward. If we did it, we’d break the versions between 100 and 109 as they still show the description on the screen. What we usually do is propagate the new version organically, and after some time (Our adoption rate grows really fast), we perform a force update. It’s a mechanism to ensure the users are using at least the version we want.
Why not use the force update always? Our deployment pace is quite high. We have Continuous Deployment, so sometimes we release 10 versions per day. You can imagine we can’t constantly bother our users with new updates. It won’t be the best experience. Furthermore, they are focused on daily work like squaring the stock, reporting breakages, adding products to be donated, and so on. That’s way more important than running the last version of the application at every single moment.
The real problem: humans
Even having force update or feature flags, we aren’t exempt from failing. “What was the version we have to set as the minimum version for [breaking change] that we did last month?“. “Can I delete the [used in old in-production versions] endpoint?” I don’t know you, but I see a lot of uncertainty in those questions. And a lot of manual work that could fail. And we failed. Of course, we did.
So yes, we had an incident, and we solved it by increasing the app’s minimum version, and that’s all. Five minutes after the incident. Nothing serious. We, as engineers, were worried about this happening again.
As we have e2e tests running before merging anything to the master branch of the Android repository, and we’re speaking about an old version, we couldn’t detect this was failing. That’s why, in the Postmortem, we talked about running the e2e tests when there’s a new backend version and executing the minimum client version, too. But after discussing it for a while, we discarded it. We didn’t want to block the backend pipeline in every pull request while the Android e2e tests ran. Too much time, too much flakiness, very little confidence. So we ended up with a new idea: try Contract Testing.
Contract testing
If you surf the Internet looking for a definition of what Contract Testing is, you’ll find different approaches, tools, and complex terms to explain something we could simplify:
A way to test that we expect A and we get A.
Let me skip the time we spent trying to add these tools and making them run. To summarize, we discarded them not only due to the steep learning curve but also to the required maintenance of those tools (We had to have specific fixtures just for this kind of test), so we created our own in-house solution.
As an engineering team, we follow the LEAN methodology and try to build things in the simplest way possible. So, we decided to cover the easiest part first. “Can we test just one endpoint?”, “What do we need?”, “Let’s do it manually, and we’ll see.” and so on were the typical sentences to start from the bottom, so this is what we did.
Requirements
By discarding those tools, we defined the bases of what we wanted. It had to be:
Easy to read
Just by taking a look at the test or the CI log, we have to know where the problem is. No spaghetti code, no magic. Just keep it simple.
Easy to maintain
They should be the single source of truth of the connection between the app and the API. Having more than one place to maintain this equals having problems in the future.
Fast
We’re going to block the backend pipeline. These tests should last as little time as possible. Launching the Android emulator or downloading Docker images and starting them was out of this equation. We had to save time by running just the tests.
Implementation
Backend
We wanted a JSON file as a single source of truth for both software pieces: backend and client. Where do we have a JSON in our backend? Exactly, as we’re using Django, it’s in the place where we check the endpoints: the view tests.
The first step was to extract the JSON of the assert to an isolated JSON file. So we passed from this:
@pytest.mark.django_db
@pytest.mark.usefixtures("center_1234")
class TestRetrieveProductView:
def test_returns_200_when_product_exists(
self, client, mock_retrieve_products, product
):
mock_retrieve_products(product=product)
response = client.get("/api/products/1234/", HTTP_CENTER_ID="1234")
assert response.status_code == 200
assert response.json() == {
"id": "1234",
"name": "HUMMUS",
"description": "Hummus de garbanzos Hacendado receta clásica",
"image_url": "https://www..."
}To this:
@pytest.mark.django_db
@pytest.mark.usefixtures("center_1234")
class TestRetrieveProductView:
def test_returns_200_when_product_exists(
self, client, mock_retrieve_products, product, expected_contract_response
):
contract_path = "src/contracts/products/retrieve_product.json"
mock_retrieve_products(product=product)
response = client.get("/api/products/1234/", HTTP_CENTER_ID="1234")
assert response.status_code == 200
assert response.json() == expected_contract_response(contract_path)As you can imagine, the content of the retrieve_product.json file is what we were asserting in the previous test. Which is no more than a plain JSON:
{
"id": "1234",
"name": "HUMMUS",
"description": "Hummus de garbanzos Hacendado receta clásica",
"image_url": "https://www..."
}Android
Two previous notes just to clarify the Android part. As tons of apps do, we’re using:
- Retrofit for the network layer.
- Kotlinx.serialization for the serialization.
As we saw at the top of the post, we have the Product model. We use this model in the endpoint to retrieve a specific product based on its ID, so we defined the interface:
interface Endpoints {
@GET("api/products/{id}/")
suspend fun getProduct(@Path("id") id: String): Product
}For those unfamiliar with Retrofit, this piece of code means that a successful network call to the api/products/<id>/ endpoint must return a Product instance. If the transformation to a Product instance fails, a SerializationException is raised.
We have the data (JSON file), and we have the requirements. What are we waiting for? Let’s test it!
First things first: we need a fake Endpoints class that will try to deserialize the JSON file properly, transforming it into a Product. As we don’t need that class in the production code, we create that implementation in the tests:
class EndpointsImpl(private val json: Json) : Endpoints {
lateinit var getProductResponse: String
override suspend fun getProduct(id: String): Product =
json.decodeFromString(getProductResponse)
}Basically, we’re injecting the JSON serializer through the constructor to use it when we invoke the getProduct method. getProductResponse var will be defined in the test. That way, we’ll be able to test different inputs.
It’s time to write the test. Let’s do it!
class EndpointsContractTest {
private val endpoints = EndpointsImpl(Json)
@Test
fun `retrieve product is ok`() = runTest {
endpoints.getProductResponse = getResponse("/contracts/products/retrieve_product.json")
endpoints.getProduct("1234")
}
}It looks quite simple, isn’t it? But it’s just like that:
- Define the endpoints instance.
- Set the response that you want in every test. In this case, we’re getting the content of the
retrieve_product.jsonfile, which was previously downloaded. - Call the
getProductmethod.
“Uhm… wait, wait, where is the assert?” There’s no assert in this case. As we’ve mentioned before, if the serialization fails, it raises a SerializationException. Anyway, you can check the returned values if you want. This was enough for us.
Running the test
It’s time to run the test:
Failure
Let’s go for the red first, so we remove the description field from the JSON file:
{
"id": "1234",
"name": "HUMMUS",
"image_url": "https://www..."
}And here we have the failure. As you can see, the error message is self-explanatory:
kotlinx.serialization.MissingFieldException: Field 'description' is required
for type with serial name 'Product', but it was missing at path: $Success
It failed as expected, so now, we restore the description field, re-run the test, and get the green!
The end?
Well, of course not. We’ll continue iterating this for sure. This is the first stone.
We’ve omitted topics out of the scope of this article, like how to download the JSON files or how to integrate this in a CI/CD service, but if you’re interested in having more info about these or any other related topic, please, write to us in the comments.
And that’s all folks. We strongly hope this article is useful to you in the same way that doing this helped us to gain confidence and protect our beloved users.
P.S. I couldn’t finish the article without mentioning that all this achievement was possible due to the huge effort of the v6 team. I’m just writing the post, but there has been a lot of work from each and everyone within this fantastic team.
