Is it an end-to-end test, a regression test, or an acceptance test?
You know what “unit test”, “integration test”, and “system test” mean. But what about “end-to-end tests”, “regression tests”, and “acceptance tests”? This post disambiguates them for you.
Speak of scopes
Let’s start with a one-paragraph refresher about unit tests, integration tests, and system tests:
First, there are unit tests. They test the smallest components (often individual functions/methods, but can also be classes/modules). At a larger scope, integration tests test how individual components work together, while system tests test the whole system. I like the explanation from this article:
If an application has three modules A, B, and C, then testing done by combining the modules A & B or module B & C or module A & C is known as Integration testing. Integrating all the three modules and testing it as a complete system is termed as System testing.
The terminology gets hazy going further.
If this one monolithic system constitutes your entire end-user workflow, then you can also refer to your system tests “end-to-end tests”. Think of fully-offline, single-purpose, desktop-only applications, such as a pomodoro timer. Chances are that your application is more complex than that.
Especially with web applications, each end-user request may involve multiple layers of services, and your system is just one of them. The term for testing the entire workflow from start to finish (as if you were an end-user) is called end-to-end testing. That’s the proper usage of this term. Let’s not waste it on monolithic applications.
Each layer may expose an API for other layers to integrate with. For example,
- your business logic layer may have RESTful or RPC APIs,
- your database may have SQL endpoints, and
- even frontends may have a mockup from the designers.
Each service has (formally or informally) promised some behavior to its users (i.e., other layers that interface with it), forming a (written-down or virtual) contract. As long as this contract is upheld, clients will be able to integrate with the service. Ensuring this promise is the goal of contract tests.
In a company with more than a dozen engineers, it’s likely that each layer is owned by a separate team. Therefore, depending on who executes those tests, there can be different flavors of “contract testing”:
- Client teams of a service may take the contract that they were informed with, test the service against this contract, and blame the service maintainers for breaching the contract. That’s called “consumer-driven contract testing”.
- Alternatively, we can do it the traditional way and have the publishers of the contract (i.e., the API maintainers) to test their API against the contract. That’s “producer-driven contract testing”.
To summarize, there are 2 sets of terminology that you may use to describe software tests with different scopes, depending on the context:
Speak of purposes
If your ever read about “software testing hierarchy” online, you may found many articles describing “acceptance testing” as one more stage after system tests (examples: 1, 2, and 3). They are not wrong, but lining up “acceptance tests” with “unit tests”, “integration tests”, and “system tests” may mislead new engineers to think that “acceptance” an synonym of “end-to-end”. It’s not.
Acceptance tests ensure system meets business/user requirements. It is often enforced with good product management (PM) practices. If your team performs agile development, you might have a checklist called “definition of done” (DoD) to complete with every project. One item on this list may read like “Feature is OK’d by stakeholders” (source: ProductPlan). That’s acceptance test.
To complete the argument, no acceptance test is involved if you are just
- scrambling up a toy project for a hackathon demo,
- developing a proof-of-concept mock that is going to be destroyed after the next brainstorming session, or
- refactoring code to clean up tech debt without modifying any observable behavior (incl. features and performance).
Since the product requirement might vary in scope, an acceptance test can also range from “unit test” to “end-to-end test”.
Another term that also emphasize on purpose rather than scope is “regression testing”. Put simply, any pre-existing test can be called a regression test.
- The scope doesn’t matter. If an unit test reported error, that unit has “regressed”. If an integration test failed, the interface has been broken.
- The aspect doesn’t matter. If a functional test started to fail, a feature is broken. If a load test reported +200% latency, then the performance has deteriorated.
There are many other types of tests that enforce different aspects of software quality, such as “smoke tests”, “security tests”, and “performance tests”. Yet some others refer to the methodology rather than scope or purpose, including fuzz tests and load tests. They are usually not as confusing as the concepts discussed above, so I’ll stop it here.
Conclusion
Software testing is an essential part of software development that helps to ensure the reliability and quality of a software product. There are different types of software tests that vary in scope and purpose. Understanding these different types of software tests and their purposes can help software developers and testers to ensure the quality and reliability of their software products.
