Fast-Check: A Comprehensive Guide to Property-Based Testing
Every once in a while, you’ve been assigned to perform some unit tests on a specific feature, and though you knew how to test it, the difficult part sometimes is to choose which test data you are going to use.
You might even have the concepts taught in the ISTQB present, and don’t misunderstand me, they are quite effective and will fulfill your needs.
But let’s imagine that your unit test, to cover the major groups of possibilities, needs for example 5 different data sets.
Having them present in your test file, depending on your implementation, can make it repetitive and huge. And also, those data sets might even not cover all the main scenarios and edge cases. Having the responsibility of finding bugs dependent on the developer to think on those combinations may be tricky.
Also, it may be a hidden cost for your company, since either you write an insufficient portion of scenarios and edge cases, leading to production bugs, or you spend too much time trying to fit in all edge cases in your test.
Results from the 2019 Diffblue developer survey revealed that the annual cost incurred by companies for writing unit tests averages £14,287 per developer.
Extrapolating this figure for a mid-size company employing around 45 developers, the total annual expense for unit testing stands at roughly £643,000.
This cost encompasses not only the initial creation of unit tests but also ongoing maintenance efforts required to ensure their relevance and effectiveness over time. Investing in quality is worth every penny, but if we could achieve the same or greater level of quality with less cost, it would be the perfect scenario, right? 😃
What if I tell you that there is a library that could generate those data sets for you, replacing your N different data sets for some few code lines? Sounds intriguing, doesn’t it?
Table of Contents
- What is fast-check?
- How to use fast-check?
- What is a runner?
- What about properties?
- What are arbitraries?
- What about running the tests?
- Conclusion
What is fast-check?
Fast-check is a Property-Based Testing (PBT) framework for JavaScript, written in TypeScript.
Basically, property-based testing is an approach that involves specifying statements that should always be true, rather than relying on specific examples.
Thus, it enables you to test functions across a large number of inputs with fewer tests.
It stands out from the traditional approaches for the following reasons:
- Generative Testing
Fast-check introduces generative testing, enabling you to define overarching properties for your functions. This departure from example-centric testing amplifies test coverage, ensuring a more comprehensive evaluation.
- Randomized Testing
The framework excels in randomized testing by generating random inputs, providing a broader spectrum of scenarios. Thus, it will unveil edge cases and potential issues that might elude conventional example-based tests. Particularly advantageous for intricate or less accessible code paths.
- Reproducibility
Fast-check boasts full determinism in property-based testing. Tests launch with a precise seed, guaranteeing consistent sets of values on each run. This reproducibility simplifies error reproduction and enhances debugging efficiency.
- Designed for Bug Detection
At its core, property-based testing aims not just to generate random data but to unearth bugs. This framework was designed to identify common problems with heightened probability, making it a potent instrument for bug discovery.
- Shrinking Capabilities
It includes a unique feature — shrinking capabilities. When a failure occurs, it tries to present a more straightforward failure scenario for analysis. This streamlines the debugging process by spotlighting the minimal input triggering the failure.
- Documentation
By articulating invariants and enduring properties, you encapsulate the fundamental expectations of your code, making it a valuable ally in maintaining a clear understanding of your code’s intended behavior.
Now that you are aware of its advantages, lets see how easy it is to use it…
How to use fast-check?
It is quite simple, you can install it in your project using either npm or yarn:
npm install --save-dev fast-check
yarn add --dev fast-checkAfterwards, lets look at a really simple test using it, that validates if a pattern is present inside a text:
import fc from 'fast-check';
// Code under test
const contains = (text, pattern) => text.indexOf(pattern) >= 0;
// Properties
describe('properties', () => {
// string text always contains itself
it('should always contain itself', () => {
fc.assert(
fc.property(fc.string(), (text) => contains(text, text)),
{ numRuns: 1000 }
);
});
})Additionally, you will need a test framework. In this example, I will use mocha.
You are seeing now methods like the assert (a runner type) and property, that I will explain later.
But just for reference, I specified that I wanted 1000 executions/combinations (the default is 100), and this was executed in 24ms… 😅
Basically, fast-check will generate 1000 different values for the text variable, and will execute the contains method, generating a broad variation of string formats, even edge cases, with the purpose to find one case that drives the assertion to fail.
Its quite a redundant example to test if a specific string contains its specific value, but for this example, it is enough to understand its purpose.
What is a runner?
A “runner” within fast-check is the component responsible for executing and testing a property.
In the example code, fc.assert serves as the runner. fc.assert is considered the most fundamental runner, as it conducts tests on the specified property and throws an error if the test fails.
While fast-check offers additional runners such as fc.check, fc.sample, and others, fc.assert suffices for the majority of use cases, proving to be a reliable solution in approximately 99% of scenarios. For further details on runners, users are encouraged to refer to the documentation.
What about properties?
The next crucial element in fast-check is fc.property, which is a function utilized for creating the cornerstone of PBT: a property.
In essence, a property is constructed by defining a list of input generators, also known as arbitraries, along with a function that accepts these inputs.
For example, consider the following line:
fc.property(fc.string(), text => contains(text, text))This line creates a property with a single string parameter generated by fc.string(), and defined by a function returning a boolean (text => contains(text, text)). Additionally, for properties requiring asynchronous behavior, fc.asyncProperty is available.
If you want detailed information, check the official Properties documentation.
What are arbitraries?
Arbitraries within fast-check are crucial components responsible for generating random inputs for properties. These arbitraries cover various data types and scenarios, providing users with a comprehensive toolkit for property-based testing.
Fast-check offers a diverse set of built-in arbitraries, where you can even define things like maximum and minimum length of the values produced and pre-conditions, and that are categorized into different types such as:
- Primitives: These include fundamental data types such as strings (
fc.string()), floats (fc.float()), booleans, and characters. - Composites: These arbitraries generate more complex data structures, such as arrays (
fc.array()), objects (fc.object()), and tuples (fc.tuple()). - Combiners: These allow users to combine or manipulate existing arbitraries, such as
fc.option(),fc.oneof(), andfc.constant(). - Fake Data: These provide generators for generating realistic but fake data, such as file contents (
fc.file()), JSON-compatible strings (fc.json()), and UUIDs (fc.uuid()). - Others: Additional types include generators for internet-related data (
fc.ipV4(),fc.emailAddress()), identifiers (fc.ulid(),fc.uuid()), and falsy values (fc.falsy()).
Users also have the flexibility to create custom arbitraries tailored to their specific testing needs. For a detailed list of available built-in arbitraries and their usage, users can refer to the documentation.
What about running the tests?
Running the tests in fast-check is a straightforward process. If you already have a mocha runner configured, simply execute yarn test as usual.
In the event of a failure, you may encounter a message similar to the following:
1) should always contain its substrings
Error: Property failed after 1 tests (seed: 1527422598337, path: 0:0): ["","",""]
Shrunk 1 time(s)
Got error: Property failed by returning falseThis error notification not only alerts you to the failure but also provides a counterexample of your property, indicating an input for which your function is incorrect. In the given example, ["","",""] represents the smallest counterexample.
It’s important to highlight that this isn’t just any counterexample; it’s the smallest one. The process by which PBT identifies the smallest counterexample from a failure is known as shrinking, which is a significant feature of PBT.
Exploring shrinking further could even be a topic worthy of its own dedicated article. And not only the shrinking capability, but also advanced usages of the tool, like:
- Fake data: replace random fake data by fake data backed by PBT;
- Fuzzing: turn fast-check into a fuzzer;
- Model based testing: also known as monkey testing to some extend;
- Race conditions: easily detect race conditions in your JavaScript code.
Conclusion
This tool is quite useful, isn’t it? 😃
It does offer a paradigm shift from traditional example-centric testing methodologies.
By leveraging generative and randomized testing, along with deterministic and reproducible properties, fast-check significantly enhances test coverage and bug detection capabilities.
Moreover, its shrinking capabilities streamline the debugging process by pinpointing minimal failure scenarios. As demonstrated, fast-check provides a diverse set of built-in arbitraries, enabling users to create comprehensive and efficient test suites.
With its intuitive usage and robust features, fast-check empowers developers to uncover complex bugs, improve code reliability, and enhance overall software quality.
As you dive deeper into its advanced functionalities, such as fuzzing, model-based testing, and race condition detection, you unlock new avenues for ensuring the robustness and resilience of your JavaScript applications.
I hope you enjoyed reading this article!
My name is João Coelho, and I am currently a QA Automation Engineer at Talkdesk. Lately, I have been writing articles regarding automation, QA and software engineering topics, that might not be known by the community.
If you want to follow my work, check my Linkedin and my author profile at Medium!
Furthermore, if you’re interested in further supporting me and my content creation efforts, you can do so by buying me a coffee! 😄👇
Your support goes a long way in helping me dedicate more time to researching and sharing valuable insights about automation, QA, and software engineering.
