What is the best testing framework for Java code nowadays? Spock or JUnit 5? Check it out in this ultimate feature comparison.

Spock was a game changer for all the people struggling with unit testing in JUnit 4. Compact syntax, parameterized tests or flexibility to mention just a few advantages. Over 10 years after JUnit 4.0, the brand new, written from scratch, Java 8 optimized Junit 5 has been released. And time has not stopped then.

In this blog post, I will compare selected areas of Spock and JUnit 5 to give you an overview how the situation looks like nowadays. I will try to answer the question if its time for Spock to fade into oblivion or maybe quite the opposite it is still light years ahead of JUnit 5.

Historical background

As a warm-up, let’s take a look at the important dates in the automatic code testing in Java.

• 2000 - JUnit - first xUnit for Java
• 2004 - TestNG - Java 5 (annotations) leveraged in tests
  • with some unique (at the time) features (such as parameterized tests, test repetition or test dependency)
• 2006 - JUnit 4 - Java 5 support
  • catching up TestNG features over the next few years
  • de facto standard (the world chose :-/ ), steady evolution rather than revolution
• 2009 - Spock - revamped test creation
  • fresh ideas and power of Groovy under the hood
• 2017 - JUnit 5 - redesigned and rewritten from scratch
  • new king of the hill? - let’s check it out!

Development comparison

Let’s start with a comparison of some biased aspects of development of JUnit 5 and Spock.

The first thing to notice is the fact that JUnit 5 is much younger project with very high development activity. There are 3 active committers with a bunch of external contributors.

One the other hand, Spock is much more mature with lower development activity. However, many of the features developed in JUnit 5 recently were already available in Spock. Therefor, the need for lots of commit in Spock is lower. Spock has currently one main maintainer and 2-3 people regularly contributing to the project (of course also with a pack of external contributors).

Partial verdict

Tool support

The next thing to cover is Java 11 compatibility and support in tools.

Both of the frameworks are currently very well supported in the Java ecosystem. However, it is worth to mention that tests (specifications) in Spock are written in Groovy (which notabene is the main power of Spock). As a downside, due to much more dynamic nature of Groovy as a language, IDEs have much harder time to provide as good support for it as for Java (e.g. refactoring or compile time error reporting).

Regarding Java 11 and Spock, with modern Groovy 2.5.x it should work flawlessly. In general, most of the things should work fine also with Java 12, 13 and (with Groovy 2.5.10+) also 14. The official Java 14+ support should be available in Spock 2.0 (still in development) with Groovy 3.0.

On the other hand, JUnit 5 is a role model on that field. It is continuously tested on a CI server with the recent Java versions (including the early access builds, such as OpenJDK 15 EA).

Partial verdict

Test structure

Let’s move on to less technical aspects of test development - test structure.

As die-hard followers of my blog may know, I am a big propagator of well structured automatic tests with the BDD-originated given-when-then concept. In short, it unifies test creation, improves their readability and makes it easier to write, especially for less experienced people. To learn more, please take a loot at a separate blog post that I wrote some time ago.

JUnit 5

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

class SimpleCalculatorTest {
    @Test
    void shouldAddTwoNumbers() {
        //given
        Calculator calculator = new Calculator();
        //when
        int result = calculator.add(1, 2);
        //then
        assertEquals(3, result);
    }
}

The given-when-then sections are marked just by plain comments in code. It is not perfect. They might be forgotten and are easy to lost/misplace during refactoring. Could it be done better?

Spock

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

class SimpleCalculatorSpec extends Specification {
    def "should add two numbers"() {
        given:
            Calculator calculator = new Calculator()
        when:
            int result = calculator.add(1, 2)
        then:
            result == 3
    }
}

Well, in Spock the (almost not used in Java) label construction is leveraged. It is crucial to highlight that they are not just comments in code - they are a part of the specification. For example, having when: removed in the example above, generates real compilation errors. Nice.

Btw, of course, in Spock it is possible to write just one-liners, if feasible. However, developers have to do it intently.

Partial verdict

Btw, with both JUnit 5 and Spock it is worth to use code templates to generate test skeleton to fill instead of writing all those given-when-then by hand :-).

Exception testing

Another topic I want to cover is exception testing. In general, to test corner cases (alternative scenarios) it is needed to verify that an exception with a given type was thrown. Usually, it is followed by some extra assertions such as an exception message and a cause.

JUnit 5

In JUnit 5, it is no longer possible to use @Test(expected = NullPointerException.class), useful for one-liners, but potentially risky with more complicated test code. Instead of, we have the assertThrows() construction, similar to catchThrowable() from AssertJ.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

@Test
void shouldThrowBusinessExceptionOnCommunicationProblem() {
    //when
    Executable e = () -> client.sendPing(TEST_REQUEST_ID)
    //then
    CommunicationException thrown = assertThrows(CommunicationException.class, e);
    assertEquals("Communication problem when sending request with id: " + TEST_REQUEST_ID,
                 thrown.getMessage());
    assertEquals(TEST_REQUEST_ID, thrown.getRequestId());
}

The syntax is very compact (thanks to the lambda expression which is used to catch an exception). In addition, it is easy to perform any required further assertion(s) on the returned exception instance.

Spock

Spock, on the other hand, still provides @FailsWith(NullPointerException) for one-liners. However, it is almost completely not used on a daily basis. All thanks to the thrown() construction.

1
2
3
4
5
6
7
8

def "should capture exception"() {
    when:
        client.sendPing(TEST_REQUEST_ID)
    then:
        CommunicationException e = thrown()
        e.message == "Communication problem when sending request with id: $TEST_REQUEST_ID"
        e.requestId == TEST_REQUEST_ID
}

It is worth to notice that there is no lambda expression to catch an exception. To implement that Spock leverages Groovy AST transformations which under the hood transparently add required code. As a result, the exception thrown in the when block is caught and returned from the thrown() method. Really smart.

In addition, it is nice that in Spock, there is a smart type inference. It is not needed to precise the exception type twice. Based on the variable type on the left side, Spock is able to create the proper try-catch block.

As a bonus, there as a separate utility class Exceptions in Spock which provides extra assertions to play with the cause chain.

Partial verdict

Conditional test execution

In multiple situation it is required to (not) execute a given test only if given condition is (not) met.

The most common cases include:

• particular Java version
  • reflection hack to use newer version features
  • compatibility testing for lower version
• specific operating system
  • notifications about changed files on Mac are much delayed
  • testing symlinks on Windows makes no sense
• tests executed only on CI server, stage environment, …

JUnit 5

JUnit 5 introduced brand new support for annotation-based @Enabled* and @Disabled* conditions.

There is a predefined set of conditions for:

• JVM version
• operating system
• system property
• environment variable

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

@Test
@DisabledOnOs(OS.WINDOWS)
void shouldTestSymlinksBasedLogic() {
    ...
}

@Test
@EnabledIfSystemProperty(named = "os.arch", matches = ".*32.*")
void shouldBeRunOn32BitSystems() {
    ...
}

The JVM version and operating system conditions are enums and thanks to that, there is a nice code completion provided by an IDE. In addition the conditions can be used as meta-annotations to make more complex scenarios easier to use in multiple places.

1
2
3
4
5
6
7

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Test
@EnabledOnOs(WINDOWS)
@EnabledIfSystemProperty(named = "os.arch", matches = ".*32.*")
@interface TestOn32BitWindows {
}

With the following usage:

1
2
3
4
5

@TestOn32BitWindows
void shouldDoStrangeThings1OnLegacyWindows() { ... }

@TestOn32BitWindows
void shouldDoStrangeThings2OnLegacyWindows() { ... }

Originally, there was also support for custom logic in script-based inline conditions. However, due to limited usability (code written in String) and deprecation of Nashorn in Java 11 it has been removed in JUnit 5.6. Writing own condition logic currently requires a custom implementation of ExecutionCondition.

Spock

Since time immemorial, Spock has provided support for annotation based conditions with @Requires and @IgnoreIf. Out of box it is possible to check:

• JVM version
• operating system
• system property
• environment variable

The same set implemented later on in JUnit 5. However, the usage is slightly different.

1
2
3
4
5

@IgnoreIf({ !jvm.java8Compatible })
def "should return empty Optional by default for unstubbed methods with Java 8+"() { ... }

@Requires({ sys["targetEnvironment"] != "prod" })
def "should execute smoke testing on non production environment"() { ... }

Instead of enums, the logic is written in Groovy Closures leveraging delegation to the jvm/os/sys/... objects providing a dedicated methods (such as isJava11Compatible() or isLinux()). Unfortunately there is no code completion available by default, but it can be enabled with a small trick (Update. Just please be aware that the linked slides are from 2016 - Spock 1.0/1.1 - and in some places are outdated).

Using Closures provides one more important benefit - custom logic can implemented inline with pure Groovy.

1
2

@Requires({ isStrongCryptographyEnabled() })    //custom static method
def "should test strong cryptography-based features"() { ... }

Partial verdict

Mocking

Mocking is a crucial part of automatic code testing. It is especially beneficial in unit testing, but can be also usable in integration testing (e.g. with Spring Framework).

JUnit 5

In JUnit 5 tests, Mockito is the first port of call when it comes to mocking.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

@Test
public void should_not_call_remote_service_if_found_in_cache() {
    //given
    given(cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER)).willReturn(Optional.of(PLUS));
    //when
    service.checkOperator(CACHED_MOBILE_NUMBER);
    //then
    then(webserviceMock).should(never()).checkOperator(CACHED_MOBILE_NUMBER);
//   verify(webserviceMock, never()).checkOperator(CACHED_MOBILE_NUMBER);   //alternative syntax
}

Mockito is an industrial standard for mocking in Java. It is very mature, well known with a rich set of features. In addition, Spring Boot provides the official support for injecting Mockito’s mocks and spies into its context, which can be very useful in integration tests.

Spock

Spock on the other hand contains a built-in mocking subsystem.

1
2
3
4
5
6
7
8

def "should not hit remote service if found in cache"() {
    given:
        cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER) >> Optional.of(PLUS)
    when:
        service.checkOperator(CACHED_MOBILE_NUMBER)
    then:
        0 * webserviceMock.checkOperator(CACHED_MOBILE_NUMBER)
}

While the syntax in Mockito is pretty readable, there are all those given/thenReturn/thenAnswer/... to write. Spock, however, leverages the power of operator overloading in Groovy (and of course its AST transformations) to make stubbing and verification as simple as possible.

1
2
3
4
5
6
7

given(cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER)).willReturn(Optional.of(PLUS)); //Mockito
// vs
cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER) >> Optional.of(PLUS) //Spock

verify(webserviceMock, never()).checkOperator(CACHED_MOBILE_NUMBER); //Mockito
// vs 
0 * webserviceMock.checkOperator(CACHED_MOBILE_NUMBER) //Spock

In a basic case, it’s enough to write a method, use an operator and provide a returned value. In verification it’s just a method call with requested cardinality. Most likely, it will not be possible to achieve that in Java in the predictable future.

Historically, the main limitation with Spock’s mocks was their tight coupling with the specification (the test itself). As a result it was not possible to use Spock’s stubs, mocks and spies in the Spring context for integration tests. However, starting with Spock 1.1 it was relaxed and Spock 1.2 introduced the first class support for mocking and spying with Spring. As a bonus, it is available also for pure Spring context (Spring Boot is not required as it takes place with Mockito’s mocks).

To be fair, it is required to admit that more magic in Spock generates some quirks with some corner cases. Nevertheless, once knowing all of them the readability and compactness definitely pay off :-).

One more thing. It is worth to remember that with Spock used for testing, we are not forced to use it also for mocking. It is perfectly fine to use Mockito if wanted (or needed).

Partial verdict

Parameterized tests

Parameterized tests in general provide a way to call one test with different set of input (and expected) parameters. It can dramatically reduce duplication in specific cases (e.g. in testing with the user defined acceptance data set).

A word of warning here. In some scenarios, using parameterized tests can hide specific business use cases which would be clearly exposed in our non-parameterized tests otherwise. There are techniques to avoid that, but it is a topic for another article.

JUnit 5

Parameterized tests were one of the key features of TestNG from its early days. In JUnit 4 the feature has been not available for year and even once implemented, the design made it very unpractical to use in most of the cases. To improve the situation the 3rd-party runners were available, but it was still sad (for JUnit 4).

Luckily, JUnit 5 is the first version of JUnit with sane built-in parameterized tests support.

1
2
3
4
5

@ParameterizedTest
@CsvSource({ "dog, 3", "wolf, 4", "hippopotamus , 12" })
void testStringLengthWithParameterizedTestInJUnit5(String word, int expectedNumberOfLetters) {
    assertEquals(word.length(), expectedNumberOfLetters);
}

For simple scenario with inline values (provided as CSV strings), there is a nice implicit argument conversion from String for various types (e.g. dates, files, path or UUID). It just requires to add @ParameterizedTest and @CsvSource (or @ValueSource, @EnumSource, etc.).

However, not everything can be put into string. The commonly encountered scenario is an object creation or calling a method defined in a test fixture. To support that JUnit 5 gives an ability to call a local method (or the one from external object) to return required input/output arguments. They should be provided as Stream (or List) of Arguments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

@ParameterizedTest(name = "value to pay for invoice {0} should be {1}")
@MethodSource("invoiceProvider")
void shouldCalculateToPayValueForInvoice(Invoice invoice, BigDecimal expectedValueToPay) {
    //when
    int valueToPay = invoice.toPayValue();
    //expect
    assertEquals(expectedValueToPay, valueToPay);
}

private static Stream<Arguments> invoiceProvider() {
    return Stream.of(
            Arguments.of(regularInvoice(), 54),
            Arguments.of(overduedInvoice(), 81),
            Arguments.of(paidInvoice(), 0)
    );
}

It is just required to use @MethodSource to point the method (or use a naming convention to resolve the correct method name automatically).

The thing I don’t like is a lack of the arguments type check. In tests with multiple parameters it can be quite confusing to determine which is which.

As a bonus it is also possible to customize a test name which will be displayed in the report (in IDE, HTML report, etc.).

Spock

In Spock, first let’s take a longer look at the simple parameterized test.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

@Unroll
def "should sum two integers (#x + #y = #expectedResult)"() {
    when:
        int result = calculator.add(x, y)
    then:
        result == expectedResult
    where:
         x |  y || expectedResult
         1 |  2 ||  3
        -2 |  3 ||  1
        -1 | -2 || -3
}

The words that come to my mind are “state of the art”. This is a place where Spocks excels.

With table-like formatting after the where keyword, input data looks very natural and is easy to read. What is worth mentioning, there is no need to define x, y, expectedResult as method arguments in a test (as it takes place in JUnit 5). The variables are added implicitly with full visibility and type inference in IDE.

The method name can be parameterized inline with the #var syntax (it’s Groovy - methods can have spaces and other “strange” characters).

In Spock it is also not a problem to use table-like syntax in a situation where constructor or methods are needed to be called (which in JUnit 5 requires using a separate provider method).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

@Unroll("Value to pay for invoice #invoice should be #expectedValueToPay")
def "should calculate value to pay for invoice"() {
    expect:
        invoice.valueToPay() == expectedValueToPay
    where:
        invoice           || expectedValueToPay
        regularInvoice()  || 54
        overduedInvoice() || 81
        paidInvoice()     || 0
}

While working with acceptance testing where results are presented to less technical users, it is also possible to distinguish a test name for developers with a scenario name for QA engineers (or product owners) by writing it in the @Unroll annotation.

Of course, for more complex/advanced use cases (e.g. generating input parameters dynamically based on exterenal source), data pipes and data providers can be used. They are not as natural as the aforementioned table, but are very powerful.

1
2
3
4
5
6
7
8
9

@Unroll("#pesel is valid (#dbId)")
def "should validate PESEL correctness (CSV)"() {
    expect:
        sut.validate(pesel)
    where:
        [dbId, _, _, pesel] << readValidPeopleFromCSVFile()
                                   .readLines().collect { it.split(',') }
                                   //ugly way to read CSV - don't do this :-)
}

Partial verdict

Migration

The next point is very important for people currently using JUnit 4.

JUnit 4 → JUnit 5

The JUnit 5 tests can co-exist with the JUnit 4 tests. There is a dedicated module junit5-vintage which executes old JUnit 4 tests on the new JUnit Platform (part of JUnit 5).

Writing new tests with JUnit 5 is also easy. The test structure is very similar. We just need to learn some new keywords, annotations and assertions (and remember to use @Test from the new org.junit.jupiter.api package :-) ).

JUnit 4 → Spock

The Spock tests can also co-exists with the JUnit 4 tests. In fact Spock 1.x is a (sophisticated, but still) JUnit 4 runner. However, besides that there is a completely new test structure which we need to get familiar with. It is easy to grasp and, in the end, it is more readable, but still, it is something new to learn.

Partial verdict

One more important migration-related thing here. During my training or consulting , I do not recommend rewriting existing hundreds and thousands of tests from JUnit 4 to the new technology, just for fun. Most of the existing tests will stay untouched for the foreseeable future. It is much better to write the new tests in the new technology and rewrite existing tests only along the way of doing some production logic related changes in them.

Summary - final verdict

Inarguably, JUnit 5 is a great progress over JUnit 4. Many features (previously) unique to Spock now are available in JUnit 5. However, Spock still excels in some areas. The best choice is no longer obvious.

Here, an anecdote, proposed my a colleague of mine. There are BMW and Audi cars. Both brands are very good and the choice usually depends on our personal preferences.

There same with Spock and JUnit 5. First, let’s take a look at the features comparison of the presented testing frameworks.

As you see, they both have their strengths and weaknesses. In addition to covered aspects, you should add to that list other aspects which are important for you (and for your team) and see which framework fits you more.

For instance.

On the other hand,

One final reflection. Preparing my presentation (and later on that blog post), I realized it would be great to have only that kind of dilemma while crafting software, choose between two pieces of good software :-).