products

Retry Or Not

How to dynamically determine if a Cypress test should be retried or not based on the error.

Cypress has built-in test retries. For example, we can let a test retry 2 more times if it fails.

1
2
3
4
5
it('retries this test', { retries: 2, defaultCommandTimeout: 1000 }, () => {
  cy.visit('/cypress-examples/')
  // this element does not exist
  cy.get('#foo')
})

The test fails because there is no element with id foo and retries 2 more times:

The test retries 2 more times

There are different kinds of errors. What if instead of the "element not found" we try to visit the page that does not exist? For example, cy.visit('/cypress-exampleZ/') would be a much worse failure, right? We probably should not even attempt to re-run the test if the cy.visit command fails. How do we change the retries dynamically based on the test error?

First, let's get the test error. In the afterEach hook we can inspect the test status and get the error object if the test has failed. The afterEach hooks do run for every test, even for the failed ones.

1
2
3
4
5
6
7
8
9
10
11
12
it('retries this test', { retries: 2, defaultCommandTimeout: 1000 }, () => {
  cy.visit('/cypress-exampleZ/')
  // this element does not exist
  cy.get('#foo')
})

afterEach(() => {
  cy.log('**after each**')
  if (cy.state('test').state === 'failed') {
    console.log(cy.state('test').err)
  }
})

The error object

Great, so if the error message includes cy.visit string, we want to disable the test retries. This is the implementation detail, but here is how Cypress v12 can do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
it('retries this test', { retries: 2, defaultCommandTimeout: 1000 }, () => {
  cy.visit('/cypress-exampleZ/')
  // this element does not exist
  cy.get('#foo')
})

afterEach(() => {
  cy.log('**after each**')
  if (cy.state('test').state === 'failed') {
    console.log(cy.state('test').err)
    if (cy.state('test').err.message.includes('cy.visit')) {
      console.error('test failed because of cy.visit')
      console.error('skipping retries')
      cy.state('test')._testConfig.testConfigList[0].overrides.retries = -1
    }
  }
})

There are no retries when the cy.visit command fails

Let's change the test to fail not because of cy.visit. The retries are working as configured.

Of course, we are using the internal implementation details to achieve this, but sometimes it is fun to sneak into the control room.

🎁 You can find the source code for this blog post in the repo bahmutov/retry-or-not.

Bonus 1: Sleep before retrying the test

Sometimes the test fails due to some temporary application hiccup. We can introduce progressively longer waits, based on the current test attempt. For example, we might sleep for 5 seconds on the second attempts, 10 seconds on the third attempt, etc - the strategy is up to you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
beforeEach(() => {
  cy.log(`current retry **${Cypress.currentRetry}**`)
  if (Cypress.currentRetry) {
    // control the sleep duration based on the retry index
    cy.wait(Cypress.currentRetry * 5_000)
  }
})

it(
  'retries the test with longer backoff',
  { retries: 2, defaultCommandTimeout: 1000 },
  () => {
    cy.visit('/cypress-examples/')
    // this element does not exist
    cy.get('#foo')
  },
)

The test sleeps depending on the attempt