Playwright’s Retry Feature for Individual Test Steps
While retries enhance test reliability, excessive use can hide issues like timeouts and race conditions. It’s critical to resolve root causes for stable tests, applying retries wisely with restrictions and logging data for unstable patterns. Playwright facilitates this with its built-in retry mechanism, automatically retrying failed assertions for improved readability. It offers better error reporting, easy integration with Playwright APIs, async/await support, and alignment with other frameworks’ patterns, thereby simplifying testing and boosting maintainability.
Unstable test environments can be mitigated with retries, which can also improve test reliability in stable environments. However, over-reliance can mask deeper issues. Identifying and addressing the root causes of test failures is vital, improving robustness by managing issues like timeouts and race conditions. Retries are a tool, not a solution, to enhance test reliability, but resolving underlying problems should be prioritized.
Best Practices
Set a retry limit to avoid endless loops from consistent failures. Retry only for temporary issue exceptions, not application bugs. Consider retries at different levels: test steps, test cases, or entire suites. Log retry data for debugging and identifying unstable test patterns. High retry volumes, which increase test execution time, may indicate deeper problems.
Retrying the Entire Test Suite
Selenium lacks a built-in retry function, needing external libraries or custom code for exception handling and test repetition, offering more control but complicating coding. Conversely, Playwright’s integrated retry mechanism simplifies flaky test handling through the hasRetries option, though it provides less control over triggering conditions and delay settings.
Retrying individual steps
To improve test reliability without having to rerun the entire test, retry only the problematic step. This strategy quickly identifies consistent or sporadic failures, thereby speeding up debugging. It saves time and resources, particularly for comprehensive tests. Selenium and Playwright can implement this retry function for individual steps by using a try-catch block for each step and defining the retry logic within the catch block. Let’s examine an example to understand the challenges of this generic approach and explore Playwright’s solution to this problem.
test('Enter the email address', async ({ page }) => {
await page.goto('https://www.voma.ai/');
const page1Promise = page.waitForEvent('popup');
await page.getByRole('link', { name: 'Try a demo' }).click();
const page1 = await page1Promise;
const emailLocator = page1.getByTestId('email');
const emailValue = 'ceroshjacob@gmail.com'
await emailLocator.fill(emailValue);
});This code snippet represents a test case. It navigates to a specific URL, clicks a link to open a popup, and enters an email address into the popup. However, the entered email address isn’t visible. This occurrence is due to the popup containing the input fields being triggered by clicking the “Try a demo” link. If you attempt to re-enter the email address, it should work. Consider using a generic try-catch to enter the email address.
const MAX_RETRIES = 3;
const RETRY_DELAY = 1000;
async function assertEmailValue(emailLocator, emailValue) {
let retries = 0;
let actualValue;
while (retries < MAX_RETRIES) {
try {
await emailLocator.fill(emailValue);
actualValue = await emailLocator.inputValue();
if (actualValue === emailValue) {
return; // Assertion passed
}
throw new Error(`Expected email value to be "${emailValue}", but got "${actualValue}"`);
} catch (error) {
retries++;
if (retries === MAX_RETRIES) {
console.error('Error:', error);
// Handle the error, e.g., mark the test as failed
} else {
await new Promise((resolve) => setTimeout(resolve, RETRY_DELAY));
}
}
}
}Start by defining constants for the maximum number of retries and the delay between attempts. Following this, create a function that persistently tries to validate the value of the email input field until it hits the retry limit, ensuring there’s a delay between each try. If the validation is successful within the specified attempts, the function will cease. Otherwise, it records an error and manages the failure. Using a generic try-catch approach to implement retries can lead to more boilerplate code and manual retry logic than using Playwright’s built-in retry mechanism with expect().toPass(). The built-in method is both more succinct and easier to set up.
const boundingBox = await emailLocator.boundingBox();
if (boundingBox) {
screenshotAfter = await emailLocator.screenshot({ path: 'emailValue.png', clip: boundingBox });
} else {
console.error('Element is not visible or does not have a bounding box');
}The entered email address becomes visible once you click the email address field before filling it in. After clicking the field and entering the email address, the above lines of code are used to capture a screenshot, which is then saved as emailValue.png. This image is used for assertions during retries.
The script below employs the Playwright’s built-in retry mechanism, expect().toPass(), to implement a particular scenario. It starts by directing to a tab and inputting an email value. Afterwards, it validates that the input field contains the correct value by capturing a screenshot of the element. If the initial validation fails, Playwright retries the validation multiple times with escalating intervals before declaring the test a failure.
By default, toPass has a timeout of 0 and disregards custom expect timeouts. The validations continue to retry until they pass or reach the validation timeout. Since these validations are asynchronous, they must be awaited.
The expect().toPass() method in Playwright lets you input an options object with various settings for the retry mechanism and timeout. The timeout option designates the maximum time (in milliseconds) that Playwright will wait for the validation to pass before failing the test. The intervals option sets the delay intervals (in milliseconds) that Playwright should follow between retries. It should be an array of ascending delay values.
test.only('Using built-in retry mechanism', async ({ page }) => {
await page.goto('https://www.voma.ai/');
const page1Promise = page.waitForEvent('popup');
await page.getByRole('link', { name: 'Try a demo' }).click();
const page1 = await page1Promise;
const emailLocator = page1.getByTestId('email');
const emailValue = 'ceroshjacob@gmail.com'
await expect(async ()=>{
await emailLocator.fill(emailValue);
await expect(emailLocator).toHaveScreenshot('emailValue.png');
}).toPass({
intervals: [1_000, 2_000, 10_000],
timeout: 60_000
})
});The HTML report for the script execution will document the retry attempt. In the first attempt to fill in the email address, the screenshot comparison failed, causing a one-second wait. However, the second attempt to fill the email address was successful, as indicated by the successful screenshot comparison.
The expect().toPass() an assertion in Playwright offers numerous advantages. These include automatic retries for failed assertions, enhanced code readability by containing asynchronous operations within the expect block, and improved error reporting with detailed failure information. It also integrates seamlessly with Playwright’s APIs, supports async/await syntax, and aligns with the assertion patterns of other popular testing frameworks. These features make the testing process smoother, improve maintainability, and ensure a robust and user-friendly experience when writing Playwright tests.
