Playwright retries without hiding flaky tests

retries in the config

Retries are off by default. Set them for the whole suite in playwright.config.ts:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

Or for one run: npx playwright test --retries=2. The command-line value overrides the config.

A failed test then runs again, up to the given number of times. When a test fails, Playwright discards the worker process it ran in and starts the retry in a new one, with a fresh browser.

Per project and per describe

Retries can differ per project, for example for a browser that is slower on CI:

projects: [
  { name: "chromium", use: { ...devices["Desktop Chrome"] } },
  { name: "webkit", use: { ...devices["Desktop Safari"] }, retries: 3 },
],

The project's value replaces the config's for that project.

And per group of tests, inside a file:

test.describe("checkout", () => {
  test.describe.configure({ retries: 1 });

  test("pays with a card", async ({ page }) => {
    // ...
  });
});

testInfo.retry

Inside a test or a hook, testInfo.retry says which attempt is running: 0 for the first, 1 for the first retry. Use it to clean up state a failed attempt may have left behind:

test.beforeEach(async ({}, testInfo) => {
  if (testInfo.retry > 0) {
    await resetTestData();
  }
});

resetTestData stands for your own clean-up.

Avoid branching on it inside the test itself: a test that behaves differently on a retry tests something else.

trace: "on-first-retry"

A trace is the most useful record of a flaky failure, and recording one for every test on every run takes time and storage. on-first-retry records a trace when a test is retried:

use: {
  trace: "on-first-retry",
},

Inside defineConfig.

The trace then belongs to the retry, not to the failure that caused it. To keep the trace of the failing attempt itself, retain-on-failure records every attempt and keeps the ones that failed.

Squally's reporter uploads traces, screenshots and browser consoles for attempts that failed or needed a retry, when they are switched on in the config.

What retries hide

A retry that passes turns a red run green. That is the point, and also the risk:

• A test that fails on every other run looks the same as one that never fails, as long as a retry passes.
• A real bug that shows up under load or with certain data can pass on the second try and ship.
• Every retry costs a full attempt of CI time, and tests that run into their timeout cost the most.

Playwright's --fail-on-flaky-tests option (failOnFlakyTests in the config) fails the run when any test was flaky. That stops the hiding, at the cost of a red run for every flaky test. The other way is to keep the run green and measure what the retries did.

How to count what retries cost: What flaky tests cost: CI time lost to retries.

Measuring runs that needed a retry

Count two things over time: how many runs needed at least one retry, and which tests needed them. The first tells you whether the suite is getting more or less reliable, the second where to start.

In Squally, the Analytics tile Runs with a retry shows the share of finished runs in which some test passed only after a retry, with the count behind it.

The Overview's Time lost tile adds up the attempts that a retry replaced, as a share of the run time.

Then the tests themselves: How to detect flaky tests in Playwright, and Turn on retries in the docs.

Retries in CI vs. locally

A common setup is retries in CI and none locally: retries: process.env.CI ? 2 : 0. On your machine a failure should stop you, and with a debugger or --ui a retry gets in the way. In CI a retry keeps an unrelated flaky test from blocking a merge, and the record of it is what you measure.

Whatever number you pick, write it down with the reason. 2 is an example, not a rule; a suite that needs 5 has a flakiness problem the retries are covering.

Squally does not retry tests itself and has no quarantine: it records what Playwright ran, attempt by attempt.