How to detect flaky tests in Playwright

Turn on retries

A flaky test fails and then passes on the same code. Playwright can see that only if it runs the failed test again. Retries are off by default; turn them on in playwright.config.ts:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

Or for one run from the command line: npx playwright test --retries=2. The details, and how to keep retries from hiding problems: Playwright retries without hiding flaky tests.

What Playwright's report marks as flaky

After a run with retries, each test ends in one of four outcomes:

• passed: passed on the first attempt;
• flaky: failed at least once, then passed on a retry;
• failed: failed on every attempt;
• skipped: not run.

The list reporter prints a count such as 1 flaky at the end, and the HTML report has a Flaky filter. Each retry is its own attempt with its own error, so you can see what went wrong the first time.

The final attempt decides

Within one run, the last attempt decides the outcome. A test that fails twice and passes on the third attempt is flaky, not failed. A test that fails on all its attempts is failed, however many retries it had.

Squally counts the same way: a test that passed on its first try is stable in that run, one that passed after at least one retry is flaky, and one whose final attempt failed is failed, retried or not.

Flaky, failed, failing in a row

Over many runs, three patterns are worth telling apart:

• Flaky: needs a retry now and then, and passes. The share of runs tells you how often.
• Failing now and then: fails on all attempts in some runs. Often the same causes as flakiness, with a bigger effect: the run is red.
• Failing in a row: failed in each of its latest runs. That is usually a real break, not flakiness.

Squally shows each per test: Flaky % and Failed % in the Test explorer, and a Failing in a row card on the Overview for tests that failed in at least their last 3 runs.

Which of them costs the most CI time: What flaky tests cost: CI time lost to retries. How to fix them: Fixing flaky Playwright tests: causes and patterns.

CI vs. local runs

Local runs are noisy: half-written tests, a debugger left open, a dev server that was not running. Counted together with CI runs, they blur the rate. Keep them apart, or look at CI runs alone.

Squally tells them apart through its reporter: it recognizes the CI system from its environment, and a run that is not on a recognized CI is local. The filters show CI runs, local runs or both; both by default.

Sharded runs count once

With --shard=1/4, one CI run is split over four machines. When you count runs per test, those four parts are one run of the suite, not four.

Squally merges the shards of one CI run and counts them once.

Outage runs left out

Sometimes most of the suite fails at once: the staging server is down, a deploy broke the login, the CI runner lost its network. Those failures say nothing about individual tests, and counted as test failures they push every rate up.

Squally leaves such runs out of the per-test rates: a run with at least 20 tests in which more than 80% failed counts as an outage.

Whatever you count with, decide on a rule like this before you look at the numbers.

Finding them in Squally

1. Add squally-reporter to the reporters in playwright.config.ts and set the ingest key in CI.
2. Keep retries on in CI, as above.
3. On the Overview, the Flaky tests tile counts the tests with at least one flaky run in the period, and the Most flaky card lists the tests with the highest share of flaky runs, once a test has at least 3 runs.
4. In the Test explorer, sort by Flaky % and open a test to see its runs and each attempt.

How Squally counts, in full: How Squally measures tests in the docs. How Squally's detection compares with other tools: Compare.