Flaky Playwright tests: find them, measure them, fix them

What a flaky test is

A flaky test gives different results for the same code. It fails, it runs again, it passes. Nothing in the test or the app changed between the two attempts; something around them did: timing, data left over from another test, a slow network call, workers running in a different order.

A flaky test is not the same as a broken one. A broken test fails every time until someone changes the code or the test. A flaky test fails some of the time, which makes it harder to notice and easier to ignore - until the team stops trusting red runs altogether.

Why retries are the signal

Playwright can run a failed test again within the same run. With retries set, a test that fails and then passes is reported as flaky; a test that fails on every attempt is reported as failed.

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

Without retries there is no second attempt, so a flaky failure looks the same as a real one. Retries are what make flakiness visible in the first place.

Squally reads the same signal: a test that passed after a retry is flaky in that run, and the final attempt decides the result.

How to set them without hiding problems: Playwright retries without hiding flaky tests.

One run vs. a rate over runs

One run tells you that a test was flaky once. Whether that is a problem depends on how often it happens. A test that needed a retry in 1 of 200 runs and a test that needs one in every fourth run call for different action.

The number to watch is a rate over many runs: of the runs in which the test ran, how many were flaky and how many failed.

Squally keeps these rates per test across all branches. A test's page shows its Stability, Flaky rate and Failure rate, and the Test explorer has a Failed % and a Flaky % column to sort by.

How to count them, and which runs to leave out: How to detect flaky tests in Playwright.

What flakiness costs

Every retry runs the test again. The pipeline takes longer, the people waiting on it wait longer, and a run that fails on a flaky test is often re-run in full.

Count the cost in time first: the minutes of CI spent on attempts that did not count. Whether that turns into money depends on your CI bill and on how you value waiting time - figures you have and a tool does not.

Squally shows this cost as time, not money. The Overview's Time lost tile adds up the attempts that a retry replaced and shows them as a share of the run time, compared with the period before.

Per test, the Test explorer's Time lost column counts every attempt of a failed run and the attempts before the pass of a flaky one, so a test that keeps failing shows up there too. The tile and the column count differently and do not add up to each other.

The Overview's Where the time goes card lists the tests whose retries and failures cost the most.

Both counts, a worked example in minutes and where to start: What flaky tests cost: CI time lost to retries.

The usual causes

• Fixed waits. page.waitForTimeout(2000) is long enough on a laptop and too short on a busy CI machine. Web-first assertions such as await expect(locator).toBeVisible() wait until the condition holds.
• A missing await. The test moves on before an action or an assertion has finished.
• Selectors that match more than one element, or that depend on layout and order. Role and text locators (getByRole, getByText) change less often than long CSS paths.
• Shared state. Tests that depend on data another test created, or on the order in which they run.
• Parallel workers. Two workers changing the same account, file or record at the same time.
• Third-party services. A slow or rate-limited API outside your control. Mocking it with page.route takes it out of the test.

Each cause with code that fixes it: Fixing flaky Playwright tests: causes and patterns.

From flaky to fixed

Start with the failed attempt, not the one that passed: its error, screenshot and trace show what was different. Playwright's trace: "on-first-retry" records a trace when a test is retried.

In Squally each attempt has its own tab - Attempt 1, Retry 1 and so on - with its error and stack trace, and a trace opens in the Playwright Trace Viewer in one click. The reporter uploads screenshots, traces and browser consoles for attempts that failed or needed a retry, when they are switched on in the Playwright config.

After a fix, keep watching the rate. One green run proves little for a test that failed one run in ten; the flaky rate over the next runs tells you whether the fix held.

Squally does not retry or quarantine tests. Retries come from your Playwright config, and a flaky test stays in the suite until it is fixed or skipped in the code.

The patterns, cause by cause: Fixing flaky Playwright tests: causes and patterns.

Step by step in the app: Flaky tests and Debugging basics in the docs.

Where your results are stored

Test results can carry personal data: names and email addresses of test accounts, screenshots of pages with customer data, URLs with tokens in them. Where a reporting tool stores them matters under the GDPR.

Squally stores test and account data in the EU: the app and its database run in Frankfurt, Germany, and screenshots, traces and browser consoles are stored in the EU. Runs are kept for at most 90 days, depending on the plan; screenshots, traces and consoles for 7 days after a first-try pass and 21 days for failures and retried tests.

What a reporter sends, masking, retention and how to send less: EU-hosted Playwright reporting and GDPR. Details: the sub-processors in the privacy policy and Hosting in the docs.

Tools that track it

Playwright's own HTML report marks the flaky tests of a single run. Tracking them across runs needs the results of every run in one place: a reporting service, or your own database fed by a custom reporter.

The comparison pages put Squally next to Allure Report, Currents and TestDino in the same thirteen areas, each based on the tool's public documentation.

Squally is one of these services. Add its reporter to playwright.config.ts, and each CI run arrives with its flaky tests, failed tests and time lost.

reporter: [
  ["html"],
  ["squally-reporter"],
],

The reporter line from the squally-reporter README; the ingest key goes into the CI environment.