Squally and Allure Report.
Side by side, area by area.

Squally

You add the npm package squally-reporter as one line in playwright.config.ts and store the project's ingest key as a CI secret. Results arrive while the suite runs, with no upload step afterwards; it needs @playwright/test 1.30 or later and Node 20.18.1 or later.

Runs are recognised on GitHub Actions, GitLab CI, Azure Pipelines, Jenkins, Buildkite, Travis CI and Drone, and the shards of one CI run become one run.

Allure Report

The allure-playwright reporter writes result files to allure-results while Playwright runs. A separate CLI step builds the HTML report: allure generate or allure serve, and in Allure 3 also allure run -- <test command>.

The docs cover Jenkins, TeamCity, Azure DevOps, Bamboo and a GitHub Action.

Squally

Squally is a hosted service. It is hosted in the EU (Frankfurt).

The privacy policy lists the providers involved and what each one processes.

Allure Report

Allure Report produces static files (in Allure 3 optionally a single HTML file) that you store and serve, for example on GitHub Pages. The data stays wherever you keep the results, reports and history.

Playwright traces open in Playwright's hosted Trace Viewer page; the docs state that the trace bytes are not uploaded.

Squally

Squally is offered as a hosted service. A self-hosted or on-premises edition is not offered.

Allure Report

Allure Report is self-hosted by design: the report goes on any web or static host.

Allure 3 adds the optional Allure Report Storage service, deployable with Docker or on Cloudflare Workers, for published reports and their history.

Squally

A test's result in a run counts as flaky when it failed first and then passed on a Playwright retry, so retries need to be turned on.

For each test and period, Squally shows runs, stability, flaky rate (flaky runs ÷ runs), failure rate and the CI time that retries and failures cost. Squally does not quarantine or skip tests.

Allure Report

Allure marks a test flaky from its history. In Allure 3 the default rule counts status changes over previous results for tests that failed or broke in the latest launch; Allure 2 looks at the latest five launches.

“Retried” and, since Allure 3.21, “Changed on retry” mark retries within one launch.

Squally

Runs and results are kept 30 days on the Free plan, 60 days on Standard and 90 days on Pro.

Screenshots, traces and browser consoles are kept 7 days for attempts that passed on the first try and 21 days for failures and retried tests, on every plan. Deletion runs once a day.

Allure Report

History is kept in files carried from build to build: Allure 2 copies a history folder and keeps up to 20 reports, Allure 3 appends to a JSONL history file.

With the Allure Report Storage service, history comes from the service, and retention per branch can be configured.

Squally

A GitHub App posts a check on every commit, one per CI job, and, if switched on for the project, one comment per pull request that is updated with each run. The Pull requests page compares each pull request's runs with its target branch.

Runs from GitLab, Azure Repos and Bitbucket are collected as well; checks and comments are posted to GitHub.

Allure Report

The allure-action GitHub Action reads an Allure 3 report and posts a pull request comment with totals and new, flaky and retried tests, and updates it on later runs.

A GitHub App is not documented; publishing the report to GitHub Pages uses community actions.

Squally

Checks: read and write. Pull requests: read and write. Metadata: read-only (mandatory). Organization members: read-only, used while connecting to confirm that the person installing the app is an owner of the GitHub organization.

No access to repository contents (code).

Allure Report

The action runs with the workflow token and the permissions pull-requests: write and checks: write.

The GitHub Pages guide uses contents: write.

Squally

Failures are grouped by a fingerprint of the normalised error message and the test file, so one error can be followed across runs and tests.

The Error explorer shows each error's runs, tests, failed and flaky rate, first and last seen, and a category derived from the error text.

Allure Report

Failures are grouped into categories defined by rules that match status, message and trace patterns; Allure 3 can also group tests by identical error message.

Allure 3 adds “resolutions” that mark known failures as an issue, muted or accepted.

Squally

The reporter uploads screenshots, Playwright traces and the browser console; video is not uploaded.

A trace opens in Playwright's Trace Viewer (trace.playwright.dev) through a link that expires after ten minutes. Attempts that did not pass on the first try also carry their recorded steps.

Allure Report

The report shows attachments, including images, image diffs, video files and Playwright traces.

Traces open in Playwright's Trace Viewer: inline in Allure 2, in a new tab in Allure 3. Each Playwright retry appears as a retry of the same test.

Squally

On the Standard and Pro plans: a read-only REST API with organization read keys, documented in OpenAPI, and the MCP server squally-mcp, which runs locally and offers 7 read-only tools.

On attempt and run pages, “Fix in your agent” builds a prompt for your own coding agent; Squally sends nothing to an AI provider. Outgoing webhooks and a write API are not offered.

Allure Report

An HTTP API or MCP server is not documented for Allure Report.

Allure 3's Agent Mode (since 3.12) runs or reads tests and writes Markdown output for coding agents.

Squally

Squally has no test case management, manual runs or test plans.

Allure Report

Allure Report links tests to external systems by test ID and supports test plans that select which automated tests run; plugins send results to Xray and Jira.

A test case repository and manual runs are not documented for Allure Report.

Squally

Squally reads Playwright results through its own reporter. Other test frameworks are not supported.

Allure Report

Adapters cover a wide range of frameworks in JavaScript/TypeScript, JVM languages, Python, .NET, PHP, Ruby, Go and others, including Playwright for JavaScript and Java.

Squally

A Slack message per CI run through an incoming webhook, and a weekly email report per project on the Standard and Pro plans.

A run report can be printed, saved as PDF or copied as Markdown. GitHub checks and pull request comments are described above.

Allure Report

Allure 3 includes a Slack plugin that posts a run summary through a webhook or a bot token, and a mail plugin that generates an HTML summary for you to send.

Exports include CSV and metrics for Prometheus or InfluxDB, and quality gates can fail the build.