A Flaky Test Quarantine Policy That Actually Holds

A flaky test quarantine policy holds when every quarantined test keeps running, stops blocking merges, has a named owner and a deadline, counts against a hard cap on how much of the suite can sit in quarantine at once, and leaves only one of three ways: fixed, rewritten, or deleted with its coverage accounted for. Drop any one of those and quarantine turns into a slower way of deleting tests. The detection side (retries, pass/fail history, a flip-rate threshold) is the easy half and most CI tools already do it. The half that fails in practice is the exit.

Why most quarantines become a graveyard

Quarantine is usually introduced on a bad week. Main has been red for two days, three tests are failing on and off, somebody adds a skip tag so the team can ship, and a ticket is opened that nobody is assigned to. Six months later the tag is on forty tests. Nobody remembers which ones were real product bugs at the time, and the checkout flow has had no automated coverage since spring.

None of that happens because people are careless. It happens because quarantine removes the one thing that forced attention: the red build. A test that no longer fails the pipeline competes for engineering time with feature work, and it loses unless the policy gives it a date, a person, and a consequence. That is why a policy document that only says "quarantine flaky tests and fix them later" does not work. "Later" is the part that needs rules.

The arithmetic that makes a policy necessary

Flakiness compounds across a suite. If a single test fails spuriously 2% of the time, it is a minor nuisance. If twenty tests in the same pipeline each have that 2% rate and fail independently, the chance that a run comes back fully green without any real bug is 0.98²⁰, about 67%. One run in three goes red for nothing. At fifty such tests it is about 36%, and the build is red more often than it is green.

The large published datasets say this is what a large suite normally looks like. In a 2016 post on the Google Testing Blog, John Micco reported that about 1.5% of all test runs at Google produced a flaky result, that almost 16% of their tests had some level of flakiness, and that 84% of pass-to-fail transitions they observed in post-submit testing involved a flaky test. Atlassian reported in December 2025 that flaky tests caused about 21% of master build failures in the Jira frontend and about 15% of repository failures in the Jira backend. Before Slack automated detection and suppression for its mobile suites, test job failures made up 56.76% of build failures and main branch stability was around 20%; after the change those figures were 3.85% and 96% (Arpita Patel, Slack Engineering, April 2022). GitHub reported going from flaky failures on 1 in 11 commits to 1 in 200 after building its own detection and assignment system (Jordan Raine, GitHub Blog, December 2020).

The cost side is better measured than most people expect. A 2024 industrial case study by Leinen, Elsner, Pretschner and colleagues at TU Munich and CQSE looked at a commercial project with roughly 30 developers and found that dealing with flaky tests took at least 2.5% of productive developer time, split into about 1.1% investigating failed pipelines, 1.3% repairing tests and 0.1% managing them. Rerunning tests cost almost nothing by comparison. That split matters for policy design: the expensive part is a person looking at a red build and trying to decide whether it is real, so a good policy makes that decision cheaper and rarer. A green build is a side effect.

The human side is the one that eventually costs you a production incident. A survey of 170 developers by Parry, Hilton, Kapfhammer and McMinn (ICSE SEIP 2022) found that developers who run into flaky tests more often may be more likely to ignore potentially genuine failures. A suite that is red a third of the time trains exactly that habit.

Entry: what earns a test a place in quarantine

The entry rule has two jobs. It has to catch flaky tests early enough that they do not poison the build, and it has to keep real regressions out, because a real regression that lands in quarantine is a bug you have decided not to see.

The cleanest evidence of flakiness is the same commit producing different outcomes. A test that fails and then passes on retry within one run is the most common version of that, and it is what Playwright reports as flaky when retries are enabled: its documentation classifies a test as passed on the first attempt, flaky if it failed and then passed on a retry, and failed if every attempt failed. Retries are off by default in Playwright, so a team that has not configured them is not collecting this signal at all.

A failure that reproduces on every attempt is a failure. It may still be caused by the test rather than the product (a changed selector, an expired fixture), but it is deterministic, and it goes to whoever owns the test today. Quarantine is the wrong place for it. Slack's system made the same distinction in a different place: before suppressing anything, it filtered out backend and API failures, crashes and infrastructure problems, so that only test-level flakiness was eligible.

For the threshold itself, published guidance clusters in a narrow band. Mergify suggests quarantining tests that fail more than about 2% of the time and blocked pull requests in the past week. minware uses a fail rate above 5% across runs as its example threshold. FlakyGuard uses a flip-rate score of 0.2 to 0.3 with at least five runs before any evaluation. Any of those is fine as a starting point. The part that matters more is the minimum sample: a test that failed once out of three runs has not proven anything, and a rule that fires on tiny samples will quarantine real bugs.

Decide who is allowed to quarantine. If anyone who is blocked can add the tag, the tag will be used to unblock, and the count will grow with delivery pressure. A small group (test owners, the automation lead, a platform team) with the rule written down is enough. Automated quarantine can work too, as Atlassian and Slack both show, but only because their systems also open the ticket, assign it, and set the due date in the same step. Automating the tag without automating the follow-up gives you a graveyard faster.

While it is in quarantine: keep it running, stop it gating

This is the rule most teams get wrong, and it is the one that separates quarantine from a skip. A skipped test produces no data. A quarantined test keeps executing and keeps recording results, it just no longer decides whether the build passes. Without those results you cannot tell whether the fix worked, whether the flakiness got worse, or whether the test has started failing for a real reason, which is the one outcome you most need to see.

Tooling does not always make this the default. Bitbucket Pipelines, for example, stores quarantine as metadata, and its documentation describes identifying and skipping quarantined tests at runtime so they do not block CI; teams wire the skip into their build scripts themselves. That is a reasonable design for unblocking a pipeline, but if you adopt it, you need a second job that runs the quarantined set, or you have skipped the tests under a different name.

The usual implementation is two CI signals from one suite. The gating job runs everything except the quarantined tests. A second, non-blocking job runs only the quarantined tests, without retries, so their real first-attempt behaviour is recorded. In Playwright, tags make this a pair of commands:

// checkout.spec.ts
test('TC-412 applies a percentage coupon at checkout', { tag: '@quarantine' }, async ({ page }) => {
  // ...
});

# gating job: everything that is not quarantined, with retries as configured
npx playwright test --grep-invert @quarantine

# quarantine job: only the quarantined tests, no retries, never blocks the merge
npx playwright test --grep @quarantine --retries=0

In GitHub Actions the second job gets continue-on-error: true. In pytest the same split is a registered marker and -m "not quarantine" for the gating job, -m quarantine for the other. The mechanism is not the interesting part. What matters is that the quarantine job's results go somewhere that keeps history per test, because a single CI log cannot tell you that a test has been clean for fifteen consecutive runs.

Retries are a measuring instrument

Automatic retries are the most common response to flakiness and they are fine, as long as the team reads a pass on retry as information. A test that needs two attempts to go green has told you something. If the reporting layer counts it as passed and moves on, the flakiness disappears from every dashboard while it keeps costing CI minutes and, eventually, trust.

The practical consequence is to track two numbers per test. Pass rate counting retries tells you whether the build would have gone green. First-attempt pass rate tells you whether the test is reliable. A test at 100% on the first number and 85% on the second is a quarantine candidate that nobody is looking at. Google's option to report a failure only after three consecutive failures is the extreme version of this trade-off, and Micco's post notes the obvious cost: it can mask real problems.

The quarantine record

A tag in the source code says a test is quarantined. It does not say why, since when, who is responsible, or what you lose while it is out. Those answers are what make an exit possible, so they need to live in a record that outlasts the pull request that added the tag.

The coverage row is where CI-only tooling runs out. A CI system knows that checkout.spec.ts › applies a percentage coupon is quarantined. It does not know that this test was the only automated check behind a regression case for discount handling, or that the discount requirement for the next release now has no automated evidence. That link sits in the test management layer, where automated results are mapped to test cases (the mechanics and the ways it breaks are covered in mapping automated test results to test cases). If your automated results do not map to cases at all, quarantining a test silently reduces coverage and no report will show it.

Capacity: a hard cap on the quarantine

Deadlines alone do not stop a quarantine from growing, because deadlines slip one at a time and each slip looks reasonable. A cap on the whole quarantine forces the trade-off to happen in the open. When the quarantine is full, adding a test means one of three things: someone fixes or deletes an existing one, the team explicitly raises the cap and says why, or the new flaky test stays in the gating job and keeps failing builds until it is fixed.

The third option sounds harsh, and it is the point. It makes flakiness a delivery problem again at the moment the backlog of flaky tests is large enough to be one. FlakyGuard treats a quarantine rate under 5% of the suite as healthy and above 10% as a sign of a systemic issue. For most suites a fixed number is easier to enforce than a percentage: it does not grow when the suite grows, and "we are at 12 of 12" is a sentence people react to.

Exit: three ways out and one way back in

A quarantined test leaves by one of three routes, and the policy should name all three so that "still quarantined" is never the default outcome of a review.

Fixed. The root cause is addressed and the test goes back into the gating job after a run of clean first-attempt results in the quarantine job. How many runs depends on how often it ran flaky before: a test that failed one run in five needs fewer clean runs to be convincing than one that failed one in fifty. Ten to twenty consecutive clean runs is the range FlakyGuard uses, and Atlassian's system returns a test once it has stayed healthy for a configured period. A test that goes back and flakes again returns to quarantine with its original date, so the clock does not reset every time the fix fails.

Rewritten. Some tests are flaky because of what they try to do: an end-to-end test that checks a price calculation through five pages of UI, or a test that depends on a third-party sandbox that is down every other night. The honest fix is to move the check to a lower layer (an API or component test) or to stub the dependency, and to delete the original. The coverage record should point to the replacement.

Deleted. If the deadline passes and neither of the above has happened, the test is removed, and the removal is recorded against the coverage it provided. Deleting a test that has been quarantined for three months costs nothing that was not already lost: it has not been protecting the build since the day it was tagged. What deletion adds is honesty. The coverage report now shows a gap instead of implying a check that is not actually running. Mergify's guidance makes the same call: past the maximum stay, fix it or retire it with a written reason.

Numbers to start with

Every value in the policy will need tuning, but a policy with numbers in it can be tuned and a policy without them cannot. These are defensible starting values for a team running its suite on every pull request, next to what the published guidance uses.

Two adjustments are worth making early. If the suite runs only nightly, count runs rather than days, because thirty days is only thirty data points and a 2% threshold cannot be measured on that. And if the product is regulated or the suite guards payments or data integrity, shorten the deadline for tests whose coverage row points at a high-risk requirement. A flaky test on a settings page and a flaky test on a refund flow should not get the same thirty days.

When quarantining tests one at a time is the wrong move

The per-test model assumes each flaky test has its own cause. Often it does not. Parry, Kapfhammer, Hilton and McMinn analysed 10,000 test suite runs across 24 Java projects (EASE 2025) and found that 75% of the flaky tests they identified belonged to clusters that failed together, with an average of 13.5 tests per cluster. The shared causes were mostly networking problems and unstable external dependencies, followed by filesystem pollution from incomplete cleanup and hard-coded timeouts. The older and much-cited study by Luo, Hariri, Eloussi and Marinov (FSE 2014), which classified 161 commits that fixed flaky tests in Apache projects, put asynchronous waiting at 45% of causes, concurrency at 20% and test order dependency at 12%. In the 2022 developer survey, respondents themselves ranked setup and teardown problems as the most frequent cause.

The practical signal is timing. When several tests enter quarantine in the same week, look for what they share before assigning them to different owners: the same fixture, the same test data service, the same environment, the same wait helper. Quarantining fourteen tests individually for one broken fixture produces fourteen tickets, fourteen owners, and one root cause that none of them owns. A cap on quarantine size helps here as well, because a cluster arriving at once will hit the cap and force the conversation.

The same logic applies to infrastructure. If the quarantine job itself fails often because a runner is starved of memory or a browser container is misconfigured, every test looks flaky and the data is worthless. Before tuning thresholds, check that the environment the quarantine job runs in is at least as stable as the gating one.

What to look at once a month

A handful of numbers show whether the policy is working, and none of them needs a dashboard project to collect.

The size of the quarantine against its cap, and the trend. A quarantine that is always full means the cap is doing its job but the fixing is not keeping up; a quarantine that is always empty with a red main branch means the entry rule is not firing. The age of the oldest entry, which tells you whether the deadline is real. The number of tests that left by each route, because a policy where every exit is a deletion is a policy that has given up on fixing. The number of tests that came back after a fix, which measures how good the fixes are. And the first-attempt pass rate of the whole gating suite, which is the number the policy exists to protect. If you already report QA metrics upward, this last one belongs next to them; QA metrics that matter for engineering leaders covers where it fits.

Where a test management system fits in the policy

CI decides what gates the build. That part of the policy lives in the pipeline config and should stay there. What CI does not keep well is per-test history across hundreds of runs, and it knows nothing about which manual cases or requirements a given automated test was covering. Those are the two things the exit rules depend on.

In QAM Hub, automated results from Playwright, Cypress and JUnit XML reports land in the same project as the manual test cases (setup for Playwright is in uploading Playwright reports). The Tests tab tracks each automated test across runs by its browser or configuration plus its full path, and reports pass rate, flakiness rate, a stability score and average duration over a window you choose, from the last 5 to 100 runs or a date range. The stability score counts only first-attempt passes, while pass rate includes tests that passed on retry, so the two numbers side by side are the retry gap described above. A Project Admin can quarantine a test with a note; the test stays visible with its history but is excluded from health metrics, and the quarantine records when it was set and by whom. Because automated tests link to manual cases through a TC- prefix in the test title, the case a quarantined test was covering is one click away, which is the coverage row in the record.

Two things it does not do, so the policy has to cover them. Quarantine in QAM Hub does not change what your CI gates on: the tag and the --grep-invert in the pipeline are still yours to maintain. And it does not set a deadline for you; the review date belongs in the note or in the linked ticket, and the monthly review is still a meeting someone has to hold.

What to prioritise

If the team has no policy today, start with two rules and add the rest later: quarantined tests keep running in a non-blocking job, and every quarantined test has an owner and a date. Those two alone stop the graveyard from forming, because the data keeps coming and someone is responsible for reading it. The cap is the third rule to add, usually the first time a review finds the quarantine has doubled.

Do not spend the first month tuning detection. Whether the threshold is 2% or 5%, or whether you use a flip rate or a failure rate, matters much less than whether anything happens after the tag goes on. Do not build a custom quarantine service before you have run the policy by hand for a few weeks with a tag, a spreadsheet or a TMS field, and a calendar reminder. And do not let the policy become a reason to stop retrying in CI. Retries are cheap, as the TU Munich study found. The mistake is treating a pass on retry as the end of the story.

For the broader picture of how automated results should flow from CI into test management, including where retries, traces and CI metadata end up, see managing automated test results in a TMS, and the runner-specific guides for Playwright and Cypress.