Flaky test
A test that passes and fails against the same code, usually because of timing, shared state, test data or an unstable environment. Flakiness costs twice: every false failure takes someone's time to investigate, and the pattern teaches the team to ignore red builds. Detecting it needs execution history across many runs, which is why a single CI report cannot show it.
Also called: Intermittent test, Non-deterministic test
Why it matters
A flaky test does two kinds of damage. The obvious one is time: someone opens a red build, reads the failure, reruns it, and finds nothing wrong. The less obvious one is trust. A team that sees false failures every week learns to rerun first and read later, and a real regression eventually gets the same treatment. At scale the numbers add up: in a 2016 post on the Google Testing Blog, John Micco reported that about 1.5% of all test runs at Google produced a flaky result and that almost 16% of their tests showed some flakiness.
Common causes
Most flakiness comes from the test or its environment, not from the product. The usual suspects are waiting for asynchronous work with fixed timeouts, tests that depend on each other's state or run order, shared test data, and unstable external services or networks. Several flaky tests failing together often point to one shared cause, such as a fixture or a service, rather than to several separate bugs.
How flakiness is detected
The clearest signal is the same code producing different outcomes. A test that fails and then passes on a retry within one run is the most common case; Playwright, for example, reports such a test as flaky when retries are enabled. Beyond one run, detection needs history: a first-attempt pass rate per test across many runs shows which tests are unreliable even when retries keep the build green.
What to do with one
Fix it if the cause is clear. If it is not, move it into quarantine with an owner and a deadline, so it keeps running without blocking merges. A full policy is in a flaky test quarantine policy that actually holds.