QA Metrics That Actually Matter for Engineering Leaders

Four numbers carry the weight of an engineering decision: escaped defects weighted by severity, the stability of your automated suite, how long it takes to get a trustworthy test result after a change, and the share of releases that go wrong in production. Everything else on a typical QA dashboard is either an input you already control, a number nobody acts on, or a target that quietly changes behaviour for the worse. This is a guide to the second group, because removing metrics is usually the higher-value move for a leader inheriting a reporting habit rather than building one.

The test a metric has to pass

A QA metric earns its place if a plausible movement in it changes what you do next week. Not what you say at the review, what you actually do: where the next hire goes, which suite gets rewritten, whether the release ships on Thursday, whether the team spends a sprint on environments instead of features.

Run that test on a real dashboard and most of it fails immediately. Test cases written per sprint moves, and nothing changes. Execution count moves, and nothing changes. Pass rate moves from 94% to 91%, and the honest answer to "what do we do" is "look at which tests failed", which means the number was a doorway to the data, not a decision input. That is fine for a QA lead triaging a run. It is not a leadership metric, and putting it on an executive dashboard invites a conversation about the wrong thing.

The second filter is manipulability. Any metric that a team is measured on will be optimised, and the cheapest way to optimise most QA metrics is to weaken the tests. Pass rate goes up when you delete the flaky ones and stop writing hard cases. Coverage goes up when you write assertions that cannot fail. Defect counts go down when triage gets stricter about what counts as a bug. None of that requires bad faith, only pressure and a deadline.

The four that hold

Escaped defects, weighted by severity

The share of defects that reached production instead of being caught before release is the closest thing QA has to an outcome measure. It answers the only question a CTO genuinely has about testing: is it working.

Three details decide whether the number is usable. Weight it by severity, because ten cosmetic issues and one payment failure are not the same signal and averaging them destroys the information. Fix the denominator and keep it fixed: defects found in production divided by all defects found in the same period is the usual formula, and it moves for two different reasons, so a team that gets better at internal testing will see the ratio improve without production changing at all. And attribute by discovery source, separating what your monitoring caught from what a customer reported, because those two numbers are about different failures. Monitoring catching a defect first is a working system. A customer catching it first is a gap in what you test, and the specific customer-found defects are the most valuable input to your test design that exists.

What it changes: a rising severe-escape rate in one product area is the strongest argument available for putting testing effort or a dedicated engineer there, and it is the argument that survives contact with a budget conversation, because it names a risk in production rather than an activity in QA.

What it will not do is give you a number to compare against the industry. Escape rates depend on how the team classifies defects, how much of the product is monitored, and how quickly users report issues. Treat it as a trend line against your own history and stop looking for a benchmark.

Suite stability

Flakiness is the metric that most teams do not track and most teams are damaged by. A flaky test produces different results on the same code, and once the rate is high enough that engineers expect red builds to be noise, the suite has stopped being a control and become a tax.

Google published the reference numbers on this from its own corpus: about 1.5% of all test runs report a flaky result, and roughly 16% of tests show some level of flakiness. In its post-submit system, about 84% of transitions from passing to failing turned out to be flaky rather than a real regression. Those figures come from an environment with dedicated infrastructure teams and heavy investment in flakiness removal, which is the part worth sitting with: this is what the problem looks like when it is being actively fought.

Small per-test rates compound with suite size, which is why the number surprises people. A suite of 500 tests each failing spuriously 0.5% of the time produces a clean run about 8% of the time. The arithmetic is unforgiving, and it is the reason a team can add tests for a year and end up with less confidence than it started with.

Track two things: the percentage of test runs that produce an inconsistent result, and the list of individual tests responsible, which is almost always short and heavily concentrated in end-to-end scenarios. What it changes is straightforward. Above a threshold you set, tests get quarantined out of the blocking path with an owner and a date rather than being left to erode trust in every other result. Quarantine is a policy decision, not a technical one, which makes it a leadership metric.

Time to trustworthy feedback

How long from a change being merged to a reliable answer about whether it broke something. This is the metric that governs everything else, because a slow signal makes every other quality practice more expensive.

Measure it end to end and include the queue: pipeline duration, environment provisioning, waiting for a manual regression pass, and the retries a flaky suite forces. Teams routinely report the 40-minute pipeline and omit the two days the change sat waiting for a shared staging environment. The gap between those two numbers is where the real problem usually lives, and it is normally infrastructure or scheduling rather than testing.

Blocked results are the companion signal here. A run with a high proportion of blocked cases is telling you about environments, test data or dependencies, not about product quality, and treating blocked as a variant of failed hides the most actionable maintenance information in the run. Keep the statuses distinct and look at the blocked share as its own trend.

What it changes: this is the number that justifies spending on environments, test data management and parallelisation, none of which have an obvious return until you can show that feedback latency is the constraint on release frequency.

Change failure rate and rework

The share of releases that cause a production failure requiring a hotfix, rollback or patch. It belongs on this list because it is the one quality metric that engineering leadership outside QA already understands, and because it forces the conversation to be about the release rather than about the test suite.

It comes from the DORA research program, alongside deployment frequency, lead time for changes and failed deployment recovery time. DORA's 2024 report added a fifth metric, rework rate, covering unplanned deployments caused by production issues, which tends to be a more sensitive instability signal than outright failed deployments because there is usually more of it.

Two cautions. Change failure rate is sensitive to release size, so a team moving to smaller, more frequent releases can show a worse rate while shipping better software, and the fix is to read it next to deployment frequency rather than alone. And it is the metric where "we do not have the data" is most often true: if failures are not consistently recorded against releases, you are estimating, and an estimated rate that people treat as measured is worse than no rate.

Metrics that break under the weight of a decision

Pass rate as a target

Pass rate is a useful reading of a single run and a destructive target across quarters. The moment a team is accountable for it, the cheapest paths to improvement are all bad: retire tests that keep finding problems, avoid writing cases for the risky paths, retry until green. A suite that finds nothing has a perfect pass rate.

Keep it as a run-level triage signal. Do not put it in a goal, a bonus, or a slide that a VP compares between teams.

Coverage as a KPI

Coverage is two different metrics with one name, and both get misused. Code coverage measures which lines executed while tests ran, which is not the same as which behaviours were verified, and a test that exercises a function without asserting anything about it raises coverage and catches nothing. Requirements coverage measures which requirements have at least one linked test case, which says nothing about whether that case is any good.

You will find published target ranges for both, typically 70% to 80% for code coverage and 80% or more for requirements. Those are conventions rather than empirical thresholds, and the last stretch is where the cost curve turns vertical while the value flattens.

Coverage is worth tracking as a gap map, not as a percentage. The useful artefact is the list of critical requirements or high-risk areas with no linked test at all, which is a work item, while the percentage is a number to defend. That distinction is the whole practical argument for maintaining a requirements traceability matrix, and it is why a matrix that is only assembled for an audit does not help anyone.

Automation percentage

The share of test cases automated is an input, not an outcome, and it is a poor input because the denominator is under the team's control. Retire 200 stale manual cases and the automation percentage jumps without a single new test. Where the question is really "are we automating the right things", the honest instrument is a look at what is automated in the highest-risk areas, which is a review rather than a number. We wrote about how to choose that order in what to automate first.

Volume metrics

Test cases written, tests executed, defects logged. These describe effort. They are legitimate for capacity planning, and they are actively misleading as quality indicators, because in each case the direction that looks good on a report is also the direction that produces a bloated test base nobody trusts. A team rewarded for cases written will write cases; a large fraction of most mature test bases is duplicated or dead, which is its own maintenance problem.

The cost argument to stop making

Somewhere in most QA budget requests is the claim that a defect costs ten times more to fix in testing and a hundred times more in production, usually attached to a tidy chart. That chart is not supported by the research it is credited to. Laurent Bossavit traced the widely reproduced version to course notes from an internal IBM training program with no published data behind them, and the claim has since propagated through secondary citation rather than replication.

Something real does sit underneath it. Boehm and Basili's Software Defect Reduction Top 10 List in IEEE Computer, January 2001, reports that fixing a problem after delivery is often about 100 times more expensive than fixing it during requirements and design, and the same paper notes that for smaller, less critical systems the ratio is closer to 5 to 1. So the effect exists, the magnitude depends heavily on system size and criticality, and the version most often quoted at executives is the one that flatters the argument most.

This matters for a practical reason rather than a pedantic one. A CFO who checks one number in your business case and finds folklore will discount the rest of it. The stronger argument uses your own escaped-defect history and your own incident record, which is specific, checkable and harder to wave away.

What AI-assisted development changes

The measurement question got sharper over the last two years, and not in the direction most tooling vendors suggest.

DORA's 2025 State of AI-assisted Software Development report describes AI as an amplifier: it magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones. Its finding on delivery is the one worth putting in front of an executive team, because higher AI adoption was associated with an increase in throughput and an increase in delivery instability at the same time. DORA's earlier work on generative AI put a figure on the direction of that trade, associating a 25% increase in AI adoption with a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability, before the throughput half of the picture turned positive in the 2025 data. The stability half did not turn.

The reading for a QA leader: more change is arriving per unit of time, and the review, testing and release machinery downstream is what decides whether that becomes speed or incidents. Which is an argument for instrumenting stability before expanding AI usage, not after. If change failure rate and escaped defects have no baseline today, they will have no baseline at the moment you most need to answer whether the AI rollout is working.

The same applies inside QA. AI-generated test cases raise the volume of the test base quickly, and volume is the input that most reliably degrades a suite. Any adoption of generated cases needs the review step and the duplicate check attached to it, and suite stability is the metric that tells you whether the generated material is helping.

How many, and how often

Four or five metrics, reviewed on a cadence that matches how fast each one can genuinely move.

Two rules about the shape of the report. Trends, not snapshots: an escape rate of 8% means nothing in isolation and means a lot if it has fallen from 15% over three months. And no cross-team league tables, because teams work on products with different risk profiles and the comparison is what turns a diagnostic into a target.

If you inherit an existing dashboard, the first useful action is subtraction. Take everything that fails the "changes what we do" test off the report, and watch how few people ask where it went.

Where the numbers actually come from

Most QA metrics programmes fail on plumbing rather than on choice of metric. Escaped defects require that production issues are linked back to the release and, ideally, to the area of the product they came from. Coverage gaps require that requirements and test cases are linked at all. Suite stability requires that automated results are stored with history rather than read off the last pipeline run. None of that is analysis, it is record-keeping, and it either happens as a by-product of how the team works or it does not happen.

This is the practical case for keeping test cases, runs, requirements and linked defects in one system rather than reconstructing the relationships in a spreadsheet each quarter. In a test management system, the trail already exists, so a metric is a report rather than a data-collection project. In QAM Hub specifically, automated results carry per-test history with a stability score and flakiness rate over 7 to 90 day windows, which is where the suite-stability number comes from without extra instrumentation; coverage of requirements is a linked view rather than a manual matrix; and reports can be saved with their filters so that the same definition is used every month. Quality and coverage analytics and scheduled email delivery of a saved report are Advanced-plan features. The relevant detail for a metrics programme is that the definition stays fixed between periods, because a metric whose definition drifts is worse than no metric.

Two KB guides cover the mechanics: automation coverage reporting and execution statuses, which is where the blocked-versus-failed distinction is enforced.

Where to start

If you are building this from nothing, start with escaped defects and suite stability, in that order. Escaped defects because it is the outcome and it is the number an executive audience will accept as evidence. Suite stability because it is usually the cheapest large improvement available and because every other automated signal depends on it.

Give both of them three months of history before you draw conclusions from either, and resist the pressure to set a target in that window. Baselines first, targets later, and only where a target does not create an incentive to weaken the tests.

What to skip this quarter: dashboards with more than six numbers, any composite "quality score" that combines metrics with invented weights, cross-team comparisons, and code coverage percentage goals. None of those will be missed.

Frequently asked questions

What should I report to the board rather than to the team?

Escaped defects by severity and change failure rate, both as trends over at least two quarters, plus the one or two production incidents behind the largest movement. Suite stability and feedback latency are engineering-internal: they explain why the first two numbers move, and they invite a tooling conversation that a board is not the right room for.

Is code coverage useless then?

It is useful as a diagnostic and harmful as a goal. Low coverage in a module that handles money or personal data is a legitimate finding worth acting on. An organisation-wide percentage target produces assertion-free tests written to move the number, and the effect is worse than not measuring it, because the resulting suite looks like protection and is not.

We have no historical data. Where do we get a baseline?

Escaped defects can usually be reconstructed backwards from the issue tracker for the last two or three releases if production bugs are labelled at all, which is often enough to start. Suite stability cannot be reconstructed and has to be collected forward, so switch on result history now even if nobody looks at it for a month. Change failure rate depends on whether hotfixes and rollbacks are distinguishable in your deployment record, and if they are not, fixing that record is a smaller job than it sounds and pays for itself the first time someone asks about release quality.

Should QA own these metrics?

QA owns suite stability and coverage gaps. Escaped defects and change failure rate belong to engineering as a whole, and assigning them to QA alone is how a quality programme turns into a blame mechanism. The distinction matters in practice: escape rate is driven by design decisions, review quality and release size at least as much as by testing, so a QA team held solely accountable for it will optimise the only lever it fully controls, which is what gets classified as a defect.