AI-Generated Test Cases: Best Practices and Common Mistakes

AI writes a usable first draft of a test case in seconds, and a wrong one just as fast. The wrong ones are the expensive part, because they read exactly like the right ones. Three practices carry most of the value: generate from the specification rather than from the working product, decide the review budget before the generation target, and cap how many cases you let into the suite per feature. The rest of the usual best-practice list is secondary to those three, and one popular habit, feeding the model a screenshot of the finished screen, actively works against you.

Most published advice on this topic stops at "always review the output". That is true and nearly useless. It does not say what to look for, how long a review should take, or when to stop generating. This article is about those three gaps, and about the failure mode that survives a careless review.

Why a plausible test case costs more than a wrong one

A generated case that says Verify the user receives a confirmation email within 30 seconds is well formed, correctly structured, and specific. If the requirement never mentioned 30 seconds, it is also an invented acceptance criterion, and it is now a permanent part of your suite.

Two things happen next. Either the case fails at some point and an engineer spends an afternoon investigating a product behaviour nobody ever asked for, or, more often, someone quietly relaxes the expected result to match what the system does and records the case as passing. The second outcome is worse: coverage reporting now counts a case that verifies nothing.

The reason this gets through review is mechanical. Reviewing well-formed prose is much faster than reviewing badly formed prose, and the reviewer's speed tracks the fluency of the text, not its correctness. Bad grammar slows a reviewer down in a way that a fabricated number does not.

There is direct measurement of the underlying effect, though it comes from the unit-testing side of the problem. In the TestPilot study of LLM-generated JavaScript unit tests (Schäfer et al., published in IEEE Transactions on Software Engineering; preprint arXiv:2302.06527), a median of 48% of generated tests passed at all, a median of 19.2% of failures were assertion errors where the model could not work out the correct expected value, and a median of 61.4% of generated tests were non-trivial, meaning they contained an assertion that actually depended on the code under test. Roughly two in five asserted nothing meaningful. That study used GPT-3.5-era models on npm packages, so the pass rates are dated, but the shape of the failure is not: the model produces the form of a test reliably and the oracle unreliably.

The functional-testing equivalent of a test with no meaningful assertion is the expected result that reads the page displays correctly. It cannot fail. It will sit in the suite for years, pass every run, and contribute to a coverage number.

The mistake that costs the most: generating from the product instead of the spec

If the model can see the implementation, it will describe the implementation. Expected results then encode what the system currently does, and the suite becomes a regression baseline of present behaviour, bugs included. A suite built that way cannot find a defect that already existed when you generated it, which is precisely the class of defect that survives longest.

Two 2026 studies measured this on code, and both point the same way.

"On the risk of coding before testing" (arXiv:2607.05139, July 2026) ran five models, GPT-5-mini, GPT-4.1-mini, DeepSeek-V4-Flash, Claude Haiku 4.5 and Llama 3.3 Instruct 70B, across HumanEval+, MBPP and BigCodeBench. Giving the model the task description together with the implementation reduced fault detection by 13.2% on average compared with the task description alone. Comparing whole workflows, a test-driven workflow that generated tests from the specification only detected 11.7% more faults on average than an agentic workflow that wrote code first and tests afterwards, 25% against 14%.

The companion finding comes from "Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests" (arXiv:2607.22883, PACMSE, July 2026), which named and measured the effect: buggy code in the prompt both increases the number of tests that assert the erroneous behaviour as correct and suppresses the generation of tests that would expose it. The authors' fix is to take the code out of the prompt and replace it with a generated specification docstring. They also note why this went unnoticed for a while: most tests generated from buggy code that pass on the buggy version also pass on the fixed version, so naive pass-rate metrics show nothing wrong.

Both studies generate unit tests from source code, and a QA engineer generating functional cases from a user story is in a different setting. The transfer has not been measured, so treat it as an argument, not a measured finding. It is a strong argument, though, and arguably sharper in the functional case: a screenshot of a built screen, or access to a staging environment, is a far more complete description of the implementation than a single function body. Generate from a screenshot and every expected result you get back is a description of the current UI, including the label that was never signed off and the validation message that fires on the wrong field.

This has a practical edge in tools that support it. QAM Hub generates test cases and checklists from either a written description or a screenshot, and the screenshot path is genuinely the right tool for one job: documenting an undocumented legacy screen, where "describe what exists" is the actual goal. For a new feature with acceptance criteria, give it the criteria and keep the screenshot out of the prompt. The step-by-step mechanics are in the knowledge base article on generating test cases with AI.

When there is no specification to generate from, write the acceptance criteria first, even if that is three lines in the ticket. If nobody on the team can write them, generation is not the bottleneck and no amount of prompting will find the missing decision.

Review is the control, and it has a price

The most useful published number on AI-generated tests comes from Meta's TestGen-LLM deployment (Alshahwan et al., arXiv:2402.09171, reported at FSE 2024). Running on Instagram and Facebook Java test classes, the system applied a chain of machine filters before a human saw anything: 75% of generated test classes contained at least one case that built correctly, 57% contained a case that built and passed reliably, and 25% contained a case that built, passed and increased line coverage. Only that last quarter was put in front of an engineer. Of the improvements that reached review, 73% were accepted by developers, and in one Instagram test-a-thon 36 of 42 submitted diffs were accepted. The authors describe the design intent plainly: the system submits for human review only test cases it can guarantee improve on the existing code base.

Read that acceptance rate carefully, because it is widely misquoted. The 73% is a property of the filter, not of the model. Three quarters of the raw output was thrown away by automation first.

Manual test case generation in a TMS has no equivalent filter. There is no compiler to reject a malformed functional case, no test run to prove it passes reliably, and no coverage delta to prove it adds anything. Every one of those judgments lands on a person, which means the review is the entire quality control mechanism and its cost is the real cost of the feature.

So decide the batch size from the review capacity rather than the other way round. Generating thirty cases for a feature takes under a minute; checking thirty cases against their acceptance criteria does not. Generate a batch you can review to completion in one sitting, finish it, then generate the next. Batches that outrun the review get approved in bulk, and bulk approval is exactly the condition under which the plausible-but-invented case enters the suite.

Input: what actually changes the output

Input quality matters more than prompt technique, and the items below are in rough order of how much difference they make.

Boundary values written out as numbers. "The reset link expires" produces a case about expiry with an invented window. Write the rule with its edges and the generator has nothing to invent:

Password reset, acceptance criteria:
- Reset link is valid for 24 hours from the moment the email is sent.
- A link opened after 24 hours shows "This link has expired" and offers to resend.
- A link that has already been used once is rejected with the same message.
- Requesting a reset for an address that is not registered shows the same
  confirmation screen as a successful request (no account enumeration).
- Roles: any authenticated or anonymous visitor. Admin accounts follow the
  same flow; SSO accounts have no reset link at all.

The last two lines are the ones teams leave out, and they are the ones that produce the cases worth having.

Negative space. What the feature must not do, which states are unreachable, which roles have no access at all. Models generate happy paths abundantly and under-generate refusals, so refusals have to be asked for by name.

Environment and integration reality. Which third-party calls are stubbed in the target environment, which test accounts exist, what data is seeded. Skip this and you get well-written cases that nobody can execute, which is the most demoralising kind of generated output.

Your own conventions. Feed it two or three of your own well-written cases as the format target, along with your template and status vocabulary. This is the cheapest lever available and the most commonly skipped; without it, every generated batch needs reformatting by hand before it fits the suite.

What to leave out: the whole PRD (it dilutes the specific criteria), the implementation, another team's test cases, and anything containing customer data. On that last point, data privacy was the most-cited barrier to AI adoption in Capgemini's World Quality Report 2025-26, named by 67% of the 2,000-plus senior executives surveyed across 22 countries and 10 sectors, ahead of integration complexity at 64% and hallucination or reliability concerns at 60%. Whether generation runs inside a tool covered by your vendor's data-processing terms or in a chat window on someone's personal account is a compliance question with a different answer in each case.

The second-order cost is volume

Generation makes test cases cheap, and a test base is one of those assets where quantity turns into liability at a predictable rate. Two mechanisms do most of the damage.

Duplicates accumulate quietly. Ten sprints of generation from overlapping stories produce near-identical cases under different titles, which inflate coverage counts and multiply execution time in every manual regression cycle. Nobody notices until a full run stops fitting in the release window.

Structural debt accumulates faster. A large study of test smells in LLM-generated tests (arXiv:2410.10628, ACM TOSEM) analysed 20,505 class-level test suites from four models across three Java benchmarks, against 14,469 EvoSuite-generated tests and 779,585 human-written tests from 34,635 open-source projects, using two independent smell detectors. Assertion Roulette, many assertions in one test with no explanatory messages, and Magic Number Test dominated the LLM output, with the pattern varying by prompting strategy, context length and model scale. The functional equivalent is the generated case that checks six unrelated things in one flow, so a failure tells you the case failed and nothing more, and the case that hardcodes a value with no stated origin.

One case, one verifiable outcome. Ask for that explicitly in the prompt, because models comply when told and consolidate by default.

Tooling helps with diagnosis and does nothing for prevention. QAM Hub's Quality Analyzer runs background analysis of the test base against run history, produces a deterministic A to D score, flags duplicates with an evidence quote for each finding, keeps a shared record of what the team has already triaged, and reports the delta against the previous analysis. It is an Advanced-plan feature, and it does not reduce the need to review generated cases at the point of generation. What it does is tell you what the last ten sprints of generation actually did to the suite, which is otherwise close to unknowable. The same ground is covered in more detail in our guide to finding duplicate and low-quality test cases.

Where to use generation first, and where not to

Start with the work where the specification is the input and the output is mechanical:

• Boundary and negative permutations from an explicit rule set. Tedious by hand, error-prone by hand, and the model has everything it needs.
• Reformatting an inherited suite into your template. The content already exists, so the oracle problem does not arise. This is the safest possible first use and it is rarely the one teams pick.
• Data permutation tables for a rule you have already written down: currencies, locales, tax regimes, plan tiers.
• Checklist expansion where a manual pass needs breadth rather than depth, such as a smoke checklist across a set of screens.

Hold off on these, and in one case indefinitely:

• Anything where the expected result is a business decision nobody has written down. The model will produce one. It will be reasonable, and it will be someone's invention.
• Exploratory testing. Generation produces the documented. Exploratory work exists to find what is not documented, and a generated "exploratory charter" is a contradiction in terms.
• Regulated evidence trails. Feasible, but the review and approval record has to exist before you generate, not be reconstructed afterwards. How strict this is depends on the regime you are audited against, and it is worth confirming with whoever signs the validation package before the first generated case enters a controlled suite.

Automation candidates are a separate decision from generation, and picking them by what is easy to generate is a reliable way to automate the wrong tests. Our guide on what to automate first covers the criteria that should drive that instead.

A review pass you can actually run

These are in order because the order saves time: each check is cheaper than the one after it, and a failure at step one makes the rest irrelevant.

1. Does the expected result come from the requirement? If it is not stated and not derivable, delete the case or fix the requirement. Most invented acceptance criteria die here, and this check alone is worth more than the other five together.
2. Could this case fail? If no observable outcome separates pass from fail, it is documentation with a pass button.
3. Is it one outcome? Split it or drop it.
4. Is it executable, by the role named, in the environment you have? Preconditions, test data, permissions.
5. Is it already in the suite? Search before saving. Deduplication at the point of entry costs seconds; deduplication in a quarterly cleanup costs days.
6. Is it linked to its requirement? Do it now. A generated case with no requirement link is unauditable six months later, when nobody remembers which story produced it. The mechanics of that link are in our requirements traceability matrix guide.

The stop rule matters as much as the checks. If two of the first three fail across a batch, the input was bad. Fix the input and regenerate rather than repairing cases one at a time, because case-by-case repair is how teams end up spending more time than writing from scratch would have taken, then concluding the technology does not work.

What to measure

Cases generated is not a metric. It measures the cheapest step in the process and nobody has ever made a decision with it.

Four numbers are worth keeping. The share of generated cases that have ever been executed, because a generated case that never ran consumed review time and returned nothing. The defects attributed to generated cases against hand-written ones over a full release, which is the only direct evidence that the output finds anything. The duplicate rate in the suite, trending. And review time per accepted case, which is the actual unit cost of the feature and the number that tells you whether to keep going.

Tagging generated cases at creation is what makes any of this measurable, and it costs one custom field. Teams that skip it cannot answer the first question three sprints later. Our write-up on QA metrics that matter for engineering leaders goes further into which of these survive contact with a management conversation.

For context on how far ahead of the evidence the category currently runs: Capgemini's World Quality Report 2025-26 found 89% of responding organisations piloting or deploying generative-AI-augmented workflows, with 37% in production and only 15% at enterprise-wide implementation, and an average reported productivity gain of 19% with roughly a third of organisations seeing minimal gains. The same report notes adoption shifting from analysing outputs, such as defect analysis and reporting, toward shaping inputs, with test case design and requirements refinement now leading. Test case generation is where the category is heading; the measured returns so far are real, moderate, and unevenly distributed.

Where this fits with the rest of your AI adoption

Test case generation is usually the first AI feature a QA team turns on, partly because it is the easiest to demonstrate and partly because the failure mode is invisible in a demo. It belongs in a sequence: generation first because the review loop is cheap to run and the blast radius is small, then AI analysis of existing artefacts, and agent-driven execution last, once the review habits are established. We set out that order and the reasoning in how to introduce AI into your QA process without breaking it, and the broader question of which QA activities AI handles reliably in AI in software testing: what it can and cannot do.

One boundary worth stating early with your team, because it prevents a specific argument later: a tool that helps you use the product is not the same thing as a tool that works on the application under test. In QAM Hub the in-app AI Assistant answers "how do I" and "where is" and deliberately refuses to do QA work, while the generation features produce test artefacts and Agent Flows, an Advanced-plan feature currently marked as coming soon, will execute against the application itself. Where an agent gets write access to the test base, the review workflow stops being a nicety, which is the subject of giving AI agents write access to your TMS.

What to prioritise

Two things carry the result: acceptance criteria with real boundary values written before generation, and a review budget that sets the batch size. Both are unglamorous and neither involves choosing a tool.

Do not over-engineer the prompt. The same study that measured the implementation-exposure effect also tested prompting strategies and found generating directly from the task description beat all three it tried, with fault detection dropping 15.5% for summarisation and 13.4% for both chain-of-thought and chain-of-verification. Elaborate prompt scaffolding is where a lot of effort goes and it is not where the return is.

Skip the labelling taxonomy, the generation policy document and the approval workflow until three sprints of real use have shown you what actually goes wrong in your own suite. One tag and a review checklist is enough to start, and enough to produce the evidence for whatever comes after.

Frequently asked questions

Does AI test case generation improve test coverage?

Not by itself. Generation increases the number of cases, which is a different thing. Coverage changes only when the new cases map to requirements or code paths that had none, so the measurement has to be traceability against the requirement set rather than case count. A batch of generated cases covering a well-covered happy path raises the count and leaves coverage where it was.

Should generated test cases be labelled as generated?

Yes, with a tag or a custom field, set at creation. It costs almost nothing and it is the only way to answer the questions that matter three sprints later: how many generated cases ever ran, how many found defects, how many were quietly abandoned. It also makes the first cleanup possible, because you can review the cohort instead of the whole suite. Nobody has ever regretted adding the field; plenty of teams have regretted not having it.

Who is the author of an AI-generated test case?

The reviewer who accepted it. Tools record whoever pressed save, and that is the right answer for practical purposes: accountability follows the judgment, and the judgment was the review. This matters most in regulated environments, where a test case needs a named author and an approval record, and where "generated" is not an acceptable entry in that field.

Is it better to generate test cases inside a TMS or in a chat window?

What separates them is what happens to the case afterwards. Generated in the tool, a case arrives with its template, its custom fields, its requirement link and its version history, and it is reviewable in place. Generated in a chat window, it arrives as text that someone reformats and pastes, and the requirement link is a step people skip when they are pasting twenty cases. The second difference is data handling: generation inside a tool is covered by that vendor's data-processing terms, and generation in a personal chat account is not covered by anything. Which vendors have built generation into the test management layer, and what their versions actually do, is compared in our review of AI test management tools.