QAM Hub QAM Hub
Home / Features / AI-assisted testing / Autonomous agent runs

AI regression testing with autonomous agentsBeta

Let an AI agent drive the browser through your regression cases, then triage the results yourself.

AI regression testing in QAM Hub means an agent reads your existing regression test cases, drives a real browser through them and reports a verdict per case, with the trace and the video to back it up. The cases are the ones your team already maintains, not a second script in a second language, so the test base stays the single source of truth. Agent Flows are in Beta: the workflow below is in the product today, and the limits are stated where they apply. Triage is deliberately left to people, which is what keeps an autonomous run from filing bugs that are really UI changes.

How an agent flow works

An agent flow is a saved set of regression test cases and the environment to run them against. You start by picking cases from your suites on the Run test cases card, and the cases keep the suite's own order, so the run goes the way a tester would read the suite. After the run you can save the selection as a flow, which runs on demand or on a schedule for a nightly or pre-release pass. Environments hold the base URL, the login and the test accounts, so the same flow can run against dev and staging.

The browser runs where you choose, through a runner installed with one command:

npx @qamadness/tms-runner start --server <url> --token <token>

The runner connects to your QAM Hub workspace, picks up the jobs for its flows and reports back as each case progresses. LLM calls go through the QAM Hub backend, so the runner holds no model key unless you choose to supply your own.

What the agent records for every case

A run produces more than a pass or fail column. For each case you get:

  • A step-by-step trace of what the agent did, matched to the steps of the test case.
  • A verdict with the agent's reasoning.
  • A video of the case, so a reviewer can watch the exact moment a check failed instead of reconstructing it.
  • A live narrative feed while the case runs, so you can follow a run in progress rather than waiting for the summary.

When the agent is unsure, it pauses and asks instead of guessing. A case that depends on an ambiguous step waits for a human answer rather than producing a confident verdict on the wrong assumption.

Recordings: replay first, AI only where the page changed

Running every case through a reasoning model on every run is slow and expensive, and it is not what regression needs most of the time. The first time a case runs, the agent walks it with AI; once it passes, its steps are recorded and the recording is replayed in a clean browser to verify it. On later runs a recorded case replays its steps without AI and falls back to the AI only where the page changed. That mode, "Replay with AI fallback", is the one scheduled runs use.

When a run had to heal a recording because a page changed, the updated recording is offered for review before it replaces the old one. The recording your flow relies on never changes behind your back.

A successful run can be saved as a Baseline, and later runs are compared against it, so a regression shows up as a difference from a known good state rather than as an isolated failure.

AI regression testing with a human in the loop

Every case that did not pass is listed under "Needs review" on the Runs tab, with the agent's reasoning next to it. Nothing is filed, rewritten or closed until a person decides, and every decision has a visible consequence.

Your decisionWhat happens
"Confirm bug"Opens the issue dialog for your tracker, pre-filled, so the bug exists in Jira, GitHub or whichever tracker the project uses, linked to the case and the run. See defect tracking and the Jira integration.
"Review and accept change"On a UI change, shows the current expected result next to the proposed one before writing it, so the case is updated deliberately and the same finding does not return on the next run.
RejectMarks the finding as not a bug and not a change, with your comment on record.

The "Needs review" list is built for a reviewer working through a night's results: cases are grouped by run, with filters by status, flow, review state and linked issue, full-text search, comments on each case and a CSV export for anyone who wants the list outside the product.

Logins, test accounts and setup

The Agent Flows section has three tabs: Flows, Runs and Setup. Setup holds the runners, the logins and test accounts, and the AI key. Credentials for the app under test are stored encrypted, and login and setup hooks can run before the cases, so a flow signs in, seeds what it needs and only then starts the first case. A flow for the admin area and a flow for the customer area can use different accounts without the test cases carrying passwords.

What the agent learns: the project knowledge base

The agent reads the project knowledge base along with the cases: PDF, TXT, Markdown, JSON and CSV documents plus text notes, organized in folders, each one enabled or disabled on its own, assembled into a context block for the run. A note about a feature flag, a test environment quirk or a domain term changes how the agent interprets a step.

After each run the agent can propose knowledge base updates based on what it saw: add this, correct that, this note is stale. A person accepts or rejects each proposal, so the knowledge base stays current without becoming a second documentation chore.

Where the results show up

Agent results live next to the rest of your testing, not in a separate tool. Each test case carries its per-case agent history on the History tab, beside its manual test run results, so a case that keeps failing under the agent is visible to whoever edits it. A bug confirmed from an agent run appears in the Defects report like any bug filed from a manual execution.

What the agent does and does not replace

Agent Flows take over the repetitive part of regression: running the same cases against every build and flagging what changed. They do not decide what a change means. A reviewer still confirms the bug, accepts the UI change or rejects the finding, and the recording model keeps the AI's share of each run small once a case has been recorded. For the broader picture of where autonomous agents fit in a regression strategy, read autonomous regression testing with AI agents.

Plans

Agent Flows are part of the Advanced plan and are in Beta.

Frequently asked questions

Do I have to rewrite my test cases for the agent?

No. An agent flow is built from the test cases already in your suites, in the suite's own order. The agent reads the steps and expected results as written, and when a step is ambiguous it pauses and asks rather than guessing. Per-case agent history appears on the test case's History tab.

Where does the browser run?

On a machine you control, through a runner installed with one command: npx @qamadness/tms-runner start with your server URL and a token. The runner connects to QAM Hub, picks up jobs for its flows and streams the trace, verdict and video back. LLM calls go through the QAM Hub backend, so the runner holds no model key unless you decide to supply your own.

Does the agent file bugs on its own?

No. Every case that did not pass is listed under "Needs review" on the Runs tab with the agent's reasoning. A person chooses "Confirm bug", which opens the issue dialog for your tracker, or "Review and accept change" on a UI change, which shows the current and proposed expected result before writing it, or rejects the finding.

Does every run cost a full AI pass over every case?

No. A case recorded with "Automate" replays its recorded steps and falls back to the AI only where the page changed, which is the "Replay with AI fallback" mode scheduled runs use. A run that healed a recording offers the updated recording for review before it replaces the old one.

Can agent runs be scheduled?

Yes. A flow runs on demand or on a cron schedule, and scheduled runs use "Replay with AI fallback". Credentials for the app under test are stored encrypted, and login and setup hooks run before the cases, so an unattended run can sign in on its own.

Read more

Try Autonomous agent runs on your own project