QAM Hub QAM Hub
Home / Blog / What to Ask a Test Management Tool Vendor About Security

What to Ask a Test Management Tool Vendor About Security

By Mike Krasnovskyi · Published
Test case card with a padlock and key next to a checklist, illustrating security questions to ask a test management vendor

Most security questionnaires sent to a test management vendor are copied from the one the company uses for its CRM, and they miss the part that is specific to testing. A test library holds things a CRM never does. It has passwords for test accounts written into steps, screenshots and videos recorded against staging with real-looking customer data, and API tokens that let the tool file bugs in Jira and accept results from CI. Ask about those first. Then ask the standard questions about certification scope, hosting and subprocessors, identity, AI processing, incident handling and exit, and read the answers for scope and dates rather than for logos.

This guide is written for the QA lead or engineering manager who has to get a TMS through a security review, and for the security reviewer who has never looked inside one. The questions are ranked. The first sections are the ones that most often decide whether a tool is acceptable, the later ones are the ones that most often get skipped.

Start with what your test library actually holds

Before sending a single question to a vendor, spend an hour inside your current test suites. The answer to "how sensitive is this data" is almost never "not very", and the reviewer on the other side will ask.

Look for these, in this order:

Two things follow. First, the vendor's answers about encryption and access control apply to more sensitive data than the procurement form assumes, so weight them accordingly. Second, some of the risk is yours to remove. Moving test credentials into a vault and referencing them by name in steps costs a day and makes every vendor question easier to answer.

Certifications: read the scope, not the badge

A vendor that lists SOC 2 or ISO 27001 has given you a starting point, not an answer. What the certificate or report covers is in the document itself, and it is the part that gets skipped.

SOC 2: which type, which period, which exceptions

A SOC 2 report is an attestation by an auditor against the AICPA Trust Services Criteria. A Type I report says the controls were suitably designed at a single point in time. A Type II report says they operated effectively over an observation period, commonly somewhere between three and twelve months. For a tool that will hold your test data for years, Type II is the one that tells you something about daily practice.

Ask for the full report under NDA, not the marketing summary, and read three parts of it: the system description (does it describe the product you are buying or a different service of the same company), the period covered (a report that ended fourteen months ago describes last year's company), and the exceptions section, where the auditor lists controls that failed testing. An exception is not automatically disqualifying. A vendor who cannot explain it is.

ISO/IEC 27001: the scope sentence is the claim

ISO/IEC 27001 certifies an information security management system, and every certificate carries a scope statement saying which organization, activities and sites the system covers. "Certified" means certified for that scope. A company can hold a perfectly valid certificate for its consulting services and still operate a SaaS product that sits outside it, so ask:

When there is no report

Smaller and younger vendors often have neither. That is not an automatic no, but the review then has to be done by questionnaire, and the burden of proof sits with the vendor. Standard formats help both sides. The Cloud Security Alliance's CAIQ, a questionnaire built on its Cloud Controls Matrix, is one many vendors have already filled in. A vendor that answers a CAIQ in specific, checkable sentences is easier to assess than one that sends a two-page security PDF.

Where the data lives and who else touches it

Third parties are a real share of the problem: Verizon's 2025 Data Breach Investigations Report found third-party involvement in 30% of the breaches it analysed, double the previous year's share. A TMS vendor is a third party, and it has its own.

The secrets the tool will hold for you

This is the section generic questionnaires leave out, and for a TMS it carries the most risk after the test data itself. A connected test management tool holds credentials for other systems. That means a Jira or GitHub token to create issues, sometimes one token per tester, and the API tokens your CI uses to post results. Tools with AI agents may also keep test-account credentials and even one-time-password seeds, so the agent can log in to the application under test.

Ask, for each kind of secret:

On your side, apply the same discipline you would to a CI secret. Use a dedicated tracker account with the narrowest project access that still lets it create issues, keep the token in the pipeline's secret store instead of a config file, and put a rotation date in someone's calendar.

Identity and access control

The questions here are familiar from any SaaS review. What changes for a TMS is who the users are. They include QA engineers who write and run tests, developers who read results, product managers and sometimes clients who only need to see a release report.

Attachments, recordings and retention

Files are where test data is least controlled. Ask how attachments are served. Short-lived signed URLs issued after an authorisation check are fine; public links that anyone with the URL can open months later are not. A public link in a bug report that was copied into a chat channel is a common way test screenshots leave the company.

Then ask what is deleted automatically and when. Large automation artefacts such as videos and traces are often removed after a retention period, while manual attachments are kept indefinitely. Both behaviours are defensible; what matters is that you know which applies to which file, because "we keep everything forever" and "we delete after 30 days" lead to different decisions about what testers may attach. If your policy requires a maximum retention for personal data, ask whether you can delete individual attachments and whole projects yourself, and how long deleted data survives in backups.

AI features and agents

Most test management tools now generate test cases, analyse failures or run AI agents against your application. Each of those sends parts of your test library to a model provider. The questions are specific and the answers should be too:

The last question is the one teams regret not asking. An agent with write access can edit or delete hundreds of cases in a minute, and the controls that matter are permissions and an audit trail, not the model's good behaviour. We cover that risk in more detail in giving AI agents write access to your TMS.

Logs, incidents and vulnerability handling

Getting out

Exit is part of security: data you cannot export is data you cannot fully control. Ask what an export contains (test cases with steps and custom fields, run history and results, attachments), whether it is available to an admin at any time without a support ticket, and whether it still works after a trial ends or a subscription lapses. Ask how deletion of the whole workspace is requested and confirmed, and whether anything happens automatically when you stop paying.

The migration side of this is covered in our guide to what you actually lose when you export test cases, which applies to any tool, not only TestRail.

A questionnaire you can send

The table condenses the questions above into the ones that most often separate an acceptable vendor from an unacceptable one. The third column is what should make you ask a follow-up, not a reason to reject on the spot.

QuestionA useful answer looks likeFollow up when
What exactly does your certification or audit report cover?The scope wording, the report type and period, and an offer to share the documentThe scope names a different service, or the report is more than a year old
Where is our data hosted, and by whom?Named providers per layer (application, database, files) and their regions"In the cloud" or "in the EU" with no providers named
Who are your subprocessors, and how do you notify changes?A list or a link to one, and a stated notification routeNo list, or a list with no change process
How are tracker and CI credentials stored?Encrypted separately from the database with a key held elsewhere; personal tokens hashed and shown onceThe answer only mentions disk encryption
Can we enforce SSO and two-factor authentication, and on which plan?Supported providers, a "require SSO" option, organization-wide 2FA enforcement, the plan namedSSO exists but cannot be enforced, or the plan is unclear
How are attachments served, and what is deleted automatically?Short-lived signed URLs; retention stated per file typePublic links, or no clear retention
What do AI features send, to whom, and is it used for training?Inputs and provider per feature, triggered only by users, processing terms stated, an off switch"Anonymised data" with no list of what is sent
Do agents and API clients act with a user's permissions, and are their changes logged?Yes to both, with read-only users staying read-onlyA shared service identity with broad rights
How fast will you notify us of a breach?A committed route and timeframe in the DPA"Promptly" with nothing in the contract
Can we export everything ourselves, including after a lapse?Self-service export with stated contents, available while blockedExport only through support, or only while subscribed

Send fewer questions and read the answers more carefully. A forty-page questionnaire answered with "Yes" in every cell tells you less than ten questions answered in full sentences with names, dates and plan tiers.

How QAM Hub answers these

We publish our answers on the QAM Hub security page, which is versioned with the product. In short, as of the page's last review on 30 October 2026:

The subprocessor list and the standard contractual clauses are available on request rather than published. If your review needs them, ask us before the pilot rather than after it.

What to do first

If you only have time for part of this, do it in this order. Clean credentials and production data out of your own test suites, because no vendor answer fixes those. Ask the secrets question and the AI question, because generic questionnaires miss both. Then read the certification scope and the subprocessor list. Leave the rest of the long questionnaire for vendors who pass those checks.

For the non-security side of choosing a tool, our test management tool evaluation checklist covers workflow, integrations and migration, and what a test management system is explains what you are evaluating in the first place.

QAM Hub is built by QA Madness, the publisher of this article.