What to Ask a Test Management Tool Vendor About Security
Most security questionnaires sent to a test management vendor are copied from the one the company uses for its CRM, and they miss the part that is specific to testing. A test library holds things a CRM never does. It has passwords for test accounts written into steps, screenshots and videos recorded against staging with real-looking customer data, and API tokens that let the tool file bugs in Jira and accept results from CI. Ask about those first. Then ask the standard questions about certification scope, hosting and subprocessors, identity, AI processing, incident handling and exit, and read the answers for scope and dates rather than for logos.
This guide is written for the QA lead or engineering manager who has to get a TMS through a security review, and for the security reviewer who has never looked inside one. The questions are ranked. The first sections are the ones that most often decide whether a tool is acceptable, the later ones are the ones that most often get skipped.
Start with what your test library actually holds
Before sending a single question to a vendor, spend an hour inside your current test suites. The answer to "how sensitive is this data" is almost never "not very", and the reviewer on the other side will ask.
Look for these, in this order:
- Credentials in steps and preconditions. "Log in as qa_admin / Winter2026!" is common in manual test cases, especially older ones. If those accounts exist on staging environments that are reachable from the internet, the test library is a credential store with no secret management at all.
- Production-like data. Test data copied from production to reproduce a bug, such as names, emails, addresses, order numbers, sometimes partial card numbers. It ends up in expected results, in attachments and in bug reports filed from failed tests.
- Screenshots and recordings. Every failed manual test that carries a screenshot, and every automated run that uploads a video or a Playwright trace, stores whatever was on screen. A trace also contains network requests, which can include session cookies and request bodies.
- Integration secrets. The tracker token the TMS uses to create Jira or GitHub issues, and the API token your pipeline uses to upload results. Both usually belong to a real person's account and carry that person's permissions.
- Product knowledge. Test cases describe unreleased features, security controls and known weaknesses in plain language. For some companies the test library is more sensitive than the code, because it explains how to break the product.
Two things follow. First, the vendor's answers about encryption and access control apply to more sensitive data than the procurement form assumes, so weight them accordingly. Second, some of the risk is yours to remove. Moving test credentials into a vault and referencing them by name in steps costs a day and makes every vendor question easier to answer.
Certifications: read the scope, not the badge
A vendor that lists SOC 2 or ISO 27001 has given you a starting point, not an answer. What the certificate or report covers is in the document itself, and it is the part that gets skipped.
SOC 2: which type, which period, which exceptions
A SOC 2 report is an attestation by an auditor against the AICPA Trust Services Criteria. A Type I report says the controls were suitably designed at a single point in time. A Type II report says they operated effectively over an observation period, commonly somewhere between three and twelve months. For a tool that will hold your test data for years, Type II is the one that tells you something about daily practice.
Ask for the full report under NDA, not the marketing summary, and read three parts of it: the system description (does it describe the product you are buying or a different service of the same company), the period covered (a report that ended fourteen months ago describes last year's company), and the exceptions section, where the auditor lists controls that failed testing. An exception is not automatically disqualifying. A vendor who cannot explain it is.
ISO/IEC 27001: the scope sentence is the claim
ISO/IEC 27001 certifies an information security management system, and every certificate carries a scope statement saying which organization, activities and sites the system covers. "Certified" means certified for that scope. A company can hold a perfectly valid certificate for its consulting services and still operate a SaaS product that sits outside it, so ask:
- What is the exact scope wording on the certificate, and does it name the product or the team that operates it?
- Can we see the certificate and the Statement of Applicability, which lists the Annex A controls the organization applies and the ones it excludes?
- When is the next surveillance audit due? Certificates are maintained through annual surveillance audits, and a certificate whose surveillance has lapsed may be suspended.
When there is no report
Smaller and younger vendors often have neither. That is not an automatic no, but the review then has to be done by questionnaire, and the burden of proof sits with the vendor. Standard formats help both sides. The Cloud Security Alliance's CAIQ, a questionnaire built on its Cloud Controls Matrix, is one many vendors have already filled in. A vendor that answers a CAIQ in specific, checkable sentences is easier to assess than one that sends a two-page security PDF.
Where the data lives and who else touches it
Third parties are a real share of the problem: Verizon's 2025 Data Breach Investigations Report found third-party involvement in 30% of the breaches it analysed, double the previous year's share. A TMS vendor is a third party, and it has its own.
- Hosting provider and region. Which providers run the application, the database and file storage, and in which region? Can you choose the region, or is it the same for every customer?
- Tenancy. Is the product single-tenant or multi-tenant, and how is one customer's data kept from another's? "Logically separated" is a normal answer for SaaS. Ask how, for example whether every request resolves the caller's organization on the server, or whether the client sends an organization ID the server trusts.
- Subprocessors. Who else processes your data (email delivery, payments, monitoring, AI model providers), and how are you told when the list changes? Under Article 28 of the GDPR a processor may not engage another processor without the controller's prior specific or general written authorisation, and under a general authorisation it has to inform you of intended changes so you can object. If the vendor has no change notification, that is a gap in the contract, not just in the documentation.
- Transfers. If any processing happens outside the EEA, under which transfer mechanism, typically the EU standard contractual clauses?
- Self-hosting. If your policy requires on-premises deployment, ask early. Many current TMS products are SaaS only, and finding that out in week three of a pilot wastes everyone's time.
The secrets the tool will hold for you
This is the section generic questionnaires leave out, and for a TMS it carries the most risk after the test data itself. A connected test management tool holds credentials for other systems. That means a Jira or GitHub token to create issues, sometimes one token per tester, and the API tokens your CI uses to post results. Tools with AI agents may also keep test-account credentials and even one-time-password seeds, so the agent can log in to the application under test.
Ask, for each kind of secret:
- Is it encrypted separately from the database's at-rest encryption, with a key held outside the database? Disk or volume encryption protects against a stolen disk. It does nothing if someone reads a database dump or a backup.
- Can it be shown again after it is saved? A tracker token that the UI can display in full to any admin is a token that leaks in a screen share.
- How are your own API tokens stored? Hashed, shown once at creation, revocable by the user and by an admin, ideally with a visible prefix so you can tell which token is which in a log.
- Whose permissions does a token carry? A CI token that acts as the person who created it stops working, or keeps working with the wrong rights, when that person changes role or leaves. Ask whether a token is attributed to its owner in the audit trail.
- For tracker integrations: can the integration use a dedicated service account in the tracker, and does the TMS support per-user credentials when you want bugs filed under each tester's own name?
On your side, apply the same discipline you would to a CI secret. Use a dedicated tracker account with the narrowest project access that still lets it create issues, keep the token in the pipeline's secret store instead of a config file, and put a rotation date in someone's calendar.
Identity and access control
The questions here are familiar from any SaaS review. What changes for a TMS is who the users are. They include QA engineers who write and run tests, developers who read results, product managers and sometimes clients who only need to see a release report.
- Single sign-on. Which protocols and providers (SAML, OIDC, Google Workspace, Microsoft Entra ID)? Can SSO be required so password sign-in is off for covered users? Which plan includes it? SSO is very often gated to the top tier, and that changes the price you are actually comparing.
- Deprovisioning. When someone leaves and is removed from your identity provider, do they lose access to the TMS at once, or do existing sessions survive? Is user provisioning automated (SCIM), or does an admin remove people by hand?
- Two-factor authentication. Can it be enforced organization-wide, or only switched on by each user?
- Roles. Are there separate organization and project roles, and is there a read-only role for stakeholders? Read-only access matters more in a TMS than in most tools, because the people who most want to see test results should not be able to change them.
- Sessions. How long does a session live, and does a password change or a two-factor reset end existing sessions?
- Vendor staff access. Which of the vendor's people can open customer data, for what reasons, and is that access logged? Every SaaS vendor has some answer to this. A vendor that says "nobody, ever" is usually describing its intentions rather than its support process.
Attachments, recordings and retention
Files are where test data is least controlled. Ask how attachments are served. Short-lived signed URLs issued after an authorisation check are fine; public links that anyone with the URL can open months later are not. A public link in a bug report that was copied into a chat channel is a common way test screenshots leave the company.
Then ask what is deleted automatically and when. Large automation artefacts such as videos and traces are often removed after a retention period, while manual attachments are kept indefinitely. Both behaviours are defensible; what matters is that you know which applies to which file, because "we keep everything forever" and "we delete after 30 days" lead to different decisions about what testers may attach. If your policy requires a maximum retention for personal data, ask whether you can delete individual attachments and whole projects yourself, and how long deleted data survives in backups.
AI features and agents
Most test management tools now generate test cases, analyse failures or run AI agents against your application. Each of those sends parts of your test library to a model provider. The questions are specific and the answers should be too:
- Which provider receives the data for each feature, and what exactly is sent? "Some context" is not an answer. A good answer names the inputs per feature, such as the description you typed, the cases under analysis, attached images, screenshots captured during an agent run.
- Is anything sent in the background, or only when a user triggers a feature?
- Under which terms is it processed, and are inputs used to train the provider's models?
- Can AI be switched off for the whole organization, and who can do it?
- For agents that act in the TMS on a user's behalf, including agents connected over MCP: whose permissions do they use, can a read-only user's agent write anything, and are an agent's changes attributed and logged like a person's?
The last question is the one teams regret not asking. An agent with write access can edit or delete hundreds of cases in a minute, and the controls that matter are permissions and an audit trail, not the model's good behaviour. We cover that risk in more detail in giving AI agents write access to your TMS.
Logs, incidents and vulnerability handling
- Audit log. Does the product record who changed what and when, across projects? Can you read it yourself, export it, and for how long is it kept? Like SSO, the organization-wide audit log is frequently a top-tier feature.
- Breach notification. How quickly, and how, will the vendor tell you about a personal data breach? Under Article 33 of the GDPR a controller has to notify its supervisory authority within 72 hours of becoming aware of a breach where feasible, and a processor has to notify the controller without undue delay. If your TMS vendor is your processor, its notification is the start of your 72 hours, so a vague answer here is your problem, not only theirs.
- Vulnerability reports. Is there a published way to report a security issue, and a response commitment? RFC 9116 defines a standard
security.txtfile for exactly this purpose; a security page with a monitored address does the same job. - Testing. Has the product had an independent penetration test, how recently, and will the vendor share a summary? May you run your own tests, and under what rules? Scanning a multi-tenant SaaS without permission usually gets your traffic blocked and can breach the terms of service.
- Backups and recovery. How often are backups taken, how far back can the vendor restore, and is availability monitored from outside the vendor's own infrastructure, ideally with a public status page?
Getting out
Exit is part of security: data you cannot export is data you cannot fully control. Ask what an export contains (test cases with steps and custom fields, run history and results, attachments), whether it is available to an admin at any time without a support ticket, and whether it still works after a trial ends or a subscription lapses. Ask how deletion of the whole workspace is requested and confirmed, and whether anything happens automatically when you stop paying.
The migration side of this is covered in our guide to what you actually lose when you export test cases, which applies to any tool, not only TestRail.
A questionnaire you can send
The table condenses the questions above into the ones that most often separate an acceptable vendor from an unacceptable one. The third column is what should make you ask a follow-up, not a reason to reject on the spot.
| Question | A useful answer looks like | Follow up when |
|---|---|---|
| What exactly does your certification or audit report cover? | The scope wording, the report type and period, and an offer to share the document | The scope names a different service, or the report is more than a year old |
| Where is our data hosted, and by whom? | Named providers per layer (application, database, files) and their regions | "In the cloud" or "in the EU" with no providers named |
| Who are your subprocessors, and how do you notify changes? | A list or a link to one, and a stated notification route | No list, or a list with no change process |
| How are tracker and CI credentials stored? | Encrypted separately from the database with a key held elsewhere; personal tokens hashed and shown once | The answer only mentions disk encryption |
| Can we enforce SSO and two-factor authentication, and on which plan? | Supported providers, a "require SSO" option, organization-wide 2FA enforcement, the plan named | SSO exists but cannot be enforced, or the plan is unclear |
| How are attachments served, and what is deleted automatically? | Short-lived signed URLs; retention stated per file type | Public links, or no clear retention |
| What do AI features send, to whom, and is it used for training? | Inputs and provider per feature, triggered only by users, processing terms stated, an off switch | "Anonymised data" with no list of what is sent |
| Do agents and API clients act with a user's permissions, and are their changes logged? | Yes to both, with read-only users staying read-only | A shared service identity with broad rights |
| How fast will you notify us of a breach? | A committed route and timeframe in the DPA | "Promptly" with nothing in the contract |
| Can we export everything ourselves, including after a lapse? | Self-service export with stated contents, available while blocked | Export only through support, or only while subscribed |
Send fewer questions and read the answers more carefully. A forty-page questionnaire answered with "Yes" in every cell tells you less than ten questions answered in full sentences with names, dates and plan tiers.
How QAM Hub answers these
We publish our answers on the QAM Hub security page, which is versioned with the product. In short, as of the page's last review on 30 October 2026:
- Certification. QAM Hub is built and operated by QA Madness Sp. z o.o., which holds ISO/IEC 27001:2022 certification for the provision of software development and software quality assurance services. The same information security management system governs the team that builds and runs QAM Hub. The certificate is available on request.
- Hosting. Application on Render, database on Neon and files on Cloudflare R2, all in Western Europe; the web app is delivered through Vercel. It is a single multi-tenant deployment in which every request resolves the caller's organization and project membership on the server. Regions are the same for every customer, and there is no per-customer region choice or self-hosted version.
- Secrets. Tracker credentials, agent-run credentials and two-factor secrets are sealed with AES-256-GCM under a key held outside the database. Personal API tokens are stored as hashes and shown once.
- Access. Organization and project roles with a read-only viewer role, and personal two-factor authentication on every plan. Enforced 2FA, Google and Microsoft SSO with a "require SSO" option, and the organization audit log are on the Advanced plan.
- Files and retention. Files go through signed URLs valid for 15 minutes. Only automation videos and Playwright traces are removed automatically, after 14, 30 or 60 days depending on the plan; everything else stays until you delete it.
- AI. Each AI feature names its provider and its inputs on the security page, nothing is sent in the background, and AI can be switched off for a workspace on request. MCP clients act as the token owner, so a viewer stays read-only.
- Exit. An organization admin can export the organization as CSV at any time, including while a workspace is blocked after a trial, and nothing is deleted when a trial or subscription ends.
The subprocessor list and the standard contractual clauses are available on request rather than published. If your review needs them, ask us before the pilot rather than after it.
What to do first
If you only have time for part of this, do it in this order. Clean credentials and production data out of your own test suites, because no vendor answer fixes those. Ask the secrets question and the AI question, because generic questionnaires miss both. Then read the certification scope and the subprocessor list. Leave the rest of the long questionnaire for vendors who pass those checks.
For the non-security side of choosing a tool, our test management tool evaluation checklist covers workflow, integrations and migration, and what a test management system is explains what you are evaluating in the first place.
QAM Hub is built by QA Madness, the publisher of this article.