Questions to ask any vendor about clinical AI

Clinical AI is moving from demonstrations into everyday tools: summaries before a consultation, drafted notes, coding suggestions, result explanations. The questions a clinic should ask have not changed as quickly. These nine apply to any vendor, including us. For each one, we describe the kind of answer that should reassure you.

1. Who signs before anything reaches the record?

Expect: a named clinician confirms every AI-drafted item before it is saved. Drafts should be visibly different from signed content, and approval should be per item, not one button that accepts everything.

2. What data does the AI read?

Expect: a clear list. Ideally the assistant reads only the open patient’s record and the clinic’s own reference material, within the permissions of the person using it. Be cautious of answers like “all available data”.

3. Is our data used to train models?

Expect: a direct no, or a clear opt-in with terms you can read. Ask whether the vendor’s model provider keeps or trains on prompts, and where that is written into the contract.

4. Where is the data processed and stored?

Expect: a named region that matches your legal obligations, and a list of the sub-processors involved, including the AI model provider. “In the cloud” is not an answer.

5. Can I see where an answer came from?

Expect: sources shown with each answer, such as the note, result or guideline it drew on, with versions where relevant. An answer you cannot trace is an answer you cannot check.

6. How do you test it, and what are its known failure modes?

Expect: a description of how outputs are evaluated, what errors have been seen and how they are handled. A vendor who says the system does not make mistakes has not tested it enough.

7. What does it refuse to do?

Expect: clear limits. For example: it does not diagnose, does not choose prescriptions, does not release lab results and does not write to the record without confirmation. Limits should be enforced by the product, not only stated in a policy.

8. What is logged?

Expect: a record of what the AI drafted, what the clinician changed and who approved it, available to your administrator. Logs are how you investigate a problem after the fact.

9. Can we switch it off?

Expect: yes, without losing the rest of the system. A clinic should be able to use the platform with AI assistance reduced or turned off for a role, a module or a period, and nothing clinical should depend on the AI being available.

How to use these questions

Ask them in writing and keep the answers with your contract. Involve a clinical lead and whoever is responsible for data protection. Ask to see the review step on a real screen, not a slide. If a vendor cannot answer one of these questions clearly, that is the answer.

Our own answers are on our AI governance and trust centre pages, and we will go through them with you in detail during an evaluation.