Questions to ask before putting an AI assistant on a clinic line

What should I ask a vendor about clinical questions?

Ask what happens when a caller describes symptoms, and whether the refusal is enforced in code or written as an instruction in a prompt. Then ask what happens after a model update.

Callers describe symptoms to whoever picks up. It is not an edge case, it is most first calls, so a vendor who has not thought about it has not thought about clinics.

An instruction in a prompt can be ignored by the model. A refusal asserted by a test suite before deployment, and re-run after every model change, cannot be quietly lost.

What should I ask about recordings and retention?

Ask whether calls are recorded, what the platform retains, and what happens to a medical detail a patient volunteers unprompted. Ask to see the retention setting rather than being told about it.

A caller can say something clinical no matter how well the assistant behaves. If the call is recorded, that is captured regardless of what the assistant does next, which is why some practices choose not to record at all.

What answer should end the conversation?

Any version of the assistant being able to help with clinical questions, or a claim of compliance without the underlying programme. Compliance is policies, training, agreements and audits, not a product feature.

A vendor claiming their software makes you compliant is describing something software cannot do on its own. Ask what specifically they are certifying and who signed it.