Author: Eunoia Consulting Co. | Published: July 29, 2026
Healthcare organisations are signing AI contracts without adequate due diligence — and the risks are different in kind, not just degree. This guide provides a structured framework covering clinical validation, data governance, regulatory compliance, and the contractual protections you must insist on.
Healthcare organisations are under significant pressure to adopt AI. Payers are deploying AI for prior authorisation and claims adjudication. Competitors are implementing AI-assisted documentation and diagnostic support. Regulators are developing AI-specific oversight frameworks. The pressure to act is real — and it is creating a procurement environment in which healthcare organisations are signing AI contracts without the due diligence those contracts warrant.
AI procurement in healthcare is not like procuring a new EHR or a billing platform. The risks are different in kind, not just in degree. An AI system that performs well in a vendor's training environment may perform poorly on your patient population. A model that is accurate on average may be systematically less accurate for specific demographic groups — creating disparate outcomes that create both ethical and legal exposure. A tool that is compliant today may require significant re-evaluation as the regulatory landscape evolves.
This guide provides a structured framework for AI vendor due diligence — the questions you should ask, the documentation you should require, and the contractual protections you should insist on before signing any AI procurement agreement.
The first and most important category of due diligence concerns the evidence base for the AI system's performance claims.
What is the training data? Ask vendors to describe the dataset on which their model was trained: its size, its demographic composition, the clinical settings it was drawn from, and the time period it covers. A model trained predominantly on data from large academic medical centres may perform differently in a community practice or rural clinic setting. A model trained on pre-2020 data may not reflect current clinical patterns or coding practices.
What is the validation evidence? Peer-reviewed validation studies are the gold standard. Ask vendors for published validation evidence, and evaluate it critically: Was the validation conducted on an independent dataset, or on a subset of the training data? Was it conducted in a clinical setting similar to yours? Were the performance metrics reported in clinically meaningful terms, or only in statistical terms that may not translate to real-world impact?
What are the known failure modes? Every AI system has conditions under which it performs poorly. Ask vendors to disclose known failure modes, edge cases, and demographic performance gaps. A vendor that cannot answer this question — or that claims their system has no failure modes — is a vendor you should not trust.
How is performance monitored post-deployment? AI models can degrade over time as clinical patterns, coding practices, and patient populations change. Ask vendors what post-deployment monitoring they provide, what performance thresholds trigger review or intervention, and what the process is for model updates.
AI systems in healthcare process protected health information. The data governance implications of AI procurement are significant and often underestimated.
What data does the system process, and where does it go? Understand precisely what patient data the AI system accesses, processes, and retains. Ask whether patient data is used to train or improve the vendor's models — and if so, under what terms and with what patient consent. Many AI vendors include model training rights in their standard contracts; this may be acceptable, but it must be a conscious decision, not an oversight.
What is the data residency and retention policy? For organisations subject to HIPAA, the EU AI Act, or other data protection frameworks, understanding where patient data is stored, for how long, and under what security controls is a compliance requirement, not a preference.
What are the Business Associate Agreement terms? For HIPAA-covered entities, a Business Associate Agreement is mandatory. Review the BAA carefully — not just for the standard provisions, but for provisions specific to AI: what happens to patient data if the vendor is acquired, what the vendor's breach notification obligations are, and what indemnification the vendor provides in the event of a data breach.
What is the vendor's security posture? Request SOC 2 Type II reports, penetration testing results, and information about the vendor's security incident history. AI systems that process clinical data are high-value targets for adversarial attacks.
The regulatory landscape for healthcare AI is evolving rapidly. Procurement decisions made today must account for the compliance obligations of tomorrow.
Is the system classified as a medical device? Under FDA guidance, AI software that meets the definition of a Software as a Medical Device (SaMD) is subject to FDA oversight, including pre-market review requirements for higher-risk applications. Ask vendors whether their system has been reviewed by the FDA, and under what classification.
How does the vendor approach EU AI Act compliance? For organisations operating in or serving EU markets, the EU AI Act's high-risk AI classification applies to many healthcare AI applications. Ask vendors how they are approaching EU AI Act compliance, what their timeline is for conformity assessment, and what documentation they will provide to support your own compliance obligations.
What is the vendor's AI governance framework? Mature AI vendors have internal governance processes for model development, validation, bias testing, and deployment. Ask vendors to describe their AI governance framework and to provide documentation of their internal processes. A vendor without a coherent answer to this question is a governance risk.
What happens when the model is updated? Model updates can materially change an AI system's behaviour. Ask vendors what their process is for notifying customers of model updates, what validation is performed before updates are deployed, and whether customers have the right to delay or decline updates.
Due diligence findings must be reflected in contractual protections. The following provisions are particularly important for healthcare AI contracts.
Performance warranties. Require vendors to warrant specific performance metrics — not just general claims of accuracy or effectiveness. Define the metrics, the measurement methodology, and the remedies available if performance falls below warranted levels.
Audit rights. Retain the right to audit the vendor's AI system performance, data handling practices, and security controls. This is particularly important for high-risk AI applications where ongoing performance monitoring is a governance requirement.
Data portability and deletion. Ensure you have the right to export your data and require its deletion if you terminate the contract. AI vendors that make data extraction difficult or expensive are creating a switching cost that reduces your negotiating leverage over time.
Liability and indemnification. Understand the vendor's liability position in the event that their AI system causes patient harm, a data breach, or a regulatory enforcement action. Standard vendor contracts typically limit liability significantly; negotiate for protections commensurate with the risk profile of the system.
Regulatory change provisions. As the regulatory landscape for healthcare AI evolves, contracts should include provisions for how compliance obligations will be managed — who is responsible for what, and what happens if regulatory requirements make the current system non-compliant.
AI procurement due diligence is not a one-time exercise — it is an organisational capability that must be built and maintained. Healthcare organisations that are serious about AI governance need procurement processes that include clinical, technical, legal, and compliance expertise; standard due diligence templates that can be adapted for different AI risk tiers; and a vendor registry that tracks the AI systems in use, their risk classifications, and their contract terms.
Eunoia Consulting Co. helps healthcare organisations build AI procurement capabilities that are rigorous, practical, and aligned with their governance frameworks. Our AI Governance service includes procurement due diligence support, vendor assessment frameworks, and contract review guidance. Book a strategy call to discuss your organisation's AI procurement challenges.
This article was produced by the Eunoia Consulting Co. Editorial Team. Eunoia Consulting Co. specialises in AI governance, healthcare operations, and data strategy for healthcare and veterinary organisations.