Author: Eunoia Consulting Co. | Published: June 29, 2026
Healthcare boards and executive teams are increasingly being asked to oversee AI — but most were not trained for it, and the governance frameworks most organisations have in place were designed for a pre-AI world. This checklist covers the questions every healthcare board member and executive should be able to answer about their organisation's AI use, and the governance structures that should be in place to support informed oversight.
Healthcare boards and executive teams are increasingly being asked to oversee AI — but most were not trained for it, and the governance frameworks most organisations have in place were designed for a pre-AI world. The result is a growing accountability gap: AI systems are being deployed in clinical and operational settings while the people responsible for organisational oversight lack the tools to evaluate whether those systems are being governed responsibly.
This checklist is designed to close that gap. It covers the questions every healthcare board member and executive should be able to answer about their organisation's AI use, and the governance structures that should be in place to support informed oversight.
The foundation of AI governance is knowing what AI you have. Many healthcare organisations are surprised to discover, when they conduct a systematic inventory, that they have far more AI in use than they realised — embedded in EHR systems, billing platforms, diagnostic tools, and administrative software.
A current inventory of all AI systems in use across the organisation must exist and be maintained. The inventory must include AI features embedded in existing software platforms, not just standalone AI tools. Each AI system must be classified by risk level — clinical AI, administrative AI, and operational AI carry different risk profiles. The organisation must know which AI systems were deployed by IT or clinical staff without formal governance review. The inventory must be reviewed and updated at least annually, and whenever new AI systems are procured.
You cannot govern what you cannot see. An AI inventory is the prerequisite for every other governance activity.
Not all AI systems carry the same risk. A clinical decision support tool that influences diagnosis carries fundamentally different risk than an AI tool that optimises appointment scheduling. Governance resources should be allocated proportionally to risk.
Each AI system in the inventory must be assessed for its potential impact on patient safety, clinical quality, and regulatory compliance. High-risk AI systems — those that influence clinical decisions or affect access to care — must have documented risk assessments. The risk assessment process must include input from clinical leadership, not just IT or operations. AI systems that would qualify as high-risk under the EU AI Act framework must be identified, even if the organisation does not operate in the EU. Risk assessments must be updated when AI systems are updated or when their use cases change.
AI governance without accountability is documentation without effect. Every AI system in use must have a named accountable owner, and there must be a governance structure with authority to make and enforce AI governance decisions.
A named executive or clinical leader must be accountable for AI governance across the organisation. Each high-risk AI system must have a named clinical owner who is responsible for monitoring its performance and appropriateness. A governance committee or equivalent body must review AI deployment decisions for high-risk systems before go-live. The governance committee must include clinical, operational, legal, and compliance representation. The board must receive regular reporting on AI governance — at minimum annually, and immediately for significant AI-related incidents.
Most healthcare AI is deployed through vendors, not built internally. Vendor oversight is therefore a core component of AI governance — and it is an area where most organisations have significant gaps.
All AI vendor contracts must include provisions for performance monitoring, incident notification, and audit rights. Vendor AI systems must be assessed against the organisation's AI governance standards before procurement. Business Associate Agreements with AI vendors must address data use for model training — the organisation must know whether its patient data is being used to train vendor models. Vendor AI performance claims must be independently validated or verified against peer-reviewed evidence.
Regulatory frameworks globally are converging on a requirement for meaningful human oversight of AI systems in high-stakes settings. Healthcare organisations must be able to demonstrate that human oversight is not just a policy statement but an operational reality.
Clinical staff using AI-assisted tools must understand that they are responsible for the clinical decision, regardless of what the AI recommends. Override mechanisms must exist for all AI systems that influence clinical decisions — staff must be able to reject AI recommendations without workflow friction. Override rates must be tracked and reviewed — a very low override rate may indicate that staff are deferring to AI inappropriately, not that the AI is always correct. Training on AI tool limitations and appropriate use must be provided to all clinical staff using AI-assisted tools.
AI systems can fail in ways that are not immediately visible — gradual performance degradation, bias in specific patient populations, or unexpected behaviour in edge cases. Performance metrics must be defined and monitored for all high-risk AI systems. Baseline performance data must be captured before deployment, enabling meaningful comparison over time. An AI incident response process must define what constitutes an AI-related incident, how it is escalated, and how it is investigated. Significant AI incidents must be reported to the board and, where required, to regulators.
The goal of AI governance is not to slow down AI adoption. It is to ensure that AI adoption creates value without creating unacceptable risk. Healthcare organisations that govern AI well will adopt it more confidently, more quickly, and with better outcomes than those that do not.
Eunoia Consulting Co. helps healthcare boards and executive teams build AI governance frameworks. Contact us to discuss your organisation's AI governance maturity.