Framework Emphasizes Safety and Clinical Oversight in Healthcare AI Deployment
Included Health has published a new peer-reviewed study in NEJM Catalyst Innovations in Care Delivery introducing a structured, clinically governed framework for deploying safe generative AI in patient-facing healthcare environments. The paper presents both a conceptual model and real-world pilot results demonstrating how healthcare organizations can responsibly integrate AI assistants while maintaining clinical safety, transparency, and appropriate human oversight.
At a time when millions of patients are increasingly turning to general-purpose AI tools for quick answers to health-related questions, the publication highlights a growing tension in digital healthcare: convenience versus clinical reliability. While public AI systems can provide fast responses, they are not designed with healthcare-specific safeguards, risk stratification, or escalation pathways. The framework introduced by Included Health is intended to address this gap by embedding clinical governance directly into AI system design and operation.
The paper describes the development and deployment of a patient-facing generative AI assistant designed to support routine health inquiries while identifying higher-risk situations that require escalation to licensed clinicians. Rather than positioning AI as a standalone diagnostic or decision-making tool, the system is structured as a triage and guidance layer within a broader care model. Its primary objective is to safely extend access to reliable information while ensuring that clinical expertise is engaged whenever needed.
A clinically governed approach to healthcare AI
A central theme of the publication is that healthcare AI must be held to standards that go beyond those applied to general consumer technology. According to Included Health leadership, the goal is not simply to make AI “helpful,” but to ensure it is safe, context-aware, and capable of recognizing its own limitations.
Dr. Ami Parekh, chief health officer at Included Health, emphasizes this distinction in the paper, noting that healthcare AI must be explicitly designed for real-world clinical complexity. This includes acknowledging uncertainty, avoiding overconfidence in responses, and ensuring that users are directed toward appropriate care pathways when risk is present.
The framework is built around the idea that AI in healthcare should function as part of a supervised clinical ecosystem rather than an isolated tool. This includes integrating governance structures, clinician oversight, and structured escalation pathways from the earliest stages of design and development.
Core principles of the framework
The published model is organized around four foundational principles that guide both system design and operational deployment:
1. Cross-functional governance with clinical leadership
The first principle establishes that AI systems in healthcare should not be developed solely by engineering or product teams. Instead, governance must include executive sponsorship alongside active clinical leadership from the beginning. This ensures that safety considerations are embedded in design decisions rather than added later as corrective measures.
Within Included Health’s approach, governance structures include clinicians, data scientists, engineers, and operational leaders working together to define system boundaries, acceptable use cases, and escalation rules. This cross-functional alignment is intended to ensure that technical capabilities are balanced with clinical responsibility.
2. Proactive risk analysis and pre-launch testing
The second principle focuses on anticipating failure modes before deployment. Rather than relying only on post-launch monitoring, the system undergoes structured risk analysis and simulation testing designed to identify where AI responses might be inaccurate, incomplete, or potentially harmful.
This includes stress-testing the assistant with high-risk clinical scenarios, ambiguous symptom descriptions, and edge cases where incorrect guidance could lead to delayed care. The goal is to define guardrails that prevent unsafe outputs and ensure consistent behavior across a wide range of patient inputs.
3. A three-tier clinical risk classification system
A key operational feature of the framework is a structured triage model that categorizes patient queries into three risk levels: standard, high-risk, and emergency.
- Standard queries involve routine health questions that can be safely addressed by the AI assistant with informational guidance.
- High-risk queries involve symptoms or contexts that may require clinician review or follow-up.
- Emergency-level situations are identified and escalated immediately to urgent care pathways or emergency services guidance.
This classification system enables the AI to dynamically determine how to respond based on potential clinical severity rather than treating all inputs uniformly. It also helps ensure that patients with urgent needs are not left to interact solely with automated systems.
4. Continuous clinician-in-the-loop review
The fourth principle ensures that clinicians remain actively involved after deployment. Rather than treating AI as a “set and forget” system, Included Health implemented continuous human oversight of AI interactions.
During the pilot, 100% of clinical interactions were reviewed daily by clinicians. This allowed the organization to monitor safety performance, identify patterns of misclassification or uncertainty, and refine the system over time. The feedback loop between clinical reviewers and system designers played a key role in improving both accuracy and user experience.
Real-world pilot and outcomes
The framework was evaluated through a seven-week pilot program in which the AI assistant was deployed to 50% of Included Health’s patient population. This allowed for controlled comparison between users interacting with the AI system and those receiving standard support pathways.
Across the pilot period, the system processed a broad range of patient questions, primarily focused on routine health concerns, symptom clarification, medication inquiries, and care navigation support. The risk-stratified design ensured that most standard queries were handled directly by the AI, while more complex or potentially concerning cases were escalated appropriately.
One of the most notable outcomes was a 65% reduction in unnecessary handoffs for standard-risk questions. This indicates that the AI was able to resolve a significant portion of routine interactions without requiring clinician intervention, improving efficiency while preserving safety boundaries.
At the same time, clinical review found that safety standards were maintained throughout the pilot. There were no reported systemic failures in risk identification, and escalation pathways functioned as intended. Importantly, patient experience metrics remained high and were comparable to the control group, suggesting that increased automation did not negatively affect satisfaction.
The results suggest that carefully governed AI systems can help reduce administrative and clinical burden without compromising care quality when properly designed and monitored.
Positioning AI within a broader care model
The publication also situates the AI assistant within Included Health’s broader “AI + EQ” approach, which emphasizes the combination of artificial intelligence with emotional intelligence and human clinical judgment. Rather than replacing human providers, the model is intended to extend their reach and improve access to timely guidance.
This approach reflects a growing recognition in healthcare AI development that purely automated systems may struggle to account for nuance, uncertainty, and patient-specific context. By contrast, hybrid systems that combine automation with clinician oversight may offer a more balanced path forward.
According to Included Health leadership, the intent is to create systems that are not only efficient but also trustworthy, transparent, and aligned with clinical realities. This includes clearly communicating what the AI can and cannot do, ensuring patients understand when human support is needed, and avoiding over-reliance on automated recommendations in high-risk situations.
Clinical and industry implications
The authors of the paper argue that the framework provides a practical blueprint for other healthcare organizations seeking to implement patient-facing AI safely. As adoption of generative AI accelerates across the industry, questions around regulation, liability, and patient safety are becoming increasingly urgent.
One of the key contributions of the framework is its emphasis on governance as a core system component rather than an external oversight layer. By integrating clinical review, risk classification, and escalation protocols directly into system architecture, the model offers a structured approach that can be replicated or adapted across different healthcare settings.
Dr. Ankoor Shah, vice president of clinical excellence at Included Health, highlights that this type of structured governance is essential for building trust in healthcare AI. Without it, organizations risk deploying tools that may be useful in some contexts but unreliable or unsafe in others.
The publication by Included Health represents a significant step in defining how generative AI can be responsibly integrated into patient-facing healthcare workflows. By combining clinical governance, structured risk classification, proactive testing, and continuous human oversight, the framework offers a detailed model for balancing innovation with safety.
As healthcare systems continue to explore the role of AI in improving access and efficiency, the findings suggest that success will depend not only on technological capability but also on the strength of the governance structures that support it. The Included Health framework positions clinical safety and accountability as foundational elements rather than optional safeguards—an approach that may shape how patient-facing AI evolves in the years ahead.
Source link: https://www.businesswire.com/
