Key Takeaways
- More than 1,200 AI-enabled medical devices have received FDA clearance, with the majority concentrated in medical imaging.
- AI adoption in medical imaging has reached 90% of U.S. health systems, making it the most mature deployment category in clinical AI.
- The health effects of AI tools are almost entirely unmeasured, even as health systems deploy them at scale.
- No single U.S. health system is currently equipped to fully validate an AI algorithm once it enters clinical use, according to a former FDA Commissioner.
For better or worse, AI will disrupt all areas of healthcare delivery. That is the central message the JAMA Summit on AI addressed in its landmark report, drawing on the contributions of more than 40 experts across medicine, law, data science, and policy.
There’s no doubt that AI in healthcare is exploding. Artificial intelligence is already embedded in more than 1,200 medical devices cleared by the FDA.
In 90% of U.S. health systems, AI tools are augmenting medical imaging interpretation.
Many EHR systems now have embedded AI-based clinical decision support tools.
But along with the benefits of AI in healthcare, there are risks.
AI tools are generating prior authorization denials that physicians cannot explain, powering wellness apps with no clinical evidence to support them, and operating within health systems that have no reliable way to measure whether they are helping or hurting patients.
Along with the good, healthcare is seeing the bad side of AI, and where it can get ugly.
The Good: The Benefits of AI in Healthcare
The clearest wins for AI in healthcare are in medical imaging. AI has fundamentally changed how radiologists and pathologists work, with meaningful improvements in diagnostic speed and accuracy. Just today, Midjourney announced its vision for tomorrow's AI-powered MRI.
Algorithmic sepsis alert systems, automated screening tools for diabetic retinopathy, and AI-assisted echocardiography interpretation represent clinical AI that has undergone genuine evaluation and, in many cases, FDA review.
Beyond imaging, healthcare AI tools enable clinicians to get quick, accurate answers to natural-language queries. As these tools have integrated more deeply with EHR platforms, both patient and clinician satisfaction have improved steadily.
Direct-to-consumer AI tools represent a third area of genuine promise. More than 350,000 mobile health apps exist today, with AI embedded in a significant number of them. Three in 10 adults worldwide have used a mobile health app, and the market exceeds $70 billion annually.
For hospitals using AI to optimize their workflows, the foundation built now determines how reliably these tools perform when lives depend on them.
The Bad: Deployment Without Evidence
Health systems are purchasing and deploying AI tools for business operations at a pace that far outstrips any evaluation of their effects on patients.
Peer-reviewed evaluations of AI for business operations are rare. Market adoption is driven primarily by vendor testimonials and use cases, often blending efficiency claims with assertions about care quality, with no independent verification of either.
The prior authorization example is particularly stark. Many U.S. physicians believe AI tools used by insurers to evaluate and deny prior authorization requests are causing widespread patient harm. However, there are no published studies on how AI has changed denial rates. Neither side has the data to settle the argument, because the evaluation requirements have never been established.
The challenge extends into the clinical space as well. EHR-embedded AI tools are widely available, but physicians are concerned about their accuracy and utility. Algorithmic bias, automation bias, poor generalizability across settings, and insufficient clinical trust all dampen adoption.
The Ugly: The Governance and Measurement Gap
The most serious problem in healthcare AI is the lack of a reliable system to determine whether a given tool is doing what it claims.
“I have looked far and wide,” Former FDA Commissioner Robert Califf said at an agency panel on AI. “I do not believe there's a single health system in the United States that's capable of validating an AI algorithm that's put into place in a clinical care system.”
A tool can be widely adopted, generating clinical recommendations every hour of every day, while no health system has the infrastructure or expertise to determine whether it is producing better, neutral, or harmful outcomes.
Monitoring standards that do exist focus on safety and process compliance. None currently address whether a tool improves clinical outcomes over time, across different patient populations and settings. There is a tacit assumption that proving effectiveness is either unnecessary or sufficiently demonstrated by pre-deployment testing, an assumption the evidence does not support.
This matters most for equity. AI models trained on historical healthcare data can encode and amplify existing disparities across race, disability, and age. An algorithm that performs well at an academic medical center may produce entirely different results at a rural critical access hospital.
For health systems exploring AI healthcare implementation, the governance and implementation questions are inseparable. Deploying a tool without measurement capability isn’t efficiency; it’s unmanaged risk.
What Responsible Implementation Requires
The JAMA Summit identifies four priorities that must advance together before healthcare AI can be trusted to deliver consistently on its clinical promise.
1. Multistakeholder engagement across the full product lifecycle
The traditional linear pathway, where a developer builds a tool and hands it to a health system for deployment, does not fit AI. Tools evolve, performance shifts, and the context of use determines outcomes as much as the underlying algorithm. Patients, clinicians, developers, regulators, and health systems all need to be involved at every stage.
2. Better measurement tools
Standards exist for evaluating the quality of AI tools. The Joint Commission's partnership with the Coalition for Health AI represents one step toward a certification process for responsible use. What is still missing is any standard for measuring effectiveness, meaning demonstrated improvement in patient outcomes, not just safety monitoring and process compliance.
3. Shared data infrastructure
Single health systems cannot generate the generalizable knowledge needed to evaluate AI tools across diverse populations and settings. A federated, nationally representative data environment would allow developers, health systems, and regulators to assess real-world performance at scale.
4. Aligned incentives
Health systems bear the full cost of standing up the digital infrastructure and expertise needed for responsible AI use, without reimbursement or market signals that reward doing it well. The industry needs federal investment comparable to the HITECH Act, which drove EHR adoption to 97% of U.S. health systems within a decade through a $35 billion investment.
The organizations that close the governance gap deliberately are the ones that will earn the clinical trust AI in healthcare still needs to establish.
Change Is Coming
We’re just now starting to see the benefits of AI in healthcare. Nonetheless, health organizations are rushing to adopt these tools in an attempt to alleviate long-standing problems in healthcare. And AI does represent an incredible opportunity to do so.
However, the AI risks in clinical systems are more consequential than in any other industry. From data privacy breaches to negative patient outcomes, the risks of AI in healthcare need to be regulated and mitigated as stringently as those of medical devices.
If your organization is building AI into clinical or operational workflows and needs a development partner who understands the healthcare environment, talk to the experts at Taazaa. We have a wealth of experience building secure, HIPAA-compliant healthcare AI solutions.
Frequently Asked Questions
What are the proven benefits of AI in healthcare right now?
Medical imaging is where AI has shown the strongest, most consistent results. Radiologists and pathologists are diagnosing faster and more accurately with AI assistance, and tools such as sepsis alert systems and diabetic retinopathy screening have strong evidence supporting them. AI scribes that handle clinical documentation are also gaining ground, with clinicians reporting real reductions in paperwork burden.
Why is AI used in healthcare business operations considered a risk?
Because no one is measuring what it is actually doing to patients. Health systems are using AI for scheduling, billing, and prior authorization at scale, but almost none of these tools have been independently evaluated for their effect on care quality or patient outcomes. Vendors offer efficiency claims and customer testimonials, but that is not the same as clinical evidence. The gap between what these tools promise and what anyone can verify is significant.
What does algorithmic bias mean in a healthcare context?
It means an AI tool can perform well for some patient populations and poorly for others, depending on the data used to train it. If a model was built mostly using data from large urban hospitals, it may not work as well at a rural clinic serving a different demographic. The danger is that this gap often goes undetected because most health systems lack the tools or resources to monitor performance across different patient groups after a tool is deployed.
How is AI in healthcare currently regulated in the United States?
The FDA oversees AI tools that qualify as medical devices, which covers most diagnostic tools and clinical decision support systems. But a large share of AI used in health systems today falls outside that definition. The 21st Century Cures Act specifically excludes AI used for administrative tasks, general wellness, and many EHR functions from FDA oversight. Consumer-facing health apps are regulated by the FTC, which focuses on privacy and false advertising rather than clinical effectiveness. The result is a patchwork with significant gaps.
What should a health system ask before deploying a new AI tool?
Start with the basics. Has this tool been tested in a setting similar to ours, with data on patient outcomes rather than just cost savings? Does the vendor provide ongoing monitoring and performance reporting after deployment? Are all implementation costs accounted for, including training, infrastructure, and maintenance, not just the license fee? And if the tool handles patient data, has a Business Associate Agreement been signed? These questions will not eliminate risk, but they will surface the gaps that most procurement processes miss.
