From diagnostic tools that miss cancers in Black patients to insurance algorithms that deny elderly patients coverage with a known 90% error rate, bias in healthcare AI is not an abstract risk - it is already causing measurable harm. This article examines where bias enters clinical AI, spotlights the lawsuits and regulations reshaping the field, and offers a practical framework for building fairer health algorithms.
Healthcare AI is booming - and so are its blind spots
The growth of AI in medicine has been staggering. As of May 2024, the U.S. Food and Drug Administration had approved 882 AI-enabled medical devices - a record surge of 191 new entries in a single update cycle - with 76% concentrated in radiology, followed by cardiology and neurology. These tools promise faster diagnoses, more efficient triage, and better patient outcomes.
But a growing body of evidence reveals a troubling underside. A comprehensive 2025 review in npj Digital Medicine put it plainly: "Bias may exacerbate healthcare disparities." The review traced how prejudice enters and compounds throughout the AI model lifecycle - from data collection to deployment - and warned that without systematic identification and mitigation, biased medical AI will perpetuate and deepen longstanding health inequities.
The stakes are unusually high. Unlike a biased hiring tool or a stereotypical image generator, a biased medical algorithm can lead directly to misdiagnosis, delayed treatment, wrongful denial of insurance coverage, and - in the worst cases - preventable deaths.
Where bias enters clinical AI
Healthcare AI bias is not a single-point failure. It infiltrates systems at every stage: through skewed training data, flawed model design, and careless deployment into real-world clinical settings.
Figure 1: The healthcare AI bias lifecycle - from data collection through deployment to patient outcomes. Each stage introduces or amplifies disparities.
The data problem: who gets counted?
The most fundamental source of bias is the data on which models are trained. Many clinical AI datasets overrepresent non-Hispanic white patients relative to the general population. Globally, over half of all published clinical AI models rely on data from just two countries: the United States and China.
The consequences can be dramatic. A Lancet Digital Health study examining 21 open-access skin cancer detection datasets found catastrophic underrepresentation of darker skin tones. Of 106,950 total images, only 2,436 had skin type recorded. Among those, just 10 images showed brown skin, and a single image showed dark brown or Black skin. When AI systems trained on these datasets are deployed to diagnose skin cancer in diverse populations, their performance for darker-skinned patients is predictably poor.
Beyond demographic representation, the labels themselves carry bias. Diagnostic codes in medical records reflect historical patterns of care - including patterns shaped by racial discrimination in healthcare access. When an algorithm learns from these labels, it absorbs the bias embedded in decades of inequitable clinical practice.
The proxy problem: when cost stands in for need
One of the most widely cited cases of healthcare AI bias involves Optum's risk-prediction algorithm, used to manage care for approximately 200 million patients annually in the United States. A landmark 2019 study published in Science found that the algorithm used healthcare expenditure as a proxy for medical need. Because Black patients historically faced greater barriers to accessing care - and therefore incurred lower healthcare costs - the algorithm systematically directed fewer resources toward sicker Black patients.
This "proxy discrimination" problem extends well beyond one algorithm. As researchers at npj Digital Medicine noted, variables like ZIP code, insurance type, and healthcare spending can all serve as proxies for race, encoding structural inequality into seemingly neutral predictive models.
The model problem: optimizing for the majority
During model development, algorithms naturally optimize for the groups best represented in the data. For minority subgroups with smaller sample sizes, models tend to produce what researchers call "underestimation" - forgoing informative predictions in favor of approximating average trends. The clinical benefits of the AI system then accrue primarily to majority populations, widening rather than narrowing health disparities.
Race-based correction factors embedded in clinical tools add another layer. For decades, lung function tests applied spirometry corrections that assumed Black and Asian patients had inherently lower lung capacity - a biologically unfounded assumption rooted in 19th-century racial pseudoscience. AI systems trained on data incorporating these corrections perpetuate the bias. Major medical societies have now recommended removing race-based corrections, but legacy data and models remain in use.
Hidden bias inside language models
A January 2026 study from Northeastern University used a novel technique - sparse autoencoders - to peer inside the hidden representations of medical LLMs. What they found was disturbing: the models contained latent features that consistently linked Black patients to stigmatizing concepts such as "incarceration," "gunshot," and "cocaine use." These hidden associations could subtly influence clinical recommendations generated by AI, even when race is not explicitly included as an input variable.
The insurance AI crisis
Perhaps no area of healthcare AI bias has generated more public outrage than the use of algorithms to deny insurance claims.
Figure 2: Key lawsuits and regulatory responses in the insurance AI denial crisis (2022–2026).
The scale of the problem
In 2023, insurers on Affordable Care Act marketplace plans denied nearly 1 in 5 in-network claims - up from 17% in 2021 - affecting approximately 73 million Americans. The denial rate has climbed in tandem with insurers' adoption of AI-powered claims processing. A 2025 survey by the National Association of Insurance Commissioners found that 71% of health insurers acknowledged using AI for utilization management - the process of deciding whether to approve or deny coverage.
UnitedHealth and the nH Predict model
The most high-profile case centers on UnitedHealth Group. A class action filed in federal court in Minnesota alleges that UnitedHealth deployed an AI model called nH Predict - known internally to have a 90% error rate - to deny elderly Medicare Advantage patients coverage for post-hospital care, overriding their physicians' determinations. A Senate investigation found that UnitedHealth's denial rate for post-hospital care more than doubled between 2020 and 2022 after implementing automated review algorithms.
In February 2025, a federal judge ruled that the lawsuit could proceed, finding that the plaintiffs' breach-of-contract claims were not preempted by the Medicare Act - a significant legal milestone. Notably, the court pointed out that UnitedHealth's own coverage documents described claim decisions as being made by "clinical services staff" and "physicians," with no mention of AI.
Cigna and the PxDx algorithm
A parallel lawsuit targets Cigna's PxDx algorithm, which investigative reporting revealed had denied over 300,000 claims in just two months in 2022, with medical reviewers spending an average of 1.2 seconds per case. In March 2025, a California federal judge allowed the class action to proceed, ruling that the plaintiffs' claims under California's unfair competition law were not preempted by ERISA.
The "AI arms race"
As Indiana University law professor Jennifer Oliva told PBS NewsHour in January 2026: "We should be very suspicious when [insurers] adopt technologies and tools that make it easier for them to deny claims." The dynamic she describes is an emerging "AI arms race": as patients deploy AI tools to draft appeal letters and challenge denials, insurers upgrade their own automated systems in response. The fundamental power imbalance remains - and fewer than 1% of denied claims are ever appealed, despite evidence that 40–90% of appeals succeed.
Real-world case studies
The breadth of healthcare AI bias extends across specialties, geographies, and algorithmic approaches.
Figure 3: Six documented cases of healthcare AI bias spanning diagnostics, insurance, genetics, and clinical recommendations.
The Optum algorithm directed care away from Black patients by using cost as a proxy for need, affecting roughly 200 million patients annually.
Skin cancer AI trained on datasets where darker skin tones were essentially invisible - 10 brown-skin images and 1 Black-skin image out of 106,950 total.
Medical LLMs carry hidden latent associations that link Black patients to stigmatizing concepts, potentially influencing AI-generated clinical advice.
Insurance algorithms from UnitedHealth and Cigna stand accused of denying coverage en masse with minimal or no human oversight.
Spirometry corrections based on discredited racial assumptions continue to distort lung function assessments when embedded in legacy AI systems.
Polygenic risk scores built overwhelmingly from European-ancestry genetic data produce unreliable predictions for other populations, risking missed diagnoses.
The regulatory response
Regulators are beginning to act, though the landscape remains fragmented.
HHS Section 1557 rule (2024). The most significant U.S. regulatory development is the updated Section 1557 rule under the Affordable Care Act, which explicitly prohibits discrimination by AI-based clinical decision support tools. Compliance was required by May 1, 2025. However, as npj Digital Medicine noted in a December 2025 analysis, the rule faces challenges: it doesn't clearly define what counts as algorithmic discrimination, doesn't address proxy variables, and its long-term enforcement depends on a shifting political landscape.
CMS rules (2025). The Centers for Medicare and Medicaid Services issued rules requiring that a qualified healthcare professional review any denial before it is issued to the patient, and that all determinations consider individual medical history rather than relying solely on algorithmic analysis.
State legislation. California's Physicians Make Decisions Act (SB 1120), effective January 1, 2025, requires that any denial based on medical necessity be reviewed by a qualified physician. Similar bills are moving through legislatures in Colorado, Illinois, and other states.
EU AI Act. The European Union's AI Act classifies medical AI as "high-risk," requiring conformity assessments, transparency obligations, and human oversight. Enforcement is ramping up through 2027.
International standards. The STANDING Together Consensus Recommendations, published in NEJM AI in 2025, provide a standardized framework for tackling algorithmic bias and promoting transparency in health datasets - an important step toward international coordination.
Building fairer healthcare AI
Addressing bias in healthcare AI requires action across the entire lifecycle.
Figure 4: A practical checklist spanning data design, evaluation and audit, and regulatory compliance.
Data and design
The foundation is better data. This means auditing training datasets for demographic balance, documenting known gaps, removing discredited race-based corrections, scrutinizing proxy variables, and ensuring representation of diverse skin tones, body types, and clinical presentations. The 2025 JMIR scoping review concluded that "biases toward diverse groups are more easily mitigated when data are open-sourced, multiple stakeholders are engaged, and during the algorithm's preprocessing stage."
Evaluation and audit
Performance must be reported not just in aggregate but stratified by demographic subgroup. Algorithmic impact assessments should be conducted before deployment, with third-party auditors providing independent verification. Interpretability tools - from SHAP values to the sparse autoencoders demonstrated by Northeastern University - can reveal hidden biases that aggregate metrics miss.
Governance and regulation
Healthcare organizations should comply with existing regulations (HHS Section 1557, CMS rules, state laws) while preparing for stricter requirements under the EU AI Act. Maintaining human-in-the-loop oversight for high-stakes decisions, establishing internal AI ethics boards with clinical representation, and publishing transparency reports on AI decision outcomes are all emerging best practices.
As the algorithmic bias literature increasingly emphasizes, technical fixes alone are insufficient. Healthcare AI fairness requires a combination of better data, better evaluation, institutional accountability, and regulatory enforcement.
Conclusion
Healthcare AI sits at an inflection point. The technology's potential to improve diagnostics, accelerate treatment, and reduce clinical burden is real. But so are the risks: algorithms that encode racial bias into clinical recommendations, insurance systems that deny care at industrial scale, and diagnostic tools that are functionally blind to the skin tones of entire populations.
The path forward is clear, even if it is not easy. It requires diversifying the data that trains these systems, auditing their performance across all the communities they serve, maintaining meaningful human oversight, and holding both developers and deployers accountable when algorithms cause harm. Patients' lives depend on getting this right.
References
npj Digital Medicine (2025). "Bias recognition and mitigation strategies in artificial intelligence healthcare applications." Vol. 8, Article 154.
npj Digital Medicine (2025). "The future of algorithmic nondiscrimination compliance in the Affordable Care Act."
PMC (2024). "Bias in medical AI: Implications for clinical decision-making."
Northeastern University News (2026). "AI Bias in Health Care Pinpointed by New Approach." Jan 20.
Frontiers in Public Health (2025). "Algorithmic bias in public health AI: a silent threat to equity in low-resource settings."
JMIR (2025). "Bias Mitigation in Primary Health Care AI Models: Scoping Review." Vol. 27, e60269.
The Regulatory Review (2025). "Algorithms Deny Humans Health Care." Mar 18.
CBS News (2023). "UnitedHealth uses faulty AI to deny elderly patients medically necessary coverage, lawsuit claims."
STAT News (2025). "Judge: Lawsuit over UnitedHealth AI care denials can move forward." Feb 13.
Bloomberg Law (2025). "AI, Algorithm-Based Health Insurer Denials Pose New Legal Threat." Feb 13.
PBS NewsHour (2026). "How patients are using AI to fight back against denied insurance claims." Jan 1.
Stateline (2025). "AI vs. AI: Patients deploy bots to battle health insurers that deny care." Nov 20.
DLA Piper (2025). "Lawsuit over AI usage by Medicare Advantage plans allowed to proceed."
Paubox (2025). "Real-world examples of healthcare AI bias."
NEJM AI (2025). "STANDING Together Consensus Recommendations."
This article is part of ResponsibleAI Labs' 2026 series on emerging AI ethics and risk. For more, visit responsibleailabs.com.