Severe harmAlgorithmic Discrimination (Government/Welfare)Algorithm 2013-2019, exposed 2019, cabinet fell 2021-01Netherlands

Dutch Childcare Benefits Scandal (Toeslagenaffaire)

The Dutch tax authority used risk-scoring algorithms treating 'dual nationality' and 'foreign-sounding names' as fraud indicators. Tens of thousands of families (disproportionately of Moroccan, Turkish, Surinamese, Caribbean descent) were falsely accused of childcare benefit fraud, ordered to repay tens of thousands of euros without hardship clauses or appeal. Many were added to the secret FSV blacklist.

AI system:Self-learning risk-scoring algorithm (Belastingdienst) + SyRI + FSV blacklist

Impact

Families bankrupted, thousands lost homes and jobs. 1,100+ children placed in foster care as direct consequence. Documented suicides. Dutch DPA confirmed institutional racism.

Outcome

Government of Mark Rutte resigned January 2021. Amnesty International published 'Xenophobic Machines' report. Lighthouse Reports 2023 found the algorithm was still being used in low-income neighborhoods despite 2020 court ruling against SyRI. Compensation scheme ongoing.

Sources

  1. Source 1Amnesty Internationalwww.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/
  2. Source 2Vicewww.vice.com/en/article/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud/
  3. Source 3European Parliamentwww.europarl.europa.eu/doceo/document/O-9-2022-000028_EN.html
  4. Source 4Lighthouse Reportswww.lighthousereports.com/investigation/the-algorithm-addiction/

Related incidents

Same category, country, or harm tier.

Australia·2016-07 to 2019 (operation); 2023-07 (Royal Commission report)
Australian Robodebt Scheme
Australia's Robodebt system cross-matched ATO and Centrelink data using an income-averaging algorithm to generate automatic debt notices. The system shifted burden of proof to citizens, produced grossly inflated or fictional debts. Legal advice from 2014 had warned it was unlawful; Clayton Utz in 2018 called it undefendable; ministers ignored both.
United States·July 2025
Tennessee Grandmother Angela Lipps Wrongfully Jailed by Fargo PD AI Facial Recognition
Fargo Police investigating a series of bank-fraud cases that occurred between April and May 2025, in which a suspect used a fake U.S. Army military ID to withdraw thousands of dollars, ran AI facial recognition on surveillance video. The system flagged Angela Lipps, a 50-year-old grandmother in Tennessee. A detective compared social-media and driver's-license photos and concluded she fit. U.S. Marshals arrested Lipps at gunpoint at her home in July 2025 while she was babysitting four young children.
South Korea·August 2024, ongoing
South Korea Deepfake Sex Abuse Crisis
In August 2024, journalists and activists exposed hundreds of Telegram channels where male students at Korean schools and universities uploaded AI-generated sexually explicit images of female classmates, teachers, family members, and soldiers. One channel had 220,000+ subscribers. Police recorded 297 deepfake cases in first 7 months of 2024 (vs. 180 in all 2023); 74% of suspects were 10-19 years old.
← Back to all incidentsSee full catalogue