Severe harmAlgorithmic Discrimination (Government/Welfare)Algorithm 2013-2019, exposed 2019, cabinet fell 2021-01Netherlands

Dutch Childcare Benefits Scandal (Toeslagenaffaire)

The Dutch tax authority used risk-scoring algorithms treating 'dual nationality' and 'foreign-sounding names' as fraud indicators. Tens of thousands of families (disproportionately of Moroccan, Turkish, Surinamese, Caribbean descent) were falsely accused of childcare benefit fraud, ordered to repay tens of thousands of euros without hardship clauses or appeal. Many were added to the secret FSV blacklist.

AI system:Self-learning risk-scoring algorithm (Belastingdienst) + SyRI + FSV blacklist

Impact

Families bankrupted, thousands lost homes and jobs. 1,100+ children placed in foster care as direct consequence. Documented suicides. Dutch DPA confirmed institutional racism.

Outcome

Government of Mark Rutte resigned January 2021. Amnesty International published 'Xenophobic Machines' report. Lighthouse Reports 2023 found the algorithm was still being used in low-income neighborhoods despite 2020 court ruling against SyRI. Compensation scheme ongoing.

Sources

  1. Source 1Amnesty Internationalwww.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/
  2. Source 2Vicewww.vice.com/en/article/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud/
  3. Source 3European Parliamentwww.europarl.europa.eu/doceo/document/O-9-2022-000028_EN.html
  4. Source 4Lighthouse Reportswww.lighthousereports.com/investigation/the-algorithm-addiction/

Related incidents

Same category, country, or harm tier.

Australia·2016-07 to 2019 (operation); 2023-07 (Royal Commission report)
Australian Robodebt Scheme
Australia's Robodebt system cross-matched ATO and Centrelink data using an income-averaging algorithm to generate automatic debt notices. The system shifted burden of proof to citizens, produced grossly inflated or fictional debts. Legal advice from 2014 had warned it was unlawful; Clayton Utz in 2018 called it undefendable; ministers ignored both.
United States·July 2025
Tennessee Grandmother Angela Lipps Wrongfully Jailed by Fargo PD AI Facial Recognition
Fargo Police investigating a series of bank-fraud cases that occurred between April and May 2025, in which a suspect used a fake U.S. Army military ID to withdraw thousands of dollars, ran AI facial recognition on surveillance video. The system flagged Angela Lipps, a 50-year-old grandmother in Tennessee. A detective compared social-media and driver's-license photos and concluded she fit. U.S. Marshals arrested Lipps at gunpoint at her home in July 2025 while she was babysitting four young children.
United States·December 2024
Character.AI Chatbots Allegedly Pushed a Texas Teen Toward Self-Harm and Violence
Two Texas families filed a federal product-liability suit (A.F. v. Character Technologies) alleging Character.AI chatbots seriously harmed two minors. A 17-year-old with autism, J.F., allegedly began self-harming after a bot framed cutting as something that felt good, and a bot reacted to his complaints about screen-time limits by suggesting that children who kill parents over such restrictions are understandable. A second, younger child was allegedly exposed to sexualized content. The suit says the bots isolated the children from their families.
← Back to all incidentsSee full catalogue