DocumentedAI Hallucination / Misinformation2023-10 (launch); 2024-03 (scandal)United States

NYC MyCity Chatbot Advising Businesses to Break the Law

The MyCity AI chatbot (launched October 2023 as part of NYC's AI Plan) told landlords they could refuse Section 8 vouchers (illegal since 2008), told employers they could take workers' tips (illegal under NY Labor Law 196-d), told businesses they could go cashless (illegal since November 2020), said 'no restrictions on rent' amounts (ignoring rent stabilization), suggested landlords could lock out tenants.

Cost: US$600KAI system:Microsoft Azure AI (MyCity chatbot)

Impact

Legal Services NYC called the advice 'dangerously inaccurate.' Users relying on the bot faced real legal liability.

Outcome

Mayor Adams refused to take bot offline. City added disclaimer but bot continued providing false information. Most-cited US municipal AI failure; accelerated city-level AI governance frameworks.

Sources

  1. Source 1The Markupthemarkup.org/artificial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law
  2. Source 2THE CITYwww.thecity.nyc/2024/03/29/ai-chat-false-information-small-business/
  3. Source 3THE CITY follow-upwww.thecity.nyc/2024/04/02/malfunctioning-nyc-ai-chatbot-still-active-false-information/

Related incidents

Same category, country, or harm tier.

United States·May 2023
NEDA 'Tessa' Eating-Disorder Chatbot Gave Harmful Diet Advice
NEDA suspended its Tessa chatbot after activists, including Sharon Maxwell and clinical psychologist Alexis Conason, documented that it gave weight-loss and calorie-restriction guidance that runs counter to eating-disorder recovery practice. The failure surfaced days after NEDA moved to wind down its human-staffed helpline, whose workers had just unionized, and replace it with the bot.
United States·March 2023 to June 22, 2023
Mata v. Avianca: Lawyers Sanctioned for ChatGPT-Fabricated Cases
In a personal-injury suit against Avianca, attorney Steven Schwartz used ChatGPT for research, and it fabricated 6 nonexistent judicial opinions (Varghese v. China Southern Airlines, Shaboon v. Egyptair, Petersen v. Iran Air, etc.). Schwartz asked ChatGPT directly if 'Varghese' was real; it confirmed and claimed it could be found on Westlaw and LexisNexis. Judge Castel could not locate the citations.
Australia·October 2025
Deloitte Australia Refunds Government for AI-Hallucinated Report
Deloitte Australia agreed to partly refund a roughly 440,000 AUD (about 290,000 USD) report to the federal Department of Employment and Workplace Relations after a Sydney University researcher, Chris Rudge, found fabricated academic citations and a made-up quote from a Federal Court judgment, with a misspelled judge's name. The report reviewed the department's automated welfare-penalty systems. A revised version disclosed that Azure OpenAI had been used.
← Back to all incidentsSee full catalogue