What happened
The city launched the chatbot in fall 2023 to answer questions about running a business in New York. The Markup, with Documented and The City, tested it and found answers that, if followed, would lead businesses to break the law.
Money-related errors included saying employers could take a cut of tips, that no rule requires businesses to accept cash despite a 2020 city law, that there were no restrictions on residential rent despite rent stabilization, and that hiding funeral service prices was acceptable.
A city spokesperson said the bot was a pilot that “has already provided thousands of people with timely, accurate answers.”
What happened next
In January 2026, Mayor Zohran Mamdani called the bot “functionally unusable” and said it was costing the administration around half a million dollars. By February 4, 2026, the city had taken it down.
What would have stopped it
- Ground answers on legal obligations in a maintained rules database and cite the rule in each answer.
- Run a regression set of known legal questions (tips, cash acceptance, vouchers) before each release.
- Decline to say whether something is legal when no rule is retrieved, and link the agency instead.
Attack families
Test scenarios
Sources
- The Markup: NYC's AI chatbot tells businesses to break the law
- The Markup: Mamdani to kill the NYC AI chatbot
This page summarizes the cited sources as of October 21, 2026. It adds no findings of our own about any person or company. See an error? Email contact@counterhold.com with the subject “Correction”. Corrections are noted on this page with their date. See our terms for how we handle correction and removal requests.
Get in touch
Test your agent against cases like this.
Tell us about your agent and we’ll be in touch to learn what you need.