New York City leaders and small business owners are learning about the pitfalls of relying on AI the hard way.
According to a report from the Daily Wire, the city introduced an AI-powered chatbot in October 2023 as a “one-stop shop” for assisting business owners with government operations. The objective was to answer questions posed by small business owners as they pertain to city laws and codes.
However, the chatbot has been giving out illegal – and rather absurd – advice, including telling employers it’s acceptable to take workers’ tips and fire pregnant workers, while advising landlords to discriminate based on income.
According to tests run by the Markup, the mistakes cover a range of topics, including housing policy, worker rights and regulations for businesses.
Errors include telling landlords they can lock out tenants and ignore income sources like Section 8 vouchers, despite these actions being illegal in New York City. The chatbot’s inaccuracies extend to other areas, wrongly informing businesses they can operate without accepting cash and misguiding them on worker tip regulations.
Some of the AI’s responses bordered on absurdity. For example, one user testing the product was informed it was legal to serve restaurant patrons cheese that had been nibbled on by rodents.
The chatbot, powered by Microsoft’s Azure AI, was expected to be a reliable source for navigating city regulations and avoiding penalties. However, its failure to provide correct information has raised concerns about its impact on small business compliance and legal standing.
Granted, like other similar ai tools, New York City’s chatbot offers a disclaimer that it can “occasionally produce incorrect, harmful or biased” information. It also includes the caveat that its answers are not legal advice. Many, however, wonder if that’s enough warning to continue employing the tool.
But NYC leaders are sticking with the program, no matter how flawed it might be.
“Anyone that knows technology knows this is how it’s done,” New York City Mayor Eric Adams said at a press conference. “Only those who are fearful sit down and say, ‘Oh, it is not working the way we want, now we have to run away from it all together.’ I don’t live that way.”
Experts disagree, however.
“They’re rolling out software that is unproven without oversight,” Julia Stoyanovich, a computer science professor and director of the Center for Responsible AI at New York University, told the Associated Press. “It’s clear they have no intention of doing what’s responsible.”
For the time being at least, the chatbot is still available to NYC small business owners. The question remains, though: Should they trust it? Like most other AI-based tools, it’s best to proceed with caution.
