On Monday, a developer utilizing the favored AI-powered code editor Cursor observed one thing unusual: Switching between machines immediately logged them out, breaking a typical workflow for programmers who use a number of gadgets. When the consumer contacted Cursor assist, an agent named “Sam” instructed them it was anticipated habits underneath a brand new coverage. But no such coverage existed, and Sam was a bot. The AI mannequin made the coverage up, sparking a wave of complaints and cancellation threats documented on Hacker News and Reddit.
This marks the newest occasion of AI confabulations (additionally referred to as “hallucinations”) inflicting potential enterprise harm. Confabulations are a sort of “creative gap-filling” response the place AI fashions invent plausible-sounding however false data. Instead of admitting uncertainty, AI fashions usually prioritize creating believable, assured responses, even when meaning manufacturing data from scratch.
For corporations deploying these programs in customer-facing roles with out human oversight, the results could be instant and expensive: pissed off prospects, broken belief, and, in Cursor’s case, probably canceled subscriptions.
How It Unfolded
The incident started when a Reddit consumer named BrokenToasterOven observed that whereas swapping between a desktop, laptop computer, and a distant dev field, Cursor periods had been unexpectedly terminated.
“Logging into Cursor on one machine immediately invalidates the session on any other machine,” BrokenToasterOven wrote in a message that was later deleted by r/cursor moderators. “This is a significant UX regression.”
Confused and pissed off, the consumer wrote an e mail to Cursor assist and shortly acquired a reply from Sam: “Cursor is designed to work with one device per subscription as a core security feature,” learn the e-mail reply. The response sounded definitive and official, and the consumer didn’t suspect that Sam was not human.
After the preliminary Reddit submit, customers took the submit as official affirmation of an precise coverage change—one which broke habits important to many programmers’ each day routines. “Multi-device workflows are table stakes for devs,” wrote one consumer.
Shortly afterward, a number of customers publicly introduced their subscription cancellations on Reddit, citing the non-existent coverage as their purpose. “I literally just cancelled my sub,” wrote the unique Reddit poster, including that their office was now “purging it completely.” Others joined in: “Yep, I’m canceling as well, this is asinine.” Soon after, moderators locked the Reddit thread and eliminated the unique submit.
“Hey! We have no such policy,” wrote a Cursor consultant in a Reddit reply three hours later. “You’re of course free to use Cursor on multiple machines. Unfortunately, this is an incorrect response from a front-line AI support bot.”
AI Confabulations as a Business Risk
The Cursor debacle recollects an analogous episode from February 2024 when Air Canada was ordered to honor a refund coverage invented by its personal chatbot. In that incident, Jake Moffatt contacted Air Canada’s assist after his grandmother died, and the airline’s AI agent incorrectly instructed him he may guide a regular-priced flight and apply for bereavement charges retroactively. When Air Canada later denied his refund request, the corporate argued that “the chatbot is a separate legal entity that is responsible for its own actions.” A Canadian tribunal rejected this protection, ruling that corporations are chargeable for data supplied by their AI instruments.
Rather than disputing duty as Air Canada had completed, Cursor acknowledged the error and took steps to make amends. Cursor cofounder Michael Truell later apologized on Hacker News for the confusion concerning the non-existent coverage, explaining that the consumer had been refunded and the difficulty resulted from a backend change meant to enhance session safety that unintentionally created session invalidation issues for some customers.
“Any AI responses used for email support are now clearly labeled as such,” he added. “We use AI-assisted responses as the first filter for email support.”
Still, the incident raised lingering questions on disclosure amongst customers, since many individuals who interacted with Sam apparently believed it was human. “LLMs pretending to be people (you named it Sam!) and not labeled as such is clearly intended to be deceptive,” one consumer wrote on Hacker News.
While Cursor mounted the technical bug, the episode exhibits the dangers of deploying AI fashions in customer-facing roles with out correct safeguards and transparency. For an organization promoting AI productiveness instruments to builders, having its personal AI assist system invent a coverage that alienated its core customers represents a very awkward self-inflicted wound.
“There is a certain amount of irony that people try really hard to say that hallucinations are not a big problem anymore,” one consumer wrote on Hacker News, “and then a company that would benefit from that narrative gets directly hurt by it.”
This story initially appeared on Ars Technica.
https://www.wired.com/story/cursor-ai-hallucination-policy-customer-service/