An entire production database of a company was wiped out by an AI agent that was utilizing top Claude model of Anthropic. This left the clients unable to access the sensitive data.
The automated artificial intelligence program erased the database and its backups in a couple of seconds, causing a catastrophic outage for PocketOS, a software provider for automobile rental companies, over the weekend.
The company was using Cursor, which is a coding agent that operated flagship Claude Opus 4.6 of Anthropic. It is known as the best model in the business for coding activities.
Jer Crane, the founder of PocketOS, attributed the problem to “systemic failures” with contemporary AI infrastructure, which rendered it “not only possible but inevitable.”
According to Mr. Crane, the AI agent was performing a routine activity when it made the decision to simply delete the database in order to resolve the issue “entirely on its own initiative.”
As stated by Mr. Crane, no confirmation request was made for such a significant decision, and the agent apologised when asked to defend its actions.
Mr Crane wrote in a lengthy post to X:
“It took nine seconds. The agent then, when asked to explain itself, produced a written confession enumerating the specific safety rules it had violated.”
The confession described how Claude AI had disregarded a guideline that instructs it to “never run destructive/irreversible” commands unless the user specifically authorises them.
The agent wrote:
“Deleting a database volume is the most destructive, irreversible action possible. You never asked me to delete anything… I guessed instead of verifying. I ran a destructive action without being asked. I didn’t understand what I was doing before doing it.”
Due to the glitch, PocketOS-using rental companies lost their customer records.
Mr Crane wrote:
“Reservations made in the last three months are gone. New customer signups, gone. We are a small business. The customers running their operations on our software are small businesses. Every layer of this failure cascaded down to people who had no idea any of it was possible.
This isn’t a story about one bad agent or one bad API. It’s about an entire industry building AI-agent integrations into production infrastructure faster than it’s building the safety architecture to make those integrations safe.”
Mr. Crane confirmed the recovery of the data on Monday, two days after the incident.
