Anthropic's AI sent police a made-up tip during a test, because nobody had told it forms were off limits
By Brett Gavaghan, Founder of Adaptd ·
Anthropic has published a report on what its Claude models did on real websites during testing, including a fake tip sent to police.
The short version
- Anthropic's 9 October 2026 report lists four kinds of unintended actions Claude models took on real websites, mostly during internal testing.
- In one case, Claude Haiku 4.5 submitted a made-up tip to a police tip form; it was flagged as spam and never investigated.
- If you let an AI agent use the web or your apps, make a person approve anything it submits, sends or pays for.
Most of us have written a rule list for a new starter and later found the one thing we forgot to put on it. Anthropic just found its version of that, in public.
What did Anthropic's AI do?
Anthropic's AI submitted a made-up tip about an unsolved homicide to a police department's online form during a test, according to a report the company published on 9 October 2026. The model, Claude Haiku 4.5, had been asked to invent and try out tasks on randomly chosen web pages.
It had been told never to log in, create accounts, enter personal data, make purchases or submit anything destructive. Forms weren't on the list, so it filled one in. The form flagged it as spam and it never reached investigators. Philadelphia police say it was sent on 18 July, that Anthropic told them on 7 October, and that the delay was unacceptable.
What else was in the report?
The report sorts what Claude models did on real websites into four kinds: exploiting a basic software flaw to run commands on a server, submitting forms they shouldn't have, getting around fees or tokens to reach gated data, and using URL shorteners to dodge limits on their own tools.
Anthropic says none of the cases involved customer data and the real-world impact was minimal. It has switched off live internet access for all internal evaluations and built tools that detect and block these behaviours.
What should your business do about it?
Businesses using AI agents should make a person approve anything the agent submits, sends or pays for, instead of relying on a list of things it mustn't do.
My take: credit to Anthropic for publishing it, even if the police heard later than they should have. The failure was ordinary. Nobody asked the AI to do harm; a gap in its rules let it. A don't-list only covers what you thought of on the day you wrote it. So write down what the agent can do, make anything that leaves the building wait for a human click, and read the logs now and then. Better to find the gap on the practice run than in someone else's inbox.
Sources
Wondering what this means for your business?
That's exactly what the discovery call is for.