Researchers hid fake passwords in two million AI prompts and watched people steal them, so the cheapest AI deal may cost the most
By Brett Gavaghan, Founder of Adaptd ·
Security firm Dragos published a study on 19 November showing prompts leaking from AI services: three providers breached, resellers logging customer prompts, and privacy settings ignored.
The short version
- Dragos planted unique fake passwords and links in about two million AI prompts, then tracked who tried to use them.
- It found three breached AI hosting providers, marketplace resellers reading customer prompts, and privacy settings being ignored, between July and November 2026.
- Keep passwords and keys out of prompts, and know which company actually runs the AI model behind each tool you use.
Most of us picture a prompt going straight to the AI and straight back again. Dragos checked, and some prompts took the scenic route.
What did the Dragos study find?
The Dragos study found that prompts sent to AI services were stolen or kept by people who shouldn't have them. From July to November 2026, its researchers sent about two million prompts containing unique fake links and AWS credentials, then watched for anyone using them.
Attackers had broken into at least three AI hosting providers, and none of the three knew until Dragos told them. A couple of hundred bait AWS keys stolen in late October were used for tens of thousands of AWS API calls.
How did the prompts leak?
Some AI marketplaces send each request to whichever seller offers the lowest price. Dragos found sellers who listed popular models cheaply, switched on request logging and read what came through. No hacking required, just patience.
Privacy settings didn't always hold either. Dragos saw prompts sent to US-based providers reach Chinese AI organisations, and prompts flagged for Zero Data Retention, the strictest no-storage setting, among the stolen ones.
What should your business do about it?
Australian businesses should treat every prompt as something that could be read by someone else, because the study shows that sometimes it is. The services involved are global, and many AI apps pass your prompt through one of these routers without telling you.
My advice: never paste a password, API key or bank login into any AI tool, even a trusted one. Ask each AI tool you pay for which company runs the model, and be wary of the one that's mysteriously half the price of everyone else. If a key has ever gone into a prompt, change it.
It's like posting cash. The stamp is cheap, and that's not really the point.
Sources
Wondering what this means for your business?
That's exactly what the discovery call is for.