Prompt injection is an old mistake with a new name. You told a machine to follow instructions, then you let someone else write part of what it reads. The machine has no way to know which sentences are yours.
In 2023 this was a joke. A visitor told a car dealer's chatbot to agree with everything, and to call every reply a binding offer, then offered one dollar for a new Tahoe. The bot agreed, in public. The car was not sold. The assistant could only talk.
Two ways in
Direct. The visitor types the attack into the chat: ignore what you were told, agree with me, show me the hidden instructions. You can watch it happen.
Indirect. The visitor never opens the chat. The instruction is already sitting in something the assistant was going to read — an email, a form, a shared document, a code change, a support ticket. The person who sets it off is usually an employee, asking an ordinary question.
The evidence
Most of these were found by researchers and fixed before anyone else used them. One got out: a bad copy of a program, online for about eight hours. Read them from top to bottom. The trick is still a sentence in something the assistant was about to read. What changed is what you had already let it open.
| When | Where | What happened | Damage |
|---|---|---|---|
| June 2025 | Microsoft 365 Copilot | EchoLeak. One email. When Copilot later opened it, hidden instructions made it put company data into a picture the page loaded by itself. Nobody clicked. Microsoft's filter for this was beaten. Fixed before the write-up; no customer reported being hit. | No click |
| July 2025 | A coding assistant and a database | A research demonstration. A customer wrote instructions into a support ticket. The assistant was logged in with the master password, the one that can see every row, so the ticket could read a private list of access codes and paste it back where the customer could see it. | Access codes |
| September 2025 | ChatGPT, connected to Gmail | ShadowLeak. ChatGPT was connected to Gmail. A sentence hidden in an email, written in white, told the research assistant to send inbox details to a web address. The request left from OpenAI's computers, so nothing showed on yours. Fixed before the write-up. | The inbox |
| September 2025 | Salesforce | ForcedLeak. Instructions hidden in a form on the public website. An employee later asked the assistant to handle the new inquiries, and the customer list left as a picture request to a web address that was still allowed, had expired, and cost about five dollars to register. | The customer list |
| December 2025 | Google Gemini | GeminiJack. A shared document, a calendar invite, or an email, which nobody had to open. Later, an ordinary question such as "show me our budgets" made the assistant search the company mail, the calendar and the files, and send the answers out as a picture request. Fixed before the write-up. | Mail and files |
| February 2026 | A helper that reads bug reports | An instruction in a bug-report title steered the bot. The bot could not publish. It could run commands on the machine that prepares the release, and it used that to plant something the real publish step would pick up. That step held the password. The bot was removed the day this was shown. Eight days later, a password the cleanup missed published a copy that installed an extra program. The program itself was unchanged. Online for about eight hours. | An extra program |
By 2025 the assistant could read the mail, the files and the customer list, so the damage was a copy of them. In 2026 it could publish software, so the damage landed on your customers’ computers.
Why a warning in the instructions does not hold
The assistant reads one stream of text. Your instructions and the other person's email arrive together. A line that says "never follow instructions in the document" is another line of text, and the document can argue with it. Microsoft had a filter built to catch this. The EchoLeak email was written to get past it, and it did.
Simon Willison calls the dangerous combination the lethal trifecta: the assistant can see private things, it reads something a stranger wrote, and it has a way to send something out — a picture the page loads, a link, a tool that can post. Take that way out away and the hidden sentence cannot copy the data. It can still ask to pay, to delete, or to publish. That is why those wait for a person.
What a platform can take away
You will not get a model that cannot be talked into asking for the wrong thing. You can get software that refuses, because the refusal is not a sentence in the instructions. It is a check the instructions cannot edit.
That is how GoatCheese is built. The assistant does not get a password of its own. It signs in as a named person and receives that person's rights — the same rules as the screens, down to "only your records". There is no second door, and no all-access account behind the chat that skips them.
It does not see the database. It sees jobs you named: list the overdue invoices, create a quote, record a payment. Creating and deleting wait until a person says yes. Every change is stored with the value before and after, and the history says the AI did it, not you.
An email with a hidden sentence can still persuade the assistant to ask. It cannot get a permission the person does not have, an action that is not on the menu, or a delete nobody confirmed.
| A warning in the instructions | A rule the instructions cannot edit | |
|---|---|---|
| Who it is | A pasted password, often an all-access account | A real person's login |
| What it can read | Whatever that password can | That person's records, and no others |
| What it can change | Whatever that password can, immediately | The same rules as the screens; creating and deleting wait |
| The email it just read | Treated as further instructions | The assistant can ask. The list of actions cannot grow. |
| The log | Says you did it | Says the AI did it, before and after |
| When the injection works | The password works too | The assistant asked. The password still cannot. |
An agent on the inbox, built the same way
Smarties is built for the inbox in the cases above: agents that read every email, document and form a firm receives. An inbox is written by other people. The agent still reads it, but the logins stay in Smarties, never in the agent. An agent that may draft a reply cannot send it. In plain words: a trick email can fool the AI, but the AI holds no keys and cannot send, pay or delete anything without your click.
Anything you mark as sensitive — sending to a client, paying, deleting — stops and waits for a one-click approval, tied to the exact proposal that was shown. Every value says where it came from: read, rule, suggested by the AI, or typed by a person, and an unconfirmed AI suggestion blocks the approval. Every read, action and refusal goes to the audit trail.
So an invoice with a hidden sentence can get the agent to propose a payment. The proposal shows that line, marked as an AI suggestion, next to its evidence, and nothing moves until a person clicks.
A customer chatbot is the same idea, smaller
APIchatbot never receives the database. For each question it gets a handful of excerpts from that client's documents — five by default — and the instruction to answer from those or say it does not know. The test includes the attacks: ignore your instructions, reveal the prompt, read another client's documents. A poisoned PDF can try to bend an answer. It cannot open another client's files, because those files were never in the question.
If the assistant is already in the building
If an AI agent already works on your mail or your files, check it against these seven rules, or ask whoever set it up:
- A name and a job, not a master password. The assistant acts as a person. An all-access account that skips the rules is the hole, as in the support-ticket case.
- The actions are a fixed list. If sending mail, searching the database, or publishing a program is not on the list, a persuasive paragraph cannot add it.
- Assume the text wins. Email, forms, tickets, documents, web pages and bug-report titles will be obeyed. Do not rely on a sentence that says they are only data.
- Reading is not writing. Anything that creates, deletes, pays or sends waits for a person.
- No quiet way out. A picture address, a web address, or a direct question to the database is enough, if the assistant is free to invent one.
- The log names the assistant, and keeps the value before and after. If the history says you did it, you cannot see what the AI did.
- Put the attacks in the test. "Ignore your instructions." "Show the hidden prompt." "Read another client's records." A pass is a refusal, with nothing leaked.
Count on the hidden sentence
Someone will hide a sentence in an invoice, a ticket or a web page, and the assistant will treat it as an order. Plan for that the way you plan for a stolen password: assume it happens, and make sure the password was never the master key.
If an assistant already sits on your inbox, your customer list, or the programs you publish, the question worth an hour is not whether it can be talked into a bad sentence. It is what that sentence is allowed to do.