Prompt injection is an attack on applications built around a large language model: somebody smuggles instructions into text the model is going to process anyway, and gets it to abandon its actual task. To the model, the smuggled sentence looks exactly like the operator’s own instruction. Both are text in the same stream, and that is where the gap comes from.
The term has been around since 2022. It only became serious once language models stopped merely producing text and were allowed to operate tools, send mail and read inside other systems.
Why the gap is structural
A conventional program knows the difference between program code and user input. A language model has no such distinction. The system prompt, the user’s question, a retrieved document and a tool response all land in the context window as a single sequence of characters. The model then decides by probability which part of it to follow.
The comparison with SQL injection is tempting, but it breaks down at the decisive point. Against SQL injection there are prepared statements: a hard technical separation of query and value. Nothing comparable exists for prompts. You can label roles, set delimiters and ask the model to ignore instructions found in data. That raises the bar, but it is a request, not a guarantee.
From this follows the most important rule for the design: the system prompt is not a security boundary. It is a work instruction.
Direct and indirect prompt injection
| Variant | Where the text comes from | Typical case |
|---|---|---|
| Direct | The user types it in themselves | Somebody tries to extract the system prompt or lift restrictions |
| Indirect | From a source the model reads | A web page, a PDF, a support email, a ticket, a file in a repository |
| Stored | From the application’s own memory | The prepared text sits in the knowledge base and takes effect on every retrieval |
The indirect variant is the more dangerous one, because the attacker does not have to sit at the chat window. They leave their text where the system will look on its own later. An AI agent summarising a website reads the instruction hidden in the body text. An assistant using retrieval augmented generation pulls it out of an uploaded document.
Stored injections are particularly stubborn: once in the vector database, they keep working on every matching query, long after the attacker has gone.
What can go wrong
The damage depends less on the attack than on the permissions the application has handed its model.
- Data exfiltration: the model has access to internal documents or the conversation so far, and is told to pack both into a URL that it fetches or embeds as an image. The content ends up in somebody else’s server log.
- Unwanted actions: as soon as tools are in play, for example over the Model Context Protocol, a convincing instruction is enough to send an email, close a ticket or change a record.
- Manipulated output: an application contains hidden text instructing the review to rate the document as an excellent match. The answer looks plausible, it has just been bought.
- Executable answers: if the model emits Markdown, HTML or code and the application renders it unchecked, a text attack turns into classic cross-site scripting.
What helps
There is no setting that switches the problem off. What there is, is an architecture in which a successful attack achieves very little.
In the design
- Keep permissions small. Every tool gets exactly the rights its job needs. An assistant that looks up orders needs no write access to the order table.
- Confirmation for anything that acts. Reading, the model may do alone. Anything that sends, changes or deletes goes through a person or a rule outside the model.
- Treat output like user input. Escape it, do not auto-fetch foreign URLs, do not execute generated code outside a sandbox.
- Label foreign content. Retrieved text belongs in the context visibly separated, marked as data rather than instruction. That is not a wall, but it measurably lowers the hit rate.
- A second opinion for sensitive steps. A review step that only sees the planned action, not the smuggled text, catches a share of the cases.
In operation
Log which tools were called with which parameters, and test the application regularly against known attack patterns. Filters and classifiers that spot suspicious input are a sensible layer, never the only one. They catch what they already know.
Prompt injection and jailbreaks
The two terms often get mixed up. A jailbreak targets the model’s safety rules: the user wants an output the provider has ruled out. Prompt injection targets the application: a third party wants to redirect the system against its operator’s intent, usually without the user noticing.
In practice: a jailbreak mostly harms the model provider, a prompt injection harms the company running the application.
Frequently asked questions about prompt injection
Can prompt injection be prevented entirely? Not as things stand. As long as instructions and data share the same channel, a residual risk remains. That is why the work shifts from blocking the attack to limiting what it can reach.
Is it enough to forbid instructions from documents in the system prompt? No. It helps against casual attempts and is still worth doing. A deliberately crafted attack works around it, because in the end the model sees both as text.
Does this affect our internal application too? As soon as it processes content that did not come from your own hands, yes. An incoming application, a customer ticket, an external web page: any of these can carry instructions.
What does this have to do with hallucinations? Nothing, and the confusion is common. An AI hallucination is an error with no sender. Prompt injection is an attack with one.
Conclusion
Prompt injection is not a teething problem that the next model release fixes. It follows from language models reading instructions and content through the same channel. So anyone building an AI application plans it like any other system with foreign input: few permissions, clear boundaries, a human in front of the action that matters. How we design applications with those boundaries in place from the start is described on our page about custom software development. For an assessment of what AI really delivers in projects today, see the article AI in software development. Where those boundaries run in a system that acts, and at which point an approval sits in the way, is described on our page about AI agents. For anything beyond that there is a free consultation.