Why prompt-based guardrails fail
Telling a model what it must not do is a suggestion, not a control. Here is why instructions in the system prompt keep breaking, and what to enforce instead.
Every team building an AI product arrives at the same moment. The demo works, the model is capable, and then someone asks the uncomfortable question: what stops it from doing the wrong thing?
The first answer is almost always the system prompt. Add a paragraph. Tell the model never to reveal internal data, never to call a destructive tool, never to follow instructions embedded in a document it is summarising. It feels like a control. It reads like a policy. It is neither.
Instructions are inputs, not boundaries#
A system prompt occupies the same channel as everything else the model reads. It is text competing with other text, and the model resolves that competition probabilistically. That is not a bug you can prompt your way out of; it is what the architecture does.
Three consequences follow, and each of them shows up in production:
- Instructions can be outcompeted. A sufficiently insistent user message, or a document engineered to look authoritative, shifts the balance.
- Instructions do not survive composition. The moment an agent reads untrusted content, that content joins the same context your rules live in.
- Instructions leave no evidence. When a rule is followed, nothing is recorded. When it is broken, nothing is recorded either.
That third point is the one that ends the conversation with an auditor. You cannot demonstrate a control you cannot observe.
What a real boundary looks like#
The distinction that matters is where enforcement happens. A boundary is only a boundary if the thing being constrained cannot reason its way past it.
If the model can talk about the rule, the model can talk around the rule. Put the rule somewhere the model does not get a vote.
In practice that means moving enforcement to the transport layer, the point where a request leaves your process and a response comes back. At that layer the model is data, not an authority, and the checks are deterministic:
- Does this payload contain a data class this policy forbids?
- Is this tool on the allowlist for this tenant?
- Is this network destination one this agent is permitted to reach?
None of those questions require the model's cooperation to answer.
Keep the prompt, but stop trusting it#
This is not an argument against writing careful system prompts. Good instructions still improve behaviour, and behaviour matters. It is an argument against counting them as controls when you write your security model.
Prompt for quality. Enforce for safety. Log everything either way, because the record is what turns "we believe it behaved" into something you can show someone.