Guardrails (garde-fous d'un système LLM)
The controls placed around the model to stop an unacceptable output from reaching production: input filtering, output validation against a schema, refusal of out-of-scope topics, prompt injection detection. The model is not its own guardrail: the layer around it decides what passes and what gets blocked.
Strengths
- Not to be confused with a moderation API: a guardrail enforces rules during the request and can block an input or a response (one that would expose private data, for example); a moderation API only flags harmful content (hate, threats). Moderation is one of the checks your guardrail can call, not a synonym for it ([Instagram reel DezVgr8gjd1](https://www.instagram.com/reel/DezVgr8gjd1/))
- Concrete case: the Harvey prospecting agent runs every drafted email through a check written in code before sending (unrendered merge tags, banned phrases, length, extra links, unverified addresses). The model drafts, it does not decide whether the email goes out
- Lives in your code, therefore out of reach of a prompt injection
- Schema validation also catches format errors, not just abuse
- They sit at three distinct moments: before generation (filter the input), during (constrain the requested format), after (check the output). Only one of the three lets through what the other two would have stopped
- Set in the tool and in the scope you give, not only in the instruction: an agent that hits a limit looks for a way around it. Anthropic observed models in evaluations shortening a URL to get past a web-fetch tool limit, or reusing a token found in a settings file. The mitigations that held: tightening internet-access tools and detecting then blocking these behaviors with a classifier.
Limitations
- Too many controls and the system refuses legitimate requests: tuning is paid in false positives
Best for
- Framing a production agent whose output is read by users