What is Guardrails?

How it works

Guardrails sit at several points in an AI application, not only inside the model:

• Input guardrails: screen what goes in. Detect prompt-injection attempts, block disallowed requests, strip personal data before it reaches a third-party API.
• Model-level behaviour: safety training by the model vendor and the instructions in the system prompt.
• Output guardrails: check what comes out. Validate JSON against a schema, check answers against retrieved sources to catch hallucinations, filter toxic or off-topic content, redact secrets.
• Action guardrails for agents: limit which tools an AI agent can use, scope its credentials, cap spending and steps, and require human approval before irreversible actions such as payments, deletions or sending email.

They are built with a mix of plain code (regex, allow-lists, schema validation), classifier models such as Meta's Llama Guard (a model fine-tuned to label prompts and responses as safe or unsafe against a policy), and frameworks such as NVIDIA's open-source NeMo Guardrails, which lets developers define programmable rails around an LLM app.

Why it matters

Models are probabilistic. A system prompt that says "never do X" lowers the chance of X but does not guarantee it, and a determined user or a malicious document can override instructions. Guardrails in code are deterministic: a tool the agent was never given cannot be called. That is why security guidance treats them as layers: no single check is enough.

Guardrails also support governance. Frameworks such as the NIST AI Risk Management Framework ask organisations to identify, measure and manage AI risks, and guardrails are a concrete way to apply those controls and log what happened.

The trade-off is friction: too-strict filters block legitimate requests and annoy users, so guardrails need testing like any other feature.

Example

An internal finance agent can read invoices and draft payments. Its guardrails: it only has read access to the accounting system plus one "create draft payment" tool; any draft over a set amount needs a named approver to click confirm; outputs are checked so the payee's bank details match the supplier record; and every action is logged for audit.