Skip to content
IAM Insights

Explainers · Glossary

Instruction hierarchy

instruction-hierarchy defense

A training technique that teaches a model to rank instructions by source, so text from a tool or web page loses to the system prompt.

In our briefings

The claim each of the newest briefings on Instruction hierarchy would have you leave with.

  1. NIST's AI standards center asked in January how to secure AI agents that act on their own, and split identity off from the question. Its request for information, open until 9 March 2026, asks about hijacking, backdoors, poisoned data, indirect prompt injection, human approval of consequential actions, sandboxing, rollback and monitoring. It asks nothing about how an agent is identified, what authority it carries, who delegated it or how its actions are attributed, and it names no one as accountable; authentication appears once, as a counterparty an agent might touch. A month later NIST's cybersecurity center put those questions in a separate concept paper, so NIST is running agent security and agent identity as two efforts.

    NIST's request on securing AI agents asks about hijacking and rollback, and never about who the agent isBI-FIND-2026-0699

  2. With its policy checks on, Microsoft's FIDES planner let 1 of 949 injection attacks succeed on gpt-4o in AgentDojo, against 163 for an undefended basic planner and 24 for FIDES without the checks. The checks do the blocking: under the same checks a basic planner let 9 through and a variable-passing planner 14, and all three reach zero once two injection tasks the authors set aside are dropped. One of those only changes what the agent says back to the user, and the authors write that "we only enforce policies upon tool calls, our planners do not stop these text-to-text attacks." In the other, the confidentiality policy allows an injected calendar event because no data leaves. So the evaluation's own claim is narrower than the introduction's "stops all prompt injection attacks": it blocks every attack that breaks its policies. That result also rests on labels the authors set, marking as untrusted every data field any benchmark injection targets.

    In Microsoft's agent tests, label checks blocked policy-breaking tool calls and its FIDES planner finished more tasksBI-FIND-2025-0454

  3. In WASP, planted instructions diverted web agents far more often than they got the attacker's goal done. Across the 13 configurations in the authors' Table 3, agents began acting on the injection in 16.7% to 85.7% of 84 tasks, as judged by a GPT-4o grader, while a rule-based check of the final state found the attacker's goal reached in 0% to 16.7%. In hand-labelled GPT-4o traces, hijacked agents often got confused and took irrelevant actions, and sometimes went back to the user's task. The authors call this "security through incompetence" and put the bottleneck on "the capability of the agent to complete their goal when hijacked". The o1 agent, which also scored highest of the five setups on clean WebArena tasks (21%), had the highest rates on both counts: 85.7% and 16.7%. URL-anchor injections diverted most agents far more often than plain text (Claude 3.5: 85.7% against 31.0%), but with plain text alone o1, its instructions in the system role, reached the attacker's goal in 23.8% of its plain-text tasks, against 9.5% with URL anchors (Table 6). The authors warn that "more capable agents might lead to more security vulnerabilities if proper mitigations are not deployed."

    Web agents in Meta's WASP test were often hijacked but seldom finished the attackBI-FIND-2025-0625

›Cited in 6 briefings

Every briefing that discusses this term, newest first.

  1. NIST's request on securing AI agents asks about hijacking and rollback, and never about who the agent is

    Article · · Federal Register (govinfo) · NIST Center for AI Standards et al. · Jan 8, 2026

  2. In Microsoft's agent tests, label checks blocked policy-breaking tool calls and its FIDES planner finished more tasks

    Paper · · arXiv · Manuel Costa et al. · May 29, 2025

  3. Web agents in Meta's WASP test were often hijacked but seldom finished the attack

    Paper · · arXiv · Ivan Evtimov et al. · Apr 22, 2025

  4. SecAlign held optimised prompt injections to 14% or less on five models; its authors say it cannot stand alone

    Paper · · arXiv · Sizhe Chen et al. · Oct 7, 2024

  5. OpenAI's instruction hierarchy ranks who may give orders by where the text arrived, not by who sent it

    Paper · · arXiv · Eric Wallace et al. · Apr 19, 2024

  6. StruQ held hand-written prompt injections to 2% or less on two small models; optimised attacks still got through

    Paper · · arXiv · Sizhe Chen et al. · Feb 9, 2024

Related terms

What appears alongside Instruction hierarchy in the same briefings.

CaMeLPrompt injectionOpenAI o1WASP