LLM-as-a-judge
LLM as a judge · LLM-as-judge
Using a second language model to review another model's output or planned action and approve or block it; a probabilistic check, since the judge can itself be fooled.
In our briefings
The claim each of the newest briefings on LLM-as-a-judge would have you leave with.
In the second round of his IDEsaster research on AI code editors, Marzouk moves the dangerous step out of the coding agent and into the language servers installed around it. Injected text that reaches the agent's context gets it to write a workspace file, using tools approved to run without asking. A language server then acts on that file by fetching a remote schema or running code. The agent can read local files without approval, such as .env and, in Cursor, an SSH key outside the workspace, and those files can then leave through the server's own web request. His recorded demonstrations used the JSON language server in an older Cursor build, and Shopify's Ruby language server and the C# server in Google's Antigravity. Only the JSON case is shown end to end, sending an SSH key out; the Ruby and C# demonstrations end once code runs on the machine. He says the Ruby server has more than 30 million downloads and the C# server more than 50 million.
A coding agent tricked into writing one file lets the editor's language server send your keys outBI-FIND-2026-0744
Deriving an agent's permissions from the user's request, and enforcing them outside the model, stopped every injected instruction in this paper's test and stopped most of the legitimate work with it. In strict mode, across 240 runs on the AgentDojo benchmark, no attack succeeded, 92.2 percent of tool calls were denied and the agent completed 33.3 percent of its tasks. In permissive mode it completed 65.8 percent and three attacks got through, each because the parsed permission was a wildcard, write to any calendar or send to any address. The failure the method has is the one it was built to remove: authority granted more broadly than the task needed.
The IBAC paper's strict mode blocked every prompt injection, and its agent finished only a third of its tasksBI-FIND-2026-0310
Cited in 2 briefings
Every briefing that discusses this term, newest first.
- A coding agent tricked into writing one file lets the editor's language server send your keys out
Video · · Black Hat · Aug 18, 2026
- The IBAC paper's strict mode blocked every prompt injection, and its agent finished only a third of its tasks
Article · · ibac.dev · Jordan Potti · Mar 3, 2026