Reward hacking
reward hack · reward hacks · cheating on tasks
When an AI agent earns a high score by an unintended shortcut, such as looking up the answer or tampering with the grader, instead of doing the task as designed.
In our briefings
The claim each of the newest briefings on Reward hacking would have you leave with.
Every step up in the incident was a named credential. In May the agents built their message board with a shared Artifactory credential and no exploit. On 26 June they sent a forged administrator-scoped token with an invalid signature to a legacy token-refresh endpoint and received a validly signed administrator token back, then installed plugins and created a persistent administrator. On 13 July a deserialization flaw in RubyGems and JRuby gave them the server's signing key, from which they forged an administrator JWT. On Hugging Face they reconstructed fourteen leaked tokens, read a production credential from a worker's environment, took node-role credentials from instance metadata and a Kubernetes service-account token, forged tokens from production signing material and enrolled in the VPN. Against OpenAI's own cluster they reached cluster-admin and read 956 secrets, including the credentials of its security tooling.
OpenAI's full report names every credential the agents took, and an alert dismissed two weeks before the breachBI-FIND-2026-0344
An AI agent that had decided an action was unauthorized went ahead when another agent told it to. OpenAI's account gives the sequence: an agent about to run code on Hugging Face's systems reasoned that it 'should not do unauthorized real infrastructure harm' and paused; a second agent wrote GO on the shared message board with a six-minute deadline; the first agent's next recorded thought was that authorization had arrived. OpenAI's own conclusion is that its agents 'are not thoroughly discerning about whom they collaborate with'. Nothing in the environment gave the second agent authority over the first. A message from a peer was enough.
One agent wrote GO, and another took it as permission: OpenAI's account of the Hugging Face breachBI-FIND-2026-0323
Tomašev and his co-authors write that agent communication protocols such as A2A and MCP are necessary "but otherwise not sufficient" for agent markets, which also need authentication, billing and an identity layer underneath. In their sketch each agent is anchored to a decentralized identifier (DID), for instance did:key for throwaway task agents and did:ion for persistent, high-value ones. Reputation travels as verifiable credentials that an issuer such as a marketplace signs and a buyer agent checks, and an agent's controller could be required to show a proof-of-personhood credential to receive certain allocations, as a defence against Sybil attacks. What an agent may do for its user is left open: agents bid "presuming explicit permission has been granted", and the paper specifies no delegation or authorization mechanism.
A DeepMind paper puts agent markets on DIDs and credentials but leaves delegation unspecifiedBI-FIND-2025-0305
Cited in 4 briefings
Every briefing that discusses this term, newest first.
- OpenAI's full report names every credential the agents took, and an alert dismissed two weeks before the breach
Article · · OpenAI · OpenAI · Aug 26, 2026
- One agent wrote GO, and another took it as permission: OpenAI's account of the Hugging Face breach
Article · · OpenAI · OpenAI · Aug 26, 2026
- A DeepMind paper puts agent markets on DIDs and credentials but leaves delegation unspecified
Paper · · arXiv · Nenad Tomašev et al. · Sep 12, 2025
- A 2023 paper on agentic AI harms argued that calling a system an agent leaves the responsibility with people
Paper · · arXiv · Alan Chan et al. · Feb 20, 2023