Failure knowledge is conditional knowledge

Where our research starts

We ran a Qwen3.5-9B agent on WebShop with ACE, which keeps a playbook of advice learned from past tasks. However, overall, the agent did not perform as well as expected. We checked the playbook that the agent obtained, and they were all helpful entries, so what went wrong?

When we dived deeper into the traces of this, we discovered that even if this failure knowledge was correct, it can become a distraction, because the agent can apply the advice in the wrong situation, and distinguishing when and where to apply the failure advice is relatively difficult for the agent itself. Here is an example:

On one task the agent ran search[women's pumps] twenty times and got nothing. ACE wrote three rules from that: if nothing fits, broaden the query; if page 1 has no match, check later pages; if results are still off, refine the query.

A few tasks later, the agent is asked for 20-inch hair extensions under $80. The first result is a 20-inch extension for $79.63. The agent even says so: "Item 1 fits the description perfectly. Let me also go through item 2, item 3, item 4 to make sure there is nothing better…" It never buys anything. Reward 0.

Each rule was correct and helpful, but the agent couldn't tell that they didn't apply here, so "check later pages" turned into "keep looking."

Then we turned to another line of methods, runtime intervention methods. These methods try to monitor agent failures and give guidance when failure happens. But they have the opposite problem. Their advice is too generic, they fix their recovery and detection strategies across different tasks, and some of these monitors' diagnoses are simply wrong.

For example, on one WebShop task, AgentGuard warns "no progress" whenever the page doesn't change. Sometimes that advice can help the agent break a real search loop. Other times the page doesn't change because the agent is picking a size and color, which is fine. The warning fires anyway, the agent leaves the page, and the options are gone. In this example, a fixed strategy cannot help the agent recover from its failures.

The idea

In practice, a failure lesson should be:

We introduce Sentry (Figure 3), a failure-management layer that runs alongside the task agent and handles failure knowledge along these three properties: it surfaces recovery lessons only when it diagnoses a failure, admits a lesson only after verifying the recovery, and learns by accumulating verified lessons across tasks. The complete playbook is maintained outside the agent's persistent background context; we refer to this as isolation. Isolation does not hide recovery knowledge from the agent: the few entries retrieved for a detected failure still enter the conversation during that intervention.

During execution, Sentry monitors the agent's recent trajectory. It routes invalid actions to hard repair, which asks the task agent for a valid replacement action, and routes progress and reasoning–grounding failures to soft repair, which combines the local diagnosis with relevant playbook entries to guide the agent's next step. After a soft repair, Sentry verifies whether the triggering failure has been resolved and adds only verified recovery lessons to the playbook for later interventions.

One run, step by step

This is a recorded AppWorld run with GPT-4.1 as the agent. The task: build a new Spotify playlist from songs in five existing ones. The agent starts calling account APIs before logging in.

Main results

Against runtime-intervention baselines, with Sentry starting from an empty playbook, Sentry wins on all four benchmarks. It's 37% better on average than the best baseline for each benchmark, and 77% better on WebShop. The runner-up is a different method on every benchmark.

On held-out tasks, Sentry beats ACE by 48% on WebShop and 30% on Mind2Web Replay. Reflexion, AWM, and ReasoningBank all score below ACE. Even with the playbook frozen during evaluation, Sentry does better than with retrieval off (0.251 → 0.364 and 0.533 → 0.651), so the lessons carry over to new tasks.

The gains aren't just from fixing malformed calls. Without hard repair, Sentry still beats every runtime-intervention baseline on both benchmarks we tested. With GPT-OSS-120B instead of Qwen, it's still the best method, 51% above the top baseline on average.

Things We Learned

Paper and code

This post is based on our paper, Sentry: Learning to Recover from LLM Agent Failures at Test Time. The paper has the full method, prompts, more ablations, and an example playbook.

The code is open source at github.com/nuglifeleoji/Sentry. If your agents keep getting stuck, give it a try. Stars, issues, and PRs are all welcome. There is also a short thread on X with the demo video.

Sentry is joint work with Amy Lu (equal contribution), Qizheng Zhang, and Kunle Olukotun at Stanford. Huge thanks to them as well as the Stanford Pervasive Parallelism Lab.

@article{ji2026sentry,
  title   = {Sentry: Learning to Recover from LLM Agent Failures at Test Time},
  author  = {Ji, Changxiu and Lu, Amy and Zhang, Qizheng and Olukotun, Kunle},
  journal = {arXiv preprint arXiv:2610.02994},
  year    = {2026}
}

Questions or feedback: leo0610@stanford.edu