Command Palette

Search for a command to run...

GitHub

Safe Retry After an Ambiguous Tool Timeout

A Hand-Checkable Conformance Suite for Agent Recovery with Provider Contracts and Webhook Evidence

Anish Prashun · Independent researcher · Technical report ·

Read the paper (PDF)Code and data (GitHub)DOI 10.5281/zenodo.24304849

Summary

When an AI agent's side-effecting tool call times out, the agent cannot tell whether the action happened. This deterministic, hand-checkable simulator scores retry policies against stale status reads and webhooks that arrive late, twice, out of order or not at all, under providers with and without idempotency-key deduplication. It ships a provider contract checklist, a decision table and commands to test your own scenarios and policies.

Key findings

  • Across 17 scripted operations without provider deduplication, immediate retry duplicated 12 and a single status check duplicated 8; the evidence-reconciling policy duplicated 1, only where the provider broke its own guarantee.
  • Stable idempotency keys removed every policy-caused duplicate, and rotating the key on each retry brought them all back.
  • Webhook delay, duplication and reordering did not affect a reconciler that only adds facts per attempt, while ignoring webhooks left 41 of 120 timing cases unresolved instead of 16.
  • Wait-then-verify is safe exactly when its wait is at least the longest pending time plus the status lag, minus the timeout.

Reproduce it

Python 3 standard library only, synthetic data, no network access. Clone the repository and run:

python3 run_sim.py

Cite

Prashun, A. (2026). Safe Retry After an Ambiguous Tool Timeout: A Hand-Checkable Conformance Suite for Agent Recovery with Provider Contracts and Webhook Evidence [Technical report]. Zenodo. https://doi.org/10.5281/zenodo.24304849

@misc{prashun2026retry,
  author    = {Prashun, Anish},
  title     = {Safe Retry After an Ambiguous Tool Timeout: A Hand-Checkable Conformance Suite for Agent Recovery with Provider Contracts and Webhook Evidence},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.24304849},
  url       = {https://doi.org/10.5281/zenodo.24304849},
  note      = {Technical report, code and data}
}

Paper and data: CC BY 4.0. Code: MIT. All data in the study is synthetic.

  • AI agents
  • Tool use
  • Idempotency
  • Retries
  • Webhooks
  • Distributed systems

More research