Safe Retry After an Ambiguous Tool Timeout
A Hand-Checkable Conformance Suite for Agent Recovery with Provider Contracts and Webhook Evidence
Anish Prashun · Independent researcher · Technical report ·
Read the paper (PDF)Code and data (GitHub)DOI 10.5281/zenodo.24304849
Summary
When an AI agent's side-effecting tool call times out, the agent cannot tell whether the action happened. This deterministic, hand-checkable simulator scores retry policies against stale status reads and webhooks that arrive late, twice, out of order or not at all, under providers with and without idempotency-key deduplication. It ships a provider contract checklist, a decision table and commands to test your own scenarios and policies.
Key findings
- Across 17 scripted operations without provider deduplication, immediate retry duplicated 12 and a single status check duplicated 8; the evidence-reconciling policy duplicated 1, only where the provider broke its own guarantee.
- Stable idempotency keys removed every policy-caused duplicate, and rotating the key on each retry brought them all back.
- Webhook delay, duplication and reordering did not affect a reconciler that only adds facts per attempt, while ignoring webhooks left 41 of 120 timing cases unresolved instead of 16.
- Wait-then-verify is safe exactly when its wait is at least the longest pending time plus the status lag, minus the timeout.
Reproduce it
Python 3 standard library only, synthetic data, no network access. Clone the repository and run:
python3 run_sim.pyCite
Prashun, A. (2026). Safe Retry After an Ambiguous Tool Timeout: A Hand-Checkable Conformance Suite for Agent Recovery with Provider Contracts and Webhook Evidence [Technical report]. Zenodo. https://doi.org/10.5281/zenodo.24304849
@misc{prashun2026retry,
author = {Prashun, Anish},
title = {Safe Retry After an Ambiguous Tool Timeout: A Hand-Checkable Conformance Suite for Agent Recovery with Provider Contracts and Webhook Evidence},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.24304849},
url = {https://doi.org/10.5281/zenodo.24304849},
note = {Technical report, code and data}
}Paper and data: CC BY 4.0. Code: MIT. All data in the study is synthetic.
- AI agents
- Tool use
- Idempotency
- Retries
- Webhooks
- Distributed systems