AgentReplay

AgentReplay is the flight recorder and control tower for AI agents. The problem. Teams are giving agents real authority - booking, buying, calling APIs, moving data. When an agent fails in production, nobody can reproduce the failure, nobody can say what the agent was allowed to do, and nobody knows who pays when it gets expensive. What we’ve built (live at agentreplay.space). A SDK that patches clients (sync and async) and records every LLM call, tool call, and state transition as a hashed, tamper-evident trace, with no changes to the agent’s code. A FastAPI ingest service and Next.js trace viewer store and display each run. Strict replay re-executes a recorded run with zero live API calls, so a production failure reproduces exactly on a developer’s machine. Regression replay runs the same case against a new agent version. A worker classifies each failure against the MAST multi-agent failure taxonomy and identifies the culprit span. A test generator then turns that failure into a pytest file with structural and semantic assertions, and a GitHub Action blocks any PR that reintroduces it. We have proven the full loop (failure, replay, generated test, CI gate) on a real bug, backed by 470+ automated tests. What we’re adding: a control plane on the same trace. Identity and scope. Every agent gets a signed identity and a scope manifest (allowed tools, spend cap, per-task limits). Each tool call is checked against it, and every decision is logged and replayable. Trust score and spend limits. A score computed from the agent’s own replay history (failure rate, regression pass rate, scope violations) sets how much it may spend. Live monitor and kill switch. An out-of-scope or over-budget action halts the run and fires an alert. Incident pipeline. One failure automatically produces a classified root cause, a regression test, a dispute packet, an insurance-claim packet, and a narrated walkthrough that highlights the exact error on screen. Multi-model replay. We replay the same recorded failure across several models to show which would have repeated it, building per-task model accuracy from real production traffic, not synthetic benchmarks. We have built a working prototype of this control plane and tested it in a simulated scenario. An agent negotiates a flight price, then books the wrong flight. When it tries to fix the mistake by buying a second ticket outside its scope, our monitor halts it and fires an alert. The same trace then automatically generates a dispute packet, an insurance claim, and a regression test that stops the mistake from recurring. The airline, payment rails, and filings are sandboxed or simulated. The recording, replay, scope enforcement, trust scoring, and halting are real. Why it matters. Observability tools show you what happened. We reproduce it, constrain it, and prevent it from happening again. As agents get authority over money, the record that proves what they did becomes the basis for trust, limits, and liability. The trace is the primitive, and every module is a read or write on it. Live and working. AgentReplay’s core loop is built and running at agentreplay.space: Recorder SDK: Python, patching the Anthropic, OpenAI, and OpenAI-compatible clients (sync and async), plus a tool decorator and a LangGraph adapter, with a redaction hook for PII. Ingest and viewer: a FastAPI ingest service on Postgres with batched export, and a Next.js trace viewer with a span timeline. Strict replay: deterministic re-execution of a recorded run with zero live API calls, with divergence detection when a new agent version departs from the recording. Failure classification: an async worker classifies failures against the MAST taxonomy and flags the culprit span. Test generation: one click turns a classified failure into a runnable pytest file with structural and semantic assertions. We proved this end to end on a real bug, not a synthetic one. 470+ automated tests across the SDK and ingest service. Newly built: the control plane prototype. We extended the trace with signed agent identity and scope manifests, per-tool-call scope enforcement, a trust score that sets spend limits, a live monitor with a kill switch, and an incident pipeline that produces a root cause, a regression test, a dispute packet, an insurance-claim packet. We ran it end to end in a simulation where an agent negotiates, books the wrong flight, and is halted when it exceeds its scope. The airline, payment rails, and filings are sandboxed or simulated, and the enforcement, scoring, replay, and halting are real. The prototype also compares multiple other models on failure classification and replays the same failure across models.