Agent Reliability Cockpit

I am a marketing data analyst; my day job is GA4 and Google Tag Manager implementations, and the recurring pain is setups failing silently — a purchase event that drops revenue with no error, an API key leaked into a commit, a tool call nobody can replay. I built the Agent Reliability Cockpit: an agentic auditor driven by Muse Spark (muse-spark-1.1) through the Meta Model API. The agent runs its own audit tools over a synthetic analytics setup — event schema checks, secret scans, replayability tests, PII scrubbing, dollar-impact math — and every fix it proposes is blocked by default behind an approval gate. A human approves; the harness, never the model, applies it. In the demo, exactly one of four fixes is approved and applied to a copy: recorded revenue for one synthetic purchase goes from $0.00 to $49.99 while the originals stay byte-identical. I tested it the way I'd want any auditor tested. On a sealed fresh holdout it had never seen: 5/5 faults detected (strict grader 5/5, typed grader 5/5; precision 6/7, disclosed). Across 10 holdout runs: 5/5 detected every time. On a clean dataset, my pre-registered test FAILED — the harness couldn't accept an empty findings list, so the model filed a placeholder; I fixed the harness and a separate post-fix re-run passed clean, and the original FAIL stays on the record exactly as it happened. Total Meta Model API spend for the entire build: $1.1534 of the $150 credit, itemized in the repo. The Cockpit composes guardrail libraries I open-sourced before the event (approval gate, secret guard, trace replay, PII redactor, judge calibration, injection firewall, MCP tool server); everything else — the agentic loop, the graders, the sealed datasets, the approval moment — was built during the event and is listed in the repo's BUILT-DURING-EVENT.md. One note for the judges: I built this as an agentic use-case — the agent autonomously drives its audit tools and proposes fixes as tool calls, and every action operates under a hard human-approval gate (the gate log is in the repo's runs/ folder).