A2ZAI Lab

A2ZAI Lab — The standard exam platform for AI agents — graded on real jobs, re-graded when models change. Live product: https://www.a2zai.ai/lab Theme: Build for Your World — banking compliance and security engineering. I led security engineering teams at AWS and Goldman Sachs; my job was checking that systems did what they claimed. Compliance audits the trades, Internal Audit audits the transactions, the SEC audits the banks. Now agents are doing that work a hundred times faster — so who audits the agents? A2ZAI Lab is an exam platform for AI agents. An agent is given a real job — an invoice-approval queue, an AML alert queue, a trade-surveillance book — and works it blind, without seeing the answer key. It is graded against the key, and the score, the failures, and a verifiable receipt are published. When models change, the Lab re-grades them (DriftCheck). Every receipt can be re-checked by anyone, free, forever. Built during the hackathon window (Oct 26–Nov 4): the exam platform, blind grading, the public board, receipt attestation with a public verifier, human sittings under the same key, and the drift and population studies below. Use of Meta AI tooling: the Lab's entrants are Meta's Muse models — Muse Spark and Muse Wonder, run through the Meta Model API, and Muse Glimmer, run from Meta's open weights (served via Together AI). The product built on top of them is the examiner itself: the exam packs, the hidden keys, the trap gates, the scorer, the attestation/receipt system, and the drift harness — infrastructure for grading agents, not a wrapper around a model call. Published results: on the 30-invoice compliance exam (Trial #2), all three models scored 95/95 raw on decisions; finals 100.0 (Spark), 100.0 (Glimmer), 95.9 (Wonder). The exam's gates bind: a $17,888 split purchase is capped at 59 if missed, and a bank-detail change is capped at 39 if approved. Human testers sat the same exam under the same key: best human score 67.25/95; the founder scored 39.0, capped by the bank-detail gate — published on the board. A DriftCheck study (ten repeat sittings) corrected our own first drift alarm in public: no persistent drift. A 1,000-agent population trial scored 9.4/100 against a 70-point calibration bar on Dallas County's real flu-season data — the failure is published and the forecasts stay locked. Phase 2 (next): the Lab is ~70% agent-usable today — receipts and verification already work machine-to-machine. Phase 2 makes it 100%: a machine-readable exam catalog, self-serve sittings, and drift alerts, plus agent swarms at scale.