formicslab

AIOps Experiment Engineer: the AI proposes the performance fix; a Rust harness proves it, or refuses it. Problem: an AI model can write a plausible performance fix in seconds, but checking that every response stays the same and that the service is really faster still takes an engineer hours. Trusting the model's own claim is the failure mode. What it does: a Meta model (Muse Spark 1.3) reads a slow service's evidence through tools: metrics, the slowest requests with their SQL counts, the source code and past experiments. It separates facts from hypotheses and proposes one hypothesis and a patch. A Rust harness then builds the patch in an isolated Cloudflare Container with no internet and no secrets, compares it with the original on golden cases, invariants and 200 sampled requests, and measures p95 latency against the unchanged service in 3 alternating pairs after measuring machine noise. A Rust evaluator, not the model, returns VERIFIED, REJECTED or INCONCLUSIVE with the measured reason. Results on the live deployment: on an order API with an N+1 query, verified patches cut SQL statements per request from 81.8 to 3.0 and p95 latency by 88-91%, with every response unchanged. On a real open-source bug (Vaultwarden issue #1402), the model reached the maintainers' own approach: 2,011 to 15 SQL statements per sync, VERIFIED at 74%. The harness is not a rubber stamp: 23 hand-written trap patches and 5 vulnerability-fix checks all got the expected verdict in CI, and 30 of 36 generated mutants were caught. Security: every patch passes a security gate, a sandbox, a read-only database role and measurement isolation. In a red team with prompt injections planted in code and logs, the injections were flagged in all 3 runs and followed in none. A weekly vulnerability hunt sends the model's attack requests to the sandboxed service; rules, not the model, decide what reproduced. It runs unattended: a daily scheduled run, retries when the machine is noisy, daily run and token budgets, Discord notifications, and pull request descriptions with the evidence.