HarnessForge - Evolutionary CI/CD for operational AI agents

HarnessForge: Evolutionary CI/CD for operational AI agents HarnessForge continuously hardens an AI agent by turning its failures into measurable harness improvements. It doesn't retrain the model or hand-edit the application. The harness itself (rules, context policy, guardrails, tool access, memory depth, model routing and retries) is a versioned "genome" stored in MongoDB Atlas, and it evolves itself (Statement 1: Recursive Harnessing), driven by hard metric signals. How it works: 1. The agent runs a deterministic eval suite 3× per task. Every run is stored in Atlas as a trajectory. 2. Failures are classified by rules (e.g. unwarranted_action, missed_approval, missing_verification) and embedded with Voyage AI. 3. A meta-agent retrieves similar past failures and distilled lessons via Atlas $vectorSearch. It then proposes exactly one structured JSON patch to the genome. It cannot write code, and safety guardrails are locked. 4. The child harness is re-evaluated. Explicit gates decide whether it's kept: the evaluation must be valid, accuracy must improve, stable passes can't regress, and cost has to be justified by the accuracy gained. Accepted genomes are then scored on a hidden holdout the meta-agent never sees. LLMs propose. Metrics decide. MongoDB remembers. Proving ground: a customer-support operations agent. It resolves tickets (duplicate charges, cancellations, billing disputes, plan changes, account access) by reading account data and taking exactly one operational action, such as refund, request approval, credit, cancel, escalate, deny or refuse. A sandboxed gateway validates the action but never executes it. The expected outcome for each ticket comes from a policy oracle over seeded Atlas data, not an LLM judge. Gen 0 is deliberately weak, and it failed exactly as a real one would. It refunded charges that weren't duplicates, issued a $499 refund without approval, cancelled annual contracts immediately, refunded a fraud claim, and tried to refund another customer's charge. Results (real run, all recorded in Atlas): - Hidden holdout accuracy 28% → 69%, and train accuracy 23% → 60%, on the same cheap 8B model, for a total cost of $0.34. - 7 accepted mutations, each fixing a different part of the harness: a policy-retrieval guardrail, refund verification, identity-before-reset, invoice access, a "verify before money moves, else deny" rule, a $200 approval guardrail, and contract-aware cancellation. - Rejected mutations are first-class. Plausible-sounding changes that didn't earn their place were rejected by the gates. - Domain-agnostic. The same engine hardened a MongoDB database-operations agent from 18% → 67% holdout: schema awareness, mandatory explain before index recommendations, and refusal of write operations. Built with: MongoDB Atlas (genome lineage, trajectory memory, $vectorSearch, event feed, the agent's operating data), Voyage AI (embeddings), OpenRouter (Llama 3.1 8B as the agent, Claude Sonnet as the meta-agent), LangSmith (tracing), and Vercel (live dashboard). - Live dashboard: https://harnessforge-one.vercel.app/?run=9fc71ae7c1 - Repo: https://github.com/Deepesh-Katudia/harnessforge

Team

+1

Placement

Participant