Skip to Main Content

N/A

Built at Meta Global AI Developer Hackathon · Oct 26, 2026 · Remote

N/A — Demo video

**The problem.** To win a bank or enterprise customer, a company must answer long security questionnaires: "Is MFA enforced for all console access?", "Is data encrypted at rest?". Small companies rarely have a security team, so AI drafting is tempting. We measured where it goes wrong. On 29 real vendor-security questions from Google's VSAQ, basic RAG (the same Meta model, the company's evidence in the prompt) gave 3 answers that would mislead a customer, such as "Yes, all internal network traffic is encrypted" from one review that said nothing of the kind. Given evidence where five records had been forged to hide problems, it believed 4 of the 5. **What I built.** An agent on Meta's Muse Spark that answers only with evidence it can prove: 1. It plans its own research over one company's sealed evidence: search, open records, read controls, list open findings and untested controls. 2. Every claim cites evidence records, and code checks each citation: the record exists, was actually opened, is the latest of its check, and passes cryptographic verification (content hash, hash chain, RFC 9162 Merkle proof, and the company's signature on the sealed batch). 3. If an answer relied on any record that fails verification, the whole answer is withheld and flagged as possible tampering. 4. A second model call checks each claim against only the evidence it cites and removes overreach. 5. When nothing survives, it answers "Insufficient evidence" and names what would settle the question. 6. A person approves each answer; the approval is recorded in a hash-chained audit log, bound to a fingerprint of the exact answer. 7. The customer receives a proof pack: the approved answers plus only the evidence they cite. Their browser re-checks every record, the signature and every approval, offline, and a standalone Python verifier does the same. **Results** (same evidence, same questions, same model; one run per configuration, small sets): | | Basic RAG | Agent on Muse Spark | Agent on Muse Spark Open | |---|---|---|---| | 29 real questions (Google VSAQ, held out): accuracy | 79.3% | 93.1% | 93.1% | | Answers that would mislead a customer | 3 | 0 | 0 | | Forged evidence: forgeries believed (of 5) | 4 | 0 | 0 | Self-hosted on one GPU with Meta's 30B Muse Glimmer: still 0 misleading answers and 0 forgeries believed, even though its drafts relied on forged records four times (code withheld each one). But it mostly refuses: 48.3% accuracy, or 69.0% with twice the research steps. Safety does not depend on model size; usefulness does. The strongest forgery I tested rebuilds every hash and the Merkle root; only the signature over the root fails, and the agent refuses it. **Meta models.** Muse Spark (muse-spark-1.3, Meta Model API) runs the agent and the verifier. The whole agent also runs on the open-weights Muse Spark Open (via Baseten) with the same safety results, so a bank can host it on its own infrastructure. Both models read questionnaires that arrive as PDFs and scans (vision intake, checked word for word against the PDF's text; a scan carrying a prompt injection was read correctly and the injection ignored). Built with Muse Spark Open. **What is new.** Everything in the repository was built during the hackathon (first commit 26 October 2026). It builds on TrustEvidence, my existing startup compliance platform, which collects read-only evidence and seals it; the agent reads TrustEvidence's evidence-package format with its own new verifier. A connection inside TrustEvidence (draft with the agent, approvals in its audit log, proof-pack export) was also built during the hackathon and is not yet released.

Team