# Semantic Release Auditor

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Jayant kumar](https://cerebralvalley.ai/u/jaykay2000)
- **GitHub:** https://github.com/jaykay2000/semantic-auditor
- **Demo video:** https://youtu.be/qUd4R_XNhvg
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1455

Semantic Release Auditor (SRA)

A release can pass every automated test and still tell its users the wrong thing.

Semantic Release Auditor (SRA) is an AI-powered release review system that checks what users actually experience across an application, not just whether individual features work.

The Problem

While building my personal finance app, Finablo, I encountered a troubling failure: a release passed all 354 automated checks, yet a 61-year-old test user received two inconsistent 401(k) contribution limits, neither of which was correct for that scenario.

Traditional tests can verify individual features, but they often miss inconsistencies in the information users encounter across screens, emails, notifications, and time.

What SRA Does

SRA combines two complementary capabilities:

1. Review the complete user experience.

SRA replays representative user journeys across application releases, capturing what users see alongside the application's underlying records. Meta's Muse Spark models evaluate this evidence against owner-defined rules including semantic requirements that involve judgment, such as whether two features offering conflicting advice adequately explain the trade-offs.

Each release produces an evidence-backed review showing which issues are new, fixed, persistent, or regressed.

2. Turn discoveries into reusable release checks.

Exploratory AI agents can discover bugs, but their findings often end as one-time reports.

SRA makes those discoveries reusable. Muse Spark Open converts a developer's or agent's finding into a replayable user journey and a semantic rule. SRA validates the generated steps against the original report, requires owner approval, and preserves the check for subsequent releases.

The result: each validated discovery can strengthen future release reviews instead of being rediscovered from scratch.

Real-World Results

Spree Commerce: SRA identified a tax-display inconsistency in an independent open-source storefront. I reported the issue, and Spree maintainers merged a fix with a unit test (PR #289).

Persistent regression checks: An imported AI-agent finding became an approved, saved check that failed on an older Spree release, passed on the corrected release, and was successfully reloaded for a fresh review.

Finablo: Comparing releases 24 versions apart, SRA identified three fixes and a regression, which I independently confirmed.

Low-cost replay: A saved check cost approximately $0.14 per release in our measured experiment.

Built with Meta AI

SRA uses Muse Spark 1.3 and 1.2 through the Meta Model API and Muse Spark Open (open weights) for evidence interpretation, semantic judgment, and converting findings into reusable checks.

Built from scratch during the hackathon and released as open source under Apache-2.0.

Try it: Run semaudit demo for a two-minute, no-API-key demonstration, or explore the live sample release review.

The agent-finding conversion capability is an early prototype: 6 of 57 held-out findings were successfully converted, with unsupported cases rejected rather than guessed. Current limitations and evaluation results are documented in the README.

The goal: Make AI-discovered problems actionable, evidence-backed, and reusable so every release benefits from what was learned before.

## More from Meta Global AI Developer Hackathon

- [Mlartab](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1452)
- [BenchMate AI](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1453)
- [GridFork](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1454)
- [Sachitha Madhuranga](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1456)
- [Xinogi](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1457)
- [Ajay Upadhyay](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1458)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1455. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt