# Shibboleth

- **Event:** [The Harness Engineering & Model Wrangling Hackathon](https://cerebralvalley.ai/e/mongodb-nyc-hackathon)
- **When:** Sat, Sep 26 at 9:00 AM – 10:00 PM (EDT)
- **Where:** New York, NY
- **Team:** [Alex Wu](https://cerebralvalley.ai/u/amadeusWoo), [Alan Wu](https://cerebralvalley.ai/u/thewoozie67), [Adam Martinez](https://cerebralvalley.ai/u/TheAdamAgenda)
- **GitHub:** https://github.com/TheApexWu/shibboleth
- **Demo video:** https://youtu.be/afkR0qdtnGk
- **Gallery:** https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/1

Shibboleth is a harness trust layer for open-weight AI models. Anyone can download a model, quietly strip out its safety ("abliterate" it), and re-upload it looking completely normal, so testing its outputs won't reveal it. Shibboleth reads the model's internal activations instead: it fingerprints how strongly the model refuses harmful prompts at each layer, then compares that to a trusted baseline.

The system runs entirely on MongoDB. A change stream fires a scan the moment a checkpoint lands, the verdict is written as a document, and $vectorSearch matches each fingerprint against the imposters already caught. That growing library is the flywheel: every catch makes the next scan sharper, so it's a recursive harness that improves its own guardrail as it runs.

It's a live eval and gate you can put in front of a model hub or a CI pipeline, so you can trust an open-weight model before you run it. On the Qwen2.5-1.5B family it caught an abliterated model at an AUC of 1.0 while clearing a legitimate finetune.

## More from The Harness Engineering & Model Wrangling Hackathon

- [Vision](https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/2)
- [APA](https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/3)
- [SubmissionAI](https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/4)
- [NYC-Flood-Map-Agent](https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/96)
- [ShadowCFO](https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/97)
- [Continuum](https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/98)

---

Markdown version of https://cerebralvalley.ai/e/mongodb-nyc-hackathon/hackathon/gallery/1. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt