FaultForge

**FaultForge - Find the failures your accuracy score misses.** FaultForge is an AI reliability testing platform built with Meta's DINOv2 vision model. It focuses on a critical challenge in industrial quality inspection: AI systems can achieve high accuracy while still missing real defects. FaultForge helps developers discover and investigate these failures through an interactive dashboard, an image-based Failure Explorer, and an anomaly heatmap. Its Red Agent tests model behavior under controlled brightness and contrast changes, revealing predictions that are sensitive to small variations. The Blue Agent evaluates an experimental multi-view majority-voting strategy to improve defect detection. Finally, the Verification Gate compares the baseline and proposed strategy across 83 evaluation images, checking accuracy, missed defects, and false alarms. In our exploratory evaluation, accuracy improved from 91.57% to 93.98%, and missed defects decreased from six to two. However, false alarms increased from one to three. Rather than accepting the higher accuracy at face value, FaultForge rejected the strategy because it introduced additional risk. FaultForge demonstrates that building more reliable AI is not just about improving accuracy. It is about discovering failures, evaluating trade-offs, and making informed decisions before deploying changes.