Ghost

A benchmark score tells you how often a model fails. It doesn't tell you what causes the failure, or whether the cause is real or noise. So I built a system that answers that question specifically targeting Astra. We run Astra as a software-engineering agent on real GitHub repositories and measured its solve rate at 0.48. Then a second adversary model (Qwen as we needed open-weights) scans through the repo to find possible things to change. It proposes a small executable generator that changes exactly one condition changed. A stale tool spec. A flaky test. An underspecified issue. A truncated observation. etc If a separate Astra model deems this a valid change (doesnt fundamentally change the objective / make it impossible) we then run evaluate Astra on the original example and the adversarially modified one. We reward the adversary based on if Astra was able to solve it originally but not with the modification and encourage diverse changes in the loss. Overall, we produce a mapping showing which perturbations Astra typically struggles with and which it succeeds on. With the goal that this could be used to better understand the model + train a better successor.