Skill Issue

AI agents can finish a task without errors and still get it wrong. Some examples are repeating an action after a timeout, reporting success on a failed step, ignoring a correction, or using stale or unauthorised information. Most agent benchmarks report one pass rate per framework, which says that something failed, not what or why. Agent frameworks also change constantly, and nothing tells a user whether a new version still handles their task correctly. We built a correctness and regression-testing framework for AI agents that works across three open-source agent frameworks: Hermes Agent, OpenHands, and DeerFlow. Benchmark harness: - 109 scenarios with 192 user turns. - Property families include idempotency, consistency, failure recovery, ordering, authorisation, confidentiality, and following user corrections. - Agents work against mock services with injected faults, such as timeouts, permission errors, and changing requirements. - Every run is checked on three levels: the final state of those services, the sequence of tool calls, and what the agent tells the user. - Controlled baseline/treatment pairs isolate a single cause. - OpenHands and DeerFlow run their own native runtimes in sandboxed, network-disabled Docker containers. Hermes runs on our own machine. Source-grounded orchestrator: - A Meta Muse Spark agent reads the framework’s actual source code at a pinned commit and writes new test scenarios. - Three Muse Spark reviewers critique each test plan (task quality, experimental validity, verification) before anything runs. - Tests run through both an edited and a frozen copy of our test checker, so a bug in a newly written check can’t pass for an agent failure. - Muse Spark then analyses the transcripts and refines the tests. Version regression mode (Hermes): - Given two framework commits, Muse Spark reads the code difference and generates tests aimed at what changed. Each test is labelled framework-level or task-level and comes with a predicted result for both versions. - The tests run on both versions. Old results are stored by commit, and each scenario is marked improved, unchanged, or regressed. We hypothesize that frameworks reliably enforce their own properties, such as approval gates and tool formats, but can’t enforce task-specific correctness. Testing several frameworks and versions shows which kinds of correctness hold up.