gradepack

gradepack is an installable registry of 42 realistic bug-fixing tasks, each with a starter, a pytest grader and a verified reference solution, for testing AI coding agents. It runs one-shot and agentic (multi-turn) benchmarks with pass@k and failure classification, and it has a leaderboard, a static API, an HTTP API and an MCP server. It solves a gap in agent evaluation: benchmark tasks are rarely checked for fairness, full specification or difficulty. gradepack includes a task factory in which Muse Spark Open drafts new tasks, a lint gate and an independent model judge review them, and probe runs keep only tasks the model sometimes fails. The review step caught an ambiguous prompt in task 030, which we fixed. Five factory tasks were added, including two of the hardest (048 and 050). Results with Muse Spark Open (42 tasks): 92.9% pass@1 and 95.2% pass@3 one-shot. Agentic mode added no gains, with 78 of 84 attempts solved at turn 1, and the remaining failures are timeouts and token limits. We report this as measured.