Task Factory

Task Factory turns real, merged LLM inference and GPU kernel speedups into verified training tasks for coding agents, graded on real Android phones. Each task gives an agent an on-device inference engine at the commit before a real optimization, for example an OpenCL GEMV or attention kernel rewrite that made SmolLM2 decode 1.5x faster on an Adreno GPU. The agent must make decode faster while producing exactly the same output tokens. Reward is the decode tokens/s measured on the phone. The problem: RL tasks for kernel and inference optimization are written by hand, slowly, and their graders are easy to game. An agent can hardcode the expected tokens, fake its own benchmark timer, or skip compute outside the checked range, and still score full marks. Task Factory automates the pipeline. Gemini 4 Argon reads a whole model port plus its git history in one call (592k tokens) and picks commits with real, measurable inference speedups. It writes and reviews a task proposal and packages it as a Harbor task. Then it tries to break the task before shipping it: - Phone checks: unchanged code must score 0 and the real optimization must score 1, timed outside the binary on Pixel 5 GPUs. - Red team: Argon managed agents try to cheat. In our first task, 2 of 4 attacks beat the grader (replaying output tokens and faking the timer). A fixer agent closed 4 holes, and the grader now scores both cheats 0. - Rollouts: agents attempt the optimization to check its difficulty. Argon reaches 1.32-1.35x of a 1.44x target (reward 0.74-0.80): partially solved, which is the learning signal labs want. Every step is in an append-only ledger, and a live monitor shows each stage, agent and phone run.

Team