# Gemini CUA Skill Loop

- **Event:** [AI Engineer World's Fair Hackathon 2026](https://cerebralvalley.ai/e/aiewf-hackathon-2026)
- **When:** Jun 27 at 9:00 AM – Jun 28 at 5:00 PM (PDT)
- **Where:** San Francisco, CA
- **Team:** [Renan Serrano](https://cerebralvalley.ai/u/renanserrano-2)
- **GitHub:** https://github.com/renantrendt/gemini-cu-skill-loop
- **Demo video:** https://youtu.be/oye4QkJkveY
- **Gallery:** https://cerebralvalley.ai/e/aiewf-hackathon-2026/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/aiewf-hackathon-2026/hackathon/gallery/48

Skill-Loop for Gemini Computer Use
A training-free loop on Gemini 3.5 Flash Computer Use. A kept skill must have passed a verified retry. We measure two things separately: (A) the reliability of a kept skill under re-rolls, and (C) whether the live distillation loop converges in one shot from scratch.

Abstract
Computer-use agents re-attempt every task from scratch and repeat their mistakes. We build a loop that watches Gemini 3.5 Flash Computer Use fail a task, distills a reusable skill from that failed trajectory, retries with the skill in context, and keeps the skill only if a verified retry passes, otherwise it is discarded. No fine-tuning, no weight updates, no reinforcement learning.

We measure two distinct things and keep their results separate. First, the reliability of a kept skill, i.e. re-rolls of the same task with the kept skill held byte-identically in context vs no skill: on ArXiv--23 from the WebVoyager benchmark, graded by WebVoyager's own automatic evaluator, the bare agent failed 3/3 trials under a fixed 30-action budget (budget-exhausted fighting an arXiv form widget) and the agent with the pre-distilled kept skill loaded byte-identically into context passed 3/3 (19–29 actions). The kept skill was earned in a separate prior live run (§3.8). The same kept skill generalised to two held-out tasks from the same template (N=1 graded run each, no held-out baseline; passes are consistent with generalisation but do not establish a pass-rate lift). Second, the boundary of where distillation can help: on Apple--0, a mechanics-bound task whose custom JavaScript dropdown cannot be reliably operated by a pixel-only agent, the keep-gate correctly refused to persist a non-working skill. The combined empirical map: strategy- fixable failures → the loop learns a fix; widget-mechanics failures → it correctly refuses to pretend it can.

A separate experiment (§5.4) running the full live loop from scratch, fresh distillation at every iteration, no pre-loaded skill, converged on iteration 3: the third distilled skill (category scroll-and-read, passed in 9 steps) was persisted by the verified keep-gate. Iterations 1 and 2 (form-filling, url-construction) failed and were discarded. The keep-gate's category-diversity prompt, which forces each iteration to propose a strategy in a different category from prior rejects, was what unblocked convergence, iters 1 and 2 in v1 (no diversity) had cycled inside the same keyboard-shortcut basin. We report both A and C honestly: A measures reliability of a kept skill on re-rolls; C measures single-shot convergence of the live loop from scratch.

To our knowledge, no open-source library combines failure-driven skill learning with a verified keep-gate wired to Gemini Computer Use; we release this as infrastructure papers and benchmarks can build on.

---

Markdown version of https://cerebralvalley.ai/e/aiewf-hackathon-2026/hackathon/gallery/48. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt