# Replay

- **Event:** [The Agent Arena Hackathon](https://cerebralvalley.ai/e/vultr-the-agent-arena)
- **When:** Sep 26 at 9:00 AM – Sep 27 at 5:00 PM (PDT)
- **Where:** San Francisco, CA
- **Team:** [pranav achar](https://cerebralvalley.ai/u/prnav)
- **GitHub:** https://github.com/PranavAchar01/agent-arena
- **Demo video:** https://replay-so101.vercel.app/replay-demo.mp4
- **Gallery:** https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/6

ChatGPT learned by reading the internet. Robots can't: every robot skill today is recorded by hand, one task at a time, and the new robot models (VLAs) are starving for that data. Replay turns the open web into robot skills from one sentence.

How it works: an agent turns the prompt into a search plan. Scrapling scrapes inside a throwaway Vultr VM per robot (Creative Commons YouTube video and web pages, licences checked), a vision model checks each clip, and MediaPipe tracks the person's hand and body. The motion is retargeted onto the SO-101's joints and replayed on hundreds of new table layouts in MuJoCo, keeping only the tries that work (the same idea as NVIDIA's MimicGen). A small policy then learns by imitation with ACT-style action chunking; for rule-based tasks the moves are planned instead, and every move is rehearsed in physics. Nothing reaches the robot until it has worked in simulation.

Results (simulated SO-101 in MuJoCo): block in the bowl 19/20 unseen layouts (policy trained in 21 s on a CPU), block tower 18/20, a dumbbell curl copied 1:1 from Creative Commons video (elbow within 1.25 degrees), Tower of Hanoi in the optimal 7 moves, a six-cup pyramid, and Deep Blue vs. Kasparov 1997 Game 6 played on both sides (37 moves, worst placement 4.9 mm). Every skill can be downloaded. Next: a real SO-101 hardware test and fine-tuning SmolVLA on Replay's data.

## More from The Agent Arena Hackathon

- [Pengwin Crew](https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/3)
- [GhostRange](https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/4)
- [Market Watch Commander](https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/5)
- [Billy, the RFP Agent](https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/7)
- [Team JC](https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/8)
- [Upgrade Chamber](https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/9)

---

Markdown version of https://cerebralvalley.ai/e/vultr-the-agent-arena/hackathon/gallery/6. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt