# Intent2Action

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Allwin Lucious](https://cerebralvalley.ai/u/AllwinLucious)
- **GitHub:** https://github.com/allwinlucious/Intent2Action
- **Demo video:** https://www.youtube.com/watch?v=F9hZ5y5LCBk
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3923

Intent2Action lets someone who knows a task teach it directly to a robot. The operator can show a short video, describe the task aloud, or type an instruction. The robot then carries it out in a new scene,
 with no robot code required from the operator.

 My starting hypothesis was that many robot-task failures come from missing physical experience. Model size and API documentation cannot tell a model everything about a particular robot. Details such as gripper width, joint limits, reach, and lag only become clear when the robot runs.

 I built Intent2Action around that idea. The model converts the demonstration and instructions into explicit rules that the operator can review. A fine-tuned model uses those rules to write a short program with actions such as find, pick, place, and verify. The robot stays still until the operator presses Run.

When an action fails, the system can retry it, skip it, patch the program, ask the operator for help, or stop. Every recovery step requires human approval.

I evaluated supervised fine-tuning on simulated robot experience, self-training, preference tuning, and reinforcement learning. I then compared base, hosted, and fine-tuned models under the same conditions on held-out tasks. Supervised fine-tuning on robot experience was the only approach that consistently improved performance. A second round of experience training taught the model another skill: pushing objects it could not grasp.

 I validate Intent2Action in two settings. The first is a benchmark against larger hosted models using tasks excluded from training. The second is a set of live demonstrations, including the submission video, recorded in real time with every model call and robot event logged.

  Code, models, complete results, and ablation studies:
  https://github.com/allwinlucious/Intent2Action

## More from Meta Global AI Developer Hackathon

- [Secure Sense Innovations](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3920)
- [Fortune Hunter](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3921)
- [Souls of Muse](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3922)
- [HOLOFIXER](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3924)
- [Mouad El Abbassi](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3925)
- [KEREN](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3926)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3923. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt