SplatForge

SplatForge is a robot manipulation agent that trains itself inside the worlds it reconstructs. Point a phone at a tabletop and it turns the scene into a photoreal 3D Gaussian Splat, a digital twin you can view from any angle, then drops a robot into a physics replica of that scene and teaches itself to "pick up the mug." Robot policies normally improve only by consuming human teleoperation and labeling, which is the slow, expensive bottleneck on every manipulation project. SplatForge's bet is that a robot can write its own training data and improve itself, closing a fully autonomous loop. The loop runs in six steps with no human in the middle. First it reconstructs a real scene into a Gaussian Splat. Then an LLM curriculum generates task variations and edge cases (the mug rotated, occluded by a bowl, lit more dimly) and emits them as structured scene specs. The policy attempts each scenario in a MuJoCo physics simulator, and a vision language critic scores every rollout with a pass or fail verdict. Successful trajectories are distilled into a LoRA fine tune of the policy, while failures become the next round's targeted curriculum, selected by vector search over embeddings of past failures. The loop repeats on its own, and a live success rate curve climbs as it goes. The central design insight is that a Gaussian Splat is a renderer, not a physics engine, so it cannot tell you whether a grasp actually succeeded. That splits the work in two. The splat is the eyes and skin, providing the photoreal views the critic scores, while MuJoCo is the body and physics, where the robot actually acts and where grasp success is measured by whether the mug clears a height threshold. The robot acts in MuJoCo and is shown in the splat, so the camera tells one coherent story: it learns inside the very scene it reconstructed. This fits the hackathon's focus on continual learning and recursive self improvement. The system measurably improves the more it runs, generates its own training data, updates its own weights, and uses the VLM critic to evaluate itself. Under the hood it runs on Python, PyTorch, and MuJoCo, which handles physics and the success signal. It runs on CPU so it works on a laptop, including Apple silicon. Gaussian splatting through gsplat and Nerfstudio builds the digital twin, and LoRA/PEFT adapters drive the fine tuning loop, trained on DigitalOcean GPU droplets. A FastAPI orchestrator runs the closed loop behind a React dashboard with a live success rate chart and a splat viewer. It leans on several sponsors: DigitalOcean for the fine tuning GPUs, Google Gemini as both curriculum generator and visual critic, MongoDB Atlas with Voyage AI as the trajectory replay buffer and failure vector search, and optionally LiveKit for voice control and MiniMax as an alternative LLM or VLM. Per the rules, the off the shelf pieces, the splat renderer and the base foundation models, are clearly separable from what we built during the event: the loop orchestration, the curriculum generator, the self critic evaluation harness, the LoRA fine tuning loop, and the dashboard that ties it all together.