AutoCut

AutoCut — an autonomous AI video editor. You upload raw footage and give a one-screen brief ("Viral Short, 9:16, 30s, loop back to the first frame"), and an agent does the rest of the editing. How it works Offline preprocess (src/preprocess/) turns raw clips into a machine-readable library: - ffprobe for duration/resolution, Whisper (via Replicate) for word-level transcripts, scene detection + a VLM for per-scene captions and tags, and now a poster frame per clip for the UI. - Output is clips.json — the structured contract the agent reasons over directly (no ffmpeg in the agent's hands). The agent loop (src/agent/, src/loop/) reads clips.json, picks segments, and emits a schema-validated EDL (edit decision list). A deterministic renderer (src/edit/, ffmpeg) compiles the EDL into the final cut, and an independent verifier returns a structured grade. Those zod schemas in src/loop/types.ts are the single source of truth at every boundary, so a bad model output fails fast instead of three modules downstream. The UI (Next.js, app/ + components/) is a two-screen flow: Setup (the footage library grid + the brief form) and Generation. Built with shadcn/ui + Tailwind. Infrastructure - Supabase Storage is the source of truth for assets: source-clips (private — videos + manifest), renders (public — final mp4), thumbnails (public — poster frames). Daytona persistent disk is a download cache. Supabase Postgres + pgvector holds caption embeddings for semantic clip search. - Models: Opus 4.8 drives the agent (not Fable 5); Replicate hosts Whisper + the VLM. Where it stands Preprocess and the core loop work and are well-tested (~193 unit tests). Known gaps: the agent server still hardcodes a SAMPLE_LIBRARY instead of loading real clips.json (agent-server.ts:141,274), the Daytona caching layer is specced but not built, and transcription is on base Whisper with WhisperX (true word-level) as the agreed next step. This session's work: rolled per-clip thumbnail generation into the preprocess pipeline and got the poster frames rendering on the Setup screen's footage grid.