Unity AI Mapper

UnityMapper runs on two Meta models, both called through the Meta Responses API (api.meta.ai/v1/responses): Muse Spark 1.3 for reasoning and SAM 3.1 for segmentation. They are the only models in the pipeline, and every request goes through a reviewed allow-list (model, fields, image count) before it is sent. Muse Spark 1.3: the agent's brain Every decision is a single multimodal call (text plus up to 12 images) with a strict JSON-schema output, so nothing reaches Unity until it has been validated against a schema. There's no hidden tool loop: one reservation, one transmission. We use it for: - Brief outline: rooms, dimensions, style and functional zones. - Catalogue review: it reads the project's asset catalogue page by page (20 pages for the demo room) and keeps or drops each prefab with a reason. - Asset review: it looks at two opposite preview renders of a prefab to find its functional front and how it must be supported (floor, tabletop, shelf, wall). - Layout and layout repair: zone anchors and rotations, re-planned when the geometry solver rejects a proposal. - Lighting intent: it reads the brief to choose a reference daylight setup taken from a real scene. - Visual inspection: it reviews Unity captures together with SAM evidence and native geometry, first zone by zone, then for the whole room. - Correction and enrichment: it proposes fixes and additions, and writes the honest list of unresolved requirements. - The project chat interface. SAM 3.1: the agent's eyes After each build step, Unity renders interior viewpoints. For each view, SAM 3.1 receives a concept prompt derived from the placed prefabs ("bed", "chest of drawers", "rug", "lamp"…) and returns one-bit masks with confidence scores. The agent: - compares masks with the bounds Unity projects for each object (IoU) as evidence of presence and placement; - renders the actual masks as colour overlays in the live inspector and in the demo video (517 segmented views in this run); - caches results by image hash and concept, and never re-sends an identical request. If a SAM result is missing, it falls back to the original pixels and native geometry instead of guessing. Cost and reliability A local ledger reserves budget before every Muse Spark or SAM call, using Meta's published per-token and per-image pricing, and records actual usage afterwards. A call with an unknown outcome is never replayed automatically.