EvalVerse

Agents are moving from reading text to seeing images, hearing voices and watching video. But most evaluation still judges only text and the final answer. Images, voice and video are reduced to text first, so a misread receipt or a wrong moment in a video still gets a passing score, and nothing checks whether the agent really understood the original input. EvalVerse is a multimodal evaluation and observability platform for AI agents. It records every agent run as a trace, with the image, audio or video kept inside it, plus each step, model call and tool call. It checks every answer against the evidence in the file, such as the region of an image or the moment in a clip, and aggregates the results. Each run gets a label that names the stage that failed: hearing, reading, locating or reasoning. Controls such as blur, noise and a no-media rerun show whether the file was needed at all, and cross-modal checks compare a spoken claim with a photo. It is a developer platform: a Python SDK, OpenTelemetry and REST ingestion, evaluators you can create (rubric or rule), offline experiments, online policies with alerts, tool-call checks, a cross-modal check (a spoken claim against a photo), and a Studio dashboard with traces, a playground, datasets and runs. Evaluation runs offline, on versioned datasets and experiments that compare models, and online, on live traffic with policies, alerts and human review. Judge models are audited with planted right and wrong answers. Scores are organised by dimension: perception, groundedness, temporal grounding, hallucination, completeness, reasoning, cross-modal consistency, media dependence, robustness, honesty, tool use and judge trust. A dimension with no data is shown as missing, never as zero. Everything rolls up into UMS, the Unified Multimodal Score: one number with a confidence interval and a release gate, always shown beside the dimensions behind it, so one score cannot hide a weak spot. Live Project: https://evalverse-one.vercel.app Docs: https://evalverse-one.vercel.app/docs/