nsega

The project "RouteCause" is an autonomous diagnostics agent for LLM inference infrastructure on Kubernetes. When shared model-serving degrades, it investigates and hands platform/infra engineers a root-cause report with a concrete fix — turning a slow, manual, four-source correlation into one automated run. The problem: LLM serving on Kubernetes (Gateway API Inference Extension / InferencePool / Endpoint Picker (EPP) scheduling, now llm-d Router) usually degrades because of routing and scheduling misconfiguration, not capacity. Symptoms — P95 Time to First Token(TTFT)/latency breaching SLO, queue depth piling on some replicas while others idle, KV-cache saturation, prefix-cache hit-rate collapse after a rollout — require cross-referencing gateway/EPP metrics, scheduler config, endpoint health, and recent manifest diffs. When you're paged on an SLO breach, that correlation is slow and error-prone. How it works: an SLO breach (or a manual POST /diagnose) triggers the agent. Context-isolated subagents each pull one evidence source — Prometheus metrics, the EPP ConfigMap, and live cluster/endpoint state — and form hypotheses independently, so no single context owns all the evidence. Each surviving hypothesis then faces an independent verifier subagent whose job is to falsify it. The output is a single root-cause report: the diagnosis, ≥2 pieces of cross-source evidence, and a fix as a manifest diff or "kubectl patch" validated with "kubectl apply --dry-run=server". It targets three fault classes — EPP scorer-weight misconfig, an unhealthy endpoint left in rotation, and prefix-cache routing silently disabled by a rollout. Stretch goal: apply the fix and confirm recovery against the SLO. Scope and provenance: the agent was built today in "nsega/routecause". The Kubernetes test environment (kind cluster + llm-d Router + InferencePool + simulated backends + fault injection) is a pre-existing project, "nsega/inference-lab", brought in as scaffolding and referenced, not rebuilt — see PROVENANCE.md for the build-today vs. brought-in boundary. The deliverable is the agent and its reports; there is no metrics dashboard.

Team

Placement

Participant