Hack Singh

AgentProof is an automated multilingual agent-evaluation harness designed to measure capability gaps, tool-calling consistency, and run-to-run variance across Indic languages (English, Hindi, and Punjabi). THE PROBLEM IT SOLVES: AI agents are overwhelmingly evaluated on English-only benchmarks. When models are deployed in multilingual environments, tool invocation degrades, date normalization fails, and run-to-run variance spikes. Without standardized multilingual evaluation harnesses, these agentic reliability gaps remain completely invisible to developers. HOW IT WORKS: 1. Standardized Benchmark: Tests the agent across 6 tasks (arithmetic planning and temporal lookup) across English, Hindi, and Punjabi, repeated 3 times each (18 runs total). 2. Dual-Scoring Harness: Every run is evaluated by both a deterministic Python code checker (normalizing numbers and multilingual scripts) and an LLM rubric judge with structured JSON output enforcing correctness, tool adherence, and language fidelity. 3. Real Empirical Findings: In our evaluation of Meta Llama 3.2 3B, English scored 50% (3/6), Hindi scored 0% (0/6), and Punjabi scored 50% (3/6). The harness captured genuine failure modes: on the English calculator, the agent hallucinated an extra ×2 factor (answering 42 km/h instead of 84 km/h); on the Hindi calendar, it failed to normalize Devanagari month strings, leading to a tool error and a hallucinated day (Saturday instead of Sunday); while on the Punjabi calculator, it passed 3/3 with 84 km/h. DELIVERABLES: - Interactive Streamlit dashboard with KPI metrics, variance charts, and full tool-call trace inspector. - Automated Model Card (MODEL_CARD.md) following Meta conventions and raw CSV data export.