FDev

NutriMind AI — Multimodal Personal Nutrition & Activity Tracking Agent Unlike generic hackathon chatbots that attempt to give risky, unqualified health advice, NutriMind AI is engineered strictly as an autonomous tracking agent and cultural data-logging utility. It eliminates daily journaling friction by turning multimodal inputs (voice notes, food photos, and natural text) into direct on-device Room DB mutations via structured tool calling (json:action). It also tackles global AI bias by accurately decomposing regional Southeast Asian dishes (Nasi Goreng, Rendang, Gado-Gado) that Western models ignore, with 100% local-first privacy. Inspiration Keeping a consistent record of meals, hydration, and physical activity is one of the most effective ways to maintain daily mindfulness, yet conventional tracking workflows present significant friction. Manually looking up ingredients, estimating serving weights, and typing every individual food item creates daily friction and tracking fatigue that causes most users to abandon tracking. In addition, traditional trackers overwhelmingly cater to standardized Western diets and frequently fail to recognize regional, Southeast Asian, and multicultural meals. Most conventional AI chatbots remain passive: they output conversational paragraphs of text but lack the agency to interact with mobile databases, forcing users to manually transcribe numbers. We built NutriMind AI to make daily lifestyle tracking intuitive, culturally inclusive, and frictionless. By leveraging Meta's multimodal vision, audio, and reasoning models, NutriMind AI connects AI-generated structured actions directly to an on-device data layer, transforming passive tracking into an autonomous, personal tracking agent that adapts to users' real daily habits. What It Does NutriMind AI is an autonomous multimodal personal nutrition and activity tracking agent for Android that simplifies lifestyle journaling directly on your device: 📸 Multimodal Food Recognition (Vision AI & Regional Cuisine): Users capture or upload meal photos. Powered by Meta Llama 3.2 Vision, NutriMind AI identifies visible food items, deconstructs complex mixed plates (separating visible proteins, carbohydrates, side dishes, and sauces), and provides approximate calorie and macronutrient estimates. It features specialized support for Indonesian and Southeast Asian dishes (such as Nasi Goreng, Gado-Gado, Satay, Rendang, Soto, Tempeh, Tofu, and Pecel). An interactive verification card (RecognizedFoodScreen) allows users to review and adjust estimates before saving them to their diary with a single tap. Defensive guardrails gracefully handle non-food images without hallucinating nutritional values. 🤖 Multimodal Personal Tracking Agent (Autonomous Tool Execution): An autonomous logging agent powered by Meta reasoning models (Llama 3.3 70B / muse-spark-1.3). Users can naturally type, speak (via Meta voice transcription), or attach food photos in English or Indonesian (e.g., "I had 2 boiled eggs and drank 500ml of water" or "Tadi sore lari 30 menit dan minum 400ml air"). The agent interprets intent, outputs structured tool directives (json:action), and autonomously logs food, water, workouts, or weight updates directly into the local database—instantly updating daily rings and progress metrics in real time through reactive Kotlin StateFlow. 🎯 Context-Aware Journal State: Dynamic context injection feeds the user’s live daily diary, remaining calorie/macro targets, hydration status, active exercise burn, and baseline biometric targets (BMR/TDEE calculated via the Mifflin-St Jeor equation) directly into the agent's prompt for situation-aware logging accuracy and multi-turn conversational memory. 🏃‍♂️ Health Connect & Smartwatch Sync: Natively integrates with Android Health Connect to automatically sync daily step counts, smartwatch activities, and active calories burned directly into the user's daily energy balance. 📊 Daily Balance Scoring & Summary: Calculates an end-of-day daily completion score (0–100) alongside a concise 3-sentence summary generated by Meta AI highlighting recorded milestones, accompanied by dynamic home dashboard summary cards that support consistent habit formation. 🎨 Shareable Milestone Card: Generates high-resolution visual snapshot cards via Jetpack Compose off-screen rendering (rememberGraphicsLayer) and Android FileProvider for one-click sharing to Instagram Stories, WhatsApp, and messaging apps. ⏰ Context-Aware Smart Reminders: Background hydration and meal reminders powered by Android WorkManager with an Intelligent Skip mechanism that automatically checks recent database records and suppresses alerts if the user has already logged their intake for that window. How We Built It Platform & Modern Android Architecture: Language & UI: 100% Kotlin 2.0 with Jetpack Compose, Material 3, and Plus Jakarta Sans typography. Architecture: Clean Architecture with unidirectional data flow (MVI/MVVM), Kotlin Coroutines, StateFlow, and Dagger Hilt for dependency injection. Persistence & Local-First: Room Database (v15) with schema migrations and DataStore Preferences, ensuring reliable offline functionality and user privacy. Sensors & Hardware: CameraX for photo captures, Google ML Kit Barcode Scanning with OpenFoodFacts API for packaged groceries, and Android Health Connect API for automated steps and active burn synchronization. AI Engine & Meta Model Ecosystem: Meta Llama 3.2 Vision / Muse Spark (muse-spark-1.3) for image analysis, portion estimation, and multi-component plate decomposition. Meta Llama 3.3 70B Instruct / Muse for contextual intent interpretation, multi-turn conversational memory, and structured json:action tool calling. Meta Voice Transcribe / MMS (muse-voice-transcribe-1.0) for hands-free voice message logging. Multicultural Cuisine Awareness: Prompted and validated across diverse regional cuisines—accurately recognizing Southeast Asian and Indonesian staples (such as Rendang, Gado-Gado, Tempeh, Tofu, Soto, and Nasi Goreng) alongside global dishes. Challenges We Overcame Multi-Component Food Recognition & Estimation Uncertainty: Real-world meals often contain several ingredients mixed on a single plate. We designed structured JSON schemas with portion bounds to help the vision model identify distinct meal components and provide realistic nutritional approximations. Because image-based estimates involve inherent uncertainty, we built an interactive verification step (RecognizedFoodScreen) so users can review and adjust values before committing them to local storage. Safe Autonomous Tool Execution: Connecting conversational LLM outputs to local database operations requires a strict validation boundary. We engineered an action parsing and execution pipeline to extract validated structured directives (LogFood, LogWater, LogExercise, LogWeight) and safely commit them to Room DB, ensuring immediate UI responsiveness through reactive StateFlow without hallucinations or UI freezes. (Note: The codebase internally uses the class identifiers CoachActionParser and CoachActionExecutor as architectural execution components; the product documentation and UX reflect their role as an autonomous tracking agent). Preventing Reminder Fatigue: Users often disable notifications if they trigger redundantly. We implemented intelligent checks in our WorkManager periodic workers to verify recent intake in the local database before scheduling alerts, maintaining a non-intrusive user experience. What We Learned & What's Next Building NutriMind AI demonstrated how Meta's vision, voice, and reasoning models can bridge the gap between passive chatbots and proactive, autonomous mobile agents that directly execute on-device data workflows. Moving forward, our roadmap includes: On-Device Inference via ExecuTorch: Bringing lightweight Llama 3.2 models directly onto Android devices using ExecuTorch for zero-latency, privacy-centric offline dietary logging. Wear OS Companion Extension: Developing dedicated Wear OS tiles and complications for glanceable calorie rings and one-tap wrist hydration logging. Scope & Non-Medical Utility Notice NutriMind AI is engineered strictly as an autonomous tracking utility and cultural dietary journaling agent designed to eliminate manual logging friction. It does not provide medical diagnoses, clinical treatment, eating-disorder intervention, or professional medical advice. Nutritional and energy values are informational approximations intended for personal lifestyle tracking and record-keeping.