Traceboard

We built a **self-improving AI agent that learns how to make machine-learning models better through experience**. Using PCB defect classification as a real-world testbed, the agent trains a model, diagnoses its failures, autonomously chooses an experiment, measures the result, and learns from it. Most importantly, it remembers successful **and failed** experiments, so the next run doesn't start from zero. The problem we're solving is bigger than PCB inspection. Today, when an ML model underperforms, an ML engineer manually studies metrics and confusion matrices, forms a hypothesis, changes the training strategy, retrains, and repeats. AI agents can automate pieces of this process, but they often have the same problem: **every new run is a cold start**. They repeat experiments and rediscover lessons they have already learned. Our agent turns those experiments into persistent experience. The system follows an autonomous loop: **Train → Evaluate → Retrieve Experience → Diagnose → Experiment → Retrain → Critique → Learn → Repeat** We demonstrate this using real PCB manufacturing defect data. The agent begins with a deliberately weak classifier and evaluates its performance across different defect classes. If, for example, open-circuit recall is significantly worse than other classes, the agent doesn't blindly tune random hyperparameters. It analyzes the evidence, determines a likely cause, searches its previous experience for similar failures, forms a hypothesis, and chooses a specific intervention such as weighted sampling, stronger augmentation, different image resolution, class weighting, learning-rate changes, or a different model strategy. After retraining, the agent compares the before-and-after results. Successful experiments become reusable knowledge, but failed experiments are equally important: the agent remembers strategies that did not work so it can avoid wasting time repeating them. ## MongoDB Atlas — Persistent Experience and Agent State MongoDB is the core memory layer of the system, not simply a database for application logs. Every experiment produces structured experience containing the model configuration, failure pattern, diagnosis, intervention, metrics before and after the intervention, whether the hypothesis was correct, and the lesson learned. For example, the agent can remember: > **Failure:** Very low recall on a minority PCB defect class > **Intervention:** Weighted sampling > **Outcome:** +9% Macro F1 and +21% minority-class recall > **Lesson:** Weighted sampling was effective for this type of class-imbalance failure. When the agent later encounters a semantically similar failure, **MongoDB Atlas Vector Search** retrieves relevant previous experiences. Those memories are provided to the AI scientist before it decides what experiment to run. This means MongoDB directly changes the agent's next action. MongoDB also persists the **LangGraph checkpoints and workflow state**, including the current experiment, iteration, best configuration, metrics, hypotheses, and experiment history. An interrupted autonomous run can therefore resume without losing what it was doing. This gives us two forms of persistence: **Short-term state:** Where is the agent in the current experiment? **Long-term memory:** What has the agent learned across previous experiments and runs? ## Voyage AI — Semantic Memory We use **Voyage AI embeddings** to turn the agent's learned experiences into semantic representations stored with its MongoDB memories. This allows the agent to retrieve experiences based on meaning rather than exact wording. For example: > “Tiny open circuits have poor recall.” can retrieve an older lesson involving: > “Small localized defects improved after increasing image resolution.” Even though the descriptions are different, their underlying failure patterns are related. Voyage therefore makes the agent's accumulated experience searchable through MongoDB Atlas Vector Search. ## Fireworks AI — The ML Scientist Fireworks is the primary reasoning engine behind the autonomous ML scientist. After every evaluation, Fireworks receives the classifier's current configuration, overall metrics, per-class precision/recall/F1, confusion matrix, dataset characteristics, recent experiment history, and relevant memories retrieved from MongoDB. It then performs: **Observation → Diagnosis → Hypothesis → Experiment** For example: > **Observation:** Open-circuit recall is dramatically lower than other classes. > **Diagnosis:** The minority class may not be receiving enough exposure during training. > **Hypothesis:** Weighted sampling should improve minority-class learning. > **Experiment:** Enable weighted sampling. Fireworks does not generate arbitrary training code. It chooses from a controlled set of executable ML experiments, which keeps the autonomous system reliable. After the experiment finishes, Fireworks acts as a **critic**. It compares the metrics before and after the intervention, determines whether the hypothesis was supported, and distills the result into a reusable lesson. That lesson is then written back to MongoDB, closing the learning loop. ## OpenRouter — Independent Second Opinion We use OpenRouter as an independent evaluator rather than duplicating Fireworks. Most experiments are handled entirely by the primary Fireworks scientist. OpenRouter is brought in when the decision deserves additional scrutiny—for example, when Fireworks has low confidence, several experiments have failed consecutively, or the agent proposes a significant change in strategy or model family. OpenRouter evaluates the current evidence and Fireworks' proposed intervention and can agree, disagree, or suggest reconsideration. This creates a lightweight **scientist + reviewer** architecture while avoiding unnecessary model calls on every iteration. ## LangGraph — Autonomous Orchestration LangGraph coordinates the complete self-improvement process: **Train → Evaluate → Retrieve Memory → Diagnose → Propose → Judge → Retrain → Critique → Store Lesson → Repeat** Rather than scripting a fixed sequence of hyperparameter changes, LangGraph allows the next step to depend on the agent's current state and previous outcomes. MongoDB backs the LangGraph checkpoints, connecting orchestration and persistence into one continuous agent workflow. ## Kaggle PCB Dataset — Real Learning Environment We use a real PCB defect dataset from Kaggle as the environment in which the agent learns. The dataset provides real PCB images and labeled manufacturing defects. We split the data into training, validation, and untouched test sets. The agent repeatedly trains on the training data and uses validation performance to decide what to improve. The final model is evaluated against the untouched test set so that improvements represent genuine generalization rather than optimization against the test data. The PCB classifier itself is intentionally not the main innovation. It gives the agent an objective environment where every hypothesis can be tested and scored. ## What We Prove Our key metric isn't simply: **Classifier F1: 61% → 86%.** That only proves that the classifier improved. The more important comparison is: **Cold-start agent:** requires multiple experiments to reach the target. **Experienced agent:** retrieves relevant MongoDB memories, avoids previously unsuccessful approaches, and reaches the same target in fewer experiments. That demonstrates that the **agent itself has improved**. The classifier learns from data. **Our agent learns from experience.** Every experiment becomes a memory, every failure becomes a lesson, and every new run starts with everything the agent learned before.