GraspGrade

OVERVIEW GraspGrade turns human egocentric hand videos into robot training data. It cuts each video at the moment a hand touches an object and scores every clip from 0 to 100 for how well a robot hand could copy the grasp, using only the video with Meta Muse Spark and SAM 3.1. We checked it against real sensors (touch gloves and Meta HOT3D motion capture) and release those sensor labels as an open benchmark. Try it yourself at https://graspgrade.vercel.app/try: pick a held-out video and follow each step, from the detected touch to the final score and the grades behind it. PROBLEM - Robot hand teams want human egocentric video because it is cheap, easy to collect and easy to scale. - Most clips cannot teach a robot hand. Fingers are hidden, objects are too large for a robot hand, or the moment of touch is unclear. In one recent study, only 83 of 2,000 internet clips (4%) passed a quality check (Do as I Do). - Today clips are picked by looking at the video. There is no check against real contact or against a real robot hand, and no shared score that teams can reuse. WHAT IT DOES - Input: a human egocentric video of hands using objects. Video is the only input. - Output: clips cut at each touch, each with a 0 to 100 copyability score and the grades behind it. HOW IT WORKS 1. Find the touch (Muse Spark): Muse Spark reads frames at 5 fps and says, for each hand, whether it touches an object. A timing shift learned on tuning data from the glove (+7 frames at the start, -7 at the end) corrects the fact that video shows a touch before pressure is felt. 2. Track (SAM 3.1): SAM 3.1 tracks both hands and every object at 10 fps with the prompts "hand" and "small object". Its masks give two grades: visibility and contact clarity. 3. Grade (Muse Spark): Muse Spark answers seven checklist questions per clip from six frames: object size next to the hand, shape, thickness, width to wrap, handle, free space around it, and hidden fingers. Each answer is high, mid or low. 4. Score: a logistic regression on twelve grades (seven Muse Spark grades, two SAM 3.1 grades, three local video checks) gives the 0 to 100 score. Weights are learned on 400 tuning clips and every weight is visible. GROUND TRUTH FROM SENSORS (used only to check, never as input) - Contact timing: OpenGraph Ego-Tactile, 72 episodes at 30 fps with tactile gloves on both hands (80 sensors per hand). A hand is in contact when the sensor sum is over 800. Gaps of 0.1 s or less are joined and pieces under 0.1 s are dropped. Split: 50 tuning, 22 held-out episodes. - Copyability: Meta HOT3D (Project Aria), motion capture of hands and 33 rigid objects. Each human grasp is moved onto an Allegro robot hand by fingertip position, with the robot wrist allowed to shift up to 2 cm. A grasp is copyable if the robot thumb and one more fingertip are within 1.5 cm of the object surface (grip), every robot fingertip is within 2 cm of the matching human fingertip that touches the object (reach), and no part of the robot hand goes more than 5 mm into the object (collision). 47% of grasps are copyable. Split by person: 2,385 tuning clips from 6 people, 1,426 held-out clips from 3 people. HOW WE EVALUATED - Every comparison, metric and pass rule was written in a measurement plan before the held-out data was scored. Each final result was computed once. - Contact timing metrics: boundary F1 at ±5 frames (0.17 s) and ±15 frames (0.5 s), contact IoU, paired bootstrap over episodes. Compared: Muse Spark alone, Muse Spark with the glove shift, a SAM 3.1 rule, the SAM 3.1 rule plus a Muse Spark boundary check, and "always in contact". - Copyability metrics: AUC with a 95% interval from a bootstrap over the 51 held-out sequences, AUC per person, share of copyable clips in the top 20%. Compared: Muse Spark asked directly with no checklist, an object name reference (the copyable rate of each object from tuning labels), and every learned part refit without the object being scored (unseen objects). RESULTS (held-out) - Contact timing (22 episodes, 574 glove boundaries): Muse Spark with the glove shift is best on all metrics. Boundary F1 0.195 at ±5 frames and 0.376 at ±15, contact IoU 0.850. Muse Spark without the shift: 0.074, 0.273, 0.821. SAM 3.1 rule: 0.090, 0.214, 0.667. The shift adds +0.121 F1 (95% interval 0.062 to 0.172). - Copyability (1,426 clips): our score AUC 0.624 (0.589 to 0.661). Muse Spark asked directly 0.501 (0.469 to 0.529). In our top 20%, 59.4% of clips are copyable against a base rate of 46.4%. - Where the gain comes from: asking through the checklist adds +0.117 (0.073 to 0.166). Learning the weights adds +0.005 (not significant). The SAM 3.1 and local checks add +0.001 (not significant). - Unseen objects: our score 0.571 (0.534 to 0.609). The object name reference cannot score an unseen object (0.5). - By failure reason: AUC 0.611 for collision, 0.636 for reach, 0.682 for grip. LIMITS WE REPORT - For objects seen in tuning, the object name reference is better (0.771), because held-out people grasp the same 33 objects. - Inside one object the score is at chance (0.506). It ranks objects, not grasps of the same object. - With a strict 2 mm collision limit the score drops to 0.521. At the 5 mm limit fixed in advance and at 10 mm (0.632) it holds. - A soft object filter (Muse Spark asked if the object is soft) failed its rule: 51.7% of soft segments caught against a rule of 70%, while only 2.1% of rigid clips were held back. - The robot check is geometric with one robot hand. Both datasets are lab recordings, and the copyability held-out set has three people. WHAT WE RELEASE - An open benchmark: contact labels for 72 episodes (CC BY 4.0) and copyability labels for 3,811 clips with the failing condition (CC BY-NC-SA 4.0), plus standalone scorers that reproduce our numbers. - The measurement plan, 42 experiment pages with setup, numbers and cost, a paper, the code, and a live site with a demo video and a step by step view of six held-out clips. - Meta Model API use: $143 of $150 credits, with no unparsed or missing answers in the final runs.