rojak-lingo

Malaysians switch between English, Bahasa Malaysia, Mandarin, Cantonese and Manglish mid-sentence, and general-purpose models handle this unevenly. Rojak-Lingo is an open benchmark and a small fine-tuned model for it. The benchmark has 300 gold items (75 per task) across four tasks: normalizing rojak text to standard English, intent classification, mixed-language instruction following with programmatically checked constraints, and translation. The gold set was drafted with AI assistance, reviewed by me, and kept out of training. I generated about 2,350 synthetic training examples with Muse Spark, verified instruction examples against their own checks, removed duplicates, and filtered anything overlapping the gold set. I then trained a QLoRA adapter on Llama 3.1 8B Instruct. The main finding is that a fair baseline matters. Zero-shot, the fine-tune beat the base model by 33 chrF points on normalization. But three examples in the prompt closed nearly all of that gap (0.608 vs 0.594 chrF). The fine-tune matches few-shot prompting on intent, normalization and translation with a plain zero-shot prompt, saving about 1,000 prompt tokens per call, but it did not improve instruction following (0.773 vs 0.907 few-shot). Translation is the open problem: only Muse Spark moved it (0.755 chrF vs 0.601). Differences of a few points are within noise at these set sizes.