Every LLM caught every planted translation error. The benchmark still found two bugs.

There is a lot of research on judging machine translation with LLMs — GEMBA-MQM and its successors are careful, serious work. But it all scores sentences. Nobody had published a benchmark for the thing that actually breaks in software: locale files — short, ambiguous UI strings full of placeholders and plural forms, where the failure modes are "the Spanish says the opposite of the English" and "the placeholder is gone and the build is green."

So I built one. This post is the v1 report: what it measures, what seven models from three providers scored, and the two things it caught that I didn't expect — one in my own gold labels, one in my own product.

The leaderboard lives at shipi18n.com/benchmark. The dataset is on Hugging Face. The harness and item-level results are in the repo.

The setup

The corpus came before the judge. UIStringBench's public split is 228 source/translation pairs (Spanish, German, Japanese): 168 clean pairs pulled from real production locale files — including adversarial pairs whose content looks like instructions or JSON — plus 54 planted errors (18 mistranslations, 18 omissions, 18 additions) and 6 glossary violations. It was committed to git before the judge that consumes it was written, and the evaluation thresholds were fixed before implementation. Labels are never revised after seeing results — you'll see why that policy matters shortly.

A private held-out split (60 pairs, disjoint domain) is scored separately and never published. If a future model has memorized the public split, the gap will show.

The judge design under test is the one that ships in shipi18n check --semantic: three passes, majority vote (single-pass LLM judgments are unstable — that's the GEMBA finding), MQM-style error categories — mistranslation / omission / addition — instead of scalar scores, because a category is checkable and a 7/10 is not.

One framing note up front: this is an engineering QA signal, not a validated academic metric. The GEMBA authors advise caution about using their own metric in academic work, and the same caution applies here, more so.

The board

Three results worth more than the table.

1. Detection is saturated — which is itself a finding

Every LLM judge caught every planted error, on both splits. On deliberate, unambiguous errors — a translation that inverts the source, drops a consequence clause, invents a feature — current models simply do not miss, down to the cheapest tier tested.

Honest corollary: my planted errors are too easy, the held-out split especially (I wrote it harder-later than I should have — a clean sweep across all seven judges says the split isn't discriminating anything). v1.1 needs subtler corruptions: partial meaning shifts, plausible tense changes, errors in one plural form out of three. If you work in MT evaluation and want to plant errors that actually separate frontier models, the repo is open.

What the saturation does establish: for this error class, the judge's catch rate is not the engineering risk. The false-positive rate is.

2. The five frontier judges never disagreed with each other — only with me

Sonnet 5, Opus 5, GPT-5.5, Gemini 3.8 Flash and Gemini Pro each flagged exactly six of my 168 "clean" pairs. Not roughly six. The identical six pairs — across five models from three different companies. Zero variance in their dissent.

Those six are the same two strings in three languages:

• Three are the corpus's blog-page title, whose "clean" translations say "Shipi18n Translation API blog" — a product that no longer exists. The judges flagged stale copy that really was wrong. The labels are wrong; the models are right. I knew about these — the original eval report called them out — but watching five independent judges converge on them without coordination is a different kind of evidence.
• Three are a marketing meta-description that was transcreated rather than translated — a legitimate judgment call about whether a paraphrase counts as an error. Reasonable people disagree; the models unanimously say "flag it."

Per the pre-registration policy, the labels stand and the "false positives" stay counted against every judge. A corpus you edit after seeing results is not a corpus. But the honest reading of the frontier models' 3.6% false-positive rate is: half of it is my labeling debt, not their error — and their unambiguous false-positive rate on this corpus is approximately zero.

The small models tell the opposite story. Haiku 4.5's extra seven flags and GPT-5.4 mini's extra one (plus its two held-out false positives) are genuinely noisy — tone nitpicks, "could be more natural" opinions. That noise is the real cost of the cheap tier, and it's why the semantic pass ships advisory-by-default: a gate that fails builds on judgment calls is a gate teams turn off.

3. The baseline row is the whole argument

The last row runs only deterministic checks — key diffs, placeholder validation across eight syntaxes, plural-form counts, glossary terms. It catches 0 of 54 semantic errors, with zero false positives, in milliseconds, for free.

Both halves of that sentence matter. Zero catches: structure cannot see meaning, and no amount of structural tooling will notice that "were saved successfully" became "could not be saved." Zero false positives: deterministic checks never cry wolf, which is why they — and only they — should fail your build. The two tiers aren't competing; they're a stack.

The bug the benchmark found on day one

Running the GPT judges required the harness to call OpenAI for the first time — and it crashed: the OpenAI adapter in @shipi18n/core sent max_tokens, which current OpenAI models reject in favor of max_completion_tokens. Which means shipi18n check --semantic -p openai was broken for every real user on a current GPT model, and no test had caught it, because every test — like every test everywhere — mocked the provider.

The fix shipped as core 2.6.1 within the hour (modern parameter for api.openai.com, legacy for OpenAI-compatible endpoints like Ollama, one-shot fallback either way). But the lesson is the better artifact: a benchmark that actually exercises real providers is an integration test you can't fool. The eval found a product bug before any user reported it.

Limitations, stated plainly

Three language pairs. One domain — UI strings, largely from one product's site. Planted errors that are (demonstrably, see finding 1) less subtle than real-world drift. A corpus and held-out split authored by the same person who built the tool being measured, with LLM assistance. A "Gemini Pro (latest)" alias that isn't version-pinned. Aggregates from single runs per model — prior repeated runs of the shipping judge reproduced catch and FP rates exactly while label accuracy moved between 98.1% and 100%, and the same variance should be assumed here.

Run it

git clone https://github.com/Shipi18n/shipi18n && cd shipi18n
node evals/semantic/leaderboard.mjs --models anthropic:claude-haiku-4-5 --split public
# or any OpenAI-compatible endpoint, e.g. a local model:
node evals/semantic/leaderboard.mjs --models openai:llama3.3@http://localhost:11434/v1 --split public

Item-level results for every model on the board are committed in the repo. If you make a judge look worse than the board says — or find the next mislabeled pair — I'd genuinely rather hear it than not. The judge itself ships in npx @shipi18n/cli check --semantic, advisory by default.