%20--%3e%3csvg%20version='1.1'%20id='Layer_1'%20xmlns='http://www.w3.org/2000/svg'%20xmlns:xlink='http://www.w3.org/1999/xlink'%20x='0px'%20y='0px'%20viewBox='0%200%20720%20720'%20style='enable-background:new%200%200%20720%20720;'%20xml:space='preserve'%3e%3cstyle%20type='text/css'%3e%20.st0{fill:%232664EB;}%20%3c/style%3e%3cpath%20class='st0'%20d='M652.1,507.8c14.1-33.4,22.3-68.7,24.4-104.9c0.3-1.4,0.5-2.9,0.5-4.5c0-0.9-0.1-1.9-0.2-2.8%20c0.1-3.7,0.2-7.5,0.2-11.3V143c0-0.3,0-0.7,0-1c0-0.1,0-0.1,0-0.2c0-0.3,0-0.6-0.1-0.8c0-0.1,0-0.1,0-0.2c0-0.3-0.1-0.6-0.1-0.8%20c0-0.1,0-0.1,0-0.2c-0.1-0.3-0.1-0.6-0.2-0.9c0,0,0-0.1,0-0.1c-0.1-0.3-0.1-0.6-0.2-0.9c0,0,0,0,0-0.1c-0.1-0.3-0.2-0.6-0.3-0.9%20c0,0,0,0,0,0c-0.6-1.6-1.3-3.2-2.3-4.6c0-0.1-0.1-0.1-0.1-0.2c-0.1-0.2-0.3-0.4-0.5-0.6c-0.1-0.1-0.1-0.2-0.2-0.3%20c-0.1-0.2-0.3-0.3-0.4-0.5c-0.1-0.1-0.2-0.2-0.3-0.3c-0.1-0.2-0.3-0.3-0.4-0.4c-0.1-0.1-0.2-0.2-0.3-0.3c-0.1-0.1-0.3-0.3-0.4-0.4%20c-0.1-0.1-0.2-0.2-0.3-0.3c-0.2-0.2-0.4-0.4-0.6-0.5c0,0-0.1-0.1-0.1-0.1l0,0c0,0,0,0,0,0L543.2,23c-3.4-2.8-7.7-4.4-12.2-4.4H188.3%20c-4.5,0-8.8,1.6-12.2,4.4L49.9,128.3c0,0,0,0,0,0l-0.1,0.1c-0.1,0-0.1,0.1-0.1,0.1c-0.2,0.2-0.4,0.3-0.6,0.5%20c-0.1,0.1-0.2,0.2-0.3,0.3c-0.1,0.1-0.3,0.3-0.4,0.4c-0.1,0.1-0.2,0.2-0.3,0.3c-0.1,0.1-0.3,0.3-0.4,0.5c-0.1,0.1-0.2,0.2-0.2,0.3%20c-0.1,0.2-0.3,0.3-0.4,0.5c-0.1,0.1-0.1,0.2-0.2,0.2c-0.2,0.2-0.3,0.4-0.5,0.7c0,0-0.1,0.1-0.1,0.1c-1,1.4-1.7,2.9-2.3,4.6%20c0,0,0,0,0,0c-0.1,0.3-0.2,0.6-0.3,0.9c0,0,0,0,0,0.1c-0.1,0.3-0.2,0.6-0.2,0.9c0,0,0,0.1,0,0.1c-0.1,0.3-0.1,0.6-0.2,0.9%20c0,0,0,0.1,0,0.1c0,0.3-0.1,0.6-0.1,0.8c0,0.1,0,0.1,0,0.2c0,0.3-0.1,0.5-0.1,0.8c0,0.1,0,0.1,0,0.2c0,0.3,0,0.7,0,1v241.4%20c0,4.2,0.1,8.5,0.3,12.7c0,0.4-0.1,0.9-0.1,1.3c0,1.1,0.1,2.1,0.3,3.1c1.9,36.7,10.1,72.4,24.5,106.3c13.3,31.4,31.3,60.1,53.8,85.6%20c0.3,0.3,0.5,0.6,0.8,0.9c4.3,4.9,8.8,9.6,13.4,14.2c29.1,29.1,63,52,100.8,67.9c39.1,16.5,80.6,24.9,123.4,24.9%20c42.8,0,84.3-8.4,123.4-24.9c37.8-16,71.6-38.8,100.8-67.9c4.8-4.8,9.4-9.8,13.9-14.8c0.1-0.1,0.1-0.2,0.2-0.2%20C620.7,568,638.8,539.2,652.1,507.8z%20M518.9,538.2c4.2-9.8,8.1-19.8,11.6-30.2c10.1-29.4,17-59.8,20.8-90.6H637%20c-6.3,52.8-27.4,101.2-59,140.8C558.6,550.5,538.9,543.8,518.9,538.2z%20M298.5,318.1c0-34,27.5-61.5,61.5-61.5%20c34,0,61.5,27.5,61.5,61.5c0,34-27.5,61.5-61.5,61.5C326,379.6,298.5,352.1,298.5,318.1z%20M341,644.3c-24.3-8.9-50.7-32.7-73.7-67.4%20c-2.8-4.2-5.5-8.6-8.2-13.1c26.9-5,54.2-7.9,81.9-8.9V644.3z%20M379,645.1v-90.1c28.4,1,56.5,4.1,84,9.3c-2.6,4.3-5.2,8.6-8,12.7%20C431.3,612.7,404,636.9,379,645.1z%20M372.5,516.7c87.6-132.1,104.3-145.3,104.3-198.6c0-64.5-52.3-116.8-116.8-116.8%20s-116.8,52.3-116.8,116.8c0,53.3,16.7,66.5,104.3,198.6c-36.2,0.7-71.8,4.7-106.7,11.9c-15.4-34.3-26.2-72.5-31.4-111.1h47%20c-9.3-14.7-16.3-26.8-21.4-38.1h-154V162h557.9v217.4h-154c-5.1,11.3-12.1,23.4-21.4,38.1h49.4c-5.2,38.7-16.1,77.1-31.6,111.6%20C445.7,521.6,409.4,517.4,372.5,516.7z%20M142,558.3c-31.6-39.6-52.7-87.9-59-140.8h88c3.7,30.8,10.7,61.2,20.8,90.6%20c3.5,10.1,7.3,20,11.4,29.6C182.5,543.4,162.1,550.2,142,558.3z%20M605.2,124h-210l-8.6-67.3h137.5L605.2,124z%20M195.2,56.7h138.1%20l-8.5,67.3H114.6L195.2,56.7z%20M169.9,588.3c16.5-6.1,33.3-11.4,50.3-15.8c4.8,8.8,10,17.3,15.4,25.4c14.6,22,30.5,40.3,47.1,54.5%20C240,640.1,201.5,617.8,169.9,588.3z%20M441,651.4c16.1-14,31.6-31.9,45.9-53.4c5.3-8,10.3-16.3,15.1-24.9%20c16.3,4.3,32.4,9.4,48.2,15.3C519.4,617,482.2,638.8,441,651.4z'/%3e%3c/svg%3e)
UIStringBench
A public benchmark for how well LLM judges detect translation errors in software UI strings — mistranslations, omissions and additions that no structural check can see.
v1 · run 2026-09-07 · shipi18n semantic judge — 3-pass majority vote, MQM categories · corpus committed before the judge was written
| Judge | Provider | Public split (228 pairs) | Held-out (60 pairs) | |||
|---|---|---|---|---|---|---|
| Catch | False positives | Label acc. | Catch | False positives | ||
| Claude Haiku 4.5small | Anthropic | 54/54 | 13/168 (7.7%) | 98.1% | 18/18 | 0/42 (0.0%) |
| Claude Sonnet 5frontier | Anthropic | 54/54 | 6/168 (3.6%) | 100.0% | 18/18 | 0/42 (0.0%) |
| Claude Opus 5frontier | Anthropic | 54/54 | 6/168 (3.6%) | 100.0% | 18/18 | 0/42 (0.0%) |
| GPT-5.5frontier | OpenAI | 54/54 | 6/168 (3.6%) | 100.0% | 18/18 | 0/42 (0.0%) |
| GPT-5.4 minismall | OpenAI | 54/54 | 7/168 (4.2%) | 98.1% | 18/18 | 2/42 (4.8%) |
| Gemini 3.8 Flashfrontier | 54/54 | 6/168 (3.6%) | 100.0% | 18/18 | 0/42 (0.0%) | |
| Gemini Pro (latest)frontier | 54/54 | 6/168 (3.6%) | 100.0% | 18/18 | 0/42 (0.0%) | |
| Structural checks only (no LLM) | — | 0/54 | 0/168 (0.0%) | — | 0/18 | 0/42 (0.0%) |
Catch = planted errors flagged (any category). Label acc. = correct MQM category among caught. The held-out split is private (contamination defense) and scored separately.
Everyone catches everything
Every LLM judge caught all 54 planted errors on the public split and all 18 on the held-out split. On deliberate, unambiguous errors, detection is saturated — the differentiation is entirely in false positives. (Which also means the held-out split is too easy; v1.1 will plant subtler errors.)
Frontier judges agree — even when dissenting
The five frontier judges — from three different providers — flagged the identical six "clean" pairs. Three of those six are the same stale sentence in three languages that really was wrong: the judges were right and the corpus label was wrong. Per the pre-registration policy the labels stand, and the disagreement is reported instead.
The baseline makes the case
Structural checks alone — key diffs, placeholders, plurals — catch 0 of 54 semantic errors, with zero false positives. That row is the whole argument for a semantic tier: deterministic checks are necessary and cannot see meaning.
Methodology
- Pre-registered: the 228-pair public corpus was committed to git before the judge existed; evaluation thresholds were fixed before implementation; labels are never revised after seeing results.
- Judge under test: each model runs the same harness — 3 passes, majority vote, MQM-style categories (mistranslation / omission / addition) instead of scalar scores.
- Splits: public (168 clean + 54 planted + 6 glossary, es/de/ja, drawn from real production locale files) and a private 60-pair held-out split in a disjoint domain, scored separately.
- Known limitations: three language pairs; one domain (UI strings); planted errors are deliberate and less subtle than real drift; corpus authored by the tool's maintainer with LLM assistance. Treat scores as an engineering QA signal, not a validated academic metric.
Run it yourself
The harness, corpus and item-level results are public. Point it at any judge — including a local model via an OpenAI-compatible endpoint.
git clone https://github.com/Shipi18n/shipi18n && cd shipi18n
node evals/semantic/leaderboard.mjs --models anthropic:claude-haiku-4-5 --split publicThe judge measured here ships in npx @shipi18n/cli check --semantic — advisory by default, because a noisy gate is a gate teams turn off.