I pointed my linter at 64 open-source repos. It was wrong 13 different ways.

I maintain a checker for locale files. It compares every translation against the source language and reports the things that break at runtime: a dropped {placeholder}, an ICU plural that will not parse, a %2$s that vanished somewhere between English and Ukrainian.

Two weeks ago I wanted to know whether these bugs actually ship, or whether I had built a tool for a problem that gets caught in review. So I ran it over 64 public repositories — apps with real users, between 844 and 132,000 stars, all of them with locale files committed in the tree.

They do ship. 21 of those 64 carry at least one broken translation, 169 strings in total, every one read against its source string by hand before I filed anything. Eight of my fixes have been merged across seven projects.

But that is not what the exercise was actually about, and it is not the interesting part. Most of what the scan found was wrong with my checker. Thirteen distinct classes of false positive, found by pointing a tool at code its author had never seen.

The first run was a lie

The first aggregate I produced claimed 1,733 invented placeholders across the corpus. One repository, Joplin, accounted for 1,495 of them on its own. Another run reported over 8,000 type mismatches in a single Solidus locale file.

Both numbers were entirely my fault, and in a way that is worth naming precisely, because it is the failure mode of every static analyser that meets real data for the first time: the tool was measuring its own assumptions, not the code.

Joplin's locale files are gettext-style JSON, where the English string is the key and the source value is empty. My extractor compared each translation's placeholders against an empty string, concluded that every one of them had been invented, and reported it with total confidence. Solidus's Bulgarian file has null values for untranslated keys. null is typeof 'object' in JavaScript, so a string source against a null translation read as a type mismatch rather than as an empty translation, 8,000 times.

Neither of those is a subtle bug. Both were invisible until the tool met a repository that did something reasonable I had not thought of.

Thirteen ways to be wrong

The classes sort into four groups, and the groups turned out to be more useful than the individual fixes.

1. The same thing, written differently

The largest group. Two strings that are semantically identical and textually different, which a naive comparison calls a defect.

ICU plural arguments. Immich's source says {count} items moved; the German translation says {count, plural, one {# element} other {# elements}}. The argument is still count — it is simply declared as a plural now, with # standing in for the value. My extractor saw {count} on one side and {count, plural, …} on the other and reported the placeholder as dropped.

Positional versus sequential printf. Phoenix's iOS source reads %@.%@ seconds; the Czech translation reads %1$@.%2$@ vteřin. Those are the same two arguments. Explicit indices exist precisely so a translation can reorder them — the French translation of another string legitimately swaps %1$@ and %2$@ because French wants the clauses the other way round. Flagging that is flagging correct work.

Whitespace inside the braces. {{ commaSeparatedUsers }} and {{commaSeparatedUsers}} are the same i18next placeholder. stream-chat-react uses both spellings, sometimes within one file.

A repeated placeholder is one variable. Chatwoot's pagination string names {totalItems} twice across the two halves of a pipe plural. I was counting occurrences, so a translation that used it once looked like it had lost one.

2. Not a placeholder at all

Rails strftime patterns. time.formats.short: "%b %d" is a date format, not an interpolation. Every Rails app has dozens, and I was comparing them as if %b were an argument.

vue-i18n literal braces. Witsy documents its own syntax: {'{'}input{'}'} will be substituted with highlighted text. That is vue-i18n's escape for a literal brace. My parser read {'{'} as a placeholder named '{'.

%{x} inside an ARB file. Ollama-app's Turkish translation of download at {percent}% is %{percent} oranında indir. In Rails, %{percent} is an interpolation. In Flutter ARB it is a literal percent sign followed by {percent}. Same characters, different grammar, opposite meaning.

ICU sub-messages. {g, select, male {He} female {She} other {They}} contains the words He, She and They inside braces. They are message text, not arguments. I was extracting them as invented placeholders.

3. The source is not what I assumed

The Joplin empty-source case above, and the Solidus null case. Both come from the same root error: treating the source language as a guaranteed, well-formed reference. It usually is. When it is not, every comparison downstream inherits the mistake and reports it against the translation, which is the one file that did nothing wrong.

4. Legitimately different, because languages are

The subtlest group, and the one where "fix it" was the wrong instinct.

CLDR singular forms may omit the count. Hebrew's one-form for "one post" is בהודעה אחת — the number is carried by the grammar, not by a digit. English 1 vote needs its {{count}}; Hebrew's singular does not. i18next's _one suffix keys do the same thing.

A sentence fragment can be deliberately empty. Hoppscotch splits one sentence across three keys, part_1/part_2/part_3. Japanese and Korean move the verb to the end, so part_1 is empty on purpose while the other two carry the whole message. An empty string there is correct translation, not a gap.

Pipes are only plural separators in some formats. vue-i18n writes plurals as one item | many items. Rails does not — it uses one:/other: keys and treats | as an ordinary character. ifme's email subject is if-me.org | Your meeting "%{meeting_name}" is tomorrow, and two translations dropped the brand prefix. My checker read the missing pipe as a collapsed plural and reported a bug in a string whose only real problem was a missing brand name.

The discipline that made this worth doing

Every class became two things: a row in a public corpus, and a test.

The corpus is evals/placeholders/corpus.jsonl — 34 rows drawn from 20 real repositories, each labelled either as a true positive or with the false-positive class it exposed, spanning eight format grammars. A harness reports precision and recall over it, currently 100% on both with every row agreeing with its label. The false-positive rows are the valuable half: they are the cases where the correct output is silence, and silence is the thing you cannot verify by eyeballing output.

Fifty-three tests guard the classes directly. The rule I gave myself after the third one was: no class gets a one-off regex. If the grammar is wrong, fix the grammar. That forced a rewrite I had been avoiding — placeholder extraction is now one named grammar per format (ICU, i18next, vue-i18n, Rails, gettext, Android, Apple, generic) with the interpolation style of JSON trees sniffed from the source strings, rather than one regex bag applied to everything. Nearly every class in group 2 above disappears once the tool knows which language it is reading.

The other thing that changed: not everything is an error. A collapsed plural where the source had two forms is a warning now, not a failure. An English plural-suffix variable that a translation drops is a warning. A deliberately empty sentence fragment whose siblings are translated is reported at info level. Being right is not sufficient for a checker — it also has to be quiet about the cases where it cannot know, or people turn it off, and then it is right about nothing.

The one I got wrong in the other direction

Two days ago I scanned a Bitcoin wallet's iOS strings and found 48 broken format specifiers, mostly in Arabic: doubled dollar signs like %1$$lld, specifiers translated into Arabic letters, a % turned into the Arabic percent sign ٪. Real, and worth reporting.

While triaging them I wrote a quick script to sort the findings into categories, and it told me that twelve of the 48 were my own fault — Czech, Spanish and French strings where the translation used %1$@ against a source %@, exactly the positional-versus-sequential case I had fixed weeks earlier. I logged a new false-positive class, wrote a retraction into my notes, and drafted an issue to the maintainer disclosing that a quarter of my findings were bogus.

All of that was wrong. The checker handled those strings correctly; I verified it three ways afterwards. The bug was in the throwaway regex I had used to triage, which had grouped the findings badly. The twelve "false positives" were real bugs: a literal % swallowed into an incomplete %2$ specifier, a placeholder dropped outright, a space inserted inside %@.

Had I filed that issue as drafted, I would have told a maintainer who had merged my work the day before that my tool was unreliable, while under-reporting nine genuine bugs in his wallet. The lesson is narrow and I have written it down where I will see it: triage the output with the tool, not with a script you wrote to read the tool's output. I trusted a throwaway regex over a component with fifty-three tests behind it, because the regex agreed with what I expected to find.

What it added up to

Of the 64 repositories, 21 ship a broken translation. The fixes went upstream rather than into a listicle: eight merged across seven projects, four of them the same day they were filed. nocodb's maintainer took a report about one locale and swept all 39. An Excalidraw contributor answered a one-character typo by adding a placeholder-parity test suite for every locale in a 132k-star repository.

The most useful sentence anyone gave me came from Phoenix's maintainer, explaining how seven locales had drifted:

It was caused by working on an lnurl fix and the IT/PL/JP/KO/UK localisations in parallel, the translation was done on older files and not reconciled later.

That is the whole failure mode in one sentence, from someone who had no interest in selling you a tool for it. Translations are written against a snapshot of the source and nothing reconciles them afterwards, because nothing in CI reads them.

If you maintain locale files and want to know where you stand, the structural checks need no key and no account:

npx @shipi18n/cli@latest check ./locales -s en

And if you build static analysis of any kind: point it at a stranger's repository before you trust a number it gives you about your own. It is the cheapest test you will ever run, and the uncomfortable part is that most of what comes back is yours.

The scoreboard of scanned repositories, with one row per project and what was wrong, is at shipi18n.com/oss. The corpus and the harness are in the repo under evals/placeholders/.