What 200+ languages cost on AI tokenizers

We sent the same 1,012 sentences, in 221 languages and English, through eight tokenizers, including Claude and Gemini. Saying the same thing costs anywhere from about the same as English to more than seventeen times as many tokens, depending on the language and the model.

Token Bench research, November 2026. Findings, by writing system, every language, method, download the data.

What we found

No tokenizer is cheapest everywhere

Among the three big commercial tokenizers, OpenAI’s o200k is the cheapest for 98 of the 221 languages, Claude’s for 62 and Gemini’s for 61. Gemini 3 has the lowest average premium, 2.11×, while o200k has the lowest median, 1.77×: it is the best choice for a typical language and weaker on rare scripts. Which model is cheapest depends on your language, so the table below shows it for each one.

The highest costs fall on languages with their own scripts

The most expensive languages are written in scripts that few tokenizers have learned: Chakma (up to 17.52×), Mongolian script (up to 16.20×), Santali in Ol Chiki, Manipuri in Meitei Mayek, N’Ko and Tifinagh. Several are languages of India: Santali and Manipuri are both among India’s scheduled languages, and Chakma is spoken in India’s north-east and in Bangladesh.

The script matters more than the language

Some languages appear twice in the corpus, written in two scripts. On every commercial tokenizer, Manipuri costs about 3× English in Bengali script but 9 to 12.5× in Meitei Mayek; Mongolian about 2× in Cyrillic but 11 to 16× in Mongolian script; and Tamasheq about 2× in Latin but 5 to 10× in Tifinagh. Same language, same sentences: the difference is entirely the writing system.

Chinese, Japanese and Korean are close to English

Mandarin costs only 1.1 to 1.4× English on the three commercial tokenizers, Korean 1.4 to 1.6× and Japanese 1.2 to 1.7×. These scripts are well represented in the text tokenizers were built from, and each character carries a lot of meaning.

The Latin alphabet isn’t free either

The 134 languages written in Latin script average about 1.8× English on the commercial tokenizers. Most of them are smaller languages, such as Yoruba or Quechua, whose words the tokenizers rarely saw, so they are split into more pieces even though the letters are familiar.

Claude is strong on Latin, Cyrillic and Chinese, weakest on Indian scripts

Claude is competitive for Latin, Cyrillic and Chinese-family text. For Indian scripts it costs the most of the three: Punjabi averages 6.47× on Claude against 2.69× on Gemini and 2.62× on GPT. Our Indian languages study looks at this in detail.

By writing system

Each cell is the average premium for the languages written in that script: how many times more tokens they need than English for identical content. Writing systems are ordered from most to least expensive on average, and tokenizers from cheapest to most expensive.

Every language

Search by language, script or code, and click a column heading to sort. The lowest premium in each row is highlighted, and the last column shows which of the three big commercial tokenizers is cheapest for that language.

Method

The corpus is the devtest split of FLORES+: 1,012 sentences professionally translated into each language and aligned line by line, so every language expresses identical content. We used every FLORES+ variety whose file had exactly as many sentences as English: 221 varieties plus English. Where FLORES+ gives two varieties the same name, such as Valencian and standard Catalan, we label them separately. For each tokenizer we counted the tokens needed for the whole set in each language and divided by the count for English.

Open tokenizers were run locally, sentence by sentence. Claude and Gemini publish no tokenizer, so we measured them through each provider’s free token-counting endpoint. To keep the run practical at this scale, each language’s sentences were sent as one request with a line break between sentences. For every language we also counted five sentences one by one to measure what a line break costs; in all 444 measurements it was exactly one token, so we subtracted one token per line break, which gives the same totals as counting each sentence separately. The fixed overhead each endpoint adds to every request was measured and subtracted too. As a check, the Indian languages in this study match our Indian languages study exactly.

Measured on 31 October and 1 November 2026.

Tokenizer	How measured	Exact source
Gemini 3	Provider counting API	gemini-3.8-flash (countTokens)
Gemma 2	Open tokenizer, run locally	mlx-community/gemma-2-2b
Claude	Provider counting API	claude-sonnet-5-5 (count_tokens)
GPT o200k	Open tokenizer, run locally	tiktoken 0.14.0, o200k_base
Qwen 2.5	Open tokenizer, run locally	Qwen/Qwen2.5-0.5B
Llama 3	Open tokenizer, run locally	Xenova/llama3-tokenizer
Mistral	Open tokenizer, run locally	Xenova/mistral-tokenizer-v1
GPT cl100k	Open tokenizer, run locally	tiktoken 0.14.0, cl100k_base

Limitations

• Gemini was measured on gemini-3.8-flash, which gave counts identical to gemini-3.6-flash on every Indian language, and Claude on claude-sonnet-5-5, which counts identically to claude-sonnet-5. Gemini 4 Argon uses a different tokenizer and isn’t in this study yet; see the Indian languages study for how it differs.
• FLORES+ is formal, news-domain text. Everyday writing, transliterated text and code-mixed text may give different premiums.
• Language and script names follow FLORES+ and ISO 15924. The Mongolian-script row is labelled Halh Mongolian, as in FLORES+.
• Community copies of open tokenizers may differ marginally from the versions vendors run in production.

Data

world-languages-premiums.csv has every premium on this page, with total token counts for every language and tokenizer: 1,776 rows. It’s released under CC BY-SA 4.0, matching the corpus it was measured on. It contains measurements only. The FLORES+ sentences themselves aren’t included: the dataset’s terms ask that they not be re-hosted where web crawlers can reach them.