What Indian languages cost on eight AI tokenizers

What we found

Gemini is the cheapest tokenizer for Indian languages

Across all fourteen languages, Gemini averages 1.85× English, the lowest of the eight. It's also the only tokenizer that makes real headway on Odia, at 3.42× against 4.99× for the next best.

The newest OpenAI tokenizer cut the premium by more than two-thirds

OpenAI's older cl100k tokenizer charges an average of 7.23× English for these languages. Its successor, o200k, charges 2.16×. Tamil fell from 7.64× to 1.98×; Malayalam from 8.94× to 1.96×. This is the largest single improvement in the table.

Claude sits in the middle, and its coverage varies by script

Claude averages 3.40×. On Hindi, Marathi, Nepali and Urdu it's competitive, around 2.2–2.4×. On Punjabi (6.35×) and Kannada (5.70×) it's roughly three times o200k's premium for the same text. No other modern tokenizer in the set swings this much between scripts.

A bigger vocabulary alone doesn't fix it

Gemma 2 has a larger vocabulary than o200k, 256,000 tokens against roughly 200,000, yet o200k is cheaper on thirteen of fourteen languages and ties on the fourteenth. What a vocabulary is spent on matters more than its size. Recent research on how tokenizers are built reaches a related conclusion: compressing text into fewer tokens doesn’t on its own mean every language is represented well (Ali et al., 2026).

Llama 3 improved Devanagari and almost nothing else

On Malayalam, Kannada and Telugu, Llama 3's numbers match cl100k to two decimal places. On Hindi, Marathi and Nepali, all written in Devanagari, they drop by roughly half. That's consistent with Meta's description of the Llama 3 tokenizer as tiktoken's 100,000-token vocabulary plus 28,000 tokens added for other languages: in this data, those additions reached Devanagari and no other Indian script.

Cost follows the writing system

Hindi, Marathi and Nepali cluster together on every tokenizer, and so do Bengali and Assamese, which share a script. Two languages written the same way cost about the same, even when they're unrelated.

Premiums hide a second gap

A premium compares each language with English on the same tokenizer. It says nothing about how many tokens a provider spends on English to begin with, and those baselines differ a lot.

For the English sentences alone, Claude counted 42,927 tokens and Gemini counted 28,125, about 1.5 times as many. A spot check on an English news paragraph agreed: 128 tokens on OpenAI's tokenizer, 133 on Gemini's, 231 on Claude's. When the Indic premium lands on top of that heavier baseline, the gap in absolute tokens grows. Anthropic’s own pricing documentation notes that its newer models use a tokenizer that produces about 30% more tokens for the same text than earlier Claude models.

Total tokens to encode the full 1,012-sentence test set in each language.

If you're building in an Indian language

• Treat the tokenizer as a pricing decision. For identical text, the models at the expensive end of this table bill three or more times what the cheapest ones do. For languages with little training data it may be a quality decision too: in a controlled study of small research models, the quality of Tamil and Bengali varied more across tokenizers than that of better-resourced languages (Meister et al., 2026). Test a few models on your own text before choosing.
• Measure your own text. These figures come from formal, news-style writing. Conversational text and code-mixed text like Hinglish behave differently. Paste a real sample into the calculator and get exact counts from Claude and Gemini before you commit.
• Remember the reply is inflated too. The same tokenizer encodes the model's answer, and output is usually priced three to five times higher than input. If your users read responses in Tamil, the premium applies to the expensive half of the bill as well.
• Keep fixed instructions in English where quality allows. A system prompt sent on every request costs less in English. User-facing text should stay in the user's language.
• Cache long prompts you repeat. Prompt caching discounts repeated input heavily, which offsets more of the premium the longer your fixed prompt is.

Method

The corpus is the devtest split of FLORES+: 1,012 sentences professionally translated into each language and aligned line by line, so every language expresses identical content. For each tokenizer we counted the tokens needed for the whole set in each language and divided by the count for English. Holding the content fixed means any difference comes from the tokenizer, not the text.

Open tokenizers were run locally. Claude and Gemini publish no tokenizer, so we measured them through each provider's free token-counting endpoint, sending each language as a single request and subtracting the fixed per-request overhead. As a check, our figure for Odia on cl100k, 12.40×, matches the more than twelvefold premium Petrov et al. reported for the same tokenizer in 2023.

We report premiums rather than tokens per word, because counting words by splitting on spaces is unreliable for scripts that don't separate words that way.

Limitations

• Claude and Gemini were each measured on one model, claude-sonnet-5 and gemini-3.6-flash, in October 2026. Other models from the same provider may use different tokenizers, and tokenizers change between generations.
• Gemma 2, Llama 3 and Mistral were measured with community copies of their tokenizers, which may differ slightly from the originals.
• FLORES+ is formal, news-domain text. Everyday writing, transliterated text and code-mixed text may give different premiums.
• We measured input tokens only. Output pricing and quality, which matter as much in practice, are outside this study.

Related work

The premium on open tokenizers has been measured before, and our results for those tokenizers largely confirm earlier findings. What this page adds is the commercial tokenizers, Claude and Gemini, measured on Indian languages, and the comparison of absolute token counts between providers.

• Petrov et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages.
• BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base (2026). Evaluates thirteen open tokenizers across twenty-two Indian languages and proposes a replacement.
• The African Language Tax (2026).
• Meister, Altıntaş and Bosselut (2026). Language-Specific Effects of Tokenizer Choice in Multilingual Language Models. Preprint. Trains 123 small models that differ only in their tokenizer, using the same FLORES+ sentences as this study, and measures quality rather than cost. Languages with less training data, including Tamil and Bengali, were the most sensitive to tokenizer choice, and giving those languages a larger share of the tokenizer’s training data did not reliably help them.
• Ali et al. (2026). Latent Core Tokenizer: Compress, but Meaningfully. Under review. A method for building tokenizers from the structure of words, tested on 104 languages. It reports fewer tokens per word and better results on multilingual benchmarks, while the cost gap between languages stays comparable to standard methods.
• The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers (2026). Measures Claude's tokenizer through its counting API, for French.

Data

language-tax-premiums.csv has every premium in the table, plus total token counts for Claude and Gemini. It's released under CC BY-SA 4.0, matching the corpus it was measured on. The FLORES+ sentences themselves aren't included: the dataset's terms ask that they not be re-hosted where web crawlers can reach them.