What is a token?

A token is a small piece of text. AI doesn’t read whole words the way you do: it chops everything into tokens first, like snapping a sentence into building bricks. Tokens are also how AI is paid for.

Type anything and watch it turn into tokens

Or try:

These are the bricks OpenAI’s GPT models use. Other AI models cut text a little differently. A faint dot shows where a brick starts with a space.

A box of 200,000 bricks

Imagine the AI has a giant box of about 200,000 different bricks. Some bricks are whole common words, like “love” or “summer”. Others are small pieces, like “un” or “es”, or even single letters. In the example above, “mangoes” needed two bricks: “mango” and “es”.

To read your sentence, the AI builds it from the fewest bricks it can find in its box. Common English words usually get one brick each. A rare or long word gets built from several smaller bricks. Anything the box has no brick for, like an unusual emoji, gets built from tiny pieces, so it can take several bricks for one picture.

The program that does the chopping is called a tokenizer. Every AI company has its own box of bricks, so the same sentence can come to a different number of tokens on different AI models.

Why tokens matter

AI reads and writes in tokens

Everything you send to an AI is turned into tokens, and its answer is written one token at a time. That’s why you sometimes see an answer appear bit by bit.

There’s a limit

An AI can only keep a certain number of tokens in mind at once, a bit like how much fits on a desk. Very long documents may not fit.

You pay per token

Companies that use AI pay for every token, both the ones they send and the ones the AI writes back. Like a taxi meter: more tokens, bigger bill.

Same message, different bill

Here is one customer-service message in Hindi, exactly the same text, counted on three AI models:

Gemini 3

150

GPT-6

172

Claude

410

Same words, but one model uses more than twice as many bricks, so it costs more than twice as much to send. That’s why comparing AI prices needs real token counts, not just the price list. See the Token Bench Index for what everyday tasks cost on each model.

Why Indian languages often need more tokens

The bricks were chosen mostly by looking at English text, because most text on the internet is in English. So English words often have their own bricks, while words in Hindi, Tamil or Bengali are more often built from smaller pieces. More pieces means more tokens, and more tokens means a bigger bill for the same message. Some newer AI models have added many more bricks for Indian languages, which narrows the gap. Our Indian languages study measures this for fourteen languages.

Handy rules of thumb

In English, one token is about 4 letters, or three-quarters of a word.
100 tokens is about 75 English words, roughly a short paragraph.
The same text in an Indian language usually takes more tokens, from a little more to several times as many, depending on the language and the model.

Words to know

Token: A small piece of text that an AI reads or writes: a word, part of a word, a space, a number or a symbol.
Tokenizer: The program that chops text into tokens. Each AI company has its own.
Input tokens: The tokens you send to the AI: your question, instructions and any documents.
Output tokens: The tokens the AI writes back. These usually cost several times more than input tokens.
Context window: The most tokens an AI can handle at once, counting what you send and what it writes.

Now try it with your own text. See how many tokens it is on every major AI model, and what it costs.

Count my tokens Or play the Token Game