What is a token in AI? A plain-English explanation

A token is a small piece of text that a language model works with: a whole word, part of a word, a punctuation mark, or a word together with the space in front of it. Models read your prompt as tokens, write the answer as tokens, and AI APIs bill by the token. So "how many tokens is this" is really two questions: will it fit, and what will it cost.

How a model cuts text into tokens#

Before a model sees your text, a tokenizer splits it into pieces from a fixed vocabulary and turns each piece into a number. Frequent words usually get a single token. Rare and long words are split into parts.

Here is a real example. We ran one sentence through two tokenizers: the one published with DeepSeek-V4-Flash-0731 and OpenAI's o200k_base encoding.

In English both tokenizers agree: 9 words, 11 tokens. The comma and the period are tokens of their own, and most words carry the space before them.

Long words are where tokenizers start to differ. "tokenization" is two tokens in both: token + ization. The Russian word «токенизация» is three tokens for DeepSeek and four for OpenAI. Same word, different count.

How many tokens is a word#

A quick test with a short paragraph about tokens, in English and in Russian:

For English text, about 1.1–1.2 tokens per word is a fair estimate. Other scripts usually cost more: the Russian version says the same thing in fewer words but takes about 20% more tokens with DeepSeek's tokenizer.

Treat these as rules of thumb. Every model family has its own tokenizer, and the exact count for your text is the one the API reports back.

Tokens are what you pay for#

API prices are quoted per million tokens, separately for input and output:

• input tokens — everything you send: the system prompt, chat history, documents and your question;
• output tokens — everything the model writes back, including its reasoning if the model reasons before answering.

The cost of a request is input tokens × input price + output tokens × output price.

Two practical consequences. A long chat gets more expensive with every message, because the whole history goes back to the model as input each time. And output usually costs several times more per token than input, so asking for shorter answers saves real money.

Four things people call a "token"#

The word is overloaded, which is why searching for it is so confusing:

If you're answering a quiz question like "a token is…", for language models the right answer is a piece of text: a word, part of a word or a symbol.

Why you might see "not enough tokens"#

The same phrase can point to four different problems, and only one of them is about money:

• The prompt doesn't fit into the context window. Every model has a limit on input plus output. At CheapTokens it's 380,000 tokens for DeepSeek-V4-Flash, 390,000 for GLM-5.3-Flash and 200,000 for MiniMax-M2.7. Shorten the history or split the document.
• The answer hit the output limit. The reply stops mid-sentence. Raise max_tokens within the model's maximum, or ask for a shorter answer.
• The balance ran out. This is the only case where topping up fixes it.
• The app has its own limit. Chat services often cap usage by their own rules, unrelated to the model itself.

How to count tokens in your own text#

The most reliable number comes from the API: every response includes a usage block with prompt_tokens and completion_tokens.

To estimate before sending, run the model's tokenizer locally. Open models publish their tokenizer next to the weights. This snippet counts the example sentence with the DeepSeek-V4-Flash-0731 tokenizer:

# pip install tokenizers huggingface_hub
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

path = hf_hub_download("deepseek-ai/DeepSeek-V4-Flash-0731", "tokenizer.json")
tok = Tokenizer.from_file(path)

text = "A neural network reads text in tokens, not words."
enc = tok.encode(text, add_special_tokens=False)
print(len(enc.ids))  # 11

For OpenAI models there's the tiktoken library. Keep in mind that a count from one tokenizer is only an estimate for any other model.

Your next step#

To estimate a budget, take a typical prompt and answer size and multiply by the price per million tokens. Current CheapTokens prices, fixed in US dollars: