Count tokens before you send a request: tiktoken in Python

You can count a prompt's tokens on your own machine before it is sent: install tiktoken, encode the text, take len() of the result. tiktoken is OpenAI's tokeniser, so for DeepSeek, GLM and MiniMax models the number is an estimate. The real figure comes back in usage.prompt_tokens after the first call, and comparing the two takes one more line.

Why bother? Because the bill and the errors both start with the input size. Paste a long contract into a script, and you find out it was too big only when the API answers with a 400. Run a batch of five thousand emails, and you find out what it cost only when the balance is lower. Counting first turns both surprises into a number you can read in advance.

Five lines that count tokens#

The tiktoken README (checked 1 November 2026) shows the whole idea: pick an encoding, encode the text, and the length of the list is the token count.

pip install tiktoken

import tiktoken

enc = tiktoken.get_encoding('o200k_base')
text = 'Hello world'
tokens = enc.encode(text)
print(len(tokens))  # how many tokens the text is

enc.decode(tokens) gives back the original text, which is a handy sanity check that you are looking at the same text the model will see. The README also mentions tiktoken.encoding_for_model('gpt-4o'), which picks the tokeniser for a specific OpenAI model. That is the catch worth remembering: tokenisers belong to model families. The README's rule of thumb is about four bytes of text per token, and it describes OpenAI's tokenisers only.

So what do you do for a DeepSeek model? Use the same count as a working estimate, and calibrate it, as shown below.

Does the prompt fit the window?#

Every model has a context window, and your input plus the answer you allow has to fit in it. The windows for the models we serve: DeepSeek-V4-Flash 380K, GLM-5.3-Flash 390K, MiniMax-M2.7 180K tokens. The longest answer is 32,768 tokens for DeepSeek-V4-Flash and 8,192 for the other two (CheapTokens model list, 1 November 2026).

A guard before the call is four lines:

CONTEXT = {
    'deepseek-ai/DeepSeek-V4-Flash-0731': 380_000,
    'zai-org/GLM-5.3-Flash': 390_000,
    'MiniMaxAI/MiniMax-M2.7': 180_000,
}

def fits(prompt_tokens, max_tokens, model):
    return prompt_tokens + max_tokens <= CONTEXT[model]

Say you have a 250,000-token document. fits(250_000, 8_192, 'MiniMaxAI/MiniMax-M2.7') is False, so you route that job to a model with a bigger window or split the document, and the 400 never happens. fits(250_000, 8_192, 'deepseek-ai/DeepSeek-V4-Flash-0731') is True. If you want the full story of what the error looks like, request too long: 400 vs 413, context window and max_tokens walks through it.

What will it cost?#

Now the budget. Take one typical item, count it, and multiply. Suppose you classify 5,000 order emails. You count one representative email and get about 800 input tokens, and you cap the label and a short reason at 200 output tokens. The totals: 5,000 × 800 = 4,000,000 input tokens and 5,000 × 200 = 1,000,000 output tokens.

The table below prices exactly that workload at live prices, with input and output counted together:

In total, with input and output counted together, the CheapTokens column is cheaper than OpenRouter for these models. Change the three numbers to your own counts and you have a budget before the first request. For the difference between the two sides of the bill, see input vs output tokens: what an AI API bills you for, and for real measured examples how many tokens an API request uses. Current prices are always on the pricing page.

Compare the estimate with usage#

The estimate is a guess about someone else's tokeniser, so check it once against the answer key. Every response carries a usage field, and usage.prompt_tokens is what the API counted for your input. Put the two numbers side by side and divide.

pip install tiktoken openai
export CHEAPTOKENS_API_KEY='your-key-here'

import os
import tiktoken
from openai import OpenAI

MODEL = 'deepseek-ai/DeepSeek-V4-Flash-0731'
enc = tiktoken.get_encoding('o200k_base')
messages = [{'role': 'user',
             'content': 'Classify this email as order, refund or other: Hi, my parcel from last Tuesday has not arrived yet.'}]

estimate = sum(len(enc.encode(m['content'])) for m in messages)

client = OpenAI(base_url='https://api.cheaptokens.dev/v1',
                api_key=os.environ['CHEAPTOKENS_API_KEY'])
r = client.chat.completions.create(model=MODEL, messages=messages, max_tokens=200)

real = r.usage.prompt_tokens
print(f'estimate {estimate}, usage.prompt_tokens {real}, ratio {real / estimate:.2f}')

From then on the routine is: estimate locally, guard the window, apply your ratio to the budget, and let usage in each response keep you honest. If you also want the running total over a whole job, track AI agent token cost in Python builds that on the same field. New to the connection itself? Your first API request with CheapTokens covers it, and the docs have the rest.

Counting is cheap, local and takes seconds. Do it before the call that costs money.