Estimate a model tournament's API cost before you run it

You can know the bill of a model tournament before the first request: count the tokens one seat sends over a whole game, multiply by seats and games, and put a hard cap in the code. The part people miss is the history. Every turn resends everything said so far, so input grows much faster than the number of turns.

What a $20,000 headline does and doesn't tell you#

On 5 November 2026 podcaster Dwarkesh Patel published Clocktower Bench, a benchmark where models play the social-deduction game Blood on the Clocktower against each other. He says he spent about $20,000 of API credits building the harness, and that much of it went to iterating on the environment design, more than the tokens he had planned.

The project's README (checked 8 November 2026) describes the setup: 11 models, 386 games for the headline ranking, one model playing every good seat and another every evil seat, 10–12 players, games that last several hours. Each deal is replayed with the sides swapped, so neither model gets the easier role. It also ships a free dry run (--model random), writes spend to games/ledger.jsonl, and has a --budget cap that defaults to $1,000.

What it doesn't give is tokens per game. So there's nothing to copy; you build the estimate yourself. That's what the rest of this article does, with stated assumptions you can swap for your own.

One seat: why the input keeps growing#

Say you run a smaller game: 6 seats, every seat is a model with its own conversation, 20 turns per seat. Assume a 2,000-token system prompt (rules and the seat's role), 600 new tokens of table talk arriving before each turn, and a 300-token reply. The API has no memory, so each request carries the whole conversation again (why a long chat costs more).

Turn 1 sends 2,000 + 600 = 2,600 tokens. Turn 20 sends 2,000 + 600 × 20 + 300 × 19 = 19,700. Try the last number yourself:

Add up all 20 turns for one seat: 2,000 × 20 + 600 × 210 + 300 × 190 = 40,000 + 126,000 + 57,000 = 223,000 input tokens. The output is only 20 × 300 = 6,000. If you counted each message once, you'd get 2,000 + 20 × 600 = 14,000 and be off by a factor of sixteen.

The output side barely moves. Input is where the money goes, and that's the side that gets worse as games get longer.

From one seat to a whole tournament#

Now scale it with three multipliers.

• Seats. 6 seats × 223,000 = 1,338,000 input tokens and 6 × 6,000 = 36,000 output tokens per game.
• Side swap. Take 3 models in a round robin: 3 pairings. Play 10 deals per pairing, each one twice with the sides swapped: 3 × 10 × 2 = 60 games.
• Games. 60 × 1,338,000 = 80,280,000 input and 60 × 36,000 = 2,160,000 output tokens for the tournament.

One game on DeepSeek-V4-Flash comes to $0.0237. The table below prices the whole 60-game tournament as if that one model played every seat, so you can compare the three models CheapTokens offers for this kind of long-conversation job. In total, with input and output counted together, each of them is cheaper than the OpenRouter reference in the last column.

Budget-wise this is the whole trick: you don't need a $20,000 line item to learn which of three models plays a negotiation game better. You need the estimate above, and then a real pilot to check it.

Two checks before you trust any such number. First, the largest request must fit the model's window. At 20 turns the biggest request is 19,700 tokens, no problem. A multi-hour game with 200 turns per seat reaches 2,000 + 600 × 200 + 300 × 199 = 181,700, about 182K, so read the context size on the model's page before you design the game. DeepSeek-V4-Flash has a 380K window and up to 32,768 output tokens; see the model page. Second, if a model emits long reasoning, your 300 output tokens become more (thinking tokens in a bulk job).

Put a hard cap in the code#

The README's --budget flag is the right idea, and it takes ten lines to copy into your own harness. The script below estimates before it runs, refuses to send a request that could push the total over the cap, and logs every call to a ledger file. The prices in it are an example: substitute the current ones from the pricing table.

import json, os
from openai import OpenAI

client = OpenAI(
    base_url='https://api.cheaptokens.dev/v1',
    api_key=os.environ['CHEAPTOKENS_API_KEY'],
)
MODEL = 'deepseek-ai/DeepSeek-V4-Flash-0731'
PRICE_IN, PRICE_OUT = 0.10, 0.40  # per 1M tokens: example, substitute current
CAP_USD = 5.00
spent = 0.0


def cost(tin, tout):
    return (tin * PRICE_IN + tout * PRICE_OUT) / 1_000_000


def estimate(seats, turns, system=2000, new_in=600, out=300):
    tin = sum(system + new_in * k + out * (k - 1) for k in range(1, turns + 1))
    return seats * tin, seats * turns * out


def ask(seat, history, max_out=400):
    global spent
    rough_in = sum(len(m['content']) for m in history) // 3  # pessimistic
    if spent + cost(rough_in, max_out) > CAP_USD:
        raise SystemExit(f'cap reached: ${spent:.4f} spent')
    r = client.chat.completions.create(
        model=MODEL, messages=history, max_tokens=max_out)
    u = r.usage
    spent += cost(u.prompt_tokens, u.completion_tokens)
    with open('ledger.jsonl', 'a') as f:
        f.write(json.dumps({'seat': seat, 'in': u.prompt_tokens,
                            'out': u.completion_tokens,
                            'spent': round(spent, 6)}) + '\n')
    return r.choices[0].message.content


print(estimate(6, 20))  # (1338000, 36000) per game

estimate(6, 20) prints (1338000, 36000), the same per-game numbers as above, so the code and the arithmetic check each other. usage.prompt_tokens and usage.completion_tokens come from the response and are what you're billed for; the pre-check uses a pessimistic character count because the real count is only known after the call. More on per-project ledgers in tracking agent token cost in Python.

A safe order for the first run#

The repo's own sequence is a good model: free first, cheap second, expensive last. Put these steps in order:

The README says a demo game on cheap models costs about $2–5. That's the author's statement and the models aren't named, so treat it as a sanity range, not a quote for your setup. Your own pilot game gives the number that matters: if its ledger shows 1,500,000 input tokens where you assumed 1,338,000, you've learned something for a few cents instead of at game 40.

For longer experiments, the usual levers are in trimming chat history: cut what each seat has to re-read, and the quadratic term shrinks with it. When you're ready to run it, the docs have the endpoint and model ids.