Compare two LLMs fairly on your task: side-swap and cost log

You ran three prompts through two models, and the second one won. Before you trust that, check which one was listed second. A fair comparison needs three things: identical settings for both models, every pair judged in both orders, and a log of tokens per call, so you know what the whole run cost. Below is a small Python script that does all three on your own task.

What a public tournament does that you can borrow#

Dwarkesh Patel's Clocktower Bench is an unofficial research project where language models play the social-deduction game Blood on the Clocktower against each other. The game is not the point here; the harness is. According to the project's About page (checked 8 November 2026), it does a few things that any head-to-head should do:

• one model plays all the good seats and another all the evil seats, and each deal is played twice with the sides swapped, so a lucky seat does not decide the result;
• all models get the same settings: the same reasoning effort, an 8,000-token ceiling per turn and a 150-word limit on speech;
• the Storyteller is scripted, so no model referees the game;
• every API call is written to calls.jsonl with its tokens and cost. The page says a single game costs from about $2 to $50, depending on the models (on OpenAI and Anthropic APIs).

The harness is built for OpenAI and Anthropic models, and your task is probably not a board game. But the three habits transfer: equal settings, swapped sides, and a cost log. The rest of this guide applies them to something ordinary.

The task: drafting replies to 30 support tickets#

Say you run a small support desk and want to know whether DeepSeek-V4-Flash or GLM-5.3-Flash writes better first drafts. You have a file tickets.jsonl with 30 real tickets, one JSON object per line: {"id": 1, "text": "..."}.

The API is OpenAI-compatible, so the official openai package works with CheapTokens at https://api.cheaptokens.dev/v1. Use the full model ids from the docs: deepseek-ai/DeepSeek-V4-Flash-0731 and zai-org/GLM-5.3-Flash. Both models also get a third model as a judge, MiniMaxAI/MiniMax-M2.7, so neither contestant grades its own work.

Start with the part that makes the log possible: one function through which every call passes.

import json, os, re
from openai import OpenAI

client = OpenAI(base_url="https://api.cheaptokens.dev/v1",
                api_key=os.environ["CHEAPTOKENS_API_KEY"])

CONTESTANTS = {"deepseek": "deepseek-ai/DeepSeek-V4-Flash-0731",
               "glm": "zai-org/GLM-5.3-Flash"}
JUDGE = "MiniMaxAI/MiniMax-M2.7"
SETTINGS = {"temperature": 0.2, "max_tokens": 800}   # same for both

def call(model, messages, role, ticket_id, **settings):
    r = client.chat.completions.create(model=model, messages=messages, **settings)
    with open("calls.jsonl", "a", encoding="utf-8") as f:
        f.write(json.dumps({
            "role": role, "ticket": ticket_id, "model": model,
            "input_tokens": r.usage.prompt_tokens,
            "output_tokens": r.usage.completion_tokens,
            "finish": r.choices[0].finish_reason}) + "\n")
    return r.choices[0].message.content or ""

def draft(model, ticket):
    msgs = [{"role": "system", "content": "You answer support tickets. "
             "Reply in under 80 words, politely, with no promises about refunds."},
            {"role": "user", "content": ticket["text"]}]
    return call(model, msgs, "draft", ticket["id"], **SETTINGS)

The fields in calls.jsonl are the ones this script chooses; the usage numbers come from the usage object of the API response. Both models get the same system prompt, the same temperature and the same max_tokens. Change one of them for one model and you are comparing settings, not models.

Swap the sides: judge every pair twice#

Now the judge. A judge model can lean toward the answer in a particular slot, so a single verdict mixes quality with position. The Clocktower trick fits neatly: show each pair in both roles. A reply wins a ticket only if the judge picks it in both orders. If the verdict follows the slot instead of the reply, the ticket is a tie.

def judge(ticket, first, second):
    prompt = (f"Ticket:\n{ticket['text']}\n\nReply A:\n{first}\n\nReply B:\n{second}\n\n"
              "Which reply is better for the customer? Finish with a last line: Verdict: A or Verdict: B.")
    out = call(JUDGE, [{"role": "user", "content": prompt}], "judge",
               ticket["id"], temperature=0, max_tokens=1000)
    letters = re.findall(r"Verdict:\s*([AB])\b", out)
    return letters[-1] if letters else None

tickets = [json.loads(l) for l in open("tickets.jsonl", encoding="utf-8")]
wins = {"deepseek": 0, "glm": 0, "tie": 0}

for t in tickets:
    a = draft(CONTESTANTS["deepseek"], t)
    b = draft(CONTESTANTS["glm"], t)
    v1 = judge(t, a, b)   # DeepSeek's reply shown as A
    v2 = judge(t, b, a)   # GLM's reply shown as A
    pick1 = {"A": "deepseek", "B": "glm"}.get(v1)
    pick2 = {"A": "glm", "B": "deepseek"}.get(v2)
    wins[pick1 if pick1 and pick1 == pick2 else "tie"] += 1

print(wins)

Two details are easy to miss. The judge runs at temperature=0 to keep its verdicts steadier; not every model honours that setting, so rerun a few pairs and compare the verdicts before you trust them. The verdict is read only from a final Verdict: line, and the caps are generous (800 for drafts, 1000 for the judge) so a model that reasons before answering is not cut off mid-thought; the estimate below uses typical sizes, not the caps. An unparsable answer gives None, which falls into a tie instead of crashing the run.

Order is only one way to tilt a test. Settings, prompts and logging can tilt it too, so check the whole setup, not just who goes first. Here is the same experiment before and after.

Log every call, then price the run#

The log is what turns the comparison into a budget. After the run, sum the tokens per model:

from collections import defaultdict

totals = defaultdict(lambda: [0, 0, 0])
for line in open("calls.jsonl", encoding="utf-8"):
    c = json.loads(line)
    s = totals[c["model"]]
    s[0] += c["input_tokens"]; s[1] += c["output_tokens"]; s[2] += 1

for model, (tin, tout, n) in totals.items():
    print(model, n, "calls", tin, "input", tout, "output")

You will get one line per model with call count, input tokens and output tokens. Multiply by the model's two prices (input and output are billed separately) or, easier, paste the totals into a live cost marker.

You can also estimate before you run. Assume each ticket plus the system prompt is about 300 input tokens, a draft is about 250 output tokens, and the judge sees roughly 800 input tokens (ticket plus both replies plus the question) and answers with about 200 tokens (the verdict line plus a little room to think):

• drafts: 30 tickets × 2 models = 60 calls, 30 × 300 = 9000 input and 30 × 250 = 7500 output tokens per contestant;
• verdicts: 30 tickets × 2 orders = 60 calls, 60 × 800 = 48000 input and 60 × 200 = 12000 output tokens on the judge.

On current prices, one contestant's 30 drafts cost $0.00089 on DeepSeek-V4-Flash and $0.00048 on GLM-5.3-Flash. The 60 verdicts on MiniMax-M2.7 cost $0.0042. Add the three and you have the price of the whole comparison before you send a single request. The same drafting workload across all three models, with input and output counted together:

The table also shows the OpenRouter reference, so you can see how the total compares. The token assumptions are yours to change: if your tickets are long, the judge input grows fastest, because it carries both replies twice. If your judge thinks aloud before answering, its output number grows too, and your log will show it. For the habit of counting tokens before a request, see count tokens before you send a request.

The same cap Clocktower uses is worth stealing. It sets 8,000 tokens per turn, and you already have max_tokens. Put it in SETTINGS and a runaway answer cannot quietly multiply your estimate.

Before you launch the full run#

That last item matters. Run three tickets, open calls.jsonl and read it. Check that finish says stop rather than length; if a draft was cut off by max_tokens, one model may be losing only because its answer was truncated. The same goes for the judge's calls: a length there turns into a silent tie, so count the None verdicts too. If your task has a single right answer, you may not need a judge at all: test your prompt on 20 cases with exact-match grading.

Reading the result without fooling yourself#

Look at the three numbers in wins. Thirty tickets is a small sample: a split like 12 to 9 with 9 ties tells you the models are close, not that one is better. The log helps here. If one model wins slightly but uses much more output, the real question is the price of the task, not the win count; price per 1M tokens versus cost per task walks through that comparison.

If the result is close, pick the cheaper model for the task, or add more tickets and rerun. Both are cheap to try because the whole machinery is already a loop. When you want to repeat the exercise on another model, the three models above sit behind one API and one key: the setup in the docs is the same for each, and the pricing page lists what each costs right now. Dwarkesh Patel's own finding on OpenAI versus Anthropic models (detection versus deception) is his result for his game; yours will come from your tickets.