How to read an AI benchmark: MTGBench and a 5-prompt test

You open a leaderboard, see model A above model B, and pick A. Then A stumbles on your task and B doesn't. Nothing is broken: a rank is an average over someone else's tasks, and your task may not look like theirs. The fix is small. Take five of your own prompts, run them on two models, read the answers and compare what each one cost per task. The rest of this page shows why that works, using a fresh paper as the example, and gives you the script.

Why a rank is not a verdict#

The paper is MTGBench: A Multi-Turn Game Benchmark for Diverse LLM Capability Assessment, published in Findings of EMNLP 2026 (checked 29 October 2026). Its authors evaluated 16 frontier models on seven games, and one of their main findings is that model rankings depend heavily on the task. Move to a different game and the order shifts.

Now think about a leaderboard with a single average at the top. The average hides the games. If your task looks like the games a model is good at, you win. If it doesn't, the number you read was about something else.

What the seven games are#

The games are Word Chain, Wordle, 20 Questions, Wheel of Fortune, Situation Puzzles, 20 Features and Alias (names from the project README on GitHub, checked 29 October 2026). They are multi-turn: the model doesn't answer once, it plays. Guess, get feedback, guess again. Wordle is the easy picture: you propose a word, learn which letters fit, and try again.

Two things in the design are worth copying in your own test.

First, who checks the answer. Four games are scored by ordinary code. The other three use a constrained LLM as the rule enforcer, and the authors validated it against human annotators. Someone always has to decide what counts as correct. In your mini-test, that someone is you.

Second, language. The suite is bilingual, English and Russian, and the episodes were generated natively in each language, not translated. The authors also collected a human baseline, 476 sessions, reported for Russian. Their summary: the strongest reasoning-enabled models match or exceed that baseline across the suite.

What a score can't tell you#

It can't tell you how to ask. MTGBench compares three setups: answer-only (the default), chain-of-thought and native reasoning. The authors report that CoT is nearly essential for letter-level games but gives uneven gains on knowledge-driven ones. So the same model looks different depending on how it was asked. When you test, try your hardest prompt both ways.

It can't tell you the cost of thinking. Thinking out loud is text the model writes, and written text is output, the pricier side of the bill (more in Input vs output tokens and Reasoning effort and thinking tokens). Say a prompt of 600 tokens gets a 100-token answer without CoT and a 600-token answer with it. On DeepSeek-V4-Flash that is $0.000019 against $0.000069 per request. Small money, until you send thousands of requests a day.

It can't tell you about your language. MTGBench builds its episodes natively in English and in Russian rather than translating, which is a hint in itself: language is part of the task. If your users write in Russian, put Russian prompts in your five.

Run the 5-prompt test#

The API is OpenAI-compatible, so one script runs any model and only the model id changes (ids are in the docs).

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.cheaptokens.dev/v1",
    api_key=os.environ["CHEAPTOKENS_API_KEY"],
)
models = ["deepseek-ai/DeepSeek-V4-Flash-0731", "zai-org/GLM-5.3-Flash"]
prompts = [open(f"prompts/{i}.txt").read() for i in range(1, 6)]

for model in models:
    tin = tout = 0
    for i, prompt in enumerate(prompts, 1):
        r = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=2000,
        )
        tin += r.usage.prompt_tokens
        tout += r.usage.completion_tokens
        print(f"--- {model} / prompt {i} / {r.choices[0].finish_reason}")
        print(r.choices[0].message.content)
    print(f"=== {model}: {tin} in, {tout} out")

The table below shows the same example, five requests of 600 tokens in and 300 out, on every model we have, with OpenRouter next to it for reference. Input and output are counted together, and the total is lower than OpenRouter's on each row.

A model that scores 9 of 10 at a fraction of the cost beats one that scores 10 of 10 at several times the price, and now you can see which case you are in. The pricing page has the live numbers for the rest of the models, and the DeepSeek page covers long answers (up to 32,768 output tokens) if your prompts need step-by-step output.

What to put in your five prompts#

The right five depend on what you build. Pick your case.

When the numbers are in, How to choose an AI model helps with the rest of the decision: context size, output length, and what to do when two models tie.