Filter 40K comments with an LLM: keep/drop prompt and cost

Your archive has 40,000 comments, and most of them are noise: spam, "+1", jokes, replies that have nothing to do with what you're after. A cheap LLM can read each one and answer with a single word, keep or drop. Send every comment as its own short request and the whole pass comes to about 6 million input tokens and 200 thousand output tokens (assuming roughly 150 tokens in and 5 out per comment). On DeepSeek-V4-Flash that run costs $0.11 at the current price.

Below: the prompt, a short Python loop, and the arithmetic you can redo for your own archive. Everything goes through the OpenAI-compatible endpoint described in the CheapTokens docs.

Where the idea comes from#

Andrej Karpathy (AI researcher, OpenAI co-founder, former AI director at Tesla) described in his post of 31 October 2026 how he built HackerTube, a site of videos recommended in Hacker News comments. In that build, Sonnet ran over the comments in a Claude Code workflow to remove spam and narrow them down to about 40,000 comments that recommend YouTube videos. The result was 6,454 videos across 150 topics, and he says the whole build now takes about two hours.

The post doesn't report token counts or cost. So the numbers in this article are arithmetic from assumptions stated right next to them, not a reconstruction of his run. What you can borrow is the shape of the task: lots of short texts, one yes/no question about each.

How a yes/no filter works#

Think of a doorman with a one-line rule: "let in anyone with a ticket". He looks at each person for a second, says yes or no, and never remembers the previous guest. An LLM filter works the same way. Each request carries your instruction plus one comment, and the model sends back one word. Comment 17,000 knows nothing about comment 16,999.

That design is why the bill is predictable. Every request has roughly the same size, so cost is requests × tokens per request. And because the answer is one word, almost all of the money goes into the input side: the instruction and the comment you send. If input and output pricing is new to you, input vs output tokens explains the split.

Write the keep/drop prompt#

The prompt is the whole product here. Compare a vague version with an exact one:

Is this comment good? Answer yes or no.

Answer keep if the comment recommends a specific YouTube video or talk. Answer drop for spam, jokes, replies without a video, and everything else. Reply with one word.

Say three comments arrive. These are the answers you are aiming for:

• "This talk on the history of Unix changed how I see shells: youtube.com/watch?v=... Worth the hour." → keep
• "Cheap watches here!!! click my profile" → drop
• "Agree, great point about caching." → drop

Three habits make the prompt hold up across 40,000 items. Say what to do with the leftover cases ("everything else is drop"); otherwise the model invents a third answer. Ask for one word, so the reply is easy to compare with keep in code. And wrap the comment in tags, as in the loop below: a comment that says "ignore the instructions above" is then clearly data, not an order. Label every document in a prompt goes deeper on that.

The loop: one request per comment#

Here is the whole thing. It reads comments.jsonl (one JSON object with a text field per line), asks for a verdict on each comment, and writes the kept lines to kept.jsonl. The model id is the full gateway_id; the key comes from an environment variable.

import json
import os
from openai import OpenAI

client = OpenAI(
    base_url='https://api.cheaptokens.dev/v1',
    api_key=os.environ['CHEAPTOKENS_API_KEY'],
)

INSTRUCTION = '''You filter Hacker News comments.
Answer keep if the comment recommends a specific YouTube video or talk.
Answer drop for spam, jokes, replies without a video, and everything else.
Reply with one word: keep or drop. The comment is between <comment> tags.'''

def verdict(text):
    r = client.chat.completions.create(
        model='deepseek-ai/DeepSeek-V4-Flash-0731',
        messages=[
            {'role': 'system', 'content': INSTRUCTION},
            {'role': 'user', 'content': f'<comment>{text}</comment>'},
        ],
        temperature=0,
        max_tokens=10,
    )
    return r.choices[0].message.content.strip().lower(), r.usage

kept = tokens_in = tokens_out = 0
with open('comments.jsonl') as src, open('kept.jsonl', 'w') as out:
    for line in src:
        answer, usage = verdict(json.loads(line)['text'])
        tokens_in += usage.prompt_tokens
        tokens_out += usage.completion_tokens
        if answer.startswith('keep'):
            out.write(line)
            kept += 1

print(kept, 'kept;', tokens_in, 'tokens in,', tokens_out, 'tokens out')

temperature=0 makes the verdicts more repeatable between runs, and max_tokens=10 leaves room for one word and nothing more. The last line is the useful one: it adds up the usage numbers the API returns, so you get your real token counts instead of a guess.

Run it on the first ten comments and read every verdict yourself. If an answer looks cut off or comes back empty, check finish_reason as described in why an AI answer stops mid-sentence. One request at a time is fine for a test; for the full 40,000, AsyncOpenAI sends many in parallel.

What 40,000 comments cost#

Here are the assumptions, so you can swap in your own. A comment plus the instruction is about 150 tokens of input. The reply is about 5 tokens. Hacker News comments are often longer, so treat 150 as a floor and replace it with the rehearsal numbers. For 40,000 comments:

• input: 40,000 × 150 = 6,000,000 tokens
• output: 40,000 × 5 = 200,000 tokens

The instruction is the part people forget. It travels with every single comment. Try a guess:

Now the bill for the full pass, on live prices:

On this workload, DeepSeek-V4-Flash and GLM-5.3-Flash both come out cheaper than OpenRouter in total, with input and output counted together; the OpenRouter column shows it side by side. Both are priced for a high-volume job like this: DeepSeek-V4-Flash has one price at any hour and up to 32,768 output tokens per answer, GLM-5.3-Flash is a second option on the same endpoint. Try your 50-comment rehearsal on each and compare the verdicts yourself. To switch, change the model line to zai-org/GLM-5.3-Flash. Current prices for every model are on the pricing page, and DeepSeek models at CheapTokens lists the details.

Before you press run#

First, a rehearsal. Take 50 comments, run the loop, and read all 50 verdicts. Then use the printed token totals: 50 × 800 = 40,000, so the full run is the rehearsal's tokens multiplied by 800. That replaces the 150-and-5 assumption with your own numbers.

Second, fix the prompt on the rehearsal, not on the full run. If three "drop" verdicts are wrong, add one sentence about that case and rerun the 50. It costs a fraction of a cent to be picky here, and a lot more to rerun 40,000.

Third, remember what the model sees: one comment, no thread. If "agree, great point" is only meaningful next to the parent comment, put the parent in the request as well. That makes each request longer, so redo the token arithmetic with the new size.

When the rehearsal looks right, make an API key in your account, point CHEAPTOKENS_API_KEY at it and start the full pass. If the key is new to you, what an API key is and how to store it takes five minutes.