Why the Same Prompt Gets Two Different Bills — And It's Not the Pricing Page's Fault

Day 9: Tokenization — Why the Same Prompt Costs Different Amounts on Different Models

You're being billed by a unit you've probably never looked at. Not words, not characters — tokens. If you've never inspected how your provider's tokenizer actually chops up text, you can't predict what a prompt will cost, you can't debug a context-window overflow when it blows up in production, and you definitely can't explain to a PM why the bill for 'basically the same feature' looks different across two providers.

The A/B test that made no sense

A team I worked with ran an A/B cost comparison: same system prompt, same user input, same output length, two different providers. One came back noticeably pricier per request — even though both charged roughly the same rate per token and the task was functionally identical. First reflex was to blame the pricing page. Wrong target. The real answer was that the two models' tokenizers split that exact same input string into a different number of tokens. Nobody had actually counted tokens — they'd counted words, or eyeballed character length, and assumed the number would carry over roughly the same everywhere. It doesn't.

Same string, three different splits

Take one input string and run it through three tokenizer families side by side: OpenAI's tiktoken (cl100k_base / o200k_base), Anthropic's Claude tokenizer, and a Mistral-style SentencePiece tokenizer you can inspect yourself via Hugging Face's AutoTokenizer. None of them agree on where a 'word' ends.

Input: "El café cuesta $3.50 🙂 def calculate_total():"

GPT-style (BPE, cl100k/o200k):
["El", " café", " cuesta", " $", "3", ".", "50", " 🙂", " def", " calculate", "_total", "():"]

Claude-style tokenizer:
["El", " caf", "é", " cuesta", " $", "3.", "50", " 🙂", " def", " calculate", "_", "total", "():"]

Mistral-style (SentencePiece):
["▁El", "▁caf", "é", "▁cuesta", "▁$", "3", ".", "50", "▁", "🙂", "▁def", "▁calculate", "_total", "():"]

Exact counts drift between tokenizer versions, so treat this as illustrative, not a benchmark — the point is structural, not the specific numbers. Look at 'café' and the emoji: they don't split the same way twice. That's the entire lesson sitting inside one string. None of these models are reasoning over 'words.' They're reasoning over whatever chunks their vocabulary happened to merge during training.

A token isn't a word. It's a BPE merge.

Every major LLM tokenizer today runs on some flavor of subword tokenization — usually Byte-Pair Encoding (BPE) or a close cousin like SentencePiece unigram. The training process itself is mechanical, almost boring: start from raw bytes or characters, repeatedly merge whatever pair shows up most often into one new symbol, and keep going until you hit a target vocabulary size — commonly somewhere between 32k and 200k+ entries. Whatever pairs were frequent in that lab's training corpus become single tokens. Everything else stays fragmented.

• ▹A token is a vocabulary entry — something a tokenizer produced by counting frequent byte/character pairs in one specific training corpus. It's not a word, not a morpheme, not any linguistic unit you'd recognize.
• ▹Every model ships its own vocabulary. OpenAI's o200k_base, Anthropic's tokenizer, Mistral's SentencePiece vocab — all trained separately, on different data mixes, at different sizes.
• ▹Different merge tables mean the same input string decomposes into a different token sequence, and a different token count, on every model.
• ▹Which means 'cost per prompt' was never a property of the prompt. It's a property of the (prompt, tokenizer) pair. Ask 'how many tokens is this?' and the only honest answer is 'according to which model?'

Where this quietly bites teams

• ▹Non-English text: vocabularies get trained on corpora skewed hard toward English web text, so morphologically rich or non-Latin-script languages — Vietnamese, Arabic, CJK, plenty of others — routinely need 2-3x more tokens to encode the same sentence. Same meaning, more billable units.
• ▹Code and identifiers: camelCase, snake_case, and the odd variable names your team actually writes weren't the dominant pattern in most pretraining corpora, so they split unpredictably. 'calculate_total' might land as one token, three tokens, or fracture at a byte boundary depending on the model — which matters a lot if you're running agents that read and write large codebases.
• ▹Emoji and rare symbols: anything outside the common merge patterns falls back toward byte-level tokens, so a single emoji can quietly cost several tokens instead of one.
• ▹None of this shows up until you actually count. Word count and character count are both bad proxies for token count, and the gap compounds fast across a long system prompt or a big RAG context.

The subtler cost: context budget and reasoning, not just price

A fixed context window — say, 200k tokens — is a fixed budget of fragments, not a fixed amount of information. If your tokenizer breaks the same document into more pieces, you fit less actual content into the same window. That hits RAG chunk sizing directly, it caps how much conversation history an agent can hold before truncation, and it eats into how much room is left for the model's own reasoning/thinking tokens in a multi-step agent loop. Denser tokenization isn't just cheaper. It's more headroom for the model to actually think.

The 5-minute check: count tokens before you ship

Don't estimate. Measure — against your actual prompts, for every provider you're evaluating.

# OpenAI-style: tiktoken (pip install tiktoken)
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
print(len(enc.encode(my_prompt)))

# Anthropic: use the Messages Token Counting API
# POST /v1/messages/count_tokens (or client.messages.count_tokens in the SDK)

# Mistral-family: Hugging Face tokenizers
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
print(len(tok.encode(my_prompt)))

The habit worth taking from Day 9: before you commit to a model for pricing or context-budget reasons, run your real prompts — the non-English strings, the code snippets, the emoji your users actually send, not a clean English test sentence — through each candidate's tokenizer and compare the counts directly. Don't infer cost from word count. And don't assume a comparison done on English-only text tells you anything about your actual traffic.

Extend your knowledge

• ▹Read the paper this all traces back to: Sennrich, Haddow, Birch, 'Neural Machine Translation of Rare Words with Subword Units' (2016) — the basis for most modern LLM tokenizers.
• ▹Pull up OpenAI's tiktoken repo (github.com/openai/tiktoken) and run it against your own prompts, not the examples in this post.
• ▹Check Anthropic's docs on the Token Counting API for the exact request/response shape before wiring it into a pre-flight cost check.
• ▹Load a Mistral-family tokenizer with Hugging Face's `transformers` AutoTokenizer and poke around its vocab directly with `tokenizer.get_vocab()`.