Tokens and Sampling

Models do not read characters; they decode one token at a time. Input-side tokenization determines the cost baseline, while output-side sampling strategies determine creativity and determinism—and by 2026, these two ends have diverged between Claude and local models.

Overview

Models do not read characters, nor do they directly spit out sentences. On the input side, text is chopped into tokens; on the output side, the model produces "the probability distribution of the next token across the entire vocabulary" in one step, and a sampling strategy picks one, appends it to the input, and generates autoregressively, one by one. These two ends are usually invisible, yet they determine three practical issues that constantly arise in development: can you calculate costs accurately (billing units are tokens), can you tune "creativity" correctly (sampling strategies), and why do identical prompts yield different results (non-determinism).

More importantly, the "common sense" regarding these two ends has diverged in 2026. Open-source/local models still rely on temperature/top_p/top_k to tune sampling (e.g., Qwen running locally); meanwhile, Claude's hosted API has completely removed these parameters starting from Opus 4.7—passing them results in a direct 400 error. It's two worlds under the same concept; this article explains the mechanisms of both ends and this divergence in depth.

Input Side: How Text Becomes Tokens

Subword Tokenization and BPE

Modern LLMs do not tokenize by character or word, but by subword. The most common algorithm is BPE (Byte Pair Encoding): during training, it starts from characters and repeatedly merges the "most frequent adjacent token pair" into a new token until the vocabulary reaches a set size (usually tens of thousands to over a hundred thousand). The result is: high-frequency words become a single token, while low-frequency words are split into several subwords.

Modern implementations mostly use byte-level BPE: performing BPE on bytes rather than Unicode characters. The benefit is no OOV (Out-Of-Vocabulary)—any input (rare characters, emojis, binary strings) can fall back to bytes and will never fail to tokenize. The cost is that non-ASCII characters (Chinese, Japanese) often occupy multiple bytes and thus multiple tokens per character.

Counter-intuitive but Critical Points

• token ≠ word ≠ character. In English, approximately 1 token ≈ 4 characters; in Chinese, one character often takes 1–2+ tokens, and punctuation/indentation in code also consumes tokens. Estimating length by character count will systematically underestimate.
• Chat templates also consume tokens. In multi-turn conversations, role markers, message separators, and tool definition schemas are all counted as input tokens—not just the body text costs money.
• Tokens are not universal across models, and they drift with versions. Claude's tokenizer ≠ OpenAI's; do not use tiktoken to estimate Claude (it undercounts by 15–20%, and the error is even worse for code/Chinese). Even within the same vendor, versions change: Opus 4.7's tokenizer might split the same text into 1.0–1.35 times the number of tokens of the previous generation. When switching models, re-baseline; do not apply old multipliers.
• For accuracy, call the API; do not use third-party approximators:

# Single-segment counting
n = client.messages.count_tokens(
    model="claude-opus-4-8",
    messages=[{"role": "user", "content": text}],
).input_tokens

# Estimate cost (based on official pricing, which may change): input_tokens × unit price
# Compare diff between two versions: count separately, then subtract—the endpoint is stateless, so do the subtraction yourself

count_tokens is stateless; to calculate "how many tokens changed," count separately and subtract.

Output Side: Sampling One Token at a Time

The model does not output words at each step, but a score vector (logits) across the entire vocabulary. These are converted into a probability distribution via softmax, then sampled according to a strategy, appended to the input, and the process repeats—autoregressively.

flowchart LR
    T["Text"] --> TK["Tokenizer → tokens"]
    TK --> M["Model (Transformer)"]
    M --> L["logits<br/>(one score per token in vocabulary)"]
    L --> SM["softmax(logits / T)<br/>→ Probability Distribution"]
    SM --> S["Sampling Strategy<br/>top_p / top_k / greedy"]
    S --> N["Next Token"]
    N -->|Append to input, Autoregressive| M

Sampling Knobs (on models that still support them)

In practice, the temperature + top_p combination is most common: temperature controls randomness, while nucleus sampling cuts off absurd long tails. top_k is coarser.

Why Chat LLMs Don't Use Beam Search

Classic machine translation uses beam search (maintaining multiple candidate paths simultaneously to find the global optimum). Chat/open-generation models almost never use it: it tends to produce high-probability but boring, repetitive text ("neural text degeneration"), sacrificing diversity and naturalness. Sampling (top_p/temperature) instead generates text that is more human-like and informative—this was the core conclusion of the 2020 nucleus sampling paper.

Why T=0 Does Not Guarantee Byte-for-Byte Reproduction

Many people believe "temperature 0 = reproducible." This is not true. The reason lies in hardware and scheduling, not sampling:

• Floating-point non-associativity: The order of accumulation for massive parallel additions on GPUs is not fixed. (a+b)+c ≠ a+(b+c) holds true in floating-point arithmetic, and tiny differences can flip the choice at the argmax threshold.
• Batch composition: Which requests are batched together and how kernels are scheduled affect numerical results.
• MoE routing: MoE models expert selection is sensitive to numerical variations.

"Determinism" is an engineering goal (fixed prompt, low randomness, accepting minor drift), not a guarantee provided by any single sampling parameter.

Divergence: Claude Removed Sampling, Local Models Still Use It

Starting with Claude Opus 4.7 / 4.8 / Fable 5, temperature, top_p, and top_k were removed—passing them results in a direct 400 error. This is not a regression; it represents a shift in design philosophy:

• On strong reasoning models, using sampling temperature to "tune creativity/determinism" yields limited and uncontrollable benefits.
• Anthropic instead provides two higher-level tools:
  • Prompting—if you want determinism, hardcode instructions; if you want divergence, explicitly request diversity in the prompt (e.g., in frontend design scenarios, ask the model to "provide 4 different directions first, then implement," see the design section in the migration docs);
  • Effort (output_config.effort)—controls how deep to think and how many tokens to spend (see Reasoning and Thinking).

Migration key points: temperature=0.7 in old code must be removed for new models. Translate intent into prompts + effort—temperature=0 (seeking determinism) → effort:"low" + hardcoded instructions; temperature=high (seeking divergence) → explicitly state "provide multiple different directions" in the prompt. This migration recipe is detailed in Prompt Engineering and Structured Output.

But on the other end, sampling is still alive. Local/open-source models (llama.cpp, vLLM) still accept these parameters—Local LLM Deployment sets them. Moreover, model cards usually provide recommended values: Qwen-AgentWorld officially recommends temperature 0.6 / top_p 0.95 / top_k 20. So "removing sampling" is a stance specific to Claude's hosted API, not an industry standard—use the rules of the world you are in.

Best Practices

• For accuracy, call count_tokens; do not use third-party approximators. tiktoken undercounts Claude by 15–20%, and the error is even worse for Chinese/code; chat templates and tool schemas also consume tokens.
• Re-baseline tokens when switching models. Tokenizers drift with versions; do not apply old multipliers; to compare differences between versions, count separately and subtract.
• Use the rules of the world you are in. For local/open-source models, continue tuning temperature/top_p/top_k (according to model card recommendations); Claude's hosted API has removed sampling, so use prompts + effort instead.
• Set sampling values for local models according to model cards. For example, AgentWorld recommends 0.6 / 0.95 / 20; do not gamble on default values for divergence.
• Do not expect T=0 to guarantee byte-for-byte reproduction. Floating-point accumulation, batching, and MoE routing all introduce jitter; to strictly reproduce, cache results.

Trade-offs and Failure Modes

• Estimating tokens by character count: Severe underestimation for Chinese/code → use count_tokens, and include chat templates/tool schemas.
• Applying token counts or old temperatures across models: Tokenizers and sampling faces change → re-baseline when switching models, remove sampling parameters during migration.
• Expecting T=0 to reproduce: Impossible to guarantee byte-for-byte consistency → fix prompt + low effort, accept minor drift; to strictly reproduce, cache results.
• Not setting recommended sampling values for local models: Using default temperature may be too divergent or too rigid → follow model cards (e.g., AgentWorld's 0.6/0.95/20).

References

• Anthropic Official Docs: Token Counting, Models API, Migration Guide (sampling parameter removal) (platform.claude.com)
• Papers: "Neural Machine Translation of Rare Words with Subword Units" (BPE, Sennrich 2016), "The Curious Case of Neural Text Degeneration" (nucleus sampling, Holtzman 2020)
• Model Cards: Qwen-AgentWorld-35B-A3B (recommended sampling values)

Keywords: tokenization, subword, BPE, byte-level BPE, vocabulary, OOV, chat template, token ≠ word, count_tokens, tiktoken, tokenizer drift, logits, softmax, autoregressive, greedy, temperature, top_p, nucleus sampling, top_k, repetition penalty, stop sequence, beam search, text degeneration, determinism, floating-point non-associativity, sampling parameter removal, effort