---
title: Tokens and Sampling
url: https://doc.liz6.com/en/ai/01-models-and-context/02-tokens-and-sampling
locale: en
area: ai
tags:
- ai
- models-and-context
date: 2026-06-30
modified: 2026-07-16
description: Models do not read characters; they decode one token at a time. Input-side tokenization determines the cost baseline, while output-side sampling strategies determine creativity and determinism—and by 2026, these two ends have diverged between Claude and local models.
---
# Tokens and Sampling
> Models do not read characters; they decode one token at a time. Input-side tokenization determines the cost baseline, while output-side sampling strategies determine creativity and determinism—and by 2026, these two ends have diverged between Claude and local models.
## Overview
Models do not read characters, nor do they directly spit out sentences. On the **input** side, text is chopped into **tokens**; on the **output** side, the model produces "the probability distribution of the next token across the entire vocabulary" in one step, and a **sampling strategy** picks one, appends it to the input, and generates autoregressively, one by one. These two ends are usually invisible, yet they determine three practical issues that constantly arise in development: **can you calculate costs accurately** (billing units are tokens), **can you tune "creativity" correctly** (sampling strategies), and **why do identical prompts yield different results** (non-determinism).
More importantly, the "common sense" regarding these two ends has diverged in 2026. **Open-source/local models** still rely on `temperature`/`top_p`/`top_k` to tune sampling (e.g., Qwen running locally); meanwhile, **Claude's hosted API has completely removed these parameters starting from Opus 4.7**—passing them results in a direct 400 error. It's two worlds under the same concept; this article explains the mechanisms of both ends and this divergence in depth.
## Input Side: How Text Becomes Tokens
### Subword Tokenization and BPE
Modern LLMs do not tokenize by character or word, but by **subword**. The most common algorithm is **BPE (Byte Pair Encoding)**: during training, it starts from characters and repeatedly merges the "most frequent adjacent token pair" into a new token until the vocabulary reaches a set size (usually tens of thousands to over a hundred thousand). The result is: high-frequency words become a single token, while low-frequency words are split into several subwords.
Modern implementations mostly use **byte-level BPE**: performing BPE on **bytes** rather than Unicode characters. The benefit is **no OOV (Out-Of-Vocabulary)**—any input (rare characters, emojis, binary strings) can fall back to bytes and will never fail to tokenize. The cost is that non-ASCII characters (Chinese, Japanese) often occupy multiple bytes and thus multiple tokens per character.
### Counter-intuitive but Critical Points
- **token ≠ word ≠ character.** In English, approximately 1 token ≈ 4 characters; **in Chinese, one character often takes 1–2+ tokens, and punctuation/indentation in code also consumes tokens.** Estimating length by character count will systematically underestimate.
- **Chat templates also consume tokens.** In multi-turn conversations, `role` markers, message separators, and tool definition schemas are all counted as input tokens—not just the body text costs money.
- **Tokens are not universal across models, and they drift with versions.** Claude's tokenizer ≠ OpenAI's; **do not use `tiktoken` to estimate Claude** (it undercounts by 15–20%, and the error is even worse for code/Chinese). Even within the same vendor, versions change: Opus 4.7's tokenizer might split the same text into 1.0–1.35 times the number of tokens of the previous generation. **When switching models, re-baseline; do not apply old multipliers.**
- **For accuracy, call the API; do not use third-party approximators:**
```python
# Single-segment counting
n = client.messages.count_tokens(
model="claude-opus-4-8",
messages=[{"role": "user", "content": text}],
).input_tokens
# Estimate cost (based on official pricing, which may change): input_tokens × unit price
# Compare diff between two versions: count separately, then subtract—the endpoint is stateless, so do the subtraction yourself
```
`count_tokens` is stateless; to calculate "how many tokens changed," count separately and subtract.
## Output Side: Sampling One Token at a Time
The model does not output words at each step, but a **score vector (logits) across the entire vocabulary**. These are converted into a probability distribution via softmax, then sampled according to a strategy, appended to the input, and the process repeats—autoregressively.
```mermaid
flowchart LR
T["Text"] --> TK["Tokenizer → tokens"]
TK --> M["Model (Transformer)"]
M --> L["logits (one score per token in vocabulary)"]
L --> SM["softmax(logits / T) → Probability Distribution"]
SM --> S["Sampling Strategy top_p / top_k / greedy"]
S --> N["Next Token"]
N -->|Append to input, Autoregressive| M
```
### Sampling Knobs (on models that still support them)
| Strategy | Mechanism | Effect |
|------|------|------|
| greedy (argmax) | Pick the highest probability token at each step | Most deterministic, but **prone to repetition/degeneration** |
| temperature `T` | Divide logits by `T` before softmax | `T>1` flattens distribution → more random; `T<1` sharpens → more focused; `T=0` degenerates to greedy |
| top_p (nucleus) | Sample only within the "smallest set of tokens whose cumulative probability reaches p" | **Dynamic** truncation of the long tail, adapting to distribution shape |
| top_k | Sample only within the top k highest probability tokens | **Fixed** truncation |
| repetition / presence penalty | Down-weight tokens that have already appeared | Suppress repetition |
| stop sequence | Stop when a specified string is matched | Control output boundaries |
In practice, the **temperature + top_p combination** is most common: temperature controls randomness, while nucleus sampling cuts off absurd long tails. `top_k` is coarser.
### Why Chat LLMs Don't Use Beam Search
Classic machine translation uses **beam search** (maintaining multiple candidate paths simultaneously to find the global optimum). Chat/open-generation models **almost never use it**: it tends to produce high-probability but boring, repetitive text ("neural text degeneration"), sacrificing diversity and naturalness. Sampling (top_p/temperature) instead generates text that is more human-like and informative—this was the core conclusion of the 2020 nucleus sampling paper.
### Why `T=0` Does Not Guarantee Byte-for-Byte Reproduction
Many people believe "temperature 0 = reproducible." **This is not true.** The reason lies in hardware and scheduling, not sampling:
- **Floating-point non-associativity:** The **order of accumulation** for massive parallel additions on GPUs is not fixed. `(a+b)+c ≠ a+(b+c)` holds true in floating-point arithmetic, and tiny differences can flip the choice at the argmax threshold.
- **Batch composition:** Which requests are batched together and how kernels are scheduled affect numerical results.
- **MoE routing:** [MoE models](/ai/01-models-and-context/03-model-architecture-dense-vs-moe.md) expert selection is sensitive to numerical variations.
"Determinism" is an engineering goal (fixed prompt, low randomness, accepting minor drift), not a guarantee provided by any single sampling parameter.
## Divergence: Claude Removed Sampling, Local Models Still Use It
Starting with Claude **Opus 4.7 / 4.8 / Fable 5**, `temperature`, `top_p`, and `top_k` were **removed**—passing them results in a direct **400** error. This is not a regression; it represents a shift in design philosophy:
- On strong reasoning models, using sampling temperature to "tune creativity/determinism" yields limited and uncontrollable benefits.
- Anthropic instead provides **two higher-level tools**:
- **Prompting**—if you want determinism, hardcode instructions; if you want divergence, explicitly request diversity in the prompt (e.g., in frontend design scenarios, ask the model to "provide 4 different directions first, then implement," see the design section in the migration docs);
- **Effort** (`output_config.effort`)—controls how deep to think and how many tokens to spend (see [Reasoning and Thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md)).
Migration key points: `temperature=0.7` in old code must be **removed** for new models. Translate intent into prompts + effort—`temperature=0` (seeking determinism) → `effort:"low"` + hardcoded instructions; `temperature=high` (seeking divergence) → explicitly state "provide multiple different directions" in the prompt. This migration recipe is detailed in [Prompt Engineering and Structured Output](/ai/03-applications-and-production/02-prompt-engineering-and-structured-output.md).
**But on the other end, sampling is still alive.** Local/open-source models (llama.cpp, vLLM) still accept these parameters—[Local LLM Deployment](/homelab/local-llm.md) sets them. Moreover, model cards usually provide recommended values: [Qwen-AgentWorld](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) officially recommends **temperature 0.6 / top_p 0.95 / top_k 20**. So "removing sampling" is a stance specific to Claude's hosted API, not an industry standard—use the rules of the world you are in.
## Best Practices
- **For accuracy, call `count_tokens`; do not use third-party approximators.** `tiktoken` undercounts Claude by 15–20%, and the error is even worse for Chinese/code; chat templates and tool schemas also consume tokens.
- **Re-baseline tokens when switching models.** Tokenizers drift with versions; do not apply old multipliers; to compare differences between versions, count separately and subtract.
- **Use the rules of the world you are in.** For local/open-source models, continue tuning `temperature`/`top_p`/`top_k` (according to model card recommendations); Claude's hosted API has removed sampling, so use prompts + effort instead.
- **Set sampling values for local models according to model cards.** For example, AgentWorld recommends 0.6 / 0.95 / 20; do not gamble on default values for divergence.
- **Do not expect `T=0` to guarantee byte-for-byte reproduction.** Floating-point accumulation, batching, and MoE routing all introduce jitter; to strictly reproduce, cache results.
## Trade-offs and Failure Modes
- **Estimating tokens by character count:** Severe underestimation for Chinese/code → use `count_tokens`, and include chat templates/tool schemas.
- **Applying token counts or old temperatures across models:** Tokenizers and sampling faces change → re-baseline when switching models, remove sampling parameters during migration.
- **Expecting `T=0` to reproduce:** Impossible to guarantee byte-for-byte consistency → fix prompt + low effort, accept minor drift; to strictly reproduce, cache results.
- **Not setting recommended sampling values for local models:** Using default temperature may be too divergent or too rigid → follow model cards (e.g., AgentWorld's 0.6/0.95/20).
## References
- **Anthropic Official Docs**: Token Counting, Models API, Migration Guide (sampling parameter removal) (platform.claude.com)
- **Papers**: "Neural Machine Translation of Rare Words with Subword Units" (BPE, Sennrich 2016), "The Curious Case of Neural Text Degeneration" (nucleus sampling, Holtzman 2020)
- **Model Cards**: [Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) (recommended sampling values)
*Keywords: tokenization, subword, BPE, byte-level BPE, vocabulary, OOV, chat template, token ≠ word, count_tokens, tiktoken, tokenizer drift, logits, softmax, autoregressive, greedy, temperature, top_p, nucleus sampling, top_k, repetition penalty, stop sequence, beam search, text degeneration, determinism, floating-point non-associativity, sampling parameter removal, effort*