--- title: Information Theory in LLMs url: https://doc.liz6.com/en/theory/01-information-theory/06-information-theory-in-llms locale: en area: theory tags: - theory - information-theory date: 2026-07-18 modified: 2026-07-29 description: 'Apply the tools from the first five chapters to LLMs: Perplexity is a human-readable wrapper for cross-entropy; pre-training uses cross-entropy loss; RLHF/DPO uses KL constraints to limit policy drift; sampling temperature modulates the entropy of the output distribution. The compression perspective also aids in understanding language modeling and scaling laws, but information content, capability, factuality, and usefulness are not the same thing.' --- # Information Theory in LLMs > Apply the tools from the first five chapters to LLMs: Perplexity is a human-readable wrapper for cross-entropy; pre-training uses cross-entropy loss; RLHF/DPO uses KL constraints to limit policy drift; sampling temperature modulates the entropy of the output distribution. The compression perspective also aids in understanding language modeling and scaling laws, but information content, capability, factuality, and usefulness are not the same thing. ## Introduction: The Same Tools, New Arena The quantities from the first five chapters—entropy, KL divergence, cross-entropy, entropy rate, and mutual information—may sound like relics of communications and compression. This chapter does one thing: it aims each of these tools at modern LLMs, and you will find that each occupies a critical position within them. Pre-training loss, evaluation metrics like perplexity, alignment constraints, and inference-time temperature knobs can all be explained using the quantities from the first five chapters. Information theory provides a powerful common language, but it is not a complete theory of model capabilities and behaviors. Below, we examine which relationships are strictly equivalent and which are merely heuristic interpretive perspectives. ## Cross-Entropy Loss: The Pre-training Objective Itself What is LLM pre-training doing? In one sentence: [Cross-entropy](/theory/01-information-theory/02-kl-divergence-and-cross-entropy.md) minimization. At each step, the model outputs a prediction distribution $q$ over the entire vocabulary for the next token. The training sample represents the observed token at this step as a **one-hot target** $p$ (probability $1$ for the true token, $0$ for all others). The loss for this step is the cross-entropy: $$H(p,q) = -\sum_x p(x) \log q(x) = -\log q(\text{true token})$$ Since $p$ is one-hot, the summation collapses to a single term—the loss is simply the "negative log probability assigned by the model to the true token." The loss for the entire sequence is the average of per-token cross-entropies. On the overall data distribution, minimizing expected cross-entropy is equivalent to minimizing $D_{KL}(p_\text{data}\parallel q)$; on a finite training set, minimizing per-sample NLL is **maximum likelihood estimation**. If the model's probabilities are combined with arithmetic coding / ANS, they can be converted into lossless compression code lengths close to NLL. Therefore, "training a language model" and "learning a good probabilistic compression model" are closely linked in terms of objective functions, but true compression systems also require entropy encoders and must account for model weight costs. As for how the vocabulary is segmented and what unit the one-hot encoding is based on, this depends on tokenization; see [Tokens and Sampling](/ai/01-models-and-context/02-tokens-and-sampling.md). ## Perplexity: A Human-Readable Wrapper for Cross-Entropy The unit of cross-entropy is bits (or nats), with small values that are not intuitive. Seeing a "loss of 2.1" offers little mental imagery. **Perplexity (PPL)** exponentiates it into a more intuitive quantity—"the average number of equally likely options the model is struggling to choose from": $$\text{PPL}=2^{H_2(p,q)}\quad\text{or}\quad \text{PPL}=e^{H_e(p,q)}$$ Here, $H_2$ uses $\log_2$, with units in bits; $H_e$ uses the natural logarithm, with units in nats. The numerical scales differ, so you cannot directly equate $\exp(H)$ and $2^H$ for the same $H$. How to interpret it? If the model, on average, "wavers between $N$ equally probable tokens" at each position, the perplexity is $N$. Perfect prediction (always determining the true value) yields PPL $=1$; guessing randomly over a vocabulary of size $V$ yields PPL $=V$. It is a one-to-one monotonic mapping with cross-entropy, so **reducing cross-entropy loss and reducing perplexity are the same thing**; PPL is simply a scale for human consumption. This also explains why PPL is the most classic intrinsic evaluation metric for language models: it is the reading of the model's ability to compress data (entropy rate estimation). Prediction Distribution → Cross-Entropy → Perplexity Model Output q Distribution over vocabulary True Value p one-hot H(p,q) = −log q(true) PPL = 2^H₂ or e^Hₑ Loss↓ ⟺ Perplexity↓ ⟺ Compression Ratio↑ ⟺ Closer to Data Entropy Rate ## The Role of KL Divergence in RLHF / DPO After pre-training comes the alignment phase. Here, [KL divergence](/theory/01-information-theory/02-kl-divergence-and-cross-entropy.md) changes its identity—from "loss" to "rein." **The PPO objective in RLHF**, in addition to maximizing the reward given by the reward model, adds a KL penalty term: $$\text{Objective} = \mathbb E[\text{reward}(x,y)] - \beta \cdot D_{KL}(\pi_\theta(\cdot \mid x) \parallel \pi_\text{ref}(\cdot \mid x))$$ Here, $\pi_\theta$ is the policy being trained, and $\pi_\text{ref}$ is the frozen reference model (usually the initial model after SFT). This KL term penalizes the policy for deviating too far from the reference. What does it prevent? It prevents the model from **reward hacking** (exploiting loopholes in the reward model) to inflate rewards, which would cause it to degenerate into gibberish or repetition. Note the direction: it is often written as $D_{KL}(\pi_\theta\parallel\pi_\text{ref})$. It directly penalizes the new policy for placing too much mass in low-probability regions of the reference model, primarily serving to limit policy drift. The mode-seeking intuition from Chapter 2 is helpful in constrained multi-modal fitting, but one should not attribute all behaviors of RLHF solely to "peak seeking." The larger $\beta$, the tighter the rein. **DPO (Direct Preference Optimization)** is more sophisticated. It mathematically proves that the optimal solution for RLHF with KL constraints has a closed-form solution. Thus, DPO **skips the explicit reward model and PPO**, training directly on preference data using a classification-style loss. Where did the KL constraint go? It wasn't lost—the $\beta \cdot D_{KL}(\pi_\theta \parallel \pi_\text{ref})$ regularization was **implicitly** folded into DPO's loss function. In other words, DPO didn't abandon the rein; it just wove it into the objective, eliminating the need for online sampling and explicit penalty terms. Two paths, the same rein. ## Scaling Law: A Compression Perspective The original analysis of the **Chinchilla scaling law** points out that within its model family, data, and fitting range, given a compute budget $C$, the model parameter count $N$ and training token count $D$ should scale roughly as $N \propto C^{0.5}$ and $D \propto C^{0.5}$ synchronously, which is closer to computational optimality than simply stacking parameters. The specific exponents are results of empirical fitting, not constants that remain unchanged across architectures and data. From an information-theoretic perspective, this empirical regularity can be understood through a **compression perspective**, although it is not a direct conclusion derived from the source coding theorem. The model's test loss is its cross-entropy given the data, tokenizer, and context settings, which also corresponds to the ideal average code length if the model were connected to an entropy encoder. Larger models and more data usually allow for better prediction of the data distribution, so the decrease in loss can be interpreted as "improved compression capability." Under idealized conditions—fixed test distribution, sufficient context, and a model family capable of approximating the true distribution—the lower bound of optimal cross-entropy corresponds to the conditional entropy rate of the data. The power-law shapes and specific exponents in reality remain empirical regularities, influenced by architecture, optimization, data quality, and evaluation metrics. Thus, the question of "optimal scaling" translates to: **Given compute, how do we minimize this entropy rate estimate?** Chinchilla's answer is that parameters and data must be balanced—this is consistent with the spirit of the source coding theorem that "code length approaches entropy." How model architecture affects this curve (e.g., [Dense vs. MoE](/ai/01-models-and-context/03-model-architecture-dense-vs-moe.md) parameter efficiency under equivalent compute) is another slice of the same picture. ## Sampling Temperature: The Entropy Knob for Output Distribution The **sampling temperature** $T$ during inference, from an information-theoretic perspective, is a knob that directly manipulates the [entropy](/theory/01-information-theory/01-entropy-and-information-measures.md) of the output distribution. The method involves dividing the logits by $T$ before softmax: - $T > 1$: The distribution is **flattened**, entropy **increases**. More tokens have significant probabilities, making the output more random and divergent. - $T < 1$: The distribution is **sharpened**, entropy **decreases**. Probability concentrates on a few high-scoring tokens, making the output more deterministic and conservative. - $T \to 0$: If the highest logit is unique, the distribution concentrates on the argmax, and entropy approaches $0$, degenerating into greedy decoding; if the highest values are tied, the limit may still retain uncertainty among the tied items. So, adjusting temperature literally means adjusting "how much uncertainty is carried by the output at each step." This information-theoretic explanation supplements [Tokens and Sampling](/ai/01-models-and-context/02-tokens-and-sampling.md): that article discusses how top_p / top_k truncate the candidate set and why the Claude hosted API removed these knobs; here we only add the sentence "what exactly does temperature do information-theoretically?"—it acts on the entropy of the output distribution. The two are complementary, not repetitive. | Information Theory Tool | Manifestation in LLMs | Source Chapter | |------------|----------------|---------| | Cross-Entropy $H(p,q)$ | Pre-training Loss | [Chapter 2](/theory/01-information-theory/02-kl-divergence-and-cross-entropy.md) | | $2^{H_2}$ or $e^{H_e}$ | Perplexity PPL | This Chapter | | KL Divergence $D_{KL}$ | Policy Regularization in RLHF/DPO | [Chapter 2](/theory/01-information-theory/02-kl-divergence-and-cross-entropy.md) | | Entropy Rate | Lower Bound for Ideal Language Modeling Compression; A Perspective for Understanding Scaling Laws | [Chapter 3](/theory/01-information-theory/03-source-coding-and-compression.md) | | Distribution Entropy | Sampling Temperature | [Chapter 1](/theory/01-information-theory/01-entropy-and-information-measures.md) | ## Counter-Intuitive but Critical Points - **Perplexity cannot be compared across different tokenizers.** PPL is a "per token" quantity, and different models [segment tokens at different granularities](/ai/01-models-and-context/02-tokens-and-sampling.md)—character-level, subword-level, and word-level models have PPL values that cannot be directly compared. They must be converted to bits per character or bits per byte for a fair comparison. Drawing conclusions based solely on two models' PPLs often amounts to comparing different units. - **Low perplexity does not equate to factual correctness or useful answers.** Cross-entropy rewards "assigning higher probability to tokens that actually appear in the data." A model can be very good at imitating common text yet still produce factual errors; conversely, rare but correct expressions may receive lower probabilities. PPL is a distribution prediction metric, not a comprehensive capability score. - **Information content does not equal value.** High entropy only indicates that the result is more uncertain, not that the content is more creative or of higher quality; high mutual information only indicates stronger statistical dependence, not the existence of causality. Information theory answers "how much predictable structure there is," while tasks and human preferences answer "whether this structure is useful." - **Compression capability is related to understanding capability, but they are not synonyms.** NLL does correspond to ideal code length, indicating that prediction and compression share an objective; however, reasoning, tool use, long-horizon planning, and alignment are also influenced by architecture, training processes, and inference mechanisms. They cannot be deduced solely from compression ratios. - **Ideal cross-entropy has a floor that cannot be beaten, but the real-world floor is not easily visible.** Under ideal settings with fixed distribution, units, and conditional information, the conditional entropy rate of the data forms a lower bound; real-world evaluations also mix in limited context, distribution drift, model capabilities, and data noise. One cannot directly declare the constant term of a fitted scaling law as the "true entropy rate." - **The $\beta$ in KL penalty is a double-edged sword.** If $\beta$ is too small, the policy may run away due to reward hacking; if too large, it is shackled by the reference model and fails to learn new things. It is not an optional regularization term but a key knob for alignment quality. - **Temperature does not modify the knowledge in the model parameters; it only rearranges the existing output distribution.** Temperature is post-processing applied to logits during inference; it does not move weights but only changes the relative concentration of candidate tokens. It may change whether a specific answer happens to be correct, but it cannot fundamentally fix missing knowledge or incorrect modeling. ## Convergence: One Language, One Network Looking back at the entire line. An LLM, from pre-training (minimizing cross-entropy), to alignment (KL constraints on policy drift), to scaling (loss reduction can be understood from a compression perspective), to inference (temperature modulating output entropy), each link can gain a clear explanation through information theory. Information theory is worth studying systematically because it is a common language for understanding modern LLMs. Holding tools like [Entropy](/theory/01-information-theory/01-entropy-and-information-measures.md), [KL / Cross-Entropy](/theory/01-information-theory/02-kl-divergence-and-cross-entropy.md), [Compression](/theory/01-information-theory/03-source-coding-and-compression.md), [Channel](/theory/01-information-theory/04-channel-capacity-and-coding.md), and [Information Bottleneck](/theory/01-information-theory/05-rate-distortion-and-information-bottleneck.md) in hand allows for more accurate reading of loss curves, alignment recipes, and sampling parameters; but when encountering issues of factuality, reasoning, and human preferences, one must introduce models beyond information theory. ## References - **Paper**: "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022 — Chinchilla scaling law) - **Paper**: "Direct Preference Optimization" (Rafailov et al., 2023 — How DPO implicitly encodes KL constraints) - **Paper**: "Language Modeling Is Compression" (Delétang et al., 2023 — Empirical evidence that LLM loss is bit rate) - **Documentation**: [Tokens and Sampling](/ai/01-models-and-context/02-tokens-and-sampling.md), [Reasoning and Thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md) (existing sampling and reasoning mechanisms in this repository) *Keywords: perplexity, cross-entropy loss, pre-training objective, one-hot, RLHF, PPO, KL penalty, DPO, reward hacking, scaling law, Chinchilla, entropy rate, compression perspective, sampling temperature, output entropy, information content vs. usefulness*