Information Theory in LLMs

Aim all the tools from the first five chapters at LLMs: Perplexity is a human-readable wrapper for cross-entropy, the pre-training objective is cross-entropy loss itself, RLHF/DPO uses KL penalties to leash the policy, scaling laws are curves of entropy rate convergence, and sampling temperature is directly the entropy knob of the output distribution. Information theory is not the backdrop of LLMs; it is their skeleton.

Introduction: The Same Tools, A New Arena

The quantities from the first five chapters—entropy, KL divergence, cross-entropy, entropy rate, mutual information—sound like relics of communications and compression. This chapter does one thing: it aims each of these tools at modern LLMs one by one. You will find that each occupies a critical position within them.

Pre-training loss, perplexity for evaluation, constraints during alignment, the shape of scaling laws, and the temperature knob during inference—none of these are new things; they are all manifestations of those same quantities from the first five chapters in different contexts. Information theory is not the backdrop of LLMs; it is the skeleton itself. Let's look at them one by one.

Cross-Entropy Loss: The Pre-training Objective Itself

What is LLM pre-training doing? In one sentence: Cross-entropy minimization.

At each step, the model provides a prediction distribution q over the entire vocabulary for the next token. The true next token is a one-hot distribution p (probability 1 for the true token, 0 for all others). The loss for this step is cross-entropy:

H(p,q) = -\sum_x p(x) \log q(x) = -\log q(\text{true token})

Since p is one-hot, the summation collapses to a single term—the loss is simply the "negative log probability assigned by the model to the true token." The loss for the entire sequence is the average of per-token cross-entropies.

As proven in Chapters 2 and 3, minimizing cross-entropy = minimizing D_{KL}(p \parallel q) = maximum likelihood, which is also equivalent to the bit rate for lossless compression using the model. Therefore, "training an LLM" and "training an extreme compressor" are two ways of saying the same thing. How the vocabulary is split and what unit the one-hot distribution is based on depends on tokenization; see Tokens and Sampling.

Perplexity: A Human-Readable Wrapper for Cross-Entropy

The unit of cross-entropy is bits (or nats), with small values that are not intuitive. Seeing a "loss of 2.1" is hard to visualize.

Perplexity (PPL) exponentiates it into a more visual quantity—"how many equally likely options are the model struggling between on average":

\text{PPL} = \exp(\text{cross-entropy}) = 2^{H(p,q)}

(Here the base is 2, consistent with the bit scale used in previous chapters; using base e yields \exp.)

How to interpret it? If the model, on average, "swings between N equally probable tokens" at each position, the perplexity is N. Perfect prediction (always identifying the true value) yields PPL =1; guessing randomly over a vocabulary of size V yields PPL =V.

It is a one-to-one monotonic mapping with cross-entropy, so reducing cross-entropy loss and reducing perplexity are the same thing; PPL is just a scale for human consumption. This also explains why PPL is the most classic intrinsic evaluation metric for language models: it is the reading of the model's ability to compress data (entropy rate estimation).

The Role of KL Divergence in RLHF / DPO

After pre-training comes the alignment phase. Here, KL divergence changes its identity—from "loss" to "rein."

The PPO objective in RLHF, in addition to maximizing the reward given by the reward model, adds a KL penalty term:

\text{Objective} = \mathbb E[\text{reward}(x,y)] - \beta \cdot D_{KL}(\pi_\theta(\cdot \mid x) \parallel \pi_\text{ref}(\cdot \mid x))

Here \pi_\theta is the policy being trained, and \pi_\text{ref} is the frozen reference model (usually the initial model after SFT). This KL term penalizes the policy for deviating too far from the reference. What does it prevent? It prevents the model from reward hacking—exploiting loopholes in the reward model to inflate scores—thereby degenerating into gibberish or repetition.

Note the direction: here we use the reverse KL divergence, echoing the mode-seeking behavior discussed in Chapter 2—the policy is allowed to seek high-reward modes near the reference model but is not permitted to stray too far. The larger \beta is, the tighter the rein.

DPO (Direct Preference Optimization) is more sophisticated. It mathematically proves that the optimal solution for RLHF with KL constraints has a closed-form solution. Thus, DPO skips the explicit reward model and PPO, directly training on preference data using a classification-style loss.

Where did the KL constraint go? It wasn't lost—that \beta \cdot D_{KL}(\pi_\theta \parallel \pi_\text{ref}) regularization was implicitly folded into DPO's loss function. In other words, DPO didn't abandon the rein; it just wove it into the objective, eliminating the need for online sampling and explicit penalty terms. Two paths, the same rein.

Scaling Laws: A Compression Perspective

The Chinchilla scaling law states: given a compute budget C, the model parameter count N and the number of training tokens D should scale roughly as N \propto C^{0.5} and D \propto C^{0.5} simultaneously to minimize loss—rather than simply stacking parameters.

From an information theory perspective, this law is a natural conclusion of the compression perspective.

The model's test loss is its cross-entropy with respect to the data, which is an estimate of the entropy rate. Therefore, the loss-compute curve is essentially a curve of entropy rate estimation converging as parameters/data increase: the larger the model and the more data it sees, the closer its compression of the true data distribution gets to the entropy rate lower bound. The loss approaches an irreducible loss (the entropy rate of the data itself, which no model, no matter how powerful, can compress below) along a power law.

Thus, the question of "optimal scaling" translates to: Given compute, how do we minimize this entropy rate estimate? Chinchilla's answer is that parameters and data must be balanced—this aligns with the spirit of the source coding theorem, where code length approaches entropy. How model architecture affects this curve (e.g., Dense vs. MoE parameter efficiency under equivalent compute) is another slice of the same picture.

Sampling Temperature: The Entropy Knob of the Output Distribution

The sampling temperature (temperature) T during inference is, from an information theory perspective, a knob that directly manipulates the entropy of the output distribution. The method involves dividing the logits by T before softmax:

• T > 1: The distribution is flattened, entropy increases. More tokens gain significant probability, making the output more random and divergent.
• T < 1: The distribution is sharpened, entropy decreases. Probability concentrates on a few high-scoring tokens, making the output more deterministic and conservative.
• T \to 0: Entropy \to 0, degenerating into argmax (greedy decoding), with almost no randomness.

So, adjusting temperature literally means adjusting "how much uncertainty each output step carries."

This information-theoretic explanation supplements Tokens and Sampling: that article discusses how top_p / top_k truncate the candidate set and why the Claude hosted API removed these knobs; here we just add one sentence: "what exactly does temperature change from an information theory standpoint?"—it changes the entropy of the output distribution. The two are complementary and not repetitive.

Counter-Intuitive but Critical Points

• Perplexity cannot be compared across tokenizers. PPL is a "per token" quantity, and different models slice tokens at different granularities—character-level, subword-level, and word-level models have PPL values that cannot be directly compared. You must convert to bits per character or bits per byte for a fair comparison. Drawing conclusions based solely on two models' PPL often means comparing different units.
• Cross-entropy loss has an irreducible floor. The data itself has an entropy rate; no matter how powerful the model, it cannot go below this (irreducible loss). Even if the loss curve appears to be still decreasing, it is approaching this physical lower bound, not 0. Treating "whether loss can still decrease" as "whether the model can still improve" leads to misjudging the ceiling.
• The \beta in KL penalty is a double-edged sword. If \beta is too small, the policy may reward hack and fly off course; if too large, it is tethered too tightly to the reference model and fails to learn anything new. It is not an optional regularization term but a key knob for alignment quality.
• Temperature does not change what the model knows, only what it dares to say. Temperature is post-processing applied to logits during inference; it does not move weights or change the model's modeling of the world, only adjusting entropy within the existing distribution. Expecting to correct factual errors by tuning temperature is taking the wrong approach.

Convergence: One Language, One Network

Looking back at the entire thread. An LLM, from pre-training (minimizing cross-entropy = extreme compression), to alignment (KL constraints leashing the policy), to scaling (entropy rate converging along scaling laws), to inference (temperature regulating output entropy)—every link is a direct application of information theory quantities.

Information theory is worth studying systematically, not because it is elegant, but because it is the common language for understanding modern LLMs. Holding these tools—Entropy, KL / Cross-Entropy, Compression, Channel, Information Bottleneck—in hand, and looking back at the model's loss curves, alignment recipes, and sampling parameters, they are all manifestations of the same things in different positions.

References

• Paper: "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022 — Chinchilla scaling law)
• Paper: "Direct Preference Optimization" (Rafailov et al., 2023 — How DPO implicitly encodes KL constraints)
• Paper: "Language Modeling Is Compression" (Delétang et al., 2023 — Empirical evidence that LLM loss is bit rate)
• Docs: Tokens and Sampling, Inference and Thinking (existing sampling and inference mechanisms in this repository)

Keywords: perplexity, cross-entropy loss, pre-training objective, one-hot, RLHF, PPO, KL penalty, DPO, reward hacking, scaling law, Chinchilla, entropy rate, irreducible loss, sampling temperature, output entropy