KL Divergence and Cross-Entropy

KL divergence is "how many extra bits you pay on average by using the wrong distribution for encoding"; cross-entropy is "true entropy + this penalty". Minimizing cross-entropy loss is equivalent to maximum likelihood—this is the direct link from information theory to deep learning loss functions. The asymmetry of forward/reverse KL determines whether the model "covers all modes" or "locks onto a single mode."

Prelude: Using the Wrong Codebook

In the previous chapter, we said that when data comes from distribution p, the optimal code tailored for it has an average codeword length equal to the entropy H(p).

But in reality, you often don't have access to the true p. What you have is an approximation q—estimated by a model or assumed. So you can only use a codebook optimized for q to compress data that truly comes from p.

What happens when the codebook is wrong? On average, each message will cost a few extra bits. This extra cost is the protagonist of this chapter.

KL Divergence: The Cost of Using the Wrong Distribution

Relative entropy / KL divergence (Kullback-Leibler divergence) precisely measures this "extra cost":

D_{KL}(p \parallel q) = \sum_x p(x) \log_2 \frac{p(x)}{q(x)}

Breaking it down using the language from the previous chapter makes it crystal clear. With the optimal code for p, the average codeword length is H(p); switching to a code optimized for q changes the average length to the cross-entropy H(p,q). The difference between the two is exactly the KL divergence:

D_{KL}(p \parallel q) = H(p,q) - H(p)

This difference is pure waste—the data hasn't changed, the information content hasn't changed; you just have to pay extra because you chose the wrong codebook.

A key property: D_{KL}(p \parallel q) \geq 0 (Gibbs' inequality), with equality if and only if p=q. It resembles a "distance," measuring how dissimilar two distributions are. But it is not a true distance—the asymmetry discussed in the next section is the reason.

Asymmetry: D_{KL}(p \parallel q) \neq D_{KL}(q \parallel p)

This is the most easily overlooked yet consequential property of KL divergence. Swapping p and q changes both the value and the behavior.

The root cause lies in the summation weights. For D_{KL}(p \parallel q) = \sum_x p(x) \log \frac{p(x)}{q(x)}, the weight of each term is p(x); for D_{KL}(q \parallel p), the weight is q(x). Whichever distribution serves as the weight determines where the penalty falls:

• D_{KL}(p \parallel q) penalizes "p has mass but q does not." Wherever p(x)>0 but q(x)\to 0, \log(p/q)\to\infty, resulting in infinite cost. Thus, q cannot afford to leave any support area of p empty, forcing it to cover all the mass of p.
• D_{KL}(q \parallel p) penalizes "q has mass but p does not." Wherever q(x)>0 but p(x)\to 0, the cost is infinite. Thus, q dares to stay only in the high-density regions of p, preferring to shrink into a single peak rather than venture out of bounds.

Because of this asymmetry, the phrase "minimizing KL" is incomplete—you must specify which distribution is in the numerator. This directly leads to the distinction between forward and reverse KL.

Let's look at concrete numbers, instead of staying abstract. Take p=(0.5,\ 0.5) and q=(0.9,\ 0.1):

D_{KL}(p \parallel q) = 0.5\log_2\frac{0.5}{0.9} + 0.5\log_2\frac{0.5}{0.1} \approx 0.74\ \text{bit}

D_{KL}(q \parallel p) = 0.9\log_2\frac{0.9}{0.5} + 0.1\log_2\frac{0.1}{0.5} \approx 0.53\ \text{bit}

For the same pair of distributions, the two directions yield different results. This is not a rounding error; it is structural. Whoever serves as the summation weight determines where the penalty falls.

Cross-Entropy and Maximum Likelihood: The Link to Deep Learning

Now, let's decompose KL, and the line connecting to deep learning becomes visible.

Cross-entropy H(p,q) = -\sum_x p(x) \log_2 q(x), by definition, can be split as:

H(p,q) = H(p) + D_{KL}(p \parallel q)

Placing this in the context of supervised learning: here p is the true distribution of the data (given by the training set, so H(p) is a constant independent of the model), and q is the distribution predicted by the model parameters \theta. Thus:

Minimizing cross-entropy loss = Minimizing D_{KL}(p \parallel q)—the constant term H(p) does not affect the gradient and is effectively discarded.

One step further. In classification tasks, p is usually one-hot (probability of the true label is 1, others are 0), so the summation collapses to a single term: H(p,q) = -\log q(\text{true class}). Summing over the entire dataset:

\sum_i -\log q(y_i \mid x_i) = -\log \prod_i q(y_i \mid x_i)

The right-hand side is exactly the negative log-likelihood. Thus, three things are nailed together:

Minimizing cross-entropy loss ⟺ Minimizing forward KL ⟺ Maximum Likelihood Estimation (MLE).

This is why almost all classification models and LLM pre-training use cross-entropy loss. It is not an arbitrary engineering trick, but the precise information-theoretic expression of the goal "to make the model distribution as close to the data distribution as possible." In LLMs, each step predicts the distribution q of the next token, and calculates the cross-entropy against the one-hot p of the true token, following exactly this logic (see Information Theory in LLMs).

Forward KL vs Reverse KL: Covering or Locking?

Asymmetry is not just theoretical; it determines what the fitted distribution looks like.

Consider a typical scenario: p is a fixed target, often bimodal; q is the approximation we can tune, often constrained, e.g., to be a unimodal Gaussian. Whether you optimize using forward or reverse KL results in drastically different shapes for the fitted q:

• Forward KL D_{KL}(p \parallel q) (mode-covering): It penalizes q for leaving empty spaces where p has mass. Thus, q is forced to spread out and cover all modes of p, even if it means placing unwanted mass in the low-density region between the two peaks. This is the behavior of maximum likelihood / supervised learning—wanting to account for every pattern in the data.
• Reverse KL D_{KL}(q \parallel p) (mode-seeking): It penalizes q for venturing into the low-density regions of p. Thus, q locks onto a single mode and shrinks into a narrow distribution, actively ignoring other peaks. This is the default behavior of Variational Inference (VI) and the tendency in RLHF policy optimization—preferring to firmly bet on a single high-reward mode rather than spreading out.

Remember in one sentence: forward KL fears missing (so it gets wider), reverse KL fears being wrong (so it gets narrower). RLHF uses reverse KL constraints to keep the policy close to the reference model; details can be found in Information Theory in LLMs.

Jensen-Shannon Divergence: The Symmetrizing Patch

Sometimes the asymmetry of KL is inconvenient—e.g., when you simply want a true "distance between distributions."

Jensen-Shannon divergence (JS divergence) is its symmetrizing solution. First, take the midpoint distribution m = (p+q)/2, then take the average of the two KL divergences:

\text{JS}(p,q) = \frac{1}{2} D_{KL}(p \parallel m) + \frac{1}{2} D_{KL}(q \parallel m)

It is symmetric (\text{JS}(p,q)=\text{JS}(q,p)), bounded (0 \leq \text{JS} \leq 1 bit when base 2 is used), and \sqrt{\text{JS}} is a valid metric. The discriminator objective in the original GAN is essentially minimizing the JS divergence.

However: when you need the physical meaning of "encoding cost" or need to correspond to MLE, people still use KL directly. Symmetry is not always a free benefit.

References

• Textbook: "Elements of Information Theory" (Cover & Thomas — Chapter 2 Relative Entropy, including the proof of Gibbs' inequality)
• Textbook: "Pattern Recognition and Machine Learning" (Bishop — Section 10.1 Variational Inference, classic illustrations of forward/reverse KL behavior)
• Paper: "Generative Adversarial Nets" (Goodfellow et al., 2014 — the connection between the discriminator objective and JS divergence)

Keywords: relative entropy, KL divergence, cross-entropy, asymmetry, maximum likelihood MLE, forward KL, reverse KL, mode-covering, mode-seeking, Jensen-Shannon divergence, Gibbs' inequality