KL Divergence and Cross-Entropy

KL divergence is "how many extra bits you pay on average by using the wrong distribution to encode"; cross-entropy is "true entropy + this penalty". Minimizing cross-entropy loss is equivalent to maximum likelihood—this is the direct link from information theory to deep learning loss functions. When the target is multi-modal and the approximating model is constrained, forward/reverse KL often exhibit different tendencies toward "covering modes" versus "locking onto modes".

Prelude: Using the Wrong Codebook

The previous chapter stated that when data comes from distribution p, the entropy H(p) is the theoretical lower bound for the average code length; tailored block coding for p can asymptotically approach it.

But in reality, you often don’t have access to the true p. All you have is an approximation q—estimated by a model or assumed. So you can only use a "codebook optimized for q" to compress data that actually comes from p.

What happens when the codebook is wrong? Each message will, on average, cost a few extra bits. This extra cost is the protagonist of this chapter.

KL Divergence: The Cost of Using the Wrong Distribution

Relative entropy / KL divergence (Kullback-Leibler divergence) precisely quantifies this "extra cost in bits":

D_{KL}(p \parallel q) = \sum_x p(x) \log_2 \frac{p(x)}{q(x)}

Viewed through the lens of the previous chapter, it becomes crystal clear. With p's optimal coding, the average code length is H(p); switching to coding optimized for q changes the average code length to the cross-entropy H(p,q). The difference between the two is exactly the KL divergence:

D_{KL}(p \parallel q) = H(p,q) - H(p)

This difference is pure waste—the data hasn’t changed, the information content hasn’t changed; you just have to pay more because you chose the wrong codebook.

A key property: D_{KL}(p \parallel q) \geq 0 (Gibbs' inequality), with equality if and only if p=q. It resembles a "distance," measuring how dissimilar two distributions are. But it is not a true distance—the asymmetry discussed in the next section is the reason.

Asymmetry: D_{KL}(p \parallel q) \neq D_{KL}(q \parallel p)

This is the most easily overlooked yet consequential property of KL divergence. Swapping p and q changes both the value and the behavior.

The root cause lies in the summation weights. D_{KL}(p \parallel q) = \sum_x p(x) \log \frac{p(x)}{q(x)}, where each term is weighted by p(x); whereas in D_{KL}(q \parallel p), the weights are q(x). Whichever distribution serves as the weight determines where the penalty falls:

• D_{KL}(p \parallel q) penalizes "p has it but q doesn't." Wherever p(x)>0 but q(x)\to 0, \log(p/q)\to\infty, resulting in infinite cost. Thus, q cannot afford to leave any support area of p empty, forcing it to cover all of p's mass.
• D_{KL}(q \parallel p) penalizes "q has it but p doesn't." Wherever q(x)>0 but p(x)\to 0, the cost is infinite. Thus, q dares to stay only in the high-density regions of p, preferring to shrink into a single peak rather than cross boundaries.

Because of this asymmetry, the phrase "minimizing KL" is incomplete—you must specify which distribution comes first. This directly leads to the distinction between forward and reverse KL.

Let’s look at concrete numbers, instead of staying abstract. Take p=(0.5,\ 0.5) and q=(0.9,\ 0.1):

D_{KL}(p \parallel q) = 0.5\log_2\frac{0.5}{0.9} + 0.5\log_2\frac{0.5}{0.1} \approx 0.74\ \text{bit}

D_{KL}(q \parallel p) = 0.9\log_2\frac{0.9}{0.5} + 0.1\log_2\frac{0.1}{0.5} \approx 0.53\ \text{bit}

For the same pair of distributions, the two directions yield different results. This is not a rounding error; it is structural. Whichever distribution serves as the summation weight determines where the penalty falls.

Cross-Entropy and Maximum Likelihood: The Link to Deep Learning

Now, let’s decompose KL, and the line connecting to deep learning becomes visible.

Cross-entropy H(p,q) = -\sum_x p(x) \log_2 q(x), which by definition can be decomposed as:

H(p,q) = H(p) + D_{KL}(p \parallel q)

Place this in the context of supervised learning. Here, p is the true distribution of the data (given by the training set, so H(p) is a constant independent of the model), and q is the distribution predicted by the model with parameters \theta. Thus:

Minimizing cross-entropy loss = Minimizing D_{KL}(p \parallel q)—the constant term H(p) does not affect the gradient and is effectively discarded.

Take one more step. In classification tasks, p is usually one-hot (the true label has probability 1, others 0), causing the summation to collapse into a single term: H(p,q) = -\log q(\text{true class}). Summing over the entire dataset:

\sum_i -\log q(y_i \mid x_i) = -\log \prod_i q(y_i \mid x_i)

The right-hand side is exactly the negative log-likelihood. Thus, three things are nailed together:

Minimizing cross-entropy loss ⟺ Minimizing forward KL ⟺ Maximum Likelihood Estimation (MLE).

This is why almost all classification models and all LLM pre-training use cross-entropy loss. It is not an arbitrary engineering trick, but the precise information-theoretic expression of the goal "to make the model distribution as close to the data distribution as possible." In LLMs, each step predicts the distribution q of the next token and calculates the cross-entropy against the one-hot p of the true token, following exactly this logic (see Information Theory in LLMs).

Forward KL vs Reverse KL: Covering or Locking?

Asymmetry is not just theoretical; it affects what the fitted distribution looks like. Below, we discuss a typical scenario with clear premises: the target distribution p is multi-modal, while the adjustable approximating distribution family q has limited expressive power, such as being able to select only a single-modal Gaussian. Outside these conditions, "forward must cover, reverse must seek peaks" is not an unconditional theorem.

Set up a typical scenario: p is a fixed target, often bi-modal; q is the approximation we can tune, often constrained, e.g., it can only be a single-modal Gaussian. Whether you choose forward or reverse during optimization results in drastically different shapes for the fitted q:

• Forward KL D_{KL}(p \parallel q) (mode-covering): It penalizes q for leaving empty spaces where p has mass. Thus, q is forced to spread out and cover all modes of p, even if it means placing undesired mass in the low-density regions between the two peaks. This is the behavior of maximum likelihood / supervised learning—every pattern in the data wants to be accounted for.
• Reverse KL D_{KL}(q \parallel p) (mode-seeking): It penalizes q for venturing into the low-density regions of p. Thus, q locks onto a single mode and shrinks into a narrow distribution, actively ignoring other peaks. This is the default behavior of Variational Inference (VI) and the tendency in RLHF policy optimization—prefer to firmly bet on one high-reward mode rather than spread out.

In this constrained fitting scenario, you can remember it with one sentence: forward KL fears missing (so it tends to widen), reverse KL fears being wrong (so it tends to narrow). RLHF commonly uses D_{KL}(\pi_\theta\parallel\pi_\text{ref}) to tether the policy near the reference model; this direction is indeed important, but its primary role is to limit policy drift and should not be explained solely by "mode-seeking" intuition. For details, see Information Theory in LLMs.

Jensen-Shannon Divergence: The Symmetrization Patch

Sometimes KL's asymmetry is inconvenient—e.g., when you simply want a true "distribution distance."

Jensen-Shannon divergence (JS divergence) is its symmetrization scheme. First, take the midpoint distribution m = (p+q)/2, then take the average of the two KLs:

\text{JS}(p,q) = \frac{1}{2} D_{KL}(p \parallel m) + \frac{1}{2} D_{KL}(q \parallel m)

It is symmetric (\text{JS}(p,q)=\text{JS}(q,p)), bounded (with base 2, 0 \leq \text{JS} \leq 1 bit), and \sqrt{\text{JS}} is a valid metric. The original GAN discriminator objective essentially minimizes JS divergence.

However, when you need the physical meaning of "encoding cost" or need to correspond to MLE, people still use KL directly. Symmetry is not always a free benefit.

References

• Textbook: "Elements of Information Theory" (Cover & Thomas — Chapter 2 Relative Entropy, including the proof of Gibbs' inequality)
• Textbook: "Pattern Recognition and Machine Learning" (Bishop — Section 10.1 Variational Inference, classic diagrams of forward/reverse KL behavior)
• Paper: "Generative Adversarial Nets" (Goodfellow et al., 2014 — the connection between the discriminator objective and JS divergence)

Keywords: relative entropy, KL divergence, cross-entropy, asymmetry, maximum likelihood MLE, forward KL, reverse KL, mode-covering, mode-seeking, Jensen-Shannon divergence, Gibbs' inequality