Entropy and Information Measures

Entropy is not some metaphysical "disorder," but a countable number of bits—on average, how many yes/no questions are needed to clarify an uncertain event. Self-information, entropy, joint entropy, conditional entropy, and mutual information are five terms in the same language; all subsequent content (compression, channels, loss functions) grows out of these quantities.

Introduction: How Many Guesses Does It Take?

Let's play a game first to build intuition before discussing definitions.

Someone has an integer in mind between 1 and 8. You can only ask yes/no questions, such as "Is it greater than 4?" On average, what is the minimum number of questions needed to guarantee guessing the number?

The optimal strategy is to halve the remaining possibilities with each question. First ask "Is it greater than 4?" Regardless of the answer, the candidates drop from 8 to 4; then to 2, then to 1. After three cuts, the unique answer is revealed. \log_2 8 = 3, exactly three times.

This is no coincidence. A binary question with equal probabilities carries exactly 1 bit of information; and entropy provides the theoretical lower bound for the "average minimum number of questions." Exact hits are only possible when the probabilities perfectly fit a binary tree; for a single game, the optimal average number of questions lies between H(X) and H(X)+1, asymptotically approaching H(X) only when many games are bundled together.

Let's modify the game. Suppose the person chooses 7 with 80% probability (thinking this number is "spiritual"). In this case, sticking to halving is foolish—the smart move is to ask "Is it 7?" directly in the first question. Most of the time, it's guessed in one shot, and the average number of questions is far less than 3.

The conclusion is simple: the more uneven and structured the distribution, the fewer questions need to be asked on average. The goal of this entire article is to turn this intuition into formulas. Ultimately, information content, entropy, and the compression discussed in later chapters are all answering the same question—how many bits, on average, are needed to clarify an uncertain event?

Self-Information: How "Surprising" Is an Event?

Let's look at a single event first. Self-information generalizes the intuition from guessing numbers to arbitrary probabilities. If the probability of event x occurring is p(x), its self-information is defined as:

I(x) = -\log_2 p(x)

The unit is bit. It measures "how many yes/no questions are equivalent to confirming that this event occurred."

Let's verify this with the game: With 8 numbers uniformly distributed, the probability of each number is 1/8, so the self-information is -\log_2(1/8) = 3\ \text{bit}, which matches the three questions counted manually.

Here are a few examples to help you build a feel for it:

• Flipping a coin and getting heads, p=0.5, self-information -\log_2 0.5 = 1\ \text{bit}.
• Rolling a die and getting a specific face, p=1/6, self-information -\log_2(1/6) \approx 2.58\ \text{bit}.
• A certain event like "the sun will rise tomorrow," p \approx 1, self-information \approx 0\ \text{bit}. You already knew it; it brings no new information, so no questions need to be asked.

Why must we use logarithms? Because information should be "additive": if two independent events occur simultaneously, the total information should be the sum of their individual information. The joint probability of independent events is the product, p(x)p(y). Logarithms naturally translate multiplication into addition:

-\log(p(x)p(y)) = -\log p(x) - \log p(y)

This "additivity" is not just an aesthetic preference; it is the foundation of the entire entropy system.

Entropy: The Expectation of Self-Information

Now that we've clarified single events, let's look at the entire random variable. Entropy H(X) is the expectation of the self-information of all possible values of X—which serves as the theoretical benchmark for "how many questions to ask on average" in the number-guessing game:

H(X) = -\sum_x p(x) \log_2 p(x)

It also answers another question: what is the average code length achievable when representing X using an optimal code? For single-symbol prefix codes, it usually satisfies H(X)\leq L<H(X)+1; for block coding of long sequences, the code length per symbol can approach H(X) arbitrarily closely, as guaranteed by the Source Coding Theorem. For now, just remember: the higher the entropy, the higher the uncertainty, and the more questions (or bits) need to be asked on average.

Binary Entropy Function

The simplest case is when there are only two possible values, with probabilities p and 1-p:

H(p) = -p\log_2 p - (1-p)\log_2(1-p)

This curve is the best entry point for understanding entropy:

Read this curve using the number-guessing analogy: Entropy is maximized (1 bit) when p=0.5. The fairest coin is the hardest to predict; no matter how you ask, you'll need to ask on average once. As you deviate to either end, entropy decreases; at p=0 or p=1, the result is determined, entropy is 0—no questions need to be asked.

Three Properties of Entropy

These three properties will be used repeatedly later, and all can be verified using the intuition of guessing numbers:

• Non-negativity: H(X) \geq 0. Since 0 \leq p(x) \leq 1, each term -p\log p is non-negative. Equality holds only when X is completely determined (some p(x)=1)—there is no uncertainty in determined events.
• Upper Bound and Maximum Entropy: H(X) \leq \log_2|X|, where |X| is the number of possible values. Equality holds if and only if X is uniformly distributed. Uncertainty is maximized when all values are equally probable, corresponding to the number-guessing scenario where "there is no bias to exploit, so you must stick to halving." This also explains why the uniform distribution is the "most ignorant" prior: without additional constraints, it is the distribution with the highest entropy (Principle of Maximum Entropy).
• Chain Rule: H(X,Y) = H(X) + H(Y \mid X). This reads as: the total uncertainty of clarifying both X and Y simultaneously equals clarifying X first, plus "how much uncertainty remains for Y given X." It is the core tool for decomposing complex joint distributions into conditional terms, and it mirrors the autoregressive decomposition of LLMs: H(\text{token sequence}) = \sum_i H(\text{token}_i \mid \text{context}). Each generated token is asking "how much uncertainty remains for this next position."

Continuous Variables: Entropy Cannot Simply Follow Discrete Intuition

The H(X) above applies to discrete values. For continuous variables, we use differential entropy:

h(X)=-\int p(x)\log p(x)\,dx

It looks similar to discrete entropy, but behaves differently: differential entropy can be negative, changes with unit and coordinate scaling, and cannot be directly interpreted as "how many bits are needed to encode a precise real number"—precisely representing continuous values usually requires infinite bits. Therefore, when comparing continuous distributions, it is often preferable to use KL divergence and mutual information, which are more stable under coordinate transformations. Another common conclusion is: given a fixed variance, the Gaussian distribution has the maximum differential entropy. This is the key property behind the AWGN Channel Capacity formula.

Intuition: How Structure Suppresses Entropy

Here we need to clarify a common misconception: entropy measures the shape of the distribution, not the number of values.

The number of values only determines the "maximum number of questions needed" (upper bound). What truly determines the "average actual number of questions needed" is the shape of the distribution—how uniform is it? Are there dependencies between values? Three examples solidify this distinction:

• Fair Coin: 2 values, uniform, H = 1 bit. Reaches the upper bound; no shortcuts are available.
• Fair Die: 6 values, uniform, H = \log_2 6 \approx 2.58 bit.
• Natural Language Characters: 26 letters. If uniform, it should be \log_2 26 \approx 4.7 bit/character. However, English letter distribution is extremely uneven (e is very frequent, z is very rare), and there are strong dependencies between letters (q is almost always followed by u). Under specific alphabets and corpus definitions, Shannon's classic estimate for the entropy rate of English is about 1.0–1.3 bit/character, far lower than 4.7; this number varies depending on whether spaces, punctuation, corpus type, and estimation methods are included.

This last example deserves a moment's thought. Guessing a random English letter under the uniform assumption requires asking nearly 5 questions on average. But if you first ask high-information questions like "Is it a vowel?" or "Was the previous letter q?", most positions are determined without much guessing. This gap caused by "structure suppressing entropy"—the difference between 4.7 and 1.2—is exactly what compression algorithms and language models rely on. What they essentially do is extract all this redundancy.

Joint Entropy, Conditional Entropy, Mutual Information: Between Two Variables

So far, we've only discussed a single random variable. When there are two variables, the language of "how many questions to ask on average" naturally grows into an entire family. Three members, one by one:

• Joint Entropy H(X,Y) = -\sum p(x,y)\log_2 p(x,y): How many questions to ask on average to clarify both X and Y simultaneously.
• Conditional Entropy H(Y \mid X) = -\sum p(x,y)\log_2 p(y \mid x): Given that X is already known, how many more questions to ask on average to clarify Y. The two extremes are very illustrative—if Y is completely determined by X, H(Y \mid X)=0 (no extra questions needed); if they are independent, H(Y \mid X)=H(Y) (knowing X helps nothing, not a single question is saved).
• Mutual Information I(X;Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X): Knowing one variable allows you to ask how many fewer questions about the other variable.

Mutual information has several beautiful properties: it is symmetric (I(X;Y)=I(Y;X)), non-negative, and I(X;Y)=0 if and only if X and Y are independent. It also has an equivalent formulation—I(X;Y)=D_{KL}(p(x,y) \parallel p(x)p(y)), which is the KL divergence between the true joint distribution and the distribution that "pretends the two are independent." In other words, mutual information measures exactly "how far the two variables deviate from independence."

Conditional Mutual Information and the Data Processing Inequality

Sometimes we care not about "how much total information X and Y share," but how much extra they share after knowing Z. This is conditional mutual information:

I(X;Y\mid Z)=H(X\mid Z)-H(X\mid Y,Z)

It separates the "correlation coming from context Z" from the "remaining connection between X and Y." Note that: mutual information describes statistical dependence, not automatically causality. Two variables might just be jointly influenced by a third variable.

The most valuable property of mutual information to use repeatedly is the data processing inequality. If X\to Y\to Z forms a Markov chain, meaning Z can only access X through Y, then:

I(X;Z)\leq I(X;Y)

The intuition is: information can only be preserved or lost through processing, never created out of thin air. Compression, summarization, quantization, and neural network intermediate representations all obey this constraint. The subsequent Channel Capacity and Information Bottleneck are essentially studying "how to discard only the unnecessary information."

Venn Diagram: Set Relationships of Information Measures

The algebraic relationships between these four quantities can be intuitively remembered using two overlapping circles. This is the most classic diagram in information theory:

All identities can be read directly from the diagram. The union is H(X,Y), the left circle is H(X), the overlap is I(X;Y), and the left crescent is H(X \mid Y). Thus, identities like H(X) = H(X \mid Y) + I(X;Y) and H(X,Y) = H(X) + H(Y \mid X) are immediately clear.

A reminder: this analogy holds strictly only for two variables. With three or more variables, "interaction information" can take negative values, and the Venn diagram fails. However, the two-circle diagram is sufficient for building intuition.

These measures are the foundation for all subsequent content: compression approaches code length to H(X) (Source Coding), channel capacity is \max I(X;Y) (Channel Capacity), and deep learning loss functions are direct applications of Cross-Entropy and KL Divergence.

One Table to Tie the Whole Series Together

References

• Textbook: "Elements of Information Theory" (Cover & Thomas — the standard textbook for information theory, Chapter 2 Entropy, Relative Entropy, and Mutual Information)
• Paper: "A Mathematical Theory of Communication" (Shannon, 1948 — the foundational work on entropy and information theory, everything starts here)
• Textbook: "Information Theory, Inference, and Learning Algorithms" (MacKay — free online, excellent intuition in Chapters 2 and 4)

Keywords: self-information, entropy, differential entropy, joint entropy, conditional entropy, mutual information, conditional mutual information, data processing inequality, Markov chain, chain rule, maximum entropy, binary entropy function, bit