Entropy and Information Measures

Entropy is not some metaphysical "disorder," but a countable number of bits—asking how many yes/no questions it takes, on average, to clarify an uncertain event. Self-information, entropy, joint entropy, conditional entropy, and mutual information are five words in the same language; all subsequent content (compression, channels, loss functions) grows out of these quantities.

Prelude: How Many Guesses Does It Take?

Let's play a game first to build intuition before discussing definitions.

Someone has an integer in mind between 1 and 8. You can only ask yes/no questions, such as "Is it greater than 4?" On average, how many questions do you need to ask at minimum to guarantee guessing the number?

The optimal strategy is to halve the remaining possibilities every time. First, ask "Is it greater than 4?" Regardless of the answer, the candidates drop from 8 to 4; then to 2, then to 1. After three cuts, the unique answer is revealed. \log_2 8 = 3, exactly three times.

This is no coincidence. The answer to one yes/no question carries exactly 1 bit of information; and "the average minimum number of questions to ask" is precisely the question entropy aims to answer rigorously.

Let's tweak the game. Suppose the person chooses 7 with an 80% probability (thinking this number is "spiritual"). In this case, sticking to halving is foolish—the smart move is to ask "Is it 7?" right away. Most of the time, you hit the mark in one go, and the average number of questions is far less than 3.

The conclusion is simple: the more uneven and structured the distribution, the fewer questions you need to ask on average. The goal of this entire article is to turn this intuition into formulas. Information content, entropy, and the compression discussed in later chapters all boil down to answering the same question—how many bits, on average, are needed to clarify an uncertain event?

Self-Information: How "Surprising" Is an Event?

Let's start with a single event. Self-information generalizes the intuition from guessing numbers to arbitrary probabilities. If the probability of event x occurring is p(x), its self-information is defined as:

I(x) = -\log_2 p(x)

The unit is bit. It measures "how many yes/no questions it is equivalent to asking to confirm that this event occurred."

Let's verify with the game: With 8 numbers uniformly distributed, the probability of each number is 1/8, and the self-information is -\log_2(1/8) = 3\ \text{bit}, which matches perfectly with the three questions counted manually.

A few examples to help you build a feel for it:

• Flipping a coin and getting heads, p=0.5, self-information -\log_2 0.5 = 1\ \text{bit}.
• Rolling a die and getting a specific face, p=1/6, self-information -\log_2(1/6) \approx 2.58\ \text{bit}.
• A certain event like "the sun will rise tomorrow," p \approx 1, self-information \approx 0\ \text{bit}. You already knew it; it brings no new information, so no questions need to be asked.

Why must we use logarithms? Because information should be "additive": if two independent events occur simultaneously, the total information should be the sum of their individual information. The joint probability of independent events is the product, p(x)p(y). Logarithms naturally translate multiplication into addition:

-\log(p(x)p(y)) = -\log p(x) - \log p(y)

This "additivity" is not merely an aesthetic preference; it is the foundation of the entire entropy framework.

Entropy: The Expectation of Self-Information

Having clarified single events, let's look at the entire random variable. Entropy H(X) is the expectation of the self-information of all possible values of X—which is the precise formula for "how many questions to ask on average" in the number-guessing game:

H(X) = -\sum_x p(x) \log_2 p(x)

It also answers another question: how many bits are needed, on average, to represent X using an optimal code. The equivalence of these two meanings is guaranteed by the Source Coding Theorem. For now, just remember: the higher the entropy, the higher the uncertainty, and the more questions (or bits) you need to ask on average.

Binary Entropy Function

The simplest case is when there are only two possible values, with probabilities p and 1-p:

H(p) = -p\log_2 p - (1-p)\log_2(1-p)

This curve is the best entry point for understanding entropy:

Reading this curve in terms of the number-guessing game: entropy is maximized (1 bit) when p=0.5. The fairest coin is the hardest to predict; no matter how you ask, you will need to ask the full question on average. As you bias towards either end, entropy decreases; at p=0 or p=1, the outcome is determined, entropy is 0—no questions need to be asked.

Three Properties of Entropy

These three properties will be used repeatedly later, and all can be verified using the intuition of guessing numbers:

• Non-negativity: H(X) \geq 0. Since 0 \leq p(x) \leq 1, each term -p\log p is non-negative. Equality holds only when X is completely determined (some p(x)=1)—there is no uncertainty in certain events.
• Upper Bound and Maximum Entropy: H(X) \leq \log_2|X|, where |X| is the number of possible values. Equality holds if and only if X is uniformly distributed. Uncertainty is maximized when all values are equally probable, corresponding to the number-guessing scenario where "there is no bias to exploit, so you must strictly halve the possibilities." This also explains why the uniform distribution is the "most ignorant" prior: under no additional constraints, it is the distribution with the highest entropy (Principle of Maximum Entropy).
• Chain Rule: H(X,Y) = H(X) + H(Y \mid X). This reads as: the total uncertainty of clarifying both X and Y simultaneously equals clarifying X first, plus "how much uncertainty remains for Y given X." It is the core tool for decomposing complex joint distributions into conditional terms, and it mirrors the autoregressive decomposition of LLMs: H(\text{token sequence}) = \sum_i H(\text{token}_i \mid \text{context}). Each generated token asks, "how much uncertainty remains for this next position?"

Intuition: How Structure Suppresses Entropy

Here, we need to clarify a common misconception: entropy measures the shape of the distribution, not the number of possible values.

The number of possible values only determines the "maximum number of questions needed" (the upper bound). What truly determines the "average actual number of questions needed" is the shape of the distribution—how uniform is it? Are there dependencies between values? Three examples solidify this distinction:

• Fair Coin: 2 values, uniform, H = 1 bit. Reaches the upper bound; no shortcuts are possible.
• Fair Die: 6 values, uniform, H = \log_2 6 \approx 2.58 bit.
• Natural Language Characters: 26 letters. If uniform, it should be \log_2 26 \approx 4.7 bit/character. However, English letter distribution is extremely uneven (e is very frequent, z is very rare), and there are strong dependencies between letters (q is almost always followed by u). Empirical measurements show that per-character entropy is only about 1.0–1.3 bit, far lower than 4.7.

This last example deserves a moment's thought. If you guess a random English letter assuming a uniform distribution, you would need to ask nearly 5 questions on average. But if you first ask high-information questions like "Is it a vowel?" or "Was the previous letter a q?", most positions are determined without much guessing. This gap caused by "structure suppressing entropy"—the drop from 4.7 to 1.2—is exactly what compression algorithms and language models rely on. What they essentially do is extract all this redundancy.

Joint Entropy, Conditional Entropy, Mutual Information: Between Two Variables

So far, we have only discussed a single random variable. When there are two variables, the language of "how many questions to ask on average" naturally grows into an entire family. Let's look at the three members one by one:

• Joint Entropy H(X,Y) = -\sum p(x,y)\log_2 p(x,y): How many questions to ask on average to clarify both X and Y simultaneously.
• Conditional Entropy H(Y \mid X) = -\sum p(x,y)\log_2 p(y \mid x): Given that X is already known, how many more questions are needed on average to clarify Y. The two extremes are very illustrative—if Y is completely determined by X, H(Y \mid X)=0 (no additional questions needed); if they are independent, H(Y \mid X)=H(Y) (knowing X helps nothing, not a single question is saved).
• Mutual Information I(X;Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X): How many fewer questions are needed to clarify one variable, knowing the other.

Mutual information has several elegant properties: it is symmetric (I(X;Y)=I(Y;X)), non-negative, and I(X;Y)=0 if and only if X and Y are independent. It also has an equivalent formulation: I(X;Y)=D_{KL}(p(x,y) \parallel p(x)p(y)), which is the KL Divergence between the true joint distribution and the distribution that "pretends they are independent." In other words, mutual information measures exactly "how far the two variables deviate from independence."

Venn Diagram: Set Relationships of Information Measures

The relationships between these four quantities can be precisely depicted using two overlapping circles. This is the most classic diagram in information theory:

All identities can be read directly from the diagram. The union is H(X,Y), the left circle is H(X), the overlap is I(X;Y), and the left crescent is H(X \mid Y). Thus, equations like H(X) = H(X \mid Y) + I(X;Y) and H(X,Y) = H(X) + H(Y \mid X) are immediately clear.

A reminder: this analogy holds strictly only for two variables. With three or more variables, "interaction information" can take negative values, and the Venn diagram fails. However, the two-circle diagram is sufficient for building intuition.

These measures are the foundation for all subsequent content: compression approximates codeword length to H(X) (Source Coding), channel capacity is \max I(X;Y) (Channel Capacity), and deep learning loss functions are direct applications of Cross-Entropy and KL Divergence.

References

• Textbook: "Elements of Information Theory" (Cover & Thomas — the standard textbook on information theory, Chapter 2 Entropy, Relative Entropy, and Mutual Information)
• Paper: "A Mathematical Theory of Communication" (Shannon, 1948 — the foundational work on entropy and information theory; everything starts here)
• Textbook: "Information Theory, Inference, and Learning Algorithms" (MacKay — freely available online, Chapters 2 and 4 offer excellent intuition)

Keywords: self-information, entropy, joint entropy, conditional entropy, mutual information, chain rule, maximum entropy, binary entropy function, bit