Theory

Information theory

Entropy, coding and the transmission of information.

Start reading
0 / 7 read

Topics & articlesContents

3 topics
01

Information & probability

0 / 2 read
  1. Entropy and Information Measures
  2. KL Divergence and Cross-Entropy
02

Coding & communication

0 / 2 read
  1. Source Coding and Compression
  2. Channel Capacity and Coding
03

Representation & applications

0 / 3 read
  1. Rate–Distortion and the Information Bottleneck
  2. Information Theory in LLMs
  3. Optional: Conditional Information and Data Processing