Information and entropy

Communication through noise forces a hard question: how much surprise does a message carry, and how many bits do you need to send it reliably? Compression, phone lines, and codes all trade in uncertainty that can be measured, not only felt.

Claude Shannon's 1948 paper founded information theory: bits, entropy, and communication through noise. MacTutor and the primary paper (linked in further reading) are the stops. Steal the ethic: playful rigor. Treat "information" as something you can define and measure, then build systems that respect the math. Entropy is expected surprisal. Cross-entropy ties that instinct to the losses classifiers use every day.

For a modern rereader, chance gave you weights and expectation gave you averages; this chapter asks how surprised you should be, then averages that surprise. Language models live here.

If an event has probability $p$, its surprisal (self-information) is

(in bits). Rarer events surprise more.

Entropy of a discrete distribution is expected surprisal:

A sure outcome has entropy 0. A fair coin has entropy 1 bit.

If true labels follow $p$ and a model predicts $q$,

is cross-entropy. When $p$ is a one-hot label, this becomes $-\log q_{\text{correct}}$: the usual classifier loss.

KL divergence (preview): $D_{\mathrm{KL}}(p \| q) = H(p, q) - H(p)$ measures extra surprise from using $q$ instead of $p$.

If a fair coin flip landing heads barely surprises you, a fair die landing on a predicted face surprises you more (probability $1/6$ versus $1/2$). Shannon turned that graded surprise into bits. Entropy averages it. Cross-entropy asks how surprised a model should be by the truth.

English text has lower entropy than random letters: compression exploits that structure. Shannon made the intuition quantitative. Noisy channels need codes that respect entropy limits; that is communication theory's heart.

Algebrica may not center Shannon entropy on one page, but its discrete random variables treatment is the right substrate. Shannon's 1948 paper (in further reading) is worth skimming for definitions alone.

Cross-entropy is merciless toward confident wrongness: if $q_{\text{correct}}$ is tiny, $-\log q_{\text{correct}}$ is huge. That pain is the training signal.

Information as "meaning." Shannon information is about surprise and coding length under a distribution, not about semantic importance. A rare nonsense string can carry many bits.

Entropy as "disorder" without a distribution. Entropy is a functional of a probability distribution. Metaphorical disorder talk is optional and often sloppy.

Cross-entropy versus accuracy. Accuracy counts winners. Cross-entropy scores probabilistic honesty. A model can be accurate yet poorly calibrated, or the reverse on hard sets.

KL as a symmetric distance. $D_{\mathrm{KL}}(p \| q)$ is not generally equal to $D_{\mathrm{KL}}(q \| p)$, and it is not a metric. It is a directed extra-surprisal score.

Calling any large file "high entropy" without a model. Entropy needs a distribution. A large deterministic file can compress to almost nothing.

Equating mutual information with causation. Dependence is not cause. Information measures statistical connection, not mechanism.

Claude Shannon's 1948 paper founded information theory: bits, entropy, and communication through noise. MacTutor's Shannon biography and accessible portraits (such as Veritasium's The Bit Player material linked in further reading) convey the playful rigor. He played as hard as he thought, building gadgets and juggling ideas.

Treat information as measurable, then build systems that respect the math. That ethic is why compression, coding, and modern probabilistic ML share a vocabulary.

Compression, noisy phone lines, and efficient codes all trade in entropy: how much uncertainty a source has, and how many bits you need. File formats, error-correcting codes, and streaming protocols are engineering descendants of the same ideas.

Whenever a system says "this message was unexpected," Shannon offers a meter.