PYTHON · NUMPY

What is information?

Put a number on surprise, so that later we can ask what a channel costs.

01Challenge

Measure the entropy of English three ways and watch it fall as you learn more about the source.

02Model

A bit is the answer to one yes/no question you could not have guessed. Shannon's entropy counts how many such questions a source is worth on average. The important part for this course is what happens as you learn the source's habits: entropy falls. The gap between what a symbol could have told you and what it actually tells you is redundancy, and redundancy is what makes communication survive noise. Information is the currency a channel carries; before we can ask how to match a channel, we need to know what flows through it.

03Guided practice

1. Knowing nothing. With 27 symbols and no knowledge, every symbol costs log2(27) = 4.75 bits. 2. Knowing the letter frequencies. E is common, Z is not; that alone drops the cost. 3. Knowing the previous letter. After a Q, a U is nearly free. Each thing you know about the source shrinks its surprise — and the amount it shrinks is exactly the redundancy you could compress away, or spend on beating noise.

04Feedback

PASS WHEN PASS when entropy falls as context grows: conditional < per-symbol < uniform.

If your run fails, check:

A cell errored — run it again.
Entropies should fall as you learn more about the source.

05Retrieve & space

What is a bit? (the answer to one yes/no question you could not have guessed).
Why does entropy fall when you condition on the previous letter? (context removes surprise).
What is redundancy for? (it is what lets a message survive noise).

06Mastery & project