PYTHON · NUMPY

The price of a token

At depth four the grammar has sixteen leaves, and every local rule has been seen by sample ten. How many samples until every distinct full string has been seen?

Measure, on a compositional grammar, the sample cost of learning surface strings against the cost of learning the local rules that generate them.

01Challenge

A depth-L grammar expands one symbol into a string of 2 to the L leaves. Count the samples needed to witness every local rule, and the samples needed to witness every distinct string, as depth grows.

Predict firstAt depth four the grammar has sixteen leaves, and every local rule has been seen by sample ten. How many samples until every distinct full string has been seen?

02Model

Why would predicting your own latents beat predicting the data? Sample-complexity theory published this year gives the mechanism, and it is countable on your own machine. A compositional grammar builds strings the way scenes and sentences are built: a symbol expands into parts, parts into subparts, depth L levels down. A learner that works at the surface, on whole strings, has to cover a space that multiplies at every level. A learner that works at the rules, level by local level, has to cover a set that only adds. The theory proves the first needs samples exponential in depth and the second constant up to logarithms, and the rules are exactly what a network's own latent representations hold. This bench counts both, empirically, on the same grammar with the same random stream. The numbers come out embarrassingly far apart, and the gap is the statistical case for why an agent should learn from what it already represents rather than from what the world happens to emit.

03Guided practice

Run the count at depths one to four. Read the rules column as the latent learner's bill and the strings column as the token learner's.

04Feedback

PASS WHEN You measured 3, 6, 8, 10 samples for the rules against 3, 17, 482, and past sixty thousand for the strings.

If your run fails, check:

The rules bill grows by twos and threes per level. The strings bill multiplies: witnessing composition at the surface means witnessing every way the parts can co-occur.
At depth four the token learner did not finish at sixty thousand samples while the rule learner finished at ten. That ratio is the theory's exponential-versus-constant gap, measured.
The rules are what latents hold. Predicting your own latents is learning at the level where the bill only adds, which is the published theorem this bench makes countable.

05Retrieve & space

Why does surface coverage multiply with depth while rule coverage adds?
What object in a modern self-supervised network plays the role of the grammar's rules?
The theory says constant in depth up to logarithmic factors. Where does the logarithm come from in your measurement?

06Mastery & project

Given a task's compositional depth, estimate the direction and rough scale of the gap between token-level and latent-level sample requirements.

Pick a robot task with obvious part structure, grasp then move then place. Sketch its depth, and ask which of your available datasets pay the strings bill and which pay the rules bill.

GO FURTHER · THE LIVE PLAYGROUND

The reading trail's JEPA line ↗ · Question one of the trail holds the latent-prediction family this result prices.

← The model that cannot exist The copy sent inward →

PYTHON · NUMPY

The price of a token

Same grammar, same sampler, two coverage questions. The string count is capped at sixty thousand samples, and depth four does not finish inside the cap.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you