PYTHON · NUMPY

Absmean: rounding a net to three states

Ternarize a Gaussian weight matrix with the BitNet b1.58 absmean rule (scale = mean(|W|)) so that every value lands in {-1,0,+1} and the zero fraction, by the rule's scale-invariance, lands inside the 0.2-0.45 band.

01Challenge

Everyone agrees three states are cheap. The open question is whether three states are ENOUGH to hold a working network — and if so, how you get a full-precision net down to them without a catastrophe. This lesson takes the crudest possible path: round an already-trained layer straight to {-1,0,+1}. You will watch the representation survive, see roughly a third of the weights fall to exactly zero for free, and feel precisely why post-hoc rounding is not the end of the story.

Predict firstIf you round a fully-trained FP32 network's weights straight to {-1,0,+1} with absmean, does its task accuracy survive intact?

02Model

Here is the move that opens the whole frontier: a weight is not a number you must store to seven decimal places — it is a DIRECTION and a magnitude, and for an edge network the direction carries nearly all the signal. BitNet b1.58 makes that concrete with the absmean rule. Compute one scalar per matrix — the mean absolute weight, scale = mean(|W|) — then divide, round to the nearest integer, and clip into {-1,0,+1}. That is the entire operation: no learned codebook, no per-channel search, one cheap statistic. And the geometry pays a dividend for free. For Gaussian weights a value rounds to zero exactly when |W| < scale/2, and because scale = sigmasqrt(2/pi) that threshold is sigma0.399 — which captures the central ~31% of any zero-mean Gaussian, independent of sigma. Roughly a third of your weights become exact STRUCTURAL zeros — skipped multiplies, not small multiplies — and every survivor collapses to a single sign times one shared scale. The information-theory reading is exact: log2(3) = 1.585 bits per weight, and the balanced, sign-symmetric alphabet {-1,0,+1} is Shannon's symmetrical notation — the same three levels JEDEC chose for GDDR7 PAM-3 because three levels buy ~50% more voltage margin than four. The honest catch, and the entire reason module 3 exists: applying this POST-HOC to a net trained in FP32 is lossy — the net never learned to live on this grid, so accuracy drops. The frontier claim is not that rounding is free. It is that three states are ENOUGH to carry the network; the only real question is HOW you arrive there.

03Guided practice

Worked: Start from what absmean actually is. Take any trained layer W. The scale is a single number, the mean absolute weight: scale = np.mean(np.abs(W)). For W ~ N(0, sigma) (sigma = std) this equals sigma * sqrt(2/pi) ~= 0.798 * sigma. Now W/scale has mean absolute value 1 by construction — so round(W/scale) sends the smallest-magnitude weights (those with |W| < scale/2) to 0, and everything else to +/-1 (with clip catching the rare tail beyond 1). That scale/2 threshold is why the zero fraction lands near 0.31 for Gaussian weights regardless of sigma: the absmean rule is scale-invariant.

Your turn: The starter divides by a hard-coded scale = 1.0. Trained weights are small (here std 0.4), so round(W/1.0) pushes ~79% of them to zero — the matrix nearly vanishes and the grader's sparsity check fails. Replace the 1.0 with the absmean statistic so the scale tracks the actual weight magnitude.

Independent: Confirm the two things that make this a valid ternarization: (1) set(np.unique(Wq)) is a subset of {-1,0,1} — the clip(...,-1,1) guarantees no +/-2 leaks through from the round; (2) the reported zero_fraction lands in the 0.2-0.45 band. Then read recon_error and ask yourself why a lossy reconstruction can still be "enough" — that tension is the setup for QAT.

04Feedback

PASS WHEN Console prints three lines then PASS: unique values [-1.0, 0.0, 1.0], zero fraction 0.308, recon MSE 0.04243.

If your run fails, check:

Zero fraction near 0.79 (grader says 'zero fraction off') means you left scale = 1.0. Because the weights have std ~0.4, dividing by 1.0 makes almost every rounded value 0 — the whole matrix collapses. Set scale = np.mean(np.abs(W)) so the threshold tracks the real magnitude and ~31% (not 79%) land at zero.
Grader reports values outside {-1,0,1}: you rounded but did not clip, so a rare large weight became +/-2. Wrap the round in clip: Wq = np.clip(np.round(W/scale), -1, 1). The clip is what enforces the three-state alphabet.
Zero fraction near 0 with a tiny recon error means your scale is far too small (e.g. you divided by std**2 or by a per-element value) so nearly every weight rounds to +/-1. absmean is a single SCALAR — one np.mean over the whole matrix — not a per-weight quantity.

05Retrieve & space

A colleague ternarizes a layer whose weights are N(0, 2.0) instead of N(0, 0.4). Will the absmean zero fraction change much, and why or why not?
State the exact condition, in terms of |W| and scale, under which a Gaussian weight rounds to 0. Use it to explain why the free sparsity sits near ~31% rather than, say, 50%.
Why does clip(...,-1,1) matter even after round — what specific weights would otherwise escape the {-1,0,+1} alphabet?

06Mastery & project

Passing proves you can turn a full-precision layer into a valid {-1,0,+1} representation with a single non-learned scalar, and that you can read the resulting ledger — the ~31% free structural sparsity and the nonzero reconstruction error — as evidence that three states hold the net while post-hoc rounding still leaves accuracy on the table.

This is the on-ramp to the arc's central claim: three states are enough to CARRY a network. You have now built the representation by hand. The nonzero recon error you measured is the crack that module 3 (quantization-aware training) pries open — training the net to live on the ternary grid instead of being rounded onto it after the fact — and the sparsity you measured is the free lunch that later modules cash in as skipped multiplies in the multiply-free datapath.

GO FURTHER · THE LIVE PLAYGROUND

Drive the live ternary policy ↗ — FP32 vs ternary vs log, trained live
Read the whitepaper ↗ — Multiply-Free Neural Computation (TR-2026-02)

← Three states delete the multiply Train for ternary: QAT matches FP32 →

PYTHON · NUMPY

Absmean: rounding a net to three states

A trained layer, simulated as a small-magnitude Gaussian matrix. Ternarize it with the absmean rule and read the honest ledger: which values survive, how many go to zero, and how much you lost. The starter divides by the wrong scale — fix the one marked line.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you