PYTHON · NUMPY

Attention is one step

Four stored patterns, a query that sits near none of them, and a sharp softmax (beta = 16). Let the machine settle all the way into the energy's minimum. Its answer differs from attention's by about:

Show that softmax attention is exactly one gradient step on an energy, and measure how far a machine that settles into that energy's minimum lands from attention's answer.

01Challenge

A thermodynamic chip settles into the minimum of whatever energy it is given, and attention can be written as an energy. Decide whether a chip that settles into attention's energy computes attention.

Predict firstFour stored patterns, a query that sits near none of them, and a sharp softmax (beta = 16). Let the machine settle all the way into the energy's minimum. Its answer differs from attention's by about:

02Model

Since 2020 it has been known that softmax attention and the modern Hopfield network are the same object: write an energy that rewards a query for overlapping the stored patterns and penalises its length, and attention's output is what you get by taking one step downhill on it. That identity is the bridge the case for running AI on thermodynamic hardware usually leans on, because a thermodynamic chip is a machine that goes downhill. The next step people take is the one to check. If attention is an energy, then a chip that settles into that energy computes attention. It does not, and the reason is in the word one. Attention is exactly one gradient step, of size exactly one. Settling is as many steps as it takes, and it ends at the energy's minimum. The two land in the same place only in one corner: when the query already sits on a stored pattern and the softmax is sharp. That corner is the assumption behind the original paper's one-step retrieval result, and it is the case where the head had nothing to do. A query near no pattern, which is what a real attention head is handed, settles somewhere else at every sharpness measured: more than half the answer's length away here at the sharpest softmax, and between forty-one and seventy-eight percent on average in sixteen dimensions. So the substitution is not a speed-up with a constant to argue about. It computes a different function. What a settling substrate can offer attention is the one step, not the settling.

03Guided practice

Write the energy, confirm by finite differences that the query minus its gradient is attention, then settle queries into the minimum and measure how far the minimum sits from attention's one step, on a pattern and away from every pattern.

04Feedback

PASS WHEN You checked the identity instead of assuming it, and then checked the step everyone takes from it. Attention is one step on an energy; a machine that settles on that energy computes something else, except in the one corner where the answer was already there.

If your run fails, check:

The log-sum-exp is ln of the sum of exp(beta times each overlap), divided by beta. Subtract the largest term before exponentiating so a sharp beta does not overflow.
If the identity check fails by a factor, look at the division by beta: without it the gradient is beta times too large.
If it fails by a sign, the log-sum-exp term enters the energy negated and the length term positive.

05Retrieve & space

Softmax attention is exactly one gradient step, of size one, on the energy minus log-sum-exp of the overlaps plus half the squared length.
A machine that relaxes into that energy returns its minimum, which is a different vector from attention's output.
The two coincide only when the query already sits on a stored pattern and the softmax is sharp, which is the assumption behind one-step retrieval.
A diffuse query, the kind a working attention head receives, settles far from attention's answer at every sharpness.
What a settling substrate can offer attention is the single step, not the settling.

06Mastery & project

Given a claim that some computation is energy-based and therefore suits a relaxing substrate, state whether the computation is the energy's minimum or a fixed number of steps on it, and name the measurement that tells them apart.

Take another model component that can be written as a step on an energy, implement both the step and full relaxation, and report how far apart they land across the inputs the component actually sees.

← What the substrate actually samples What a factor describes →

PYTHON · NUMPY

Attention is one step

Four stored patterns, attention over them, and a machine that settles. One function, the energy, connects all three.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you