PYTHON · NUMPY

Motion needs memory

A VLA policy sees ONE camera frame and must act. Why does stacking a few recent frames nearly double its success (46.6%→82.9%)?

See why a single frame under-determines the action, and why stacking frames, the 46.6%→82.9% jump on real pixels, recovers it.

01Challenge

Show that velocity is unrecoverable from one frame but easy from two, the reason real-pixel VLAs stack frames.

Predict firstA VLA policy sees ONE camera frame and must act. Why does stacking a few recent frames nearly double its success (46.6%→82.9%)?

02Model

Our policy trained on real MuJoCo pixels jumped from 46.6 percent to 82.9 percent the moment we stacked three frames instead of one. The reason is physical: the action often depends on motion, how fast and which way things are moving, and a single still frame simply does not contain that. Two frames do: the difference between them is velocity. This is the last idea of the course, and it closes the loop back to the real thing. You have built vision, language, and action; now you understand why the released real-pixel VLA looks at time, not just space. Then go run all of it.

03Guided practice

1. The task. Predict velocity, a stand-in for a motion-dependent action. 2. One frame. Position alone can't recover velocity; the error stays high. 3. Two frames. Position now and position then makes the motion explicit, and the error collapses. That is exactly why nano-vla-pixels stacks three real frames.

04Feedback

PASS WHEN PASS when the two-frame prediction is far better than the one-frame one.

If your run fails, check:

A cell errored, run it top to bottom.
Two frames should make the velocity recoverable.

05Retrieve & space

Why can't a single frame predict a motion-dependent action? (it contains no velocity information).
What does stacking two frames reveal? (velocity, the difference between them).
What real jump did frame-stacking give our pixel VLA? (46.6% → 82.9% variance explained).

06Mastery & project

You understand why real-pixel VLAs look across time, not just at one frame.

The whole arc, yours: vision, language, action, and why the real thing watches motion, now run the released models.

GO FURTHER · THE LIVE PLAYGROUND

Run every released VLA ↗ · The nano-VLA family and the real 450M SmolVLA, all runnable on your device, the trained versions of everything you built.
The model family ↗ · Seven open releases on Hugging Face: weights, cards, and the capture recipe.

← When words have order The frontier: four models, two bets →

PYTHON · NUMPY

Motion needs memory

Compare one-frame and two-frame prediction to see why motion needs memory.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you