Vision-Language-Action: Build a Policy You Can Talk To
Build a vision-language-action policy from the ground up, because to judge the incumbent path of embodied AI, you must first build it. You make vision that turns pixels into places, language that grounds open-vocabulary instructions, and an action head that joins them; you prove it obeys words rather than memorizing a target; and you meet the costs a VLA pays a body (it is data-hungry, it fails out of distribution, and it carries no guarantee before it acts) plus the two problems we hit shipping our own: compositional order, and why motion needs more than one frame. Build the incumbent, know exactly what it costs, then see the physics-first inversion in the Energy First Architecture course, where one energy is the controller and its own proof.
▶ Start the course ← All coursesWhere this sits, and what moves it.
Binding constraint · The gap between a prior and a bound. A language model supplies a probability over what to do; a body needs a limit on force, reach and velocity, and no amount of fluency measures one. That gap is the quantity, and it is measurable the moment you write the envelope down.
Talking to a robot meant a fixed command vocabulary someone had enumerated in advance. Anything outside the list was not a hard request -- it was not a request at all.
A vision-language-action stack takes instructions it was never given a slot for, and this course builds one you can talk to. What it does not give you is a guarantee: the language model supplies a prior over what to do, and a prior is not a bound. Treating fluency as reliability is the specific mistake this material is built to prevent.
The step that matters is composing the prior with a checkable envelope, so the language layer proposes and a physics layer disposes -- which is exactly the seam PAI-250's certificate and PAI-210's runtime envelope are built on. A student who takes those three together is standing on the open problem, not reading about it.
Every hard thing was impossible until the constraint that made it impossible was named. How we read a frontier →
See and be told
Turn pixels into places and ground open-vocabulary instructions.
- L3See: pixels to a placeThe first job of a VLA's vision is not recognising what a thing is. What is it?Turn a camera frame into where things are, the first job of a VLA's vision.→
- L3Be told: ground the instructionYou ground colour words in an embedding space rather than a lookup table. What does that buy?Ground colour words in an embedding space so novel phrasings still land on the right target, open-vocabulary language in miniature.→
Act
Join vision and language into an action, and prove it obeys the words.
- L4Act: build the VLAYou combine vision and language into a policy that sees two targets and is told which one to reach. What has to be true of its action?Combine vision and language into a policy that sees both targets, obeys the instruction, and acts toward the named one.→
- L4Does it obey?A VLA learned to 'pick the red block' from demos. You say 'pick the BLUE block,' but blue never appeared in training. It will most likely…Prove the policy follows the instruction rather than chasing a fixed target, by flipping which colour you ask for.→
Harder, and real
Meet compositional order and motion, the problems behind the released VLAs, then see how 2026's frontier answers them, and where it still disagrees.
- L4When words have order'Put the cup on the plate' and 'put the plate on the cup' use the same words. Can a bag-of-words model tell them apart?Confront the compositional-order problem, 'red on green' vs 'green on red', and apply the fix our stacking VLA uses.→
- L4Motion needs memoryA VLA policy sees ONE camera frame and must act. Why does stacking a few recent frames nearly double its success (46.6%→82.9%)?See why a single frame under-determines the action, and why stacking frames, the 46.6%→82.9% jump on real pixels, recovers it.→
- L4The frontier: four models, two betsGR00T N1.7 swapped its old vision-language backbone for Cosmos-Reason2, a model built for step-by-step reasoning, feeding the same flow-matching action head as before. What does NVIDIA report this buying them?Place 2026's frontier VLAs on the predict-vs-control map, and see in miniature why factoring channels lets a policy reach combinations it never saw.→
- L4What a parallel decoder costsA decoder writes 512 tokens in blocks of 32, and at every step re-reads everything from the start of the text to the end of the current block. What decides how many positions it reads in all?Count the two bills a decoder that writes a whole block at once runs up, sequential calls and positions read, from its released code, and find the one number that decides both.→
Anatomy demonstrations
The machines behind this course, taken apart three ways, the body, the one rule, and the small learned brain. Guess before you look; an open core proves every number on the page.