PAI-180 · Education

Vision-Language-Action: Build a Policy You Can Talk To

Build a vision-language-action policy — the model class organizing the frontier of embodied AI — from the ground up, in your browser. You make vision that turns pixels into places, language that grounds open-vocabulary instructions, and an action head that joins them; you prove the policy obeys words rather than memorizing a target; and you meet the two honest problems we hit shipping our own released VLAs: compositional order, and why motion needs more than one frame. Every lesson runs on your device and ends by pointing at the real, released models doing the same thing at scale.

Skilled·3 modules·6 lessons
▶ Start the course ← All courses
Module 1

See and be told

Turn pixels into places and ground open-vocabulary instructions.

  1. L3See: pixels to a placeTurn a camera frame into where things are — the first job of a VLA's vision.→
  2. L3Be told: ground the instructionGround colour words in an embedding space so novel phrasings still land on the right target — open-vocabulary language in miniature.→
Module 2

Act

Join vision and language into an action, and prove it obeys the words.

  1. L4Act: build the VLACombine vision and language into a policy that sees both targets, obeys the instruction, and acts toward the named one.→
  2. L4Does it obey?Prove the policy follows the instruction rather than chasing a fixed target, by flipping which colour you ask for.→
Module 3

Harder, and real

Meet compositional order and motion — the honest problems behind the released VLAs.

  1. L4When words have orderConfront the compositional-order problem — 'red on green' vs 'green on red' — and apply the honest fix our stacking VLA uses.→
  2. L4Motion needs memorySee why a single frame under-determines the action, and why stacking frames — the 46.6%→82.9% jump on real pixels — recovers it.→