PAI-220 · Education

Perception to Policy: From a Sensor to a VLA

The ladder from a raw sensor to a vision-language-action policy, one rung at a time, with a model you train on-device at every step: turn a sensor stream into features, train a kilobyte classifier, wire perception into a decision, clone a policy from demonstrations, hit the generalization wall and see what a VLA adds, learn a world model and plan a policy inside it, then learn a reward model to judge and improve any behaviour. The bridge nobody teaches whole.

L2·2 modules · 7 labs·7 lessons
▶ Start the course ← All courses
FROM THE OPEN-SOURCE ROBOTICS WAVE

The credential the ecosystem leaves on the table.

The $100 SO-101 arm, Hugging Face's LeRobot, and the worldwide hackathons made teleoperate → record → train → deploy something anyone can do. What they do not give you is a credential — the LeRobot course issues none, by design. This course credentials the same skills, assessed on your own runs: pass the labs and claim a verifiable, standards-aligned certificate you can take anywhere. We certify what you demonstrate here, not a course we did not run.

Run the loop live →How the credential verifies →
Module 1

Perception, learned on-device

Turn a raw sensor into features, then train a tiny classifier on them the way a $16 vision module does.

  1. L1The sensor speaks in numbersTurn a stream of raw 8-bit sensor readings into a small, centred feature vector a model can learn from.→
  2. L2A kilobyte classifierTrain a tiny softmax classifier on features -- on-device learning, the TinyML rung -- and see it generalize to views it never saw.→
Module 2

From perception to a general policy

Wire perception into a decision, clone a policy from demonstrations, then hit the generalization wall and see what a VLA adds.

  1. L2Perception becomes a decisionClose the loop: turn the classifier's output into an action so the agent reaches its goal.→
  2. L2Clone the expertFit a policy to recorded demonstrations and deploy the clone on goals it never saw.→
  3. L3The generalization wall, and what a VLA addsSee a cloned policy fail outside the world it was trained in, then fix it with diverse data -- the honest reason VLAs exist.→
  4. L3Learn in imaginationLearn a model of the world's hidden dynamics from random play, then plan a policy that acts correctly in reality -- without ever training in the real world.→
  5. L3Learn the rewardLearn a reward model from preferences, so you can judge and improve any behaviour without hand-writing what good looks like.→