Perception to Policy: From a Sensor to a VLA
The ladder from a raw sensor to a vision-language-action policy, one rung at a time, with a model you train on-device at every step: turn a sensor stream into features, train a kilobyte classifier, wire perception into a decision, clone a policy from demonstrations, hit the generalization wall and see what a VLA adds, learn a world model and plan a policy inside it, then learn a reward model to judge and improve any behaviour. The bridge rarely taught whole.
▶ Start the course ← All coursesThe credential the ecosystem leaves on the table.
The $100 SO-101 arm, Hugging Face's LeRobot, and the worldwide hackathons made teleoperate → record → train → deploy something anyone can do. What they do not give you is a credential, the LeRobot course issues none, by design. This course credentials the same skills, assessed on your own runs: pass the labs and claim a verifiable, standards-aligned certificate you can take anywhere. We certify what you demonstrate here, not a course we did not run.
Where this sits, and what moves it.
Binding constraint · Information per joule at the sensor. Every rung of the ladder is a decision about how much of the raw stream to spend and how much to discard, and discarding early is what makes the rest affordable.
Perception meant a workstation. A sixteen-dollar module that classifies on-device was not a smaller version of that pipeline -- it was a different thing that did not exist.
The ladder from a raw sensor to a VLA runs end to end on-device, and this course trains at every rung. The generalisation wall is real and this material walks you into it deliberately, because meeting it in a lab is cheaper than meeting it on a robot.
What is coming is the wall moving rather than a trick that avoids it -- and the thing that moves it is a world model, which is why this course ends where PAI-190 begins. The measurable signal is how much unlabelled interaction it takes to beat a labelled set; when that ratio inverts, the economics of the field invert with it.
Every hard thing was impossible until the constraint that made it impossible was named. How we read a frontier →
Perception, learned on-device
Turn a raw sensor into features, then train a tiny classifier on them the way a $16 vision module does.
- L1The sensor speaks in numbersYou feed raw 8-bit sensor readings, values from 0 to 255, straight into a learner. What goes wrong?Turn a stream of raw 8-bit sensor readings into a small, centred feature vector a model can learn from.→
- L2A kilobyte classifierBefore any training, the classifier's weights are all zero. What accuracy will it get on 3 classes?Train a tiny softmax classifier on features -- on-device learning, the TinyML rung -- and see it generalize to views it never saw.→
From perception to a general policy
Wire perception into a decision, clone a policy from demonstrations, then hit the generalization wall and see what a VLA adds.
- L2Perception becomes a decisionYour classifier is accurate and the agent still fails to reach the goal. Where is the problem?Close the loop: turn the classifier's output into an action so the agent reaches its goal.→
- L2Clone the expertYou clone a policy from demonstrations and test it on goals it never saw. What is the honest measure?Fit a policy to recorded demonstrations and deploy the clone on goals it never saw.→
- L3The generalization wall, and what a VLA addsYou train the policy only in one corner of the world. How will it do everywhere else?See a cloned policy fail outside the world it was trained in, then fix it with diverse data -- the reason VLAs exist.→
- L3Learn in imaginationThe world secretly rotates every action 120 degrees. A naive policy that ignores this -- does it reach the goal?Learn a model of the world's hidden dynamics from random play, then plan a policy that acts correctly in reality -- without ever training in the real world.→
- L3Learn the rewardYou learn a reward model from preference pairs rather than hand-writing one. What does that let you do?Learn a reward model from preferences, so you can judge and improve any behaviour without hand-writing what good looks like.→