Programming Physical AI
One simulated differential-drive rover, commanded at three rising depths (visual blocks, Python scripting, and bare-metal MCU), re-solving the same tasks at each depth until you can choose the right level of abstraction for a physical-AI task and prove it.
▶ Start the course ← All coursesWhere this sits, and what moves it.
Binding constraint · Loop time under a bounded worst case. A rover's behaviour is decided less by which language you wrote it in than by whether the loop closes before the world has moved.
Choosing an abstraction level used to mean committing to a toolchain and losing the ability to compare. Blocks, scripting, and firmware lived on different machines with different clocks, so 'is the metal actually faster here?' was an argument rather than a measurement.
Compiling all three depths to one target makes the comparison honest, and the answer is often that the high level is fast enough -- which is a real result, not a concession. Where it stops being true is the part worth knowing: the tail, not the average, is what breaks a control loop.
The interesting frontier is a compiler that chooses the depth for you against a stated timing budget, lowering only the loop that misses it. That needs a cost model for real peripherals that does not exist yet in any portable form, and building one is a student-scale contribution.
Every hard thing was impossible until the constraint that made it impossible was named. How we read a frontier →
One Loop, Visible: Visual Control
Make the rover sense, decide, and act with drag-and-drop blocks, and see the Python each block runs.
- L0The Embodied LoopYou wire a sense, decide, act loop to drive a rover to a goal. You give it a forward command and a 15 second budget. What happens?Wire a sense->decide->act loop that drives the rover forward and halts it within 0.25 m of the goal with no overshoot.→
- L1React and AvoidWith only a proximity sensor and if/else reactions (no map, no planning) can the rover get around the wall to the goal with zero collisions?Build a reactive rule-based behavior using conditionals so the rover avoids an obstacle and still reaches the goal with zero collisions.→
- L2Blocks Are PythonA block program generates Python. You raise the rover's speed by editing one generated line, leaving the 0.25 m stop check alone. What happens to the stop?Read the Python generated by a block program and correctly predict the effect of editing a single generated line before running it.→
Into the Code: Classical Control
Re-implement and surpass block behaviors in real Python: read sensors into a state estimate, close a PID loop, plan with a state machine.
- L2Reading the WorldYour rover drives on a fixed command and never reaches the pad. What is the smallest change that fixes it?Write control(obs) that reads the rover's goal_bearing each step and steers toward the goal pad, turning the differential-drive wheels so the rover reaches the goal within tolerance instead of driving blind.→
- L2The PID LoopThe starter reflex always commands a hard, full-tilt turn toward the goal (bang-bang). What does that do to the rover’s path?Write a proportional heading controller in Python control(obs) that eases the rover onto a goal pad within a tight 0.12 m tolerance, with no overshoot or weave.→
- L2Plan With a State MachineYour state machine has SEEK, ARRIVE and AVOID, and the rover consistently ends up far from the pad. What is the most likely cause?Design a finite-state controller in Python control(obs) that switches modes (SEEK far away, ARRIVE up close, AVOID near a wall) to drive the real MuJoCo rover onto the goal pad and settle within tolerance.→
Make It Learn: Policies in Sim
Replace hand-tuned control with learned behavior: classify, train an RL policy, and clone a demonstration in-sim.
- L3Learn From DataYou train a decision tree on sensor readings and it splits on front distance only. Where does it fail?Train a simple classifier mapping sensor readings to a discrete action and run it on the rover, meeting held-out accuracy >= 0.85 and goal completion with zero collisions.→
- L3Reinforcement LearningYou define a reward that pays out only on reaching the goal, and train. What does the learning curve look like?Define a reward function and train an RL policy in-sim to convergence (smoothed reward >= 6.0) with a greedy rollout that reaches the goal safely and efficiently.→
- L3Imitation and Sim-to-RealYou teleop the rover cleanly through the S-curve once and clone that drive. Will the clone reliably repeat the S-curve on its own?Clone a demonstrated trajectory with behavioral cloning to RMSE <= 0.12 m and correctly explain why a sim-perfect policy can fail on real hardware.→
Down to the Metal: Embedded Rust
Drop below Python: write real no_std embedded Rust firmware against the rover's peripheral HAL, compiled to WASM and run as the actual control loop.
- L4GPIO, Peripherals & the Control LoopYour first no_std firmware boots, owns its peripherals and blinks the LED. The rover does not move. Why?Write your first no_std Rust firmware: own the rover's peripherals, blink a GPIO LED, and drive the motors from the control loop to reach the goal.→
- L4Timers, PWM & Proportional DriveYou replace bang-bang control with PWM but keep a constant forward speed. Where does the rover end up?Use PWM motor channels and the millisecond clock to replace bang-bang control with a smooth proportional controller that settles in the pad without overshoot.→
- L4Async Tasks & the Real-Time LoopYour firmware navigates to the pad correctly, but the status LED never toggles. What went wrong?Structure firmware the Embassy way, with concurrent concerns on one real-time loop, by running a navigator AND a fixed-rate LED heartbeat off the millisecond clock, without blocking.→
Capstone: Autonomy, Then Optimized
Build full rover autonomy in Python, then move the critical control loop to the metal and prove the timing win.
- L3Full Autonomy in PythonYour combined perception and control loop drives the rover past the goal and times out. Which term is missing?Combine perception and control into one control(obs) that reliably drives the differential-drive rover to the goal pad within tolerance on real MuJoCo physics, fixing a partial starter controller.→
- L4Optimize the Critical Loop to the MetalYou port the whole autonomy loop to no_std Rust against the rover HAL. What has to be true for it to reach the pad?Re-implement the full autonomy control loop as no_std embedded Rust firmware against the rover HAL, with proportional steer-and-ease plus a non-blocking heartbeat, and prove it lands the rover inside a tight 0.12 m tolerance running directly on the metal.→
Anatomy demonstrations
The machines behind this course, taken apart three ways, the body, the one rule, and the small learned brain. Guess before you look; an open core proves every number on the page.
From the interactive textbook
The ideas under this course as live explorables, each runs the real Rust library and re-derives its own result.