PAI-190 · Education

World Models: Learn a Simulator You Can Imagine In

Build a world model, a learned simulator that predicts what happens next, from the ground up. You fit the smallest one to a real system, roll it forward to imagine the future and watch error compound, make it nonlinear and latent the way Dreamer and DIAMOND do, then plan with it and finally play a pixel-space world the way Genie 3 does. It ends where the 2026 frontier actually is: Meta's V-JEPA 2 (physics learned from a million hours of video, then robot control from a little real data) and NVIDIA's Cosmos world-foundation models, the same imagine-then-act loop the Institute's own runs live and on-device. And it meets that frontier's limits: benchmarks like PhysicsMind show today's world models lean on superficial cues and routinely violate balance, torque, and inertia, which is why structured, physics-grounded models (port-Hamiltonian dynamics, PH-Dreamer) are the live research direction; and a rollout's score is not proof it is safe to run for real, which is why runtime verification of an imagined plan, before you commit it, is being built now, ours included.

Skilled·4 modules·7 lessons
▶ Start the course ← All courses
THE HORIZON

Where this sits, and what moves it.

Binding constraint · Compounding error over a rollout. A learned simulator is useful exactly as far as its predictions stay inside the tolerance the task needs, and that horizon shrinks geometrically with per-step error.

Was impossible

Planning required a simulator someone had written by hand, which meant planning was available only where a physicist had already modelled the domain. Anything with contact, deformation, or an unmodelled actuator was out.

Is probable

Learning the simulator from observation moves that boundary, and imagining inside it makes planning cheap. the limit is the horizon: rollouts are trustworthy for a while and then confidently wrong, and current practice mostly manages this by not looking too far ahead.

Becomes possible

The direction with physics behind it is a world model that carries a conserved quantity by construction rather than by penalty, so error cannot accumulate in the directions the physics forbids -- the architectural injection PAI-230 builds. A model that cannot drift in energy is a model whose horizon is set by something other than luck.

Every hard thing was impossible until the constraint that made it impossible was named. How we read a frontier →

Module 1

Predict the next state

A world model is next-state prediction learned from data; fit the smallest one and roll it forward.

  1. L2Predict the next stateTo predict where a pendulum is a moment from now, how well does simply assuming it does not move at all do?Meet the world model in one line: a map from (state, action) to the next state, learned from experience.→
  2. L2Imagine a trajectoryA learned world model predicts one step with a small error ε. Over a 50-step imagined rollout, the error…Roll the model on its own predictions to imagine the future, and watch prediction error compound.→
Module 2

Learn to imagine

Make the model nonlinear and latent, the representations modern world models are built on.

  1. L3A model that bendsYou fit a world model with a linear basis and it tracks straight motion well but misses the curves. What is the smallest fix?Give the world model a nonlinear basis so it can fit the curved parts of physics a straight line misses.→
  2. L3Imagine in a latentYou compress a high-dimensional observation to a small latent and imagine the future inside it. Why imagine in the latent rather than in the observation?Compress a high-dimensional observation to a small latent, then imagine the future inside it, the core trick of modern world models.→
Module 3

Imagine, then act

Turn a world model into a controller and into a playable world.

  1. L4Plan by imaginingA world model lets an agent plan by imagining rollouts. How many REAL environment steps does that planning cost?Use the world model to choose actions, search plans in imagination, commit only the best to the real world.→
  2. L4A playable worldYou build a pixel-space, action-conditioned world model and play it. What makes it playable rather than merely predictive?Build a pixel-space, action-conditioned world model and play it, the exact principle behind Genie, Oasis, and our on-device world model.→
Module 4

Certify before you commit

Contrast scoring an imagined plan by its predicted outcome with certifying it never left what the model actually knows, the gap between every current sample-and-score gate and a formal certificate-before-commit.

  1. L4Certify, don't just scoreA candidate plan gets the BEST (lowest) predicted cost from the world model, but it also uses actions far outside anything the model was trained on. Should you commit it?See the difference between trusting a plan because the model scored it well, and trusting it because it was certified to stay inside what the model actually knows.→
See it live

Anatomy demonstrations

The machines behind this course, taken apart three ways, the body, the one rule, and the small learned brain. Guess before you look; an open core proves every number on the page.

DemonstrationThe learned brainsOn-device policies you can watch decide →The whole series →
Foundations

From the interactive textbook

The ideas under this course as live explorables, each runs the real Rust library and re-derives its own result.

Ch. 9 · ModelingMake it linearOpen the chapter ↗The full textbook →