PYTHON · NUMPY

Imagine in a latent

Compress a high-dimensional observation to a small latent, then imagine the future inside it — the core trick of modern world models.

01Challenge

Encode a 12-dim observation to a 2-dim latent, learn the dynamics in the latent, and imagine a rollout that beats freezing the first frame.

02Model

A robot never sees clean state; it sees a flood of pixels or sensors. Modern world models — Dreamer, and the latent side of DIAMOND — don't predict in that raw space. They learn an encoder to a compact latent, run the dynamics there, and decode only when they need to look. Imagining in a small latent is cheaper, more stable, and it forces the model to keep what matters. Here you take a needlessly high-dimensional, noisy observation of a pendulum, find its true low-dimensional subspace, and roll the dynamics forward entirely in that latent. Decode, and the imagined observations track the real ones — from just two numbers.

03Guided practice

1. A big observation. The pendulum, lifted into 12 noisy dimensions. 2. Encode. PCA finds the 2-D subspace it truly lives on. 3. Latent dynamics. Learn z → z' and imagine in the latent, decoding only to compare. That encode → imagine-in-latent → decode loop is the backbone of Dreamer-style world models.

04Feedback

PASS WHEN PASS when the latent rollout, decoded, beats freezing the first frame.

If your run fails, check:

A cell errored — run them in order.
The decoded latent rollout should beat the freeze-frame baseline.

05Retrieve & space

Why do world models imagine in a latent instead of raw observations? (cheaper, more stable, keeps what matters).
What does the encoder here do? (find the low-dimensional subspace the observation really lives on).
When do you decode? (only to look — the imagination happens in the latent).

06Mastery & project

You can build the encode → imagine-in-latent → decode loop that modern world models are built on.

Faced with high-dimensional observations, ask what small latent the dynamics really live in.

← A model that bends Plan by imagining →

PYTHON · NUMPY

Imagine in a latent

Encode, imagine in a 2-D latent, decode, and beat the baseline.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you