Methods note — curiosity for learning a world model
What Directed Curiosity Is For
Three ways a curious agent should learn a world model better than a random one. We tested each. All three fell. What survived is worth stating plainly.
Charlot Lab, Institute for Physical AI @ BMI
1 · The question
A free-energy robot that learns to ground verb–adjective–object commands entirely through curiosity-driven self-exploration reaches its accuracy criterion in roughly half the epochs of a non-curious control, and reproduces a developmental signature — the U-shaped error curve of a toddler who overgeneralizes ("goed") before mastering the exception.1 Its curiosity is precise: the complexity term of variational free energy, $-D_{\mathrm{KL}}\!\big[q(z\mid o)\,\Vert\,p(z)\big]$ — the agent is rewarded for observations that force its beliefs to update, i.e. for expected information gain.
That result motivates a narrower, buildable question. A world model is a forward map the agent will plan and act with; learning it well is a data-collection problem. So: does an agent that chooses informative experience learn a better world model — one that rolls out and controls well — than an agent that simply moves at random? We built this up at nano-to-small scale, each rung a prediction and its measured verdict. The reproducible scripts are the Institute's research record.5
2 · First, that it develops
Before asking whether curiosity helps, we confirmed our nano learners reproduce the phenomenon that makes OIST's agent look like an entity rather than a controller: the developmental U. A small network learns a mostly-regular mapping with a few high-frequency exceptions. Under the documented driver — a vocabulary spurt2 — it memorizes the frequent exceptions in "childhood", then the flood of regulars crystallizes the rule and, with no spare capacity, overwrites the exceptions (the dip, "goed"), and finally re-carves them (recovery, "went"). The U appeared on 10 of 10 seeds (median trough depth 0.62). The tell that this is development and not random forgetting is built in: the same spurt that dips the exceptions makes the regulars leap (≈0.3 → ≈1.0). One event, opposite effects. A controller cannot show a U; a system reorganizing its own representation as it learns can.
3 · Which surprise? — a proxy, and its trap
"Curiosity = seek surprise" hides a choice. Prediction error — high wherever outcomes are unpredictable — is a proxy. OIST's actual signal is information gain — high where beliefs would change. On a static target with a high-noise band (a Bayesian regressor, info gain in closed form, $g(x)=\tfrac12\log(1+\phi(x)^{\!\top}\Sigma\,\phi(x)/\sigma^2)$), the two come apart:
| selection policy | test error | reading |
|---|---|---|
| prediction-error (the proxy) | worst | trapped chasing the noise band — irreducible error stays high |
| pure info-gain (max variance) | good, but < random on rollout | over-weights uncertain boundaries |
| random | strong baseline | uniform coverage hits the region that matters |
| info-gain, visitation-weighted | best | info gain over states actually visited |
Two things already. Prediction-error curiosity falls into the noisy TV3 — the classic failure, here in miniature. And even the right signal, taken as pure maximum-variance, loses to random for multi-step rollout, because rollout error accumulates where the agent goes, not at the boundaries uncertainty maximizes. Weighting information gain by the visitation distribution recovers it to best. The hopeful reading was: an embodied agent gets that weighting for free, since it can only sample where it can move. That was the prediction to test.
Both signals are computed from the same posterior. Prediction error is epistemic plus aleatoric, so it rises with the noise band and keeps pointing there no matter how many samples are spent — the noisy-TV trap, and you can spend the whole budget without moving it. Information gain is $\tfrac12\log(1+\phi^{\!\top}\Sigma\phi/\sigma^2)$, and the division by $\sigma^2$ is the whole difference: irreducible noise discounts the reward instead of inflating it, so once the band is sampled the signal leaves and goes where beliefs can still change. Raise the noise and watch the red marker lock onto the band while the teal one walks away.
4 · Does embodiment rescue it? — no
We scaled up: a neural ensemble forward model, an embodied agent moving through a 2-D force-field world with a noisy-TV disk, scored by downstream control — reaching goals by planning with the learned model. Ensemble disagreement is the epistemic signal (it collapses in the noisy TV once sampled — all members agree it is noise — so it does not trap). Eight seeds:
| embodied policy | control | learnable RMSE | noisy-TV visits |
|---|---|---|---|
| random | 100% | 0.0301 | 13 |
| visitation-weighted epistemic (the fix) | 100% | 0.0309 | 16 |
| epistemic (pure disagreement) | 93% | 0.0432 | 9 |
| prediction-error | 84% | 0.0660 | 98 |
Prediction-error is trapped by the noisy TV (98 visits versus ~13) and learns the worst world model and the worst controller — the trap survives the jump to embodied, neural, control-scored. But embodiment did not rescue directed curiosity: pure epistemic still loses to random, and the explicit visitation-weighted fix only ties it (0.0309 vs 0.0301; control saturates at 100%). Embodiment constrains where you can go; it does not, by itself, supply the objective that weights uncertainty-reduction by where control happens.
5 · Does dimension rescue it? — no
The last escape hatch: in higher dimensions random coverage becomes exponentially sparse, so directed coverage should pull ahead. We swept $d=2,4,6$ with a proportional distractor (a fixed-radius ball becomes measure-zero in high $d$ and the trap would vanish for a trivial reason — we ruled that out). Five seeds, learnable-region RMSE:
| $d$ | visitation-weighted epistemic | random | prediction-error (TV visits) |
|---|---|---|---|
| 2 | 0.0252 | 0.0233 | 0.0531 (198) |
| 4 | 0.0337 | 0.0340 | 0.0467 (51) |
| 6 | 0.0378 | 0.0378 | 0.0438 (13) |
Both effects are stable. The noisy-TV trap persists — prediction-error is worst and most-trapped at every dimension (the gap narrows, 2.3× → 1.16×, as a higher-$d$ agent passes through more space en route and the smooth field generalizes from sparser data). And directed curiosity ties random at every dimension — it never pulls ahead. The dimensionality prediction is false.
6 · What survives
Three hopeful predictions, all fallen: info-gain beats everything (it needed a fix to beat the proxy, and only tied random); embodiment rescues it (no); higher dimension rescues it (no). One fact survived every stage, from static regression to embodied control across dimensions:
Prediction-error curiosity is captured by a noisy TV and learns the worst world model. Directed curiosity's real, dimension-independent value is robustness to that distractor — not beating a strong random baseline on the learnable dynamics.
For an energy-first world-model learner that is concrete guidance, not a slogan: prefer disagreement (reducible uncertainty, which the noisy TV collapses) over prediction error (which noise makes irreducible); weight exploration toward where control actually happens; and do not expect directed curiosity to out-sample a strong random baseline on the dynamics themselves. The value it does buy — never falling into the noisy TV — is real, and in a world with genuine irreducible-noise distractors it is the difference between a usable world model and a useless one.
7 · Limits
These are nano-to-small experiments in simulation — Bayesian and small-MLP models, 2–6 state dimensions, budgets of hundreds to low-thousands of steps — and every claim is a mechanism, not a scaled system. They are seed-robust (§2, 10/10; §3–§5, 5–8 seeds) but confined to a small family of environments (force-field dynamics with a localized distractor) and one downstream task (greedy model-based reaching). A negative — "directed curiosity does not beat random here" — is exactly the kind of claim a different regime can overturn, so we tried to overturn our own. We built the regime our limits nominated first: rare, localized learnable structure — a small "current" the controller must model to reach goals inside it, with no noise anywhere — where random should undersample and directed curiosity should finally win. It did not. Random still reached 99% of goals inside the current while sampling it only ~5 times, and prediction-error curiosity did worse (90%) despite visiting it the most — over-concentration unbalances coverage and hurts, the mirror of the noisy-TV failure, and this time with no noise to blame. A small generalizing model plus a closed-loop planner that corrects on real feedback simply does not need dense coverage of the rare region. We then turned the one knob left — region complexity, making the rare dynamics high-frequency so they genuinely need dense sampling and do not generalize from outside; if coverage ever mattered it should there. It still did not (directed 96% vs random 95%; prediction-error worst at 91%). That failed falsification strengthens the negative rather than qualifying it. Two regimes we have not tested are genuinely different mechanisms, not variations of this one, and could still restore an advantage: partial observability, and a distractor that is informative rather than noise. We make no claim about frontier-scale curiosity methods, and we credit the lineage this rests on: OIST for the developmental result,1 and the noisy-TV analyses of intrinsic motivation.3,4 What we claim is narrow and, we believe, correct: for learning a world model for control in these settings, coverage dominates, random is hard to beat, directed curiosity's over-concentration tends to hurt, and its one robust contribution is distractor-robustness — reported with the hopeful predictions it cost us, intact.
References
- Tinker, Doya & Tani, Curiosity-Driven Development of Action and Language in Robots Through Self-Exploration, Science Advances (2026), DOI 10.1126/sciadv.aee7533; preprint arXiv:2510.05013 (simulation; curiosity = the KL/complexity term of variational free energy).
- Plunkett & Marchman, U-shaped learning and frequency effects in a multi-layered perceptron, Cognition (1991); Rumelhart & McClelland, past-tense model (1986); Karmiloff-Smith, representational redescription.
- Pathak, Agrawal, Efros & Darrell, Curiosity-driven Exploration by Self-supervised Prediction, ICML 2017 — introduces and later analyzes the noisy-TV failure of prediction-error curiosity.
- Burda, Edwards, Pathak et al., Large-Scale Study of Curiosity-Driven Learning (2019) and Exploration by Random Network Distillation (2019); Sekar et al., Plan2Explore (2020) — disagreement-based (epistemic) exploration for world models.
- Charlot Lab, companion research record (research/oist-curiosity): five reproducible scripts — the U-curve, the KL vs prediction-error regressor, the nano world-model rollout study, the embodied ensemble control experiment, and the dimensionality sweep.