PYTHON · NUMPY

Learn the reward

Learn a reward model from preferences, so you can judge and improve any behaviour without hand-writing what good looks like.

01Challenge

You have built policies every way -- a rule, a clone, a world model. But which behaviour is good? Hand-writing a reward is brittle, and a rule as blunt as did-it-reach ties every success. Instead, learn the reward: compare pairs of behaviours, record which you prefer, and fit a model that scores any behaviour the way your preferences do. The Bench gives nine behaviours (features: reached, smoothness), a set of pairwise preferences from a hidden expert, and a reward model whose update is missing. Fill it in, and the learned reward ranks the whole set -- and picks the same best the expert would. This is RLHF for robots: the 2026 way policies are judged and improved (Robometer, TOPReward, LeRobot's reward-models API).

02Model

A reward model turns comparisons into a score. Bradley-Terry says the probability you prefer i over j is the sigmoid of the score gap, so ascending the log-likelihood nudges the preferred behaviour's score up: w += lr*(1 - sigmoid(w . d))*d, where d is the feature difference. From a few dozen preferences the model recovers what you value -- reaching, then smoothness -- and can then score behaviours it never saw. That is how you improve a policy without ever writing down the reward by hand.

03Guided practice

Step A (worked): the pairs are (i, j, was-i-preferred), labelled by a hidden expert who values reaching then smoothness. Step B: the update line is a bare pass, so w stays zero and its ranking is a coin flip -- the FAIL. Step C (the fix): w = w + lr*(1 - sigmoid(w @ d))*d, the Bradley-Terry gradient step. Step D: the learned reward now agrees with the expert on fresh pairs and picks the same best behaviour -- a naive did-it-reach reward could never separate the successes.

04Feedback

PASS WHEN PASS when the learned reward agrees with the expert on held-out pairs and picks the same best behaviour.

If your run fails, check:

Read the bench’s printed result — it passes when the learned reward agrees with the expert on held-out pairs and picks the same best behaviour. Run the cells top to bottom; if one errors, the FAIL line names the step to fix.
The reward model never learned -- add the Bradley-Terry step w = w + lr*(1 - sigmoid(w @ d))*d.

05Retrieve & space

What does a reward model turn into a score?
Why can a preference-learned reward separate successes that did-it-reach cannot?
In one line, what is Bradley-Terry preference learning?

06Mastery & project

Pass the lab: from a handful of preferences, a reward model that judges any behaviour the way you do -- the evaluation axis that improves every policy you build. But the reward model only SCORES; the rung above it is the certificate that GATES an action before it commits -- the same energy the body descends, proving it won't diverge. That is the physics-first inversion, in Energy First Architecture (PAI-250).

The live 'Learn the reward' lab lets you provide the preferences yourself and watch the model rank a whole gallery of behaviours.

GO FURTHER · THE LIVE PLAYGROUND

Do it live: Learn the reward ↗ — compare behaviours, train a reward model on your preferences, and watch it rank them.
The whole ladder, live ↗ — sense, think, act, imagine -- and the evaluation axis, on Connection Systems.

← Learn in imagination Back to course →

PYTHON · NUMPY

Learn the reward

Learn a reward model from preferences, so you can judge and improve any behaviour without hand-writing what good looks like.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you