PYTHON · NUMPY

Crossing the Reality Gap

A policy is perfect in simulation and brittle on hardware. You train it across gains from 0.5 to 2.5 instead of one nominal value. What does that buy?

See why a policy that is perfect in simulation can fail on real hardware, and make it robust with domain randomization.

01Challenge

A policy tuned at the nominal simulator gain blows up when the real actuator gain differs. Randomize the training gains until the policy survives the whole real range.

Predict firstA policy is perfect in simulation and brittle on hardware. You train it across gains from 0.5 to 2.5 instead of one nominal value. What does that buy?

02Model

Every policy you have trained so far lives in a simulator, and the released checkpoint says so plainly: sim-to-real transfer needs care. The reason is the reality gap. The numbers a simulator assumes, an actuator's gain, the friction, the mass, are never exactly what the real hardware has. Tune a policy hard against one set of those numbers and it can be perfect in sim and unstable on the robot. Domain randomization is the workhorse fix, and it is almost embarrassingly simple: instead of training against one set of parameters, train across a whole range of them, so the policy learns to work no matter which reality it meets. It is the same idea behind the Institute's sim-to-real toolkit, where a policy you train is meant to drive a real arm.

03Guided practice

1. The gap. The controller assumes the simulator's actuator gain. Real hardware has a different one, somewhere in a range.

2. The failure. A policy tuned only at the nominal gain is aggressive; at a higher real gain it overshoots and diverges, its worst-case cost explodes.

3. The fix. Set TRAIN_GAINS = np.linspace(0.5, 2.5, 21) and re-run. Training across the range yields a more conservative policy that stays stable everywhere reality might land.

04Feedback

PASS WHEN PASS when the domain-randomized policy's worst-case cost across the real range is far below the sim-only policy's.

If your run fails, check:

Still brittle - set TRAIN_GAINS to np.linspace(0.5,2.5,21) and re-run.
Read the bench’s printed result, it passes when the domain-randomized policy's worst-case cost across the real range is far below the sim-only policy's. Run the cells top to bottom; if one errors, the FAIL line names the step to fix.

05Retrieve & space

What is the reality gap? (the simulator's parameters never exactly match the real hardware's).
Why does a sim-only-tuned policy fail? (it is optimized for one parameter set and is brittle to others).
What does domain randomization do? (trains the policy across a range of parameters so it is robust to whichever reality it meets).

06Mastery & project

You can explain the reality gap and use domain randomization to train a policy that transfers.

The transfer reflex: before trusting a sim-trained policy on hardware, randomize the physics it saw so it is robust to the gap.

GO FURTHER · THE LIVE PLAYGROUND

See the sim→real gap in Forge ↗ · a generated robot learns to walk in sim, then meets the gap.

← Did It Learn, or Memorize?Beyond Frozen: The On-Robot Flywheel →

PYTHON · NUMPY

Crossing the Reality Gap

Watch a sim-tuned policy fail across the reality gap, then use domain randomization to make it robust.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you