PYTHON · NUMPY · IN-BROWSER

Crossing the Reality Gap

See why a policy that is perfect in simulation can fail on real hardware, and make it robust with domain randomization.

01Challenge

A policy tuned at the nominal simulator gain blows up when the real actuator gain differs. Randomize the training gains until the policy survives the whole real range.

02Model

Every policy you have trained so far lives in a simulator, and the released checkpoint says so plainly: sim-to-real transfer needs care. The reason is the reality gap. The numbers a simulator assumes, an actuator's gain, the friction, the mass, are never exactly what the real hardware has. Tune a policy hard against one set of those numbers and it can be perfect in sim and unstable on the robot. Domain randomization is the workhorse fix, and it is almost embarrassingly simple: instead of training against one set of parameters, train across a whole range of them, so the policy learns to work no matter which reality it meets. It is the same idea behind the Institute's sim-to-real toolkit, where a policy trained in the browser is meant to drive a real arm.

03Guided practice

1. The gap. The controller assumes the simulator's actuator gain. Real hardware has a different one, somewhere in a range.

2. The failure. A policy tuned only at the nominal gain is aggressive; at a higher real gain it overshoots and diverges, its worst-case cost explodes.

3. The fix. Set TRAIN_GAINS = np.linspace(0.5, 2.5, 21) and re-run. Training across the range yields a more conservative policy that stays stable everywhere reality might land.

04Feedback

PASS WHEN PASS when the domain-randomized policy's worst-case cost across the real range is far below the sim-only policy's.

Still brittle - set TRAIN_GAINS to np.linspace(0.5,2.5,21) and re-run.
A cell errored - run them top to bottom.

05Retrieve & space

What is the reality gap? (the simulator's parameters never exactly match the real hardware's).
Why does a sim-only-tuned policy fail? (it is optimized for one parameter set and is brittle to others).
What does domain randomization do? (trains the policy across a range of parameters so it is robust to whichever reality it meets).

06Mastery & project

You can explain the reality gap and use domain randomization to train a policy that transfers.

The transfer reflex: before trusting a sim-trained policy on hardware, randomize the physics it saw so it is robust to the gap.

← Did It Learn, or Memorize?Run the Released Checkpoint →

1e6: return 1e12 # the policy diverged\n return J\nREAL=np.linspace(0.5,2.5,21) # what reality's gain might actually be\nKgrid=np.linspace(-1.2,0.0,121)\ndef best_K(train_gains):\n tg=np.atleast_1d(np.asarray(train_gains,float))\n J=[np.mean([cost(K,g) for g in tg]) for K in Kgrid]\n return float(Kgrid[int(np.argmin(J))])\ndef worst_real(K): return max(cost(K,g) for g in REAL)\nprint(\"cost, the reality range, and a policy search are defined\")","label":"Setup · cost, the reality range, a policy search"},{"code":"# Train the policy in SIMULATION only, at the nominal gain g=1. It looks perfect...\nKnom=best_K([1.0])\nprint(f\"sim-tuned policy K={Knom:.2f} cost at nominal g=1: {cost(Knom,1.0):.3f}\")\n# ...until reality's gain differs. Check the WORST case across the real range:\nprint(f\"worst-case cost on real hardware: {worst_real(Knom):.3g} (huge = it blew up)\")","label":"Tune in sim only -> brittle on hardware"},{"code":"# Domain randomization: train the SAME policy across a whole RANGE of gains, so it\n# is robust to whichever one reality hands you. Set the training range.\nTRAIN_GAINS = [1.0] # <- TODO: nominal only is brittle. Randomize: np.linspace(0.5,2.5,21)\nKdr=best_K(TRAIN_GAINS)\nprint(f\"policy K={Kdr:.2f} worst-case real cost {worst_real(Kdr):.3g} (nominal-only was {worst_real(Knom):.3g})\")\nassert worst_real(Kdr) < 0.5*worst_real(Knom), \"randomize the training gains (np.linspace(0.5,2.5,21)) so the policy survives reality\"\nprint(\"PASS - domain randomization made the policy robust to the reality gap.\")","label":"Domain randomization -> robust · set TRAIN_GAINS"}],"intro":"Watch a sim-tuned policy fail across the reality gap, then use domain randomization to make it robust.","key":"robot-learning/sim-to-real-gap","kind":"python","title":"Crossing the Reality Gap"}">

PYTHON · NUMPY · IN-BROWSER

Crossing the Reality Gap

Watch a sim-tuned policy fail across the reality gap, then use domain randomization to make it robust.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you