PYTHON · NUMPY

What a simulator is worth

Two simulators. One overstates success by 0.20 but usually agrees with hardware scene by scene. One is exactly right on average but its per-scene verdicts are independent of reality. Which saves more hardware trials?

Price a simulator in the only unit that matters for evaluation, hardware trials saved, and show that its bias is not what you are paying for.

01Challenge

You can run a policy in simulation as often as you like and on hardware almost never. Combine the two so the answer is still about hardware, then work out which property of the simulator decides how many hardware trials you owe. It is not accuracy.

Predict firstTwo simulators. One overstates success by 0.20 but usually agrees with hardware scene by scene. One is exactly right on average but its per-scene verdicts are independent of reality. Which saves more hardware trials?

02Model

The previous lesson sized a ranking against a variance term. This one sizes the hardware bill. Policies are usually reported as a success rate from a handful of trials with no interval at all, and the fix everyone reaches for is a simulator, which immediately raises the objection that the simulator is wrong. Prediction-powered inference makes that objection the wrong one. You take the mean over a very large simulated run, then add the average gap between real and simulated outcomes measured on a small paired set where the same scene was run both ways. That second term is the rectifier, and it removes the simulator's bias whatever the bias happens to be, so a systematically optimistic simulator is not a problem. What survives is a question about spread rather than accuracy. The estimate's error is governed by the variance of the per-scene disagreement, not by the variance of the real outcome, and those two can go in opposite directions. A simulator that is wrong in a consistent way has a small disagreement variance and buys you a lot of hardware trials. A simulator that is right on average but uncorrelated scene by scene has a larger disagreement variance than the real outcome itself, and using it costs you more trials than throwing it away. You will compute both, and then check the argument by measuring the estimator instead of trusting the algebra.

03Guided practice

1. Read the four simulators first. Note that two of them have zero bias and that this turns out not to be the useful thing to know about them. 2. The rectifier takes three values. It is +1 on q10, -1 on q01, and 0 on the two agreeing cells. Both non-zero values square to 1, so the second moment is just q10 + q01. 3. Subtract the square of the mean. The mean is q10 - q01, and it is only zero when the two disagreement cells are equal. The case where it is not zero is the one that catches a missing correction. 4. Then read the last block. It does not assert the variance argument, it measures it: the estimator is run six thousand times and its spread is compared against what your formula predicted.

04Feedback

PASS WHEN You priced four simulators in hardware trials, found that the biased one is the bargain and the unbiased one is worse than useless, and then confirmed the variance argument by measuring the estimator rather than trusting the algebra.

If your run fails, check:

A simulator that never disagrees must give exactly zero. If yours does not, the mean term is missing.
If the biased case returns 0.20 rather than 0.16, you computed the second moment and stopped. Subtract the square of the mean.
Define rectifier_variance(q11, q10, q01, q00) returning one number.

05Retrieve & space

The rectifier removes the simulator's bias. Say exactly what that leaves you paying for, and name the quantity.
One of the four simulators makes the estimate worse than hardware-only evaluation. State the condition on the disagreement cells under which that happens.
You have a simulator whose per-scene verdicts you cannot pair with hardware runs of the same scene. What does that cost you, and why is it not a small loss?

06Mastery & project

You can state what a simulator is worth as a number of hardware trials, decide from paired data whether adding it helps or hurts, and explain why its accuracy is the wrong property to argue about.

Take any sim-to-real result you can find that reports a success rate. Write down the number of hardware trials behind it and the interval it implies, then state what you would need in order to shrink that interval without more hardware: specifically, how many paired scenes, and what you would have to measure about the per-scene disagreement before you were entitled to claim a saving.

GO FURTHER · THE LIVE PLAYGROUND

The method ↗ · Badithela, Snyder, Zha, Mikhail, O'Kelly, Dixit and Majumdar: SureSim, non-asymptotic confidence intervals on real policy performance, reported to save over 20 to 25% of hardware evaluation effort on a diffusion policy and a multi-task fine-tuned pi0.
The variance term ↗ · The same kind of accounting applied to whether a leaderboard's ranking can be resolved at all.
How many seeds ↗ · The sample-size question asked of training rather than of evaluation.

← Who reset the scene: the variance that governs every ranking A world model can look right and move wrong →

PYTHON · NUMPY

What a simulator is worth

Four simulators, one estimator, and the property that decides your hardware bill.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you