PYTHON · NUMPY

Who reset the scene: the variance that governs every ranking

RoboChallenge measured a real success rate moving from 0% to 100% with task, props and model held fixed, varying only which class of human reset the scene. Compute what sample size a ranking would need to survive that, and find out which lever actually moves it.

01Challenge

Take the published leaderboard, work out the smallest gap its 10 rollouts per task can resolve, then add the reset-class term and count how many independent resetters a ranking would need.

Predict firstThe leaderboard runs 10 rollouts per task and its widest gap is 32 percentage points. What is the smallest gap that budget can actually resolve at 80% power?

02Model

Three classes of human reset the scene between rollouts on two tasks, stacking bowls and pouring fries: experienced testers who had collected the training data, ignorant testers on first exposure, and adaptive testers who were the model's own authors. Task, props and model were held fixed. The measured success rate moved from 0% to 100%. Adaptive testers scored highest, and the report documents a sweet-spot effect in which the adaptive tester located and reused favourable box positions. That term is wider than almost every effect reported anywhere in this material, which puts one rule under everything the rest of the course does: a ranking is admissible only where the between-item signal exceeds the within-cell spread. The arithmetic in this lab is the ordinary two-proportion sample size, and it earns its place here twice. First it turns an anecdote about testers into a number of rollouts. Then it shows that rollouts are the wrong lever, because a reset class is a cluster rather than a coin flip, and the only thing that moves the requirement is shrinking the spread.

03Guided practice

1. Read the span, then the board. Three reset classes, with task, props and model held fixed, and a success rate that moved 100 percentage points. Against that, the published ranking from the same instrument: pi0.5 task-specific at 43.7%, pi0 at 28.3%, CogACT at 11.7%, at 10 rollouts per task. Print the pairwise gaps and divide each by the span.

2. Ask the ordinary question. How many rollouts per cell would resolve each gap at 80% power and two-sided 5%, with the reset-class term assumed away entirely? The 15.4 point gap needs 149. Ten were run.

3. Turn it round. Fix the budget at 10 rollouts and solve for the smallest gap it can resolve. You get 56.2 points, which is wider than the whole board from 43.7% down to 11.7%. TR-2026-35's own threshold of 30 rollouts per policy-task cell resolves 32.4 points, which covers the widest gap and nothing narrower.

4. Add the reset term as a cluster. One cell is run by one tester, so every rollout inside it carries that tester's bias and no number of rollouts samples a second tester. Count classes instead. At the uncontrolled spread the design needs 124 independent reset classes per cell; the experiment names three.

5. Write the rule down. Report the within-cell spread beside the between-item signal, always, and refuse to rank when the spread wins. Every later module in this course is held to it, including the Institute's own numbers.

04Feedback

PASS WHEN PASS when cell 2 prints 149 rollouts per cell needed for the 15.4 point gap against the 10 that were run, and a smallest resolvable gap of 56.2 points at 10 rollouts against a widest published gap of 32.0 points, and cell 3 prints 124 independent reset classes needed against the 3 the experiment names.

If your run fails, check:

If your rollout counts come out roughly four times too small, you have used a one-sided z or 50% power. This lab uses two-sided 5% and 80% power, so (z_a + z_b)^2 is 7.8489.
If the minimum detectable effect looks small, check that the 2 is in the square root. Two cells are being compared, so the variance of the difference is twice the variance of one cell.
Cell 3 counts reset classes and not rollouts on purpose. If you find yourself dividing the between-class variance by the number of rollouts, re-read what a cell is: one tester runs all 10, so the tester's bias never averages out inside it.

05Retrieve & space

In the RoboChallenge reset experiment, what was held constant and what varied?
At 10 rollouts per cell and 80% power, what is the smallest gap in success rate the design can resolve, and how does that compare with the widest gap on the board?
Why does adding rollouts inside one tester's cell not reduce the reset-class variance?
State the admissibility rule for a ranking in one sentence.
What residual spread would make a drift protocol implementable at the benchmark's own 10 rollouts per task, and did this review locate a published one?

06Mastery & project

You can compute the minimum detectable effect of an evaluation budget, recognise a clustered variance term that no sample size absorbs, and refuse a ranking whose within-cell spread exceeds its between-item signal.

Apply it to the widest number in your own field of view. One vendor reports 98% zero-shot first-grasp success with no real-machine training data across 100 or more unseen objects, and the reports located state no trial count and no reset procedure. Write down the sample size that claim would need to mean what it appears to mean, and write down what you would have to be told about the resets before the 98% carries any information at all.

GO FURTHER · THE LIVE PLAYGROUND

RoboChallenge, the instrument that measured itself ↗ · The 0% to 100% reset experiment, the three tester classes, the grading rule and the Visual Task Reproduction fix. Read Section 2.3 for the fix, then look for a residual spread under it.
RoboDojo, 180 real trials per policy, scored double-blind ↗ · 18 real tasks at 10 trials each, three reviewers per trial, human teleoperators at 100.0% against a best policy of 12.8%. A second published control list to hold your own protocol against.
TR-2026-35, section 8, transfer evidence and who evaluates it ↗ · Where this variance term is set against every ranking in the review, including the ones the review itself drew, and where the 30-rollouts-per-cell threshold for an Institute run is stated.

← Four walls, and which one nobody is watching A world model can look right and move wrong →

PYTHON · NUMPY

Who reset the scene: the variance that governs every ranking

The published rates, the ordinary sample-size arithmetic, and one variance term the arithmetic cannot absorb. Cell 2 needs cell 1, and cell 3 needs cell 2.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you