PYTHON · NUMPY

How many runs before a difference is real?

A bench reports task error 1.078 against 0.973 from sixteen seeds, with a per-episode spread of 0.25. What is that result?

Compute the standard error of a bench comparison and decide whether an effect the bench reports is resolved by its own sample.

01Challenge

The table in the last lesson said attitude alone made task error worse than no sensing at all. Before you check it: at 128 seeds, is that difference larger than the uncertainty on it?

Predict firstA bench reports task error 1.078 against 0.973 from sixteen seeds, with a per-episode spread of 0.25. What is that result?

02Model

Every mean on a bench has a spread behind it. Task error here varies about 0.25 from episode to episode, so the standard error at n runs is 0.25 over the square root of n. At sixteen runs that is 0.06, and effects of 0.03 are invisible inside it. The fix is not subtle and not expensive: print the interval next to the mean, and raise n until the interval is smaller than the effect you are claiming.

03Guided practice

Run the sample-size sweep. Watch the gap's sign at small n. Then find the n where the interval finally clears the effect.

04Feedback

PASS WHEN You measured your own instrument and found what it can and cannot see.

If your run fails, check:

The gap is around +0.068 at 128 seeds against a 2 s.e. interval of about 0.06, so it is barely resolved and the magnitude is not trustworthy. At 512 seeds it settles near +0.033 with an interval of 0.031.
This is not hypothetical. This bench published that gap as 1.078 against 0.973 from sixteen seeds. The DIRECTION survived, and is still teachable: attitude without its rate is underdamped and does make tracking worse. The MAGNITUDE was about three times too large and was retracted.
The fix is one line: print the standard error beside every mean. Then a reader can see the size of the effect and the resolution of the sample in the same glance, and no result can outrun its own n.

05Retrieve & space

What is the standard error of a mean over n runs, given a per-episode spread?
A bench reports an effect of 0.03 from sixteen runs with a per-episode spread of 0.25. What can you conclude?
What single addition to a results table makes an under-powered claim visible?

06Mastery & project

Given any bench result, state whether its own sample resolves the effect it reports, and how many runs would.

Take a results table from a robot-learning paper. Determine whether it reports dispersion, and if not, work out what effect size its sample could have resolved.