PYTHON · NUMPY

How Many Seeds Does a Ranking Need?

Compute how much evidence a claim like 'policy A beats policy B' actually requires, and what a multi-task suite does to that number. Pass = you derive the seed count for a 0.8-versus-0.7 comparison, confirm it empirically, and state the cost of comparing 31 tasks at once.

01Challenge

You have two policies and a five-task suite. Run the suite at 3 seeds and at 10 seeds, which is what papers typically report, and look at the measured gaps. Some of them have the wrong sign. Now compute how many seeds the comparison actually needs, check that number against a simulation, and work out what happens when the suite has 31 tasks instead of 5.

02Model

You have shipped a policy. Someone asks whether it is better than the last one. That is not one measurement, it is the difference between two measurements, and a difference needs far more evidence than either of the things being differenced. Here is the number nobody likes. To tell a seventy percent task from an eighty percent one at ninety-five percent confidence takes about two hundred and eighty seeds per policy. Per task. The field usually reports three to ten. Which means most reported rankings between close methods are not measurements, they are coin flips with a table around them. And a suite makes it worse, not better. Thirty-one tasks compared at ninety-five percent confidence each: the chance that every single ranking is sound is not ninety-five percent, it is zero point nine five to the thirty-first power, which is about twenty percent. Four times out of five, at least one row of your table is noise. You expect about one and a half false rankings from chance alone. None of this says do not build suites. It says a suite spends confidence to buy breadth, at an exchange rate you can compute -- and you should compute it before you draw a conclusion, not after someone questions one.

03Guided practice

Step A (worked): run cell 1 at 3 and 10 seeds and compare the measured gaps to the true ones printed underneath. Re-run it. Watch signs flip between runs on the tasks with small gaps. Step B (worked): cell 2 gives you the two-proportion formula and applies it to the same five tasks. Read off which task needs the most seeds and notice it is not the one with the smallest gap -- it is the one with the worst combination of small gap and high variance. Step C (you drive): cell 3 checks the formula empirically. Before running it, predict what fraction of rankings will be correct at 3 seeds. Then look: it is about 41%, which is below a coin flip, because exact ties are common at 3 seeds and a tie does not recover the ordering. Step D (you drive): cell 4 prices breadth. Compute the seed count for one task, then for the same claim across 31 tasks, and state the ratio. Step E: decide what you will report. The honest options are a wider confidence interval, fewer tasks, or many more seeds -- and picking one is the deliverable.

04Feedback

PASS WHEN PASS when cell 2 gives 283 seeds for 0.8 versus 0.7, cell 3 shows ranking accuracy of about 41% at 3 seeds rising to 99.7% at 283, and cell 4 reports P(every ranking sound) = 0.2039 with 733 seeds needed across 31 tasks.

If your run fails, check:

Your cell-1 gaps look clean and all the signs are right. Re-run it -- you drew a lucky seed. That is the point: at 3 seeds the answer changes between runs, so it is not a measurement of the policies.
seeds_to_rank returned None for walk-flat. Correct: both policies succeed 100% of the time there, and no number of seeds distinguishes equal rates. A tie is a real result and should be reported as one, not broken by rerunning.
You got about 41% ranking accuracy at 3 seeds - BELOW a coin flip. That is not a bug: at 3 seeds the two policies frequently tie exactly, and a tie is scored as failing to recover the true ordering. Being worse than chance is what a comparison with no evidence behind it looks like.
The task needing the most seeds is not the one with the smallest gap. Variance matters too: sqrt(p(1-p)) peaks at p = 0.5, so a 0.5-versus-0.4 comparison is dearer than a 0.95-versus-0.85 one at the same gap.
0.95 to the 31st power is 0.2039, not 0.95. Simultaneous comparisons multiply; that is why the per-task confidence has to rise to 0.99839 to hold the family at 95%.

05Retrieve & space

The number (typed): roughly how many seeds per policy are needed to distinguish an 80% task from a 70% one at 95% confidence? -> about 280. The field typically reports 3 to 10.
Family error (short answer): 31 tasks each compared at 95% confidence -- what is the probability that EVERY ranking is sound? -> 0.95^31 = 0.2039, so four times in five at least one is noise.
Which is dearer (multiple choice): distinguishing 0.50 from 0.40, or 0.95 from 0.85? Same 0.10 gap. -> 0.50 vs 0.40, because sqrt(p(1-p)) is largest near 0.5 and the variance enters the seed count.
The trade (short answer): what does a multi-task suite buy and what does it spend? -> It buys breadth and spends confidence; the exchange rate for a 0.8-vs-0.7 claim across 31 tasks is 2.59x the seeds.

06Mastery & project

You can compute the seed requirement for a claimed ranking before running the campaign, you report ties as ties, and you state the family-wise correction whenever you compare more than one task. You never quote a suite headline without the per-task intervals underneath it.

Feeds the capstone directly. Whatever policy you release in the final lesson, its claim has to come with the seed count that supports it -- and if you cannot afford that many, the claim narrows to what you can afford.