PYTHON · NUMPY

A world model can look right and move wrong

Rank the seven EWMBench models on appearance and on action-conditioned dynamics, correlate the two orderings, and test the correlation against a null you enumerate rather than assume.

01Challenge

Compute Spearman rho and Kendall tau between EWMBench scene consistency and dynamic consistency across all seven published models, then enumerate all 5,040 orderings of seven items to see whether the correlation beats chance.

Predict firstSeven world models are scored on appearance and on action-conditioned dynamics. How much of the dynamics ordering does the appearance ordering account for?

02Model

EWMBench scores seven embodied world models on one protocol and keeps appearance and dynamics in separate columns. Scene consistency is a DINOv2 patch cosine similarity, so it answers whether the frames look like the world. Dynamic consistency is a Wasserstein distance on velocity and acceleration, so it answers whether the motion the model produces under an action behaves like the real motion. OpenSora scores 0.9210 on scene against 0.9427 for EnerVerse_FT, the best model overall, and 0.0474 on dynamics against 0.5363. Two point three percent behind on appearance, an 11.3x gap on dynamics. Read down the whole table and the appearance column spans 1.18x while the dynamics column spans 13.07x, which is why a pixel score has almost no room to rank dynamics with even when its ordering happens to agree. The benchmark maximum on dynamics is 0.6197, held by LTX_FT at scene 0.9436, and the review sets the threshold that would retire this finding: a published model reaching dynamic consistency near 0.9 at scene parity. Until then, appearance fidelity and action-conditioned dynamics are empirically decoupled, and a leaderboard that scores pixels is scoring a different quantity from the one a policy depends on.

03Guided practice

1. Load the table. All seven EWMBench models, with scene consistency and dynamic consistency side by side. Rank each column with 1 as best and print the rank shift per model. Watch where OpenSora lands in each ordering.

2. Correlate the orderings. Compute Spearman rho from the sum of squared rank differences and Kendall tau from concordant against discordant pairs. Square rho to read how much of the dynamics ordering the appearance ordering accounts for.

3. Build the null before you believe the number. Seven items admit 5,040 orderings, so enumerate all of them and locate your rho in that distribution. Report the one-sided p and the 95th percentile of the null together.

4. Look at the ranges. Divide the maximum of each column by its minimum. The appearance axis has little room to separate models on, and the dynamics axis has a great deal, which is the mechanism behind the disagreement rather than a restatement of it.

5. Write both halves. The orderings disagree, and with seven models the residual agreement cannot be called real. A report that gives only the first half is selling you a result.

04Feedback

PASS WHEN PASS when your run prints a Spearman rho of 0.3929 and a Kendall tau of 0.2381 between the appearance and dynamics orderings, a one-sided permutation p of 0.1978 over 5,040 enumerated orderings, an appearance spread of 1.18x against a dynamics spread of 13.07x, and an 11.3x dynamics gap for OpenSora at 2.3 percent behind on appearance.

If your run fails, check:

If rho comes out negative, check the direction of your ranking. Both metrics are higher-is-better here, so rank 1 goes to the largest value in each column.
A sum of squared rank differences of 34 gives rho 0.3929. If you get 0, you have correlated a column with itself.
If the permutation p reads 0.0000, you are probably comparing against fewer than 5,040 orderings. itertools.permutations over seven items gives all of them.
The dynamics spread is 13.07x because the smallest entry is 0.0474. A metric whose column spans an order of magnitude and one whose column spans 18 percent cannot rank each other well.

05Retrieve & space

What does EWMBench measure with a DINOv2 patch cosine similarity, and what does it measure with a Wasserstein distance on velocity and acceleration?
OpenSora is 2.3 percent behind the best overall model on appearance. How far behind is it on dynamics, as a ratio?
What is the Spearman rho between the appearance ordering and the dynamics ordering of the seven models, and what one-sided p does it carry against the exact null?
Which of the two columns has the wider dynamic range, and by roughly how much?
What published result would retire the claim that appearance and action-conditioned dynamics are decoupled?

06Mastery & project

You can take a benchmark that reports several axes, rank the entries on each, and state how much one axis tells you about another, with a null you enumerated rather than assumed.

Find a world-model or video-generation leaderboard you have trusted and look for its dynamics column. If it has none, write down the metric it would have to add and how you would compute it. If it has two or more axes, run the rank test in this lab on them and report rho beside the enumerated null for that number of entries.

GO FURTHER · THE LIVE PLAYGROUND

TR-2026-35, section 4 ↗ · The action-conditioned transition function and action representation, with every quantity in this lesson graded and cited.
EWMBench ↗ · The benchmark itself, with the protocol in the same document as the scores: 10 tasks by 10 ground-truth episodes, 7 models by 3 candidates, 2,100 generated videos.
RoboChallenge ↗ · Where the run-to-run variance term comes from, and the temporal difficulty tag at 5 percent success against a 22 percent all-task mean.

← Who reset the scene: the variance that governs every ranking Angle is the integral of angular velocity, so say so in the loss →

PYTHON · NUMPY

A world model can look right and move wrong

The seven published EWMBench rows, ranked twice and correlated, with an exact permutation null. Standard library only.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you