How to tell whether a model obeys physics.
A model can look real and still create energy from nothing, pass objects through each other, or slide where friction says stick. This open benchmark scores what appearance hides: whether a dynamics model, sim engine, or world model honors the conservation laws — measured against analytic ground truth, with a verified rigid-body engine as the reference. A model earns trust here by obeying physics, not by looking real.
Three habits that transfer past this benchmark.
The benchmark is the vehicle; the habits are the point. Each was learned here by getting a confident answer wrong first, and each applies to any measurement you will make of a learned system.
1. Score structure, not accuracy.
You will be able to show that a model can be accurate at every step and still violate a conservation law over a rollout, and to say which of the two a proposed fix actually addresses.
2. Measure your instrument before your subject.
You will be able to establish a perception floor on data whose answer you already know, and treat only violations beyond it as the model's.
3. Build a scorer that can fail out loud.
You will be able to add checks that catch your own errors — impossible positions, events that never occurred, rates that move when the sample grows.
Every probe below is runnable from source in pure Rust, with no weights, GPU or dataset to download, so the exercises can be repeated rather than read about. The related course is PAI-270, which teaches the measurement discipline this benchmark depends on.
Conservation laws, with a known answer.
Most probes roll a model forward and check an invariant against analytic ground truth. The last two are different: they read the force field itself and answer whether the model can conserve, or whether it invents energy, before any simulation is run.
Energy is conserved
In a frictionless system total mechanical energy stays put; with damping it only falls. A model can be accurate step to step and still pump energy from nothing over a rollout.
reference: 0.15% drift
Momentum is conserved
With no external force, linear and angular momentum are fixed by Newton's third law. Correct physics holds them to machine precision; a per-body learned map leaks them because its errors don't cancel.
reference: 1e-15
No penetration, no free energy
Bodies do not pass through each other, and a contact never creates energy. A model that never learned the hard contact sinks through the floor; a livelier-than-real bounce invents energy.
reference: 0 / 0
The friction cone
Static friction holds a block below the cone angle; above it, the block slides at exactly g(sinθ − μcosθ). This is the invariant manipulation lives on — grasping, pushing, placing.
reference: 0.03% error
Recover the real dynamics
From a few samples, structure identifies the true parameters and extrapolates to conditions it never saw. A structure-free fit matches in-distribution and does not transfer.
reference: exact from 40 samples
On the real arm, not a toy
Every invariant also runs on the real SO-101 arm inside the trusted engine — the same conservation checks, on the actual multi-body robot, not just analytic systems.
reference: 0.13% drift
Can it conserve at all?
A force conserves energy only if it is the gradient of a potential — which shows up as a symmetry in how the force changes across space. Reading that symmetry answers the question from the force alone, with no simulation run at all.
predicts failure before any rollout
Does it invent energy?
The mirror question, for forces that depend on velocity. A real passive force may drain energy, never create it. One number bounds the worst rate at which a model manufactures energy — and catches a field that looks like harmless rotation but quietly pumps.
passive systems: never positive
Structure beats data — measured.
We put a genuinely trained neural world model on the stand: gradient-checked, so the training is verifiably real, and per-step accurate. Same data, same network capacity, same step size throughout.
A black-box next-state model
It learns to predict the next state accurately, one step at a time. Rolled forward, its energy explodes — it never learned to conserve, only to imitate.
one-step MSE 1.9e-4 · rollout energy drift 86% · FAIL
A structured model, same data
It learns only the force and lets a symplectic integrator carry the conservation law. It is more accurate and it conserves energy — it obeys physics by construction while still learning from data.
one-step MSE 2.1e-6 · rollout energy drift 4.2% · PASS
The difference is not data and not per-step accuracy. It is structure. You can build a learned model that obeys physics — by building the physics in.
Non-conservativeness runs away. Structure removes it.
The same test on the real five-joint arm is subtler, and it is the deeper lesson. Two learned models fit the same gravity data with the same true inertia and the same integrator. The only difference is how the force is parameterized. Both are gradient-checked.
The energy runs away
The learned force is a hair away from conservative, but that hair compounds: the energy it invents speeds the arm into larger errors, which invent more energy. The drift grows without bound.
energy drift 26% → 2,497% → 12,637% · over 1–5 s
The runaway is gone
Write the same force as the gradient of a learned potential and it is conservative by construction, at any accuracy. The runaway disappears, leaving a bounded offset rather than a growing leak — and chasing that leftover down turned out to be the more instructive half of the story.
energy drift 49% → 100% → 100% · bounded
We assumed that leftover was ordinary fit error and would train away. It did not: a network fitted 95 times more accurately still drifted 127%, barely better than before. So we instrumented the rollout and found the real cause — the arm swings to 27 radians on one joint, more than four full revolutions, far outside both the range it was trained on and the physical limits of the real hardware. The learned force was being asked for values in a regime it had never seen and the robot cannot reach.
Tell it that angles wrap
Gravity repeats every full turn of a joint, but a network reading raw angles has no way to know that 27 radians and 1.87 radians are the same pose. Feed it the sine and cosine of each angle instead and that fact is built in, so the force stays valid wherever the arm swings. Energy drift falls from 162% to 2.4%, against 1.8% for exact physics — and it tracks that reference across every starting condition we tried.
162% → 2.4% · reference 1.8% · verified across 4 starts
Twice, structure beat accuracy
Ninety-five times more fit accuracy bought almost nothing. Two facts about the physics — the force is a gradient, and the torque repeats every turn — bought everything. That is the same lesson the benchmark was built to test, arriving unbidden in its own open problem.
95× accuracy: no help · 2 structural facts: 65× better
Structure does not make the model perfect; it removes the failure modes accuracy cannot reach. One caution earned along the way: this only helps because the arm leaves its training range. We tried the same encoding on a second, gentler system that never does, and it made things slightly worse — so the question to ask is not whether a symmetry is real, but whether your trajectory goes where the fit stops meaning anything.
Built to conserve, not fit to conserve.
The deepest version: let the model learn the arm's inertia itself, and it still cannot leak energy. A mechanical system's Coriolis force is fixed by its mass matrix. Compute it from the same learned matrix that defines the energy, and the energy leak is zero — as an algebraic identity, not a fitted number.
Zero leak, by construction
The learned inertia is symmetric and positive-definite by construction, so kinetic energy can never go negative, and the Coriolis force is its exact geometric term. The instantaneous energy it injects is machine-zero — at any accuracy, in the training region or outside it.
energy injection rate ~1e-15 · conserved by construction
A free-form force leaks
Learn that same Coriolis force as a free field, fit to the same data, and it does real work on the model's energy — over a hundred percent of the dynamics' own power scale. Accuracy does not buy conservation.
energy injection rate ~4.0 · drifts 5% and climbing
Rolled forward on a real swing, the built-in model holds its energy to the same zero as the exact physics, while the fitted one drifts. That is the whole thesis in one line: conservation is a property you build into the model, not a number you fit toward. A learned world model can obey physics — mass, energy, and all — because the physics is in its structure.
The same test, run through a real world model's codec.
A video world model outputs pixels, not physics. To ask whether it obeys the laws, you read physical state from its frames — and first measure the tracker's own error, so only violations beyond that floor count. We ran the open method through a frontier 4B video world model's actual encoder and decoder, in pure Rust, on a laptop GPU. No cloud.
The whole model, in pure Rust
The 4B diffusion transformer and its video VAE run on a laptop GPU, matched to the reference implementations to about one part in a hundred thousand. A frontier video world model, on your own hardware, no server.
transformer 2e-5 · VAE encode 3e-6 · VAE decode 1e-5
Physics preserved, violations still caught
Round-tripped through the model's real latent bottleneck, an object's position is preserved to about a millimetre on a ten-metre scene. A scene with the wrong gravity is still flagged after passing through it — the verdict survives the codec.
position ±1 mm · correct 9.6 vs wrong-gravity 3.0 m/s²
The scorer works end to end through the model's own encoder and decoder, not just on clean rendered frames — the prerequisite for trusting any verdict about what a video model generates.
It generates a real fall
Given a still image of a ball in the air, a frontier video world model generated what happens next: the ball descends with motion blur, meets the floor, and stays one continuous object throughout. Watched frame by frame, the fall is there and it looks right. Measuring how right is where it gets hard.
ball in mid-air → motion blur → resting on the floor
A flat image cannot measure a 3D world
Tracking the object in the picture gives you its position on screen, not in space. In a generated room with perspective, a ball rolling away along the floor climbs toward the horizon and shrinks — indistinguishable, in screen coordinates, from a ball rising into the air. Our ball appeared to ascend from rest, which would be energy from nothing. It was not: it lost 80% of its area as it climbed, and a depth model run over the same frames shows it moving steadily away from the camera. Three independent signals agree it was receding, not levitating — no violation, and no way to read gravity off the screen either.
apparent "ascent" · area 12,000 → 2,200 px · depth confirms receding
Every clean number here was wrong once
A weaker model produced abstract smears, and the trajectory math still reported a tidy fall — there was no object at all. A clip with no collision in it still yielded a restitution figure. A rate measured on nine samples moved when we ran twenty-one. And a ball that was moving away read as a ball flying upward. Each number looked publishable. Each was caught by a check that could fail out loud.
inspect the frames · grow the sample · make the scorer able to fail
So the honest answer is narrower than we first wrote, and more useful. Qualitatively, the model imagines a real fall: object permanence, continuous motion, contact with the floor, all visible frame by frame. Quantitatively, we cannot yet score it — the invariants that work on a known 2D scene do not survive contact with a generated 3D one, because screen position is not physical position. Adding a depth model got us part of the way — enough to prove the suspicious moment was recession rather than a violation — but it returns relative depth, not metres. We then tried to close that last gap and could not, which turns out to be the more interesting result. A fall can be measured without metres if you use the object itself as the ruler and check whether the implied object size is believable, but that only works while the object holds still in depth. It would not: asked for a ball falling against a flat wall at a fixed distance, with the request for perspective and size change explicitly negated, the model returned a ball that hovered and shrank to a quarter of its width. Reaching for a metric depth model instead only moves the problem, since its metre estimate on invented footage is its own prior, not the video model's physics. We concluded a trustworthy number was therefore out of reach, and that was too pessimistic — the ball turns out to carry its own ruler. For a sphere, apparent size falls off with distance exactly as position on screen does, so dividing one by the other cancels both the unknown distance and the unknown lens, leaving height measured in ball-radii. On synthetic footage with a known answer the estimator recovers gravity to five decimal places even while the ball recedes — the very motion that defeated raw pixel tracking. Turned on real generated frames it splits the question in two, and the split is the result. The **shape** replicates: across seeds every fall we could measure is a clean parabola, R² between 0.992 and 0.995, which is what constant downward acceleration looks like. The **rate** does not: the same scene and prompt yield accelerations differing by more than sixfold, one seed accelerating upward, and one producing no measurable fall at all. A world with real gravity has one value of it, so a model that draws a convincing parabola at a different rate each time has learned the form of falling without a fixed strength behind it. We could quote a number from any single run — the first one implied a six-metre ball — and it would mean nothing. This paragraph is the ninth revision of this result. Each earlier version read as a clean verdict, and each was corrected by looking harder — twice by larger samples, once by reading the model's documentation, and finally by noticing that a ball which shrinks as it climbs is not climbing at all.
Put your model on the stand.
The benchmark is a scoreboard, not a paper. Any model that can step a state forward is scored on the same invariants and ranked — by the harness, never by the submission, so the board cannot be gamed. This board is the live, CI-scored standings on the frictionless pendulum, where energy is constant and the flow is exactly reversible.
| Model | Energy drift | One-step accuracy | Verdict |
|---|---|---|---|
| velocity-verlet (structured) | 0.07% | 4.6e-5 | PASS |
| symplectic (structured) | 2.64% | 3.4e-3 | PASS |
| structured-net (learned) | 2.88% | 3.7e-3 | PASS |
| lossy (wrong invariant) | 30.83% | 4.0e-3 | FAIL |
| explicit-euler (no structure) | 59.81% | 3.4e-3 | FAIL |
| black-box-net (learned) | 8550.72% | 2.7e-2 | FAIL |
The tell is in two rows. The symplectic integrator and explicit-euler have the same one-step accuracy and opposite verdicts. Accuracy per step does not buy conservation over a rollout — structure does. That is the axis the whole benchmark scores. Two rows above are real trained neural networks, gradient-checked: a structured net that learns only the force and lets a symplectic step conserve (PASS), and a black-box next-state net that is accurate step to step yet pumps energy on a rollout (FAIL). Same one-method interface, weights embedded — a learned model earns its place on the board exactly the way an integrator does.
Submit: fork the repo, add submissions/<model>.rs implementing Model::step, open a PR — CI scores it and the standings update automatically.
Pure Rust, no weights, no GPU, no data.
The reference implementation runs from source with nothing to download — the physics is the whole point, and the physics is free.
- Eight probes with analytic or engine-verified ground truth, plus the real SO-101.
- A model is any next-state predictor; drop yours in and it prints the same scorecard next to the reference.
- Standings and method are open; the axis is physics fidelity, and everyone is invited to run it.
v1 covers analytic systems and the real SO-101. Scoring a frontier video world model adds a render-and-perceive pipeline to read physical state from generated frames — the next step.