Compute · substrate
The Physical AI Hardware Lottery
Energy-based models, post-von-Neumann architecture, and how a substrate shapes which ideas get developed.
The Charlot Lab · Institute for Physical AI @ Bailey Military Institute
Builds on TR-2026-01, which surveys the substrates. This report asks which algorithm class they suit. Sweep conducted in thirteen vocabularies.
1. The mechanism this report traces
Hooker's essay describes how hardware and software have repeatedly shaped which research directions advanced, and observes that domain-specialised hardware raises the cost of working away from the established path 1. Her formulation is that a hardware lottery can delay progress by making a productive idea look like a failed one.
The subject here is specific and narrow. Energy-based models require an operation that the dominant hardware generation did not accelerate. Section 3 establishes that this is a property of the operation and can be measured; Sections 4 through 7 follow what happened to the idea afterwards. The report describes a relationship between an algorithm and a machine, and it does not make a claim about what any research community concluded or intended.
2. Scope, method, and the prior report this one continues
TR-2026-01 already surveys the substrates. It organises roughly forty methods into nine families across two axes, from energy-harvesting microcontrollers to orbital datacentres, and establishes that the roots of nearly every method are decades old and openly published: balanced ternary hardware from 1958, the memristor from 1971, the Ising model from 1925, reversible computing from 1961 15. Its closing observation is that the near-term advance is a convergence of existing paradigms rather than a single new one, and specifically the pairing of a multiply-free deterministic substrate with a physics-based stochastic one.
What follows contributes no new measurement. It reads measurements made by others, together with two computations from the Institute's own benches which are marked where they appear. The literature sweep covered thirteen vocabularies: English, Chinese, Japanese, Korean, German, French, Spanish, Russian, Portuguese, Swedish, Dutch, Italian and Indian-institution English. Each figure is given with what was measured to obtain it, so that a reader can see which quantity a number describes. Where the sweep did not return a treatment of something, the text says that it did not locate one, which describes the search rather than the literature.
3. The operation, and how the hardware met it
An energy-based model defines a distribution in which probability falls exponentially with an energy. Fitting it by maximum likelihood requires the partition function, the integral of that exponential over the whole answer space, which grows as grid resolution raised to the number of dimensions and is out of reach for a robot action. The approach developed from 1985 onward estimates the gradient using negative samples drawn by Markov chain Monte Carlo 3.
Three properties of that approach interact with parallel hardware in a way that can be stated exactly.
| Property of the method | Behaviour on massively parallel hardware |
|---|---|
| A Markov chain is sequential: each state depends on the last | Parallel lanes accelerate work that divides into independent pieces. A chain does not divide, so lane count does not enter its wall-clock time. |
| A short chain carries bias; a long chain costs proportionally more | The trade between bias and cost is set by chain length, which lane count does not change. |
| Contrastive divergence is not the gradient of an objective function | Its convergence behaviour is studied empirically rather than derived, so tuning it is slower work. |
Backpropagation has a different structure: a forward pass and a reverse pass, both dense linear algebra, both dividing perfectly across lanes, with an objective whose gradient is exact. When general-purpose graphics processors arrived, the cost of backpropagation fell by orders of magnitude and the cost of a Markov chain did not, because the chain's limit is its dependency structure rather than arithmetic throughput.
4. What the record shows about the idea
4.1 The physics lineage
The 2024 Nobel Prize in Physics was awarded jointly to John Hopfield and Geoffrey Hinton for foundational discoveries enabling machine learning with artificial neural networks. The citation identifies the work specifically: Hinton, between 1983 and 1985, used tools from statistical physics to create the Boltzmann machine 2. The line originates in statistical mechanics, and that origin is what makes a physical substrate a natural question to ask about it.
4.2 Diffusion models are energy-based models
In a diffusion model the noise-prediction network is the gradient of an energy with respect to its input, and when data is perturbed with Gaussian noise the denoising score-matching loss coincides with the diffusion training loss. The relationship is stated directly in the literature: an energy-based model and a plain diffusion model differ in parameterisation, energy against score 5. Lesson 6 of the accompanying course recovers one form from the other numerically and shows the residual falling with the integration step, which is the behaviour of a discretisation error rather than a difference between the objects.
The same sources record that score parameterisation is dominant in practice, citing flexibility and efficiency, and that pre-trained energy-parameterised models are not publicly available at scale. Both forms describe the same distribution. They differ in what it costs to obtain a gradient: the score form returns it from a single network evaluation, and the energy form obtains it through sampling.
4.3 Results on embodied tasks
Implicit Behavioral Cloning reported that energy-based policies often outperform explicit mean-squared-error and mixture-density policies on robot learning tasks, including high-dimensional action spaces and visual inputs, with physical robots learning contact-rich behaviours at one millimetre precision 4. Its theoretical argument concerns representing functions that are discontinuous and multi-valued, which is the property Section 5 develops.
4.4 What is established at what scale
The Institute's own bench series places the energy-based advantage on specific properties: out-of-distribution generalisation, composition by summing energies, verification by scoring candidates, and adaptive computation at inference. Every result in that series is at or below 800 million parameters 14. Section 3 concerns the cost of an operation and the scale at which a parameterisation is represented; the question of capability at frontier scale is a separate one, and this report treats it as open rather than settled in either direction.
5. Why Physical AI is where this becomes concrete
A robot's correct action is frequently a set rather than a point. There is an obstacle; going left is correct and going right is correct. A policy trained by minimising squared error against demonstrations of both returns their mean, and the mean of left and right passes through the obstacle. Additional demonstrations move the mean closer to the centre, because the mean is what the loss is defined to return.
The Institute's bench reproduces this on a multivalued algebraic system, where a fairly supervised feed-forward network scored zero at every model size tried 14. An energy over the action space holds each valid answer as a separate minimum, and selection is a separate step. Lessons 1 and 2 of the course build both objects and compare them.
5.1 The dominant policy class samples an energy at inference
Diffusion Policy is described in its literature as the dominant paradigm for representing multimodal action distributions in robot learning, and its inference procedure iteratively optimises with respect to a learned gradient field through a series of stochastic Langevin dynamics steps 6. The same literature states the cost and its consequence: these methods require typically ten to one hundred network function evaluations, and that requirement is the constraint on real-time deployment.
5.2 The controller and the contact model share the same form
Model Predictive Path Integral control, the sampling controller used on contact-rich manipulators and off-road vehicles, weights sampled rollouts by the exponential of negative cost over a temperature, which is a Boltzmann distribution over trajectories. The control literature derives this directly, obtaining MPPI as a preconditioned gradient step on a Kullback-Leibler-regularised free energy objective and as expectation maximisation with Boltzmann reweighting 7. A mapping of MPPI onto Ising hardware has been published, so far without hardware results or energy measurements 8.
Below the policy, contact dynamics is formulated as a complementarity problem, and modern treatments express contact constraints as energy terms in an objective and integrate variationally. The policy, the controller and the contact model are therefore all stated as energy objects, and all three are currently evaluated on hardware that represents an energy landscape numerically rather than physically.
6. What a settling substrate changes
A Boltzmann machine, an Ising model and a probabilistic-bit fabric describe one object. A fabric held at a given inverse temperature occupies configurations with probability falling exponentially in their energy, so its resting distribution is the distribution a sampler is written to produce. Lesson 7 of the course verifies this against an exact enumeration and reports the sampling noise floor alongside the residual.
The same applies to fitting. Equilibrium propagation estimates gradients by comparing a freely settled state with one settled under a small nudge toward the target, using only local quantities, with no separate reverse pass and no stored activations 10. Hardware demonstrations exist in analog, Ising, oscillator and memristive devices. Lesson 8 derives the estimate on a case where the exact gradient is available and shows the two converging as the nudge shrinks.
Both loops therefore have a physical realisation on this class of device. What such a device requires in exchange is the subject of Section 7.
7. Where the two costs meet
Both sides of the comparison now carry a number. The published operating range for diffusion policies is ten to one hundred network evaluations per action, evaluated in sequence 6. An embedded field-programmable gate array running simulated bifurcation over 2,048 spins has a measured end-to-end latency of approximately 100 milliseconds, independent of problem size, obtained with 32-fold lossless compression of the coupling matrix and a learned parameter estimator in place of per-problem tuning 11.
One cost grows with the number of steps because the steps are sequential. The other does not vary with step count, because the settling substrate has no steps. The figure below crosses them.
Figure 1. One published range and one measured latency. The 3 millisecond network forward pass is this report's assumption and is the quantity a reader is most likely to have better information about; changing it moves the crossing point proportionally. The shaded band marks the published ten to one hundred evaluation range.
At the measured latency and a 3 millisecond forward pass the costs meet near 33 evaluations, inside the published range. The crossing point is therefore within the operating region already in use rather than beyond it.
8. Two workloads on one substrate
Transformer attention is being mapped onto the same class of device. An analog in-memory architecture built on gain cells reports up to a 70,000-fold reduction in energy and a hundred-fold speed-up against graphics processors for a 1.5 billion parameter model, reaching text-processing performance comparable to GPT-2 through an initialisation algorithm rather than retraining 9. Attention accounts for roughly seventy to eighty percent of large language model inference energy.
The two workloads use the device differently, and the difference is physical.
| Attention in an analog memory array | An energy-based model on a sampling fabric | |
|---|---|---|
| What the device performs | analog multiply-accumulate | occupation of states with Boltzmann probability |
| Relation to the algorithm | acceleration: the arithmetic is faster and the algorithm is unchanged | realisation: the resting distribution is the sampled distribution |
| What it asks of the device | linearity, precision, low drift | statistics that match the intended distribution |
| Role of device fluctuation | a source of error, to be suppressed | the source of randomness, to be characterised |
That last row is a statement about what each workload needs from the same silicon. Analog computing has a long record of difficulty in production, and the reported causes are consistent across two decades of reviews: conductance variability, non-linear and asymmetric switching, drift, mismatch, and the integration of emerging non-volatile memory into standard flows. Digital neuromorphic parts are closer to commercial availability; analog crossbar in-memory computing is at an earlier stage.
9. Sampling as a general-compute workload
Inference-time sampling has become routine across the field: generate candidates, score them, select. Coverage rises log-linearly with the number of samples, and verifier-based selection scales more robustly than verifier-free selection. Purpose-built inference silicon has begun to include dedicated acceleration for token sampling.
The Institute's benches show the same shape from the other side. Using an energy to select among candidates rather than to descend a landscape reached complete accuracy on conjunctions where gradient descent on the same landscape was sensitive to step size 14. Generate-and-verify is an energy-based procedure in structure.
Beyond machine learning the sampling class is long established: Bayesian inference, Monte Carlo, uncertainty quantification, combinatorial optimisation. And the economics of the substrate transition are measurable at the level of the memory hierarchy. A dynamic-memory access costs on the order of three thousand times a sixteen-bit multiply at a 45 nanometre node, data movement runs roughly a hundred times the arithmetic, and a large majority of energy in a conventional machine is spent moving operands rather than operating on them. These figures describe machine-level energy per operation, which is one of the three denominators separated in Section 10.
10. Three denominators, and what each supports
Efficiency in this field is stated in at least three different denominators, and a figure is only comparable to another figure in the same one. Naming them is what allows the datacentre results of Section 9 and the embodied results below to be read together.
| Denominator | What a figure in it describes | What it supports |
|---|---|---|
| Energy per operation | the cost of an arithmetic result or a sample on a given device | device and architecture comparisons, including the memory-wall figures of Section 9 |
| Energy per completed task on a body | compute and actuation together, integrated over a closed loop until the task succeeds | deployment planning for a robot, and bounded as below |
| Time per action | latency against a control period | whether a policy can run at a given control rate, which is Section 7 |
In the second denominator, compute is one term of a sum. On a reaching task the compute share runs from five percent at one watt of compute to fifty-nine percent at thirty watts, and on a mobile robot under autonomous navigation the graphics processor alone draws 37.3 percent while the motors draw 16.6 percent 13. Improving compute efficiency by a factor leaves the non-compute term unchanged, so total task energy falls to the non-compute share plus the compute share divided by that factor, and the limit is the reciprocal of the non-compute share.
| Compute share of a completed task | Limit on whole-robot saving |
|---|---|
| 37.3 percent, mobile robot, GPU alone | 1.60x |
| 42 percent, reaching at 15 W | 1.72x |
| 59 percent, reaching at 30 W | 2.44x |
| 74 percent, reaching at 60 W | 3.85x |
At the 59 percent share a thirty-fold compute improvement delivers 95.4 percent of what an unbounded improvement could deliver in this denominator. A figure of 70,000-fold stated in energy per operation and a limit of 2.44x stated in energy per completed task are therefore both correct and describe different quantities.
11. What is established, and what is open
Established. That the sampling operation is sequential and that lane count does not enter its wall-clock time, which Section 3 states and Lesson 5 measures. That the score and energy parameterisations describe the same object, which Lesson 6 verifies numerically. That the dominant visuomotor policy class samples an energy at inference at ten to one hundred evaluations per action, which its literature states 6. That an embedded sampler has a measured end-to-end latency of about 100 milliseconds independent of problem size 11. That total task energy on a body bounds any compute improvement to between roughly 1.6 and 2.4 times 13.
Open. Whether energy-parameterised models are competitive at frontier scale, which the Institute's own results at 800 million parameters and below do not settle in either direction 14. Whether conditioning latency reaches control rates, where roughly one to two orders remain at 100 Hz and the published improvements are compression and estimation rather than new physics 11. Whether analog device statistics can be characterised closely enough to serve as a sampling source at scale, which is an active materials and circuits programme.
Prior art. Boltzmann machines, Ising annealing, equilibrium propagation, MPPI, diffusion policies and in-memory computing are established fields with substantial literatures. What this report contributes is their joining: the mechanical reading of the operation in Section 3, the acceleration and realisation distinction in Section 8, conditioning bandwidth as the governing quantity for embodied use in Section 7, and the separation of denominators in Section 10. This review did not locate these three stated together in the literature it read, and the field's consensus vision document, which makes intelligence per joule its stated north star and treats embodied AI at length, did not return energy-based models or thermodynamic computing as a named workload direction on the seeds used 12.
12. Conclusions
Energy-based models require sampling, sampling is sequential, and the hardware generation that carried the field forward accelerates work that divides into independent pieces. That is a relationship between an operation and a machine, and it is measurable. Over the same period the underlying idea continued to develop in the parameterisation whose gradient costs one forward pass, and it is that parameterisation which carries the field's public models at scale. Physical AI makes the cost visible, because an embodied policy needs to represent a set of correct actions and must produce one within a control period, and the dominant policy class pays ten to one hundred sequential network evaluations to do so. A substrate whose resting distribution is the sampled distribution removes the sequence rather than shortening it, and on one published range and one measured latency the two costs meet inside the region already in use. The quantity that distinguishes this case from the parallel case of analog attention is what each asks of device fluctuation: one characterises it, the other suppresses it.
13. The forcing function
| What is bounded | The physics that sets it | The engineering change that moves it | What becomes possible |
|---|---|---|---|
| Ten to one hundred network evaluations per action | Sampling an implicit landscape by iterated denoising on a machine that represents the distribution numerically | A substrate whose resting distribution is the sampled one, so a single settling replaces a sequence of passes | Multimodal policies at full control rate, with the latency term removed rather than reduced |
| Conditioning bandwidth, about 100 ms per new problem | A fabric re-equilibrates after each new set of boundary conditions | Resident models with fast clamp ports; coupling compression, measured at 32-fold; estimated rather than tuned parameters | Roughly one to two orders remain to robot control rates, and each is an interface and memory-hierarchy problem |
| Fitting requires a sequential sampler | The intractable partition function | Equilibrium propagation, where the device supplies its own gradient by settling twice | Learning on the device at the edge, without a stored activation tape |
| Analog device statistics vary and drift | Nanoscale device physics | For arithmetic, tighter linearity and lower drift. For sampling, characterisation of the distribution the device produces | A device can become usable for the sampling case at a maturity earlier than the arithmetic case requires |
| Whole-robot energy saving, limit 1.6x to 2.4x | Total task energy is compute plus actuation | Work on the actuation term: winding and gearing design, thermal management, trajectory shaping against an identified power model | The larger term of the sum becomes addressable, and the embodied case is developed in latency, duty cycle and thermal envelope where no such limit applies |
Energy-based models were out of reach at scale when every gradient step required a sequential sampler on a machine built to divide work across lanes. They are demonstrably productive today, in score parameterisation, at the head of generative modelling and of robot policy learning, with a physics lineage recognised by a Nobel Prize and a sampling cost that the robotics literature identifies as its real-time constraint. They become ordinary when a substrate holds the distribution physically and can be conditioned at control rate, which requires no new physics: it requires conditioning bandwidth, device characterisation, and an interface, which is the most tractable class of obstacle there is.
References
- Hooker, S. The Hardware Lottery. arXiv:2009.06489 (2020). read · framing
- The Nobel Prize in Physics 2024, press release and popular science background, Royal Swedish Academy of Sciences. read · primary
- Ackley, D., Hinton, G. and Sejnowski, T. A Learning Algorithm for Boltzmann Machines. Cognitive Science 9, 147 (1985). cited via prior Institute record
- Implicit Behavioral Cloning. Florence et al., Conference on Robot Learning 2021, arXiv:2109.00137. read · abstract and result summary
- Denoising score matching and energy parameterisation: the equivalence of diffusion training and EBM score matching, and the dominance of score parameterisation in practice. Composite of the review literature located in this sweep. read · secondary review
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv:2303.04137; International Journal of Robotics Research 44 (2025), with its subsequent survey literature for the ten to one hundred evaluation figure. read · abstract and survey
- Generalized Model Predictive Path Integral Control as Expectation Maximization, arXiv:2606.00317; Model Predictive Path Integral Control as Preconditioned Gradient Descent, arXiv:2603.24489; Model Predictive Control via Probabilistic Inference: A Tutorial and Survey, arXiv:2511.08019. read · abstracts
- Werthen-Brabants, L. and Simoens, P. Ising Machines for Model Predictive Path Integral-Based Optimal Control. arXiv:2512.15533 (2025). read · full record checked for hardware and energy content, both absent
- Analog in-memory computing attention mechanism for fast and energy-efficient large language models. arXiv:2409.19315; Nature Computational Science (2025). reported · vendor-independent journal, figures not reproduced here
- Scellier, B. and Bengio, Y. Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation. arXiv:1602.05179 (2016), with 2026 hardware implementations including arXiv:2606.13454 and arXiv:2606.09112. read · abstracts
- Hamakawa, Y., Kashimata, T., Yamasaki, M. and Tatsumura, K. (Toshiba). Machine Learning-assisted High-speed Combinatorial Optimization with Ising Machines for Dynamically Changing Problems. arXiv:2503.23966; Nature Communications (2026). read · full record, latency and compression figures verified
- AI+HW 2035: Shaping the Next Decade. arXiv:2603.05225 (2026). read · full record, checked for energy-based model content
- Liu, Shi and Shin, arXiv:2511.20467, mobile robot power decomposition, via TR-2026-36. measured · third party
- Institute for Physical AI @ BMI, Charlot Lab, internal energy-based model bench series and total task energy bench. measured · ours
- Institute for Physical AI @ BMI, Post-von Neumann and Energy-Efficient Computing Paradigms for Physical AI at the Edge: A Survey, Technical Report TR-2026-01. The substrate taxonomy this report builds on and does not repeat. ours · prior report