PYTHON · NUMPY

The machine that follows gradients

Watch a learner improve by computing the exact gradient of its error and stepping downhill — and meet the engine that makes it exact: reverse-mode automatic differentiation, the keystone under every method in this course.

01Challenge

A learner has one move: measure how wrong it is, then change its parameters in the direction that reduces the error fastest — the negative gradient. But where does the gradient come from? Guessing it by nudging each parameter (finite differences) is slow and noisy. The Bench drives ferromotion's real reverse-mode autodiff Tape over a simple surface f(x,y) = (x−3)² + 2(y+1)². Read the two gradient vectors — the tape's exact ∇f and the finite-difference ∇f — and confirm they agree; then run the descent and drive the point to the minimum at (3, −1). This one move — exact gradient, step downhill — is how every network in this course learns.

02Model

Learning is minimizing a loss. To minimize, you need the gradient — the direction of steepest increase — and you step the other way. Reverse-mode autodiff (backpropagation) computes that gradient exactly and cheaply: it records every operation of the forward pass on a tape, then walks the tape backward applying the chain rule, so ONE backward sweep gives the derivative of the loss with respect to every parameter at once. Finite differences would need one extra forward pass per parameter and still carry step-size error. The valley here is steeper in y than in x, so the exact gradient points slightly off the straight line to the minimum — and the descent path curves. That curve is the whole reason gradients, not values, are what learning follows.

03Guided practice

Step A (read): the Bench shows f, the tape gradient ∇f, and the finite-difference gradient side by side — note that ‖tape − finite-diff‖ is ~1e-6, not zero, because finite differences are only approximate. The tape is exact. Step B (feel the direction): the gold arrow is −∇f, the descent direction; it does not point straight at the minimum because the bowl is anisotropic. Step C (descend): press Step, then Run 60. Watch the value fall and the point spiral into (3, −1). Step D (tune): raise the learning rate and re-run — too large and the steps overshoot and zig-zag; too small and it crawls. The step size is the first hyperparameter you will ever tune.

04Feedback

PASS WHEN PASS when the descent converges to the minimum — meaning the exact tape gradient guided every step correctly.

If your run fails, check:

Press Step, then Run 60, to walk the point downhill along the negative gradient.
If the path zig-zags and diverges, lower the learning-rate slider — the steps are overshooting the valley.

05Retrieve & space

Why is one reverse-mode backward pass enough to get the gradient with respect to every parameter?
Why does the tape gradient differ from the finite-difference gradient by ~1e-6 instead of exactly zero?
The descent path curves instead of heading straight at the minimum — what property of the surface causes that?

06Mastery & project

You drove the real autodiff engine: exact gradients, one backward pass, step downhill. Every learner in this course — PINNs, Lagrangian nets, Neural ODEs, differentiable controllers — is this same loop wrapped around a physical prior.

Feeds the next lesson, where these gradients train a small network (an MLP), and underlies every module after: the tape you just watched is what makes a physics loss or a learned Lagrangian trainable.

GO FURTHER · THE LIVE PLAYGROUND

ferromotion-learn on the ferromotion library ↗ — the reverse-mode Tape and forward-mode Dual/HyperDual you just used, in pure Rust.

← Physics-Informed Physical AI The universal approximator →

PYTHON · NUMPY

The machine that follows gradients

Live ferromotion-wasm: reverse-mode autodiff driving gradient descent. Run the descent to the minimum; watch the exact tape gradient track the finite-difference gradient the whole way.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you