PYTHON · NUMPY

A Neural Policy, Trained by Gradient Descent

Your multi-layer policy trains but lands short of the target. The architecture matches the released checkpoint. What is the first thing to change?

Build a small multi-layer policy with the same shape as the released checkpoint, and train it by gradient descent until it beats the linear floor.

01Challenge

The learning rate is too small to converge in the given epochs. Raise it until the neural policy explains at least 95% of the action variance.

Predict firstYour multi-layer policy trains but lands short of the target. The architecture matches the released checkpoint. What is the first thing to change?

02Model

The expert's law is curved, and a straight line can only approximate it. A multi-layer network with a tanh hidden layer can bend, so it captures what the linear policy misses. You will implement the whole thing by hand: the forward pass, the backward pass, and plain gradient descent, in a few lines of numpy. It is the same architecture as the released checkpoint, only smaller. The one knob that decides whether it learns in time is the learning rate.

03Guided practice

1. Z-score both ends. Normalize the inputs and the actions, so the loss is well-scaled.

2. The MLP. Seven inputs to a hidden layer of tanh units to three outputs. The forward pass is two matrix multiplies; the backward pass is the same multiplies in reverse.

3. Tune the rate. With LR=0.01 it barely moves in 400 epochs. Raise it (try 0.2) until the held-out variance-explained clears 95%.

04Feedback

PASS WHEN PASS when the trained MLP explains at least 95% of the held-out action variance (held-out MSE under 5% of baseline).

If your run fails, check:

Still short, the learning rate is too small; raise it toward 0.2.
A cell errored, run them in order.
If it diverges, ease the rate back down a little.

05Retrieve & space

Why does an MLP beat a linear policy here? (a tanh hidden layer can bend to the expert's curved law).
What does the learning rate control? (how far each gradient step moves the weights).
What shape is this policy? (7 → hidden → 3, same as the released checkpoint).

06Mastery & project

You can implement and train a neural policy from scratch and tune it to convergence.

The core skill of modern robot learning: fit a differentiable policy to demonstrations by gradient descent.

← Linear Behavior Cloning When Cloning Drifts: DAgger →

PYTHON · NUMPY

A Neural Policy, Trained by Gradient Descent

Implement and train a tiny MLP policy by gradient descent. Raise the learning rate until it converges.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you