PYTHON · NUMPY · IN-BROWSER

A Neural Policy, Trained by Gradient Descent

Build a small multi-layer policy with the same shape as the released checkpoint, and train it by gradient descent until it beats the linear floor.

01Challenge

The learning rate is too small to converge in the given epochs. Raise it until the neural policy explains at least 95% of the action variance.

02Model

The expert's law is curved, and a straight line can only approximate it. A multi-layer network with a tanh hidden layer can bend, so it captures what the linear policy misses. You will implement the whole thing by hand: the forward pass, the backward pass, and plain gradient descent, in a few lines of numpy. It is the same architecture as the released checkpoint, only smaller. The one knob that decides whether it learns in time is the learning rate.

03Guided practice

1. Z-score both ends. Normalize the inputs and the actions, so the loss is well-scaled.

2. The MLP. Seven inputs to a hidden layer of tanh units to three outputs. The forward pass is two matrix multiplies; the backward pass is the same multiplies in reverse.

3. Tune the rate. With LR=0.01 it barely moves in 400 epochs. Raise it (try 0.2) until the held-out variance-explained clears 95%.

04Feedback

PASS WHEN PASS when the trained MLP explains at least 95% of the held-out action variance (held-out MSE under 5% of baseline).

Still short — the learning rate is too small; raise it toward 0.2.
A cell errored — run them in order.
If it diverges, ease the rate back down a little.

05Retrieve & space

Why does an MLP beat a linear policy here? (a tanh hidden layer can bend to the expert's curved law).
What does the learning rate control? (how far each gradient step moves the weights).
What shape is this policy? (7 → hidden → 3, same as the released checkpoint).

06Mastery & project

You can implement and train a neural policy from scratch and tune it to convergence.

The core skill of modern robot learning: fit a differentiable policy to demonstrations by gradient descent.

← Linear Behavior Cloning When Cloning Drifts: DAgger →

H -> 3 with a tanh hidden layer, trained by plain gradient\n# descent. This is the same shape as the released checkpoint (just smaller H).\nrng=np.random.default_rng(0); H=32\nW1=rng.normal(0,0.4,(7,H)); b1=np.zeros(H)\nW2=rng.normal(0,0.4,(H,3)); b2=np.zeros(3)\ndef fwd(Xb):\n z1=Xb@W1+b1; h=np.tanh(z1); return z1,h,h@W2+b2\nLR=0.01 # <- TODO: too small to converge in time. raise it (try 0.2)\nEPOCHS=400\nXtr,Ytr=X[tr],Y[tr]\nfor e in range(EPOCHS):\n z1,h,yh=fwd(Xtr); err=yh-Ytr\n gW2=h.T@err/len(Xtr); gb2=err.mean(0)\n gh=(err@W2.T)*(1-h**2)\n gW1=Xtr.T@gh/len(Xtr); gb1=gh.mean(0)\n W1-=LR*gW1; b1-=LR*gb1; W2-=LR*gW2; b2-=LR*gb2\n_,_,yhv=fwd(X[val]); val_mse=float((((yhv*ys+ym)-act[val])**2).mean())\nbase=float(((act[val]-act[tr].mean(0))**2).mean())\nprint(f\"MLP held-out MSE {val_mse:.2e} variance explained {100*(1-val_mse/base):.1f}%\")","label":"A tiny MLP, trained by gradient descent · tune LR"},{"code":"# The MLP's tanh layer captures the curved expert law a straight line cannot,\n# but only once it has actually converged (>= 95% of the action variance).\nassert val_mse < 0.05*base, \"raise LR (try 0.2) until the MLP explains >=95% of the variance\"\nprint(\"PASS - a trained neural policy beats the linear one.\")","label":"It beats linear"}],"intro":"Implement and train a tiny MLP policy by gradient descent. Raise the learning rate until it converges.","key":"robot-learning/a-neural-policy","kind":"python","title":"A Neural Policy, Trained by Gradient Descent"}">

PYTHON · NUMPY · IN-BROWSER

A Neural Policy, Trained by Gradient Descent

Implement and train a tiny MLP policy by gradient descent. Raise the learning rate until it converges.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you