PYTHON · NUMPY · IN-BROWSER

Program vs. Policy

See why a hand-written controller falls short on a reach task, and why the data itself becomes the program.

01Challenge

A hand-coded controller uses one fixed gain on the target error. Tune the gain K to make it as good as you can, then read the gap to the expert it is trying to copy.

02Model

The old way to move a robot is to write the rule yourself: if the target is to the right, turn right. It works until the task has any subtlety, an obstacle to curve around, joints that couple. The new way is to show the robot what to do and let it fit the rule from the demonstrations. This is imitation learning, and its program is a table of what the robot saw and what an expert did. You will feel the ceiling of hand-coding here, so the rest of the course has something to beat.

03Guided practice

1. The data. Each row is one control tick: a 7-number observation (three joint angles, the target offset, the obstacle offset) paired with the expert's 3-number joint move.

2. The hand rule. It only looks at the target error and ignores the joints and the obstacle, so it can never match the expert exactly. Raising K helps up to a point, then overshoots.

3. The lesson. No single gain wins. That gap is the room a learned policy will fill.

04Feedback

PASS WHEN PASS once the cells run: you have the observation→action data and a baseline error to beat.

A cell errored — run them top to bottom.
Try a few values of K and watch the hand-coded MSE move.

05Retrieve & space

What are the two halves of one row of imitation-learning data? (an observation and the expert's action).
Why can't the hand rule match the expert? (it ignores the joints and the obstacle).
What does the mean-baseline MSE represent? (the error of always predicting the average action).

06Mastery & project

You can frame a control task as observation→action data and state why hand-coding hits a ceiling.

The habit the course is built on: before writing a controller, ask whether the rule should be learned from demonstrations instead.

← Robot Learning: From Demonstrations to a Released Policy The Dataset Is the Program →

what the expert DID (3).\nobs, act = make_reach(seed=1)\nprint(\"observations:\", obs.shape, \" actions:\", act.shape)\nprint(\"a joint delta stays inside the +/-0.04 rad limit:\", float(np.abs(act).max()) <= 0.041)","label":"The reach task, as data"},{"code":"# A hand-written controller: \"turn each joint toward the target by a fixed gain.\"\n# This is the code you'd write WITHOUT learning. Tune K to make it as good as you can.\nK = 0.010 # <- TODO: try 0.02, 0.04, 0.08 ...\ndef hand_policy(o):\n # use only the target-error columns (3,4); ignore joints & obstacle\n te = o[:, 3:5]\n a = np.zeros((o.shape[0], 3))\n a[:,0] = K*te[:,0]; a[:,1] = K*te[:,1]; a[:,2] = K*(te[:,0]+te[:,1])\n return np.clip(a, -0.04, 0.04)\nhand_mse = float(((hand_policy(obs) - act)**2).mean())\nmean_base = float(((act - act.mean(0))**2).mean())\nprint(f\"hand-coded MSE {hand_mse:.2e} vs predict-the-mean {mean_base:.2e}\")","label":"A hand-written controller · tune K"},{"code":"# The point of the course: a LEARNED policy will beat both of these without you\n# hand-designing the rule. We just need the data to teach it.\nprint(\"hand-coded error is real, and hand-tuning K only goes so far.\")\nprint(\"PASS - you have the data and a baseline to beat.\")","label":"Why we'll learn instead"}],"intro":"Load the reach task as data, then hand-code a controller and measure how far it falls short of the expert.","key":"robot-learning/program-vs-policy","kind":"python","title":"Program vs. Policy"}">

PYTHON · NUMPY · IN-BROWSER

Program vs. Policy

Load the reach task as data, then hand-code a controller and measure how far it falls short of the expert.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you