PYTHON · NUMPY · IN-BROWSER

When Cloning Drifts: DAgger

See why a cloned policy drifts on the states it visits, and fix it with DAgger — training the policy on its own distribution.

01Challenge

Behavior cloning drifts badly from a hard start it never saw. Run DAgger by raising ROUNDS until it at least halves the drift.

02Model

Behavior cloning has a quiet failure mode. It learns from the expert's states, but the moment the policy makes a small error it lands in a state the expert never visited, where it has no guidance, so it errs again, and the errors compound. This is why a cloned policy can look perfect on the demos and then drift off a cliff in the real rollout. DAgger fixes it with one idea: let the policy drive, gather the states it actually reaches, ask the expert what it should have done there, add those to the data, and retrain. Round after round, the policy learns on its own distribution, and the drift stops. This is exactly the ladder Forge walks you through: clone, watch the tail fail, then DAgger to recover.

03Guided practice

1. The trap. Clone from easy demonstrations near the goal, then start the policy far away. It visits states the demos never covered and drifts, its error measured on its own visited states is large.

2. The idea. DAgger runs the current policy, collects the states it reaches, and has the expert label them. Those labelled states join the dataset.

3. Aggregate and refit. Set ROUNDS to 3 and re-run. Each round trains the policy on the distribution it actually induces, and the drift at least halves.

04Feedback

PASS WHEN PASS when DAgger at least halves the cloned policy's drift on the states it visits.

Still drifting - raise ROUNDS (try 3) and re-run.
A cell errored - run them top to bottom.

05Retrieve & space

Why does behavior cloning compound errors? (a small mistake leads to states the expert never demonstrated).
What does DAgger collect and relabel? (the states the policy itself visits, labelled by the expert).
On whose distribution does DAgger train the policy? (the policy's own induced distribution).

06Mastery & project

You can explain and run DAgger to stop the compounding drift that plain behavior cloning suffers.

The robustness reflex: when a cloned policy drifts, gather the states it actually reaches and relabel them, don't just collect more expert demos.

← A Neural Policy, Trained by Gradient Descent Did It Learn, or Memorize? →

PYTHON · NUMPY · IN-BROWSER

When Cloning Drifts: DAgger

Watch behavior cloning drift off-distribution, then run DAgger to train the policy on its own states.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you