82.9% when we STACKED frames. Why? The action\n# depends on MOTION, and one still frame can't show it. Here's the mechanism.\nr=np.random.default_rng(1); N=500\nposn=r.uniform(0,1,(N,2)); vel=r.uniform(-.1,.1,(N,2)) # the action to predict = velocity\nprev=posn-vel # where it was one frame ago\nfrom numpy.linalg import lstsq\nA1=np.c_[posn,np.ones(N)]; w1,*_=lstsq(A1,vel,rcond=None); e1=float(((A1@w1-vel)**2).mean())\nA2=np.c_[posn,prev,np.ones(N)]; w2,*_=lstsq(A2,vel,rcond=None); e2=float(((A2@w2-vel)**2).mean())\nprint(f\"one frame -> velocity MSE {e1:.3e}\")\nprint(f\"two frames -> velocity MSE {e2:.3e}\")\nassert e2 < 0.2*e1, \"stacking frames should make the motion recoverable\"\nprint(\"PASS - a single frame can't show motion; stacked frames reveal velocity. That is exactly why the real-pixel VLA stacks 3 frames. Now open /research/vla and run the released models.\")","label":"Why one frame isn't enough — stack them"}],"intro":"Compare one-frame and two-frame prediction to see why motion needs memory.","key":"vision-language-action/motion","kind":"python","title":"Motion needs memory"}">
PYTHON · NUMPY · IN-BROWSER

Motion needs memory

Compare one-frame and two-frame prediction to see why motion needs memory.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you