82.9% when we STACKED frames. Why? The action\n# depends on MOTION, and one still frame can't show it. Here's the mechanism.\nr=np.random.default_rng(1); N=500\nposn=r.uniform(0,1,(N,2)); vel=r.uniform(-.1,.1,(N,2)) # the action to predict = velocity\nprev=posn-vel # where it was one frame ago\nfrom numpy.linalg import lstsq\nA1=np.c_[posn,np.ones(N)]; w1,*_=lstsq(A1,vel,rcond=None); e1=float(((A1@w1-vel)**2).mean())\nA2=np.c_[posn,prev,np.ones(N)]; w2,*_=lstsq(A2,vel,rcond=None); e2=float(((A2@w2-vel)**2).mean())\nprint(f\"one frame -> velocity MSE {e1:.3e}\")\nprint(f\"two frames -> velocity MSE {e2:.3e}\")\nassert e2 < 0.2*e1, \"stacking frames should make the motion recoverable\"\nprint(\"PASS - a single frame can't show motion; stacked frames reveal velocity. That is exactly why the real-pixel VLA stacks 3 frames. Now open /research/vla and run the released models.\")","label":"Why one frame isn't enough — stack them"}],"intro":"Compare one-frame and two-frame prediction to see why motion needs memory.","key":"vision-language-action/motion","kind":"python","title":"Motion needs memory"}">
PYTHON · NUMPY · IN-BROWSER
Motion needs memory
Compare one-frame and two-frame prediction to see why motion needs memory.
You read
the arrays and values already in scope
You change
the code you write in each cell
Fixed
the dataset and the checks that grade you