PYTHON · NUMPY · IN-BROWSER

See: pixels to a place

Turn a camera frame into where things are — the first job of a VLA's vision.

01Challenge

Build a visual feature that locates the target in a 16×16 scene, and confirm its centroid sits on the target.

02Model

A vision-language-action policy starts with seeing. Its vision half turns a camera frame into a field of features — a learned stack of convolutions in the real thing, but the idea is simple enough to hold in your hand: a filter that responds to what matters, pooled down to a location. Here you build the smallest version, a colour detector, and watch it find the target. Once vision can say where things are, language can say which one, and the policy can act. That is the whole loop, and it begins here.

03Guided practice

1. The scene. A 16×16 RGB frame with one green target blob. 2. The feature. Green-minus-red-blue isolates the target. 3. Pool to a place. The centroid — the weighted centre of the feature — is the target's location. That pixels→location step is what every VLA's vision does, just with learned filters instead of a hand-written one.

04Feedback

PASS WHEN PASS when the feature centroid lands on the target (within a pixel).

A cell errored — run them top to bottom.
The centroid should sit on the green blob.

05Retrieve & space

What does a VLA's vision half produce? (features that say where things are).
Why pool to a centroid instead of a single peak? (it's robust to the blob's extent).
In a real VLA, what replaces the hand-written filter? (a learned convolution stack).

06Mastery & project

You can turn a scene into a location — the first half of perception in a VLA.

The habit under every VLA: before language or action, vision has to answer 'where is what?'

← Vision-Language-Action: Build a Policy You Can Talk To Be told: ground the instruction →

where things are.\ngreen = np.clip(img[:,:,1] - 0.5*(img[:,:,0]+img[:,:,2]), 0, None)\nys,xs = np.mgrid[0:16,0:16]; tot = green.sum()\ncx = float((green*xs).sum()/tot); cy = float((green*ys).sum()/tot)\nprint(\"feature centroid at\",(round(cx,1),round(cy,1)),\" true target\",(tx,ty))\nassert abs(cx-tx)<=1 and abs(cy-ty)<=1, \"the feature centroid should sit on the target\"\nprint(\"PASS - a visual feature turns pixels into a location. That is what 'vision' does in a VLA.\")","label":"A visual feature → a location"}],"intro":"Build a colour feature and locate the target by its centroid.","key":"vision-language-action/vision","kind":"python","title":"See: pixels to a place"}">

PYTHON · NUMPY · IN-BROWSER

See: pixels to a place

Build a colour feature and locate the target by its centroid.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you