PYTHON · NUMPY

See: pixels to a place

The first job of a VLA's vision is not recognising what a thing is. What is it?

Turn a camera frame into where things are, the first job of a VLA's vision.

01Challenge

Build a visual feature that locates the target in a 16×16 scene, and confirm its centroid sits on the target.

Predict firstThe first job of a VLA's vision is not recognising what a thing is. What is it?

02Model

A vision-language-action policy starts with seeing. Its vision half turns a camera frame into a field of features, a learned stack of convolutions in the real thing, but the idea is simple enough to hold in your hand: a filter that responds to what matters, pooled down to a location. Here you build the smallest version, a colour detector, and watch it find the target. Once vision can say where things are, language can say which one, and the policy can act. That is the whole loop, and it begins here.

03Guided practice

1. The scene. A 16×16 RGB frame with one green target blob. 2. The feature. Green-minus-red-blue isolates the target. 3. Pool to a place. The centroid, the weighted centre of the feature, is the target's location. That pixels→location step is what every VLA's vision does, just with learned filters instead of a hand-written one.

04Feedback

PASS WHEN PASS when the feature centroid lands on the target (within a pixel).

If your run fails, check:

Read the bench’s printed result, it passes when the feature centroid lands on the target (within a pixel). Run the cells top to bottom; if one errors, the FAIL line names the step to fix.
The centroid should sit on the green blob.

05Retrieve & space

What does a VLA's vision half produce? (features that say where things are).
Why pool to a centroid instead of a single peak? (it's robust to the blob's extent).
In a real VLA, what replaces the hand-written filter? (a learned convolution stack).

06Mastery & project

You can turn a scene into a location, the first half of perception in a VLA.

The habit under every VLA: before language or action, vision has to answer 'where is what?'

← Vision-Language-Action: Build a Policy You Can Talk To Be told: ground the instruction →

PYTHON · NUMPY

See: pixels to a place

Build a colour feature and locate the target by its centroid.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you