PYTHON · NUMPY

Act: build the VLA

You combine vision and language into a policy that sees two targets and is told which one to reach. What has to be true of its action?

Combine vision and language into a policy that sees both targets, obeys the instruction, and acts toward the named one.

01Challenge

Assemble a vision-language-action policy and confirm its action points at the target you named.

Predict firstYou combine vision and language into a policy that sees two targets and is told which one to reach. What has to be true of its action?

02Model

Now the loop closes. You have vision that finds targets and language that names one; the action half joins them into a single function: from a frame and an instruction, out comes a move. This is the definition of a VLA, and you are about to build one in a dozen lines. It is deliberately transparent (the vision is a colour locator, the action a step toward the chosen target) so you can see every part. Our released nano-VLAs learn this same mapping from data with a convolution stack and an MLP; the shape of the thing is exactly what you build here.

03Guided practice

1. The scene. Two targets, red and green, plus the end-effector. 2. Locate both. The vision half finds each colour's position. 3. See → be told → act. Pick the target the instruction names, and step toward it. That composition, vision and language into one action, is a vision-language-action policy.

04Feedback

PASS WHEN PASS when the action aligns with the named target (cosine > 0.9).

If your run fails, check:

Read the bench’s printed result, it passes when the action aligns with the named target (cosine > 0.9). Run the cells top to bottom; if one errors, the FAIL line names the step to fix.
The action should point at the target you named.

05Retrieve & space

What are the three parts of a VLA? (vision, language, action).
What does the action half output here? (a step toward the chosen target).
How do our released nano-VLAs learn this mapping? (from data, with a conv stack + MLP).

06Mastery & project

You built a working vision-language-action policy, see, be told, act.

The whole idea in one function: a frame and an instruction go in, a move comes out.

GO FURTHER · THE LIVE PLAYGROUND

Run the released nano-VLA ↗ · Type an instruction and watch the real released policy drive the arm, the trained version of what you just built.

← Be told: ground the instruction Does it obey? →

PYTHON · NUMPY

Act: build the VLA

Compose vision + language into an action and verify it aims at the named target.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you