An open-vocab robot arm you drive with words.
A vision-language-action policy maps what it sees and what it's told into what it does. This one is tiny enough to run its whole forward pass in your browser, its language is open-vocabulary — type any instruction and a real language model embeds it on your device — and it controls an articulated arm through its joints. Tell it which block to reach and watch it look, then move. No server, no GPU, no install.
Now make it manipulate: “stack the red on the green.”
Reaching is one step. This policy does two: it grasps the block you name, carries it, and places it on the other one — a real manipulation sequence. The held block is drawn at the gripper, so the policy can see which phase it's in. One honest detail: mean-pooled sentence embeddings barely encode word order (“red on green” and “green on red” land at ~0.67, near chance), so the source block is parsed from the instruction and the policy is conditioned on the resolved source and target.
Held-out, full-episode: it grasps, carries, and stacks the correct block 94.0% of the time. Flip which block is the source and success against the original target drops to 22.8% — the language is doing the work. 55,666 parameters; the whole two-phase forward pass runs in plain JavaScript. Weights: ↗ nano-vla-stack.
Trained on the actual sim, not a drawing.
Every policy above learns from a renderer we wrote. This one learns from the real thing: 2,500 frames captured live from the Forge MuJoCo arm running on WebGPU, each paired with the action the checkpoint took. It reads three stacked frames — motion matters — and predicts the joint move from pixels alone. On held-out reaches it explains 82.9% of the checkpoint's action variance (46.6% from a single frame; the jump is the velocity you can only see across frames).
An honest limit: this is a replay on captured frames, not a live loop. Pulling the render back into JS is the catch — the 2D/screenshot path reads blank on a WebGPU canvas (it's double-buffered), and WebGPU's proper readback (copyTextureToBuffer) is real but currently unreliable in three.js's WebGPU backend under a live render loop. So external capture is today's dependable path; a live in-page loop is impractical, not impossible. Weights + the capture recipe: ↗ nano-vla-pixels.
Now run the real 450M SmolVLA in your browser.
The policy above is tiny, to teach the idea. But the actual frontier is also runnable here: SmolVLA — Hugging Face's 450-million-parameter vision-language-action model — running its full flow-matching pipeline entirely on your GPU through WebGPU. No server touches it. The weights (831 MB, fp16) stream from our Hugging Face repo, onnxruntime-web runs them on WebGPU, and it produces a real 50-step action trajectory. Verified end-to-end in the browser.
See, be told, act.
Three modalities in one policy. This one is built so it genuinely needs all three.
It sees the arm and the blocks
A 40×40 perspective camera view — a receding tabletop with a 2-link arm, a gold gripper, and a red and a green block at positions that change every scene. Foreshortened, camera-like, not a flat schematic. It has to look to find them.
It obeys what you type
Any instruction works. It's embedded on your device by a real language model (all-MiniLM) — the same encoder used at training time — so paraphrases it never saw, “grab the green one”, “head toward crimson”, still land.
It drives the arm's joints
Out come two joint commands. The arm is stepped forward and the loop closes in the browser — the whole forward pass, three convolution layers and all, in plain JavaScript. No server, no GPU.
Tiny, and honestly measured.
Evaluated closed-loop — driving the arm joint by joint — on held-out scenes, with instructions it never trained on.
The novel-instruction number is the open-vocab proof: unseen phrasings still work, because the on-device text encoder maps them near the ones it learned. The flipped number is the language-grounding proof: with the wrong instruction it drives to the other block, so success at reaching the named one collapses to zero. It was trained by behavior cloning on a scripted Jacobian-IK expert and made robust to closed-loop drift with DART-style state-noise injection — the same idea as DAgger. A small reference policy on a rendered task, not a foundation model.
Weights, and the family it belongs to.
nano-vla-arm-3d
Weights, config (incl. the baked perspective homography), and the model card. The flat-camera predecessor is nano-vla-arm.
forge-arm-reach-bc
Our first released policy — the state-based reach controller, runnable on the page.
Vision-Language-Action
The hands-on course: make vision, language, and an action head yourself, in your browser — the exact arc behind these models, honest findings and all.