Physical AI Agents: VLM/LLM-Powered Systems for Robot Planning and Code Generation
SUMMARY
A Physical AI Agent is a VLM/LLM system that turns a natural-language prompt into a runnable robot program. Following the Code-as-Policy paradigm, an Agent composes existing Skills into executable Python code — and generates new Skills when one doesn't exist yet. The result is readable code you can inspect, debug, and version-control, not opaque low-level commands.
Tzara executing vision-based pick and place from natural language—fully autonomous, no task-specific training.
What is a Physical AI Agent?
Physical AI Agentfrom telekinesis import cornea
from telekinesis.synapse import robot, gripper
# Perceive: segment the target part
mask = cornea.segment_image_using_sam(image, color="red")
pose = estimate_pose_from_mask(mask, intrinsics)
# Act: pick and place by composing Skills
robot.set_cartesian_pose(pose, speed=0.25)
gripper.close(force=50.0)
robot.set_cartesian_pose(place_pose, speed=0.25)
gripper.open()- A natural-language prompt and a robot view are sent to the agent.
- The agent reasons with a VLM / LLM, composing from 200+ Skills — and generating new Skills when one doesn't exist yet.
- It outputs readable, auditable Code-as-Policy you review before running.
A Physical AI Agent turns a natural-language goal into an executable robot program. Given a prompt — plus optional images and robot context — it:
- Interprets the instruction and the scene
- Reasons about the task with a Vision-Language Model (VLM) or Large Language Model (LLM)
- Generates executable Python that composes Telekinesis Skills — and writes new Skills when an existing one doesn't cover the task
TIP
The output of a Physical AI Agent is a production-grade robot program that can be deployed in real-world manufacturing, aerospace, logistics environments.
What is Code-as-Policy?
Code-as-Policy (CaP) is the paradigm behind our Physical AI Agents, introduced by Google Research (Liang et al., 2022): instead of a model driving motors directly, a VLM/LLM writes general-purpose Python code that controls the robot. The policy is code — the model reasons about the task, then generates a program that calls perception, motion planning, and control libraries to carry it out.
Here is how the framework works and its key features:
- Language to code, not language to actions. Rather than mapping an instruction straight to motor commands, the model writes executable Python (using the Telekinesis libraries) that processes perception and drives the hardware.
- Hierarchical generation. The model recursively defines helper functions and builds up custom libraries, scaling from a simple "stack the blocks" to complex logic, loops, and feedback.
- Spatial & semantic reasoning. Because it builds on a pretrained VLM/LLM, it resolves ambiguous descriptions and unseen instructions, translating them into parameterized robot primitives — with no task-specific training.
Built for production, not demos. Deploying robots in manufacturing, logistics, or aerospace demands systems an engineer can verify and stand behind — not a black box that works in a video and fails on the line. Because a Physical AI Agent's output is ordinary code, robot behavior inherits the full discipline of production software engineering:
- Auditable & testable — read the exact behavior, unit- and integration-test it, and validate it against your acceptance criteria before it ever touches hardware.
- Deterministic & reviewable — the same program runs the same way every time; commit it to version control, review the diff, and roll back on regression.
- Fits your existing stack — code review, CI/CD, linting, and simulation apply directly, so robot behavior is held to the same bar as the rest of your software.
- Human sign-off by default — nothing runs on a real robot until an engineer has read and approved the code.
- Portable across the fleet — the same program transfers to another robot once its Skills are implemented, instead of being retrained from scratch.
An opaque end-to-end policy offers none of this: you can't code-review a weight matrix, unit-test a black box, or explain to a safety auditor why it will behave. Code-as-Policy is engineered for teams that need repeatable, production-grade reliability in the field — where "it worked once in a demo" is not good enough.
For the full formalization and baselines, see Google's Code as Policies paper and interactive demo.
What Differentiates Code-as-Policy from VLAs & World Models?
Both take the same prompt and produce a policy — the difference is what that policy is made of.
A Vision-Language-Action (VLA) model or a World-Model-based controller is an end-to-end policy: a single neural network that maps observations (camera frames, language, robot state) straight to a stream of low-level actions. That behavior lives implicitly in the network's weights — an opaque, locked box, learned by training on large demonstration datasets.
Code-as-Policy makes the policy explicit — as code. Rather than generating raw actions, the Agent writes a human-readable program that composes modular Skills, so it can be inspected, tested, versioned, and changed without retraining. As shown above: one hands you an opaque policy of actions; the other, an editable policy as code.
On the axes that decide whether a robot actually ships, Code-as-Policy comes out ahead:
| What you need | End-to-End Policy (VLA / World Model) | Code-as-Policy (Telekinesis Agent) |
|---|---|---|
| Stand up a new task | Weeks — collect demonstrations, then train | Minutes — write a prompt |
| Fix a failure | Opaque — retrain and hope it generalizes | Open the code, find the line, edit it |
| Sign off before it runs | Can't audit network weights | Review and diff the program first |
| Data to get started | Large, task-specific datasets | Little to none — a pretrained VLM/LLM plus existing Skills |
| Hold up under change | Brittle outside the training distribution | Recombines Skills and re-plans over novel goals |
| Reuse across robots & tasks | Re-train per embodiment | Same program — swap in that robot's Skills |
| Get better over time | Frozen at training | Compounds — every new model becomes another Skill |
End-to-end models still win at one thing: low-level, dexterous, contact-rich control. Code-as-Policy doesn't fight that — it absorbs those models as Skills, which is exactly why it comes out ahead as the top-level controller. That's the subject of the next section.
Why Code-as-Policy Wins General-Purpose Robotics
End-to-end models, namely VLAs and World Action Models, are improving fast — and will commoditize. As these models proliferate across labs and open source, they will become interchangeable. The durable advantage moves up a layer — to whatever can reason and select the best model to solve the application.
Code-as-Policy is that layer. It doesn't compete with learned models; it ingests them as Skills. A VLA (or any learned visuomotor policy) is exposed as a Skill the Agent calls for the sub-tasks where end-to-end learning wins — dexterous, contact-rich control — while perception, sequencing, safety checks, and error handling stay as explicit, inspectable code. A World Model likewise becomes a Skill the Agent queries to predict dynamics or plan ahead.
model = Pi05()model.act(obs)Code-as-PolicyThat turns model progress into a compounding flywheel:
More models → More Skills → More applications
Every new model the field ships becomes another Skill in the library; every new Skill unlocks more tasks an Agent can compose; every deployed application feeds data back to sharpen the Skills. A single monolithic policy can't ride that curve — a composition layer over an open, growing Skill library can.
THE EVIDENCE
This isn't only a design bet — it measures better. NVIDIA's ASPIRE (2026), an agent that writes and refines its own Skills, reaches 31% on the perturbed LIBERO-Pro Long benchmark versus 4% for prior methods — with double-digit gains across bimanual and long-horizon tasks. See the research ↓
Tzara: The First Physical AI Agent
Tzara is Telekinesis's general-purpose coding agent that translates natural language instructions into executable Skill code. It operates in two modes:
Chat Mode
Interactive coding assistant for iterative refinement, debugging, and explanation of generated code.
- Best for: Exploring capabilities, debugging generated code, learning by example
Code Generation Mode
Direct conversion of prompts into Python code for fast pipeline generation.
- Best for: Quick prototyping, batch code generation, CI/CD pipelines
Tzara integrates directly into Visual Studio Code via a native extension, enabling seamless workflows. It sits on top of the Skill Library — orchestrating Skills into programs rather than replacing them, so every behavior stays a composition of tested, reusable capabilities.
Limitations
An Agent's authority is deliberately bounded — it produces code, not motion:
- Never executes on hardware directly. A human runs the reviewed code — safety stays with the operator.
- Requires human review before execution.
- Depends on the Skill Library. It composes and extends Skills, but can't reach hardware or capabilities that aren't exposed as Skills.
- Needs context to be correct. Vague prompts or missing robot context can yield incorrect code — clear instructions and calibration/hardware details matter.
Next Steps
Research Foundation
Optional reading — the peer-reviewed work behind Code-as-Policy and agentic Skill discovery.
Three papers, in chronological order, trace the arc from "language models can write robot code" to "agents can discover and refine their own Skills" — the exact approach Telekinesis is built on.
| Paper | Group · Year | What it established |
|---|---|---|
| Code as Policies | Google · 2022 | Language models can turn natural language into executable robot-control code |
| CaP-X | Berkeley · Stanford · NVIDIA · 2026 | Coding agents benchmarked at scale — and why abstractions (Skills) drive reliability |
| ASPIRE | NVIDIA · 2026 | Agents that autonomously write, refine, and reuse Skills |
Code as Policies — Google, 2022
Liang et al. showed that code-trained language models can be repurposed to write robot policies directly from language — composing perception and control libraries (NumPy, Shapely), reasoning about geometry, and generalizing to new instructions via few-shot prompting. It reached 39.8% on the HumanEval code-generation benchmark (state-of-the-art at the time) and was validated on real reactive and pick-and-place robots. This paper originated the Code-as-Policy paradigm.
CaP-X — Berkeley, Stanford & NVIDIA, 2026
CaP-X (Fu et al., with Fei-Fei Li, Ken Goldberg, and NVIDIA's Jim Fan) benchmarks coding agents for manipulation using CaP-Gym and CaP-Bench across 12 frontier language and vision-language models. Its central finding is the principle Telekinesis is built around: performance improves with well-designed abstractions and degrades as they are removed — a strong, typed Skill library is what makes coding agents reliable. It further shows that added test-time compute (multi-turn interaction, visual differencing) closes much of the remaining gap, with its CaP-Agent0 reaching human-level reliability on several tasks in simulation and on real robots.
ASPIRE — NVIDIA, 2026
ASPIRE (NVIDIA GEAR) takes the next step: an agent that autonomously writes and refines robot programs and accumulates a reusable Skill library, using execution monitoring, failure diagnosis, skill refinement, and evolutionary search. The gains under distribution shift and on long-horizon tasks are substantial:
| Benchmark | Prior methods | ASPIRE |
|---|---|---|
| LIBERO-Pro Long (perturbed) | 4% | 31% |
| LIBERO-Pro (under perturbation) | baseline | +77% |
| Robosuite bimanual handover | baseline | +72% |
| BEHAVIOR-1K (long-horizon household) | baseline | +32% |
Together, these results validate the Telekinesis architecture: Code-as-Policy as the control substrate, a typed Skill Library as the abstraction that makes it reliable, and agents that generate new Skills when the library falls short.