ACT: Action Chunking Transformer
SUMMARY
ACT predicts a short chunk of future actions from the current camera images and joint state, instead of one action at a time. Chunking absorbs the non-Markovian pauses and corrections in human teleoperation, and executing the overlapping chunks with temporal ensembling smooths the result into a single continuous trajectory.
That makes it the algorithm to reach for first: it fine-tunes on as few as 10-50 demonstrations per task and trains in minutes to a few hours on a single GPU.
When to Use ACT?
Reach for ACT when:
- The task is a single, well-defined manipulation skill - a pick, an insertion, a bimanual hand-off
- Demonstrations come from teleoperation on the robot itself, so there is no embodiment gap to bridge
- The demonstration set is small (tens, not thousands) and collecting more is expensive
- Fine-grained, contact-rich motion matters more than following open-ended language instructions
For a single pretrained policy that generalizes across tasks and takes a free-form language instruction, see π0.5 instead. For a task with real dynamics to get wrong - contact, momentum, deformable objects - see Fast-WAM.
Requirements
| Requirement | Value |
|---|---|
| Python | 3.11-3.12 |
| GPU | Recommended for training; a single consumer GPU is enough for most tasks |
| Demonstrations | A teleoperated dataset for the target task - camera frames, joint state, and the joint-space action taken at each step. Currently, only the LeRobot dataset format is supported |
Quick Installation
pip install "telekinesis-cerebellum[act]"The act extra brings the ACT model and its training loop. Everything else - dataset loading, checkpointing, and the deployment interface shared with π0.5 - comes with the base package.
Quickstart
Load your demonstration data from LeRobot
Pull a dataset of teleoperated episodes - local or from the LeRobot Hub - into the layout Cerebellum trains on:
from telekinesis.dataengine import load_lerobot_dataset
# Load a LeRobot dataset of teleoperated episodes
dataset = load_lerobot_dataset(
repo_id="telekinesis-ai/insert_gear", # local path or LeRobot Hub repo id
)
print(dataset)Create config.yaml
task: insert_gear
dataset:
path: demos/
cameras: [cam_high, cam_wrist]
chunk_size: 100 # actions predicted per forward pass
policy:
class_name: ACT
hidden_dim: 512
num_encoder_layers: 4
num_decoder_layers: 7
kl_weight: 10.0 # CVAE regularization
training:
num_epochs: 4000
batch_size: 8
learning_rate: 0.00001
device: auto
logger:
log_dir: logs
experiment: insert_gear_act
save_interval: 500Deploy
A finished run leaves a checkpoint under logs/insert_gear_act/. Load it and act in a closed loop: feed the current camera frames and joint state in, apply the next action from the predicted chunk, and re-plan as new observations arrive.
from telekinesis.cerebellum.policy import Policy
# Load Synapse robot
# Connect to the robot controller at the specified IP address
robot.connect(ip=robot_ip)
# Or connect to the matching articulation in a running Isaac Sim stage
robot.connect(simulation_prim_path=prim_path)
# Disconnect from the robot controller to cleanly release the session
robot.disconnect()
policy = Policy("logs/insert_gear_act/checkpoint_best.pt")
observation = robot.get_observation() # camera frames + joint state
action_chunk = policy.get_action(observation)
robot.execute(action_chunk)Tuning Notes
- Chunk size trades off reactivity against smoothness - shorter chunks correct sooner, longer chunks ride out sensor noise and mid-episode hesitation in the demonstrations more smoothly.
- KL weight controls how much the CVAE's latent style code is allowed to vary - raise it if the policy copies demonstration-specific quirks instead of the underlying skill.
- ACT overfits happily on small datasets; watch held-out rollouts rather than training loss alone to decide when to stop.