telekinesis-illusion: Synthetic Data Generation for Physical AI
SUMMARY
telekinesis-illusion generates physically simulated, perfectly labeled synthetic datasets. You describe how a scene may vary – which parts appear, where they land, what they are made of, what is behind them, where the camera looks – and it renders that distribution into training-ready COCO or YOLO datasets.
What is telekinesis-illusion?
telekinesis-illusion is the synthetic data generation engine of the Data Engine. It solves the cold-start problem in Physical AI: a new Skill needs training data, but with no deployment yet there is nothing to produce it. Instead of collecting and annotating images by hand, you define a randomizer tree – a graph of randomizer nodes, each responsible for one axis of variation – and generate as many labeled scenes as the model needs.
Because the scene is authored rather than captured, every annotation is exact: instance masks, bounding boxes, and categories come out of the renderer, not out of a labeling tool. It renders through a bundled, modified BlenderProc that uses the bpy package, so no separate Blender installation is required.
telekinesis-illusion is designed for physical AI and computer vision applications, including:
- Bin picking and part detection: instance segmentation of mechanical parts settled by physics inside a container.
- Cold-start model training: a labeled dataset for a new Skill before any deployment exists to collect from.
- Domain randomization: instance count, object pose, material, background HDRI, and camera pose as independent axes of variation.
- Rare and unsafe configurations: scenes that are expensive, dangerous, or impractical to stage physically.
- Dataset scaling: sharded generation across a full use case, merged and split into train, valid, and test without a manual annotation pass.
Why telekinesis-illusion?
If you have hit these walls collecting real training data, telekinesis-illusion addresses them directly:
- Manual Annotation Cost – Every real image has to be segmented by hand before it can train anything. Masks, boxes, and categories come straight out of the renderer, so annotation is not a step in the pipeline at all.
- Label Noise – Hand-drawn masks disagree between annotators and drift on thin or occluded geometry. Segmentation output is keyed on
category_id,instance, andname, so a label is exact by construction. - No Data Before Deployment – A new Skill needs a model, and the model needs data that only a running deployment produces. A randomizer tree produces the dataset from the CAD models you already have.
- Long-Tail Coverage – Heavy clutter, unusual lighting, and rare part combinations are the cases a model fails on and the hardest to capture. Each is a bound on a randomizer node.
- Separate Blender Install – Rendering pipelines usually mean installing and version-matching Blender across every machine. Rendering goes through a bundled
bpypackage, sopip installis the whole setup and Blender is only needed for the optional tuning extension. - Reproducibility – A dataset assembled by hand cannot be regenerated. A spec YAML is the dataset definition: the same spec produces the same distribution, and a changed bound is a diff.
If you have felt these pains too, telekinesis-illusion is the way to go for you.
What Pain Points does telekinesis-illusion Solve?
| Pain Point | telekinesis-illusion Solution | Engineering Benefit |
|---|---|---|
| Manual Annotation Cost | Real datasets require a human segmentation pass per image. Masks, boxes, and categories are written by the renderer. | No annotation budget, no annotation backlog between a model idea and a trained model. |
| Label Noise | Hand-drawn masks are inconsistent on thin, reflective, or occluded geometry. Segmentation is keyed on category_id, instance, and name. | Pixel-exact ground truth, so model error is model error rather than label error. |
| No Data Before Deployment | A new Skill has no deployment to collect from. A randomizer tree generates the dataset from existing CAD models. | Removes the cold-start dependency between deployment and training. |
| Long-Tail Coverage | Clutter, lighting, and rare part combinations are hard to stage and rare to capture. Each is a bound on a randomizer node. | Failure cases become a parameter change rather than a data collection campaign. |
| Rendering Setup | Rendering pipelines normally require a matched Blender install per machine. Rendering runs through a bundled bpy package. | Reproducible environments, and CI or headless machines need no GUI application. |
| Dataset Reproducibility | A hand-assembled dataset cannot be rebuilt. A spec YAML defines the whole dataset. | Datasets are versioned, diffable, and regenerable at any size. |
| Scaling a Dataset | Long runs risk losing finished work and outgrowing disk. Generation is sharded, size-capped, and merged afterwards. | Interruptions never discard completed scenes, and dataset size stays bounded. |
Install
telekinesis-illusion is installed from source and ships with a small default asset collection, so the examples run without any downloads.
git clone https://github.com/telekinesis-ai/telekinesis-illusion.git
cd telekinesis-illusion
# Install the bundled BlenderProc, then telekinesis-illusion
pip install -e ./BlenderProc
pip install -e .Components
| Component | Description |
|---|---|
| Context | The scene state shared by every randomizer node: registered models, categories, materials, background, and camera. |
| Randomizer | The randomizer graph executed over the context, including per-node configuration and stage-scoped re-randomization. |
| SyntheticDataGenerator | The generation loop: randomize, optionally simulate physics, render, and write. |
| Randomizer Nodes | The individual axes of variation: instance count, object pose, material, background, and camera pose. |
| Workers | Spec-driven, sharded dataset generation for a complete use case, configured entirely in YAML. |
| Dataset Utilities | Merge shards, split into train/valid/test, convert to COCO or YOLO, and inspect the result in FiftyOne. |
Output Formats
| Format | Layout | Use With |
|---|---|---|
| COCO with RLE masks | merged_coco_annotations.json + per-shard images/ | Any COCO-compatible trainer, instance segmentation |
"coco" split | train/, valid/, test/, each with _annotations.coco.json | RF-DETR, COCO-compatible trainers |
"yolo" split | <split>https://assets.telekinesis.ai/telekinesis-documentation/public/images/, <split>/labels/, data.yaml | Ultralytics YOLO-seg |
How to Use telekinesis-illusion?
To use the synthetic data engine, simply use:
from telekinesis import illusionHere is a minimal example. A dataset is built in four steps: register the models, describe how the scene varies, generate, and inspect.
Register models on a Context
The Context holds the scene. add_model() loads a model, assigns it a COCO category, and pre-creates linked duplicates so instance counts can be randomized without reloading geometry.
from telekinesis.illusion.core.context import Context
from telekinesis.illusion.utils.assets import resolve_asset_dir
context = Context()
assets_dir = resolve_asset_dir()
context.add_model(
str(assets_dir / "models" / "mechanical_parts" / "gearwheel_1.glb"),
object_name="part_1",
min_number_instances=1,
max_number_instances=3,
)
context.add_model(
str(assets_dir / "models" / "mechanical_parts" / "pipe_1.glb"),
object_name="part_2",
min_number_instances=1,
max_number_instances=1,
)Build the randomizer tree
One node per axis of variation, appended in the order they should run: instances first, then poses, then appearance, then the camera.
import numpy as np
from telekinesis.illusion.randomizer.randomizer import Randomizer
from telekinesis.illusion.randomizer.randomizer_node import (
ObjectInstanceRandomizer,
ObjectPoseRandomizer,
MaterialRandomizer,
BackgroundRandomizer,
CameraPoseRandomizer,
)
from telekinesis.illusion.sampler.camera_pose_sampler import shell_sampler
from telekinesis.illusion.types.object import Object
def sample_pose(obj: Object):
"""Sample a 6-DoF pose uniformly inside a box around the origin."""
obj.set_location(np.random.uniform((-0.1, -0.1, 0.1), (0.1, 0.1, 0.1)))
obj.set_rotation(np.random.uniform((-180, -180, -180), (180, 180, 180)))
randomizer = Randomizer()
randomizer.add_randomizer(
randomizer_node=ObjectInstanceRandomizer(
target_objects=["part_1", "part_2"],
min_num_total_objects=2,
max_num_total_objects=4,
),
node_name="instance_randomizer_objects",
)
randomizer.add_randomizer(
randomizer_node=ObjectPoseRandomizer(
pose_sampling_function=sample_pose,
target_objects=["part_1", "part_2"],
),
node_name="pose_randomizer",
)
randomizer.add_randomizer(
randomizer_node=MaterialRandomizer(
target_objects=["part_1", "part_2"], types=["metal"], context=context
),
node_name="material_randomizer",
)
randomizer.add_randomizer(
randomizer_node=BackgroundRandomizer(
categories=["indoor/industrial", "indoor/studio"]
),
node_name="background_randomizer",
)
randomizer.add_randomizer(
randomizer_node=CameraPoseRandomizer(
pose_sampling_function=shell_sampler,
number_of_views=2,
radius_min=0.5,
radius_max=0.7,
),
node_name="camera_pose_randomizer",
)Generate
The generator runs the loop: randomize the scene, render it, write the annotations. With number_of_views=2, every randomized scene is rendered from two viewpoints.
from telekinesis.illusion.core.synthetic_data_generator import SyntheticDataGenerator
from telekinesis.illusion.writer.writer import CocoWriter
writer = CocoWriter()
data_generator = SyntheticDataGenerator(
context=context, randomizer=randomizer, writer=writer
)
data_generator.generate(num_images=5, save_blender_scene=False)Inspect the dataset
from telekinesis.illusion.viewer.shard_viewer import view_coco
view_coco(writer.get_output_dir())The dataset is written to ./output/synthetic_data_<timestamp>/, containing the rendered PNGs and a coco_annotations.json with RLE instance masks.
For a complete dataset – sharded generation, merging, and a train/valid/test split – drive a Worker from a spec YAML instead of assembling the tree in Python.
Tutorials
Flying Things
Scatter parts in mid-air against randomized backgrounds and render a labeled COCO dataset.
Explore →Parts in a Bin
Drop parts into a bin, settle them with physics, and render a bin-picking dataset.
Explore →Generating a Dataset with a Worker
Run a spec-driven dataset end to end: shards, merging, and train/valid/test splits.
Explore →Tuning a Randomizer Tree in Blender
Load a spec, edit the randomizer tree with a live preview, and export the tuned YAML.
Explore →