Illusion - Generating a Dataset with a Worker

Goal: Generate, merge, and split a complete bin-picking dataset from a spec YAML.

Level: Intermediate

Time: ~20 minutes

Background

The quickstarts build a scene in Python, which is the right way to experiment. A production dataset is different: it runs for hours, has to survive interruptions, and has to end up in a format a trainer can read.

A Worker covers that. The whole dataset – models, categories, placement, camera, physics, output format – lives in one spec YAML. The worker generates the images in shards, merges them into a single annotation file, and splits them into train, valid, and test.

In this tutorial you will run the bundled bin-picking spec, then change it to produce your own dataset.

This tutorial uses telekinesis-illusion 0.2.6.

Quickstart

Create `configs/quickstart_bin_picking.yaml`

yaml

metadata:
  asset_directory: ../assets
  dataset_name: quickstart_bin_picking
  num_images: 4
  base_output_directory: output
  info:
    description: "Quick bin-picking dataset."
    version: "1.0.0"
    year: 2026
    contributor: "Telekinesis"
  licenses: []

shard:
  size: 2

models:
  - name: gearwheel_2
    id: 1
    supercategory: part
    category_name: gear
    path: models/mechanical_parts/gearwheel_2.glb
    instances: { min: 1, max: 2 }
    simulation: { active: true, collision_shape: CONVEX_HULL }
    scale: 1.0
    preprocess_model: true
    shading: AUTO_SMOOTH

  - name: plastic_bin_2
    id: null
    supercategory: container
    category_name: distractor
    path: models/bins/plastic_bin_2.glb
    instances: { min: 1, max: 1 }
    simulation: { active: false, collision_shape: MESH }
    scale: 1.0
    preprocess_model: true

instance_randomizer:
  part: { min: 1, max: 2 }
  container: { min: 1, max: 1 }

pose_sampling:
  strategy: grid
  params:
    min_height: 0.05
    max_height: 0.08
    face_sample_range: [0.1, 0.9]
    grid:
      rows: 1
      cols: 2
      layers: 1
      layer_spacing: 0.03
      xy_jitter: 0.05
      z_rotation_range: [-180.0, 180.0]
      shuffle: true

material_randomizer:
  part: [metal]
  container: [plastic]

background_randomizer:
  categories: [indoor/industrial]

camera:
  field_of_view: 0.691111
  image_width: 128
  image_height: 128

camera_pose_randomizer:
  sampler: volume_sampler
  number_of_views: 2
  params:
    distance_range: [0.5, 0.8]
    inplane_rot_min: -15.0
    inplane_rot_max: 15.0
    point_of_interst: [0.0, 0.0, 0.0]

renderer:
  device: CPU
  samples: 8
  image_format: JPEG
  jpg_quality: 90

physics_simulator:
  active: true
  min_simulation_time_range: [0.1, 0.2]
  max_simulation_time_range: [0.3, 1.0]
  check_object_interval: 0.5
  object_stopped_location_threshold: 0.01
  object_stopped_rotation_threshold: 1.0
  substeps_per_frame: 10
  solver_iters: 10
  verbose: false
  use_volume_com: false

output:
  max_size_gb: 1
  dataset_format: null

Create `generate_bin_picking_dataset.py`

Create the runner in the telekinesis-illusion repository root. It is intentionally small because the YAML owns the dataset configuration:

python

"""Generate a bin-picking dataset using a YAML configuration."""

import argparse
from collections.abc import Sequence
import os
from pathlib import Path
import sys


def main(argv: Sequence[str] = ()) -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument(
        "--spec_file",
        type=Path,
        default=Path("configs/quickstart_bin_picking.yaml"),
        help="Path to the bin-picking YAML configuration.",
    )
    parser.add_argument(
        "--preview",
        action=argparse.BooleanOptionalAction,
        default=True,
        help="Open the generated dataset in the interactive viewer.",
    )
    args = parser.parse_args(argv)

    os.environ["OPENCV_IO_ENABLE_OPENEXR"] = "1"

    from telekinesis.illusion.workers.bin_picking_worker import BinPickingWorker

    worker = BinPickingWorker(args.spec_file)
    worker.generate()
    worker.merge_shards(preview=args.preview)


if __name__ == "__main__":
    main(sys.argv[1:])

Generate and merge the example dataset

bash

python generate_bin_picking_dataset.py \
  --spec_file configs/quickstart_bin_picking.yaml --no-preview

Walkthrough

Command options

--spec_file selects the YAML file containing the scene, generation, and output configuration.
--no-preview skips the built-in preview; use --preview to open the merged dataset after generation.
Omitting --spec_file uses configs/quickstart_bin_picking.yaml.

The quickstart generates four images in two shards. Increase metadata.num_images and shard.size only after this small run succeeds.

Generate and merge from Python

The runner has three responsibilities:

Parse the spec path and preview flag.
Set OpenCV's EXR switch before importing Illusion, then construct BinPickingWorker from the YAML spec.
Generate restart-safe shards and merge them, optionally opening the merged dataset preview.

The YAML holds the actual scene and dataset configuration. The quickstart spec defines the assets, categories, instance counts, grid placement, camera, renderer, physics, sharding, and output behavior used below.

This generates four images of gearwheels in a plastic bin, writes two COCO shards, and merges their annotations.

The relative metadata.asset_directory path is resolved from the spec file's directory, so ../assets points to the repository's sample assets.

Read the output layout

The run writes shards under the dataset root, then a merged annotation file across them:

output/
└── quickstart_bin_picking/
    ├── merged_coco_annotations.json
    ├── shard_20260903_101500_a1b2c3d4/
    │   ├── coco_annotations.json
    │   └── images/
    │       ├── 000000.jpg
    │       └── ...
    └── shard_20260903_104512_e5f6a7b8/
        ├── coco_annotations.json
        └── images/

Each shard is a self-contained mini-COCO dataset, so a run that stops early still leaves usable data behind. Train against merged_coco_annotations.json, whose image paths resolve relative to the dataset root – not against the individual shards.

The quickstart uses output.dataset_format: null, so it stops after producing merged_coco_annotations.json. Set the format to coco or yolo to create training splits as described below.

Register models and categories

Copy the quickstart spec and change the target model. Every path in models[].path is relative to metadata.asset_directory:

yaml

metadata:
  asset_directory: ../assets
  dataset_name: my_bracket_dataset
  num_images: 200
  base_output_directory: output

shard:
  size: 50

min_number_visible_models: 1
max_number_visible_models: 6
models:
  - name: bracket_01
    id: 1
    supercategory: part
    category_name: bracket
    path: models/mechanical_parts/bracket_01.glb
    instances: { min: 0, max: 6 }
    simulation: { active: true, collision_shape: CONVEX_HULL }
    scale: 1.0
    preprocess_model: True

  - name: plastic_bin_2
    id: None
    supercategory: container
    category_name: distractor
    path: models/bins/plastic_bin_2.glb
    instances: { min: 0, max: 1 }
    simulation: { active: false, collision_shape: MESH }
    scale: 1.0
    preprocess_model: True

The supercategory is what routes an asset through the randomizers: part objects are placed in the container, container objects are picked one per scene, and distractor objects use their own visibility bounds.

Setting id: None with category_name: distractor keeps the bin out of the annotations. Give a distractor a real id and category_name instead if you want the model to learn it as a class.

Configure randomization and physics

The same spec controls how instances are placed and rendered. This excerpt uses the current grid sampler, two camera views, per-role instance and material policies, two randomized lights, and physics:

yaml

pose_sampling:
  strategy: grid
  params:
    min_height: 0.05
    max_height: 0.08
    face_sample_range: [0.1, 0.9]
    grid:
      rows: 3
      cols: 2
      layers: 1
      layer_spacing: 0.03
      xy_jitter: 0.1
      z_rotation_range: [-180.0, 180.0]
      shuffle: true

camera_pose_randomizer:
  sampler: volume_sampler
  number_of_views: 2
  params:
    distance_range: [0.2, 1.5]
    inplane_rot_min: -30.0
    inplane_rot_max: 30.0
    point_of_interst: [0.0, 0.0, 0.0]

instance_randomizer:
  part: { min: 1, max: 5 }
  container: { min: 1, max: 1 }
  distractor: { min: 5, max: 10 }

material_randomizer:
  part: [metal]
  container: [plastic]
  distractor: [metal]

background_randomizer:
  categories: [indoor/industrial, outdoor/day]

physics_simulator:
  active: true
  min_simulation_time_range: [0.1, 0.2]
  max_simulation_time_range: [0.3, 1.0]
  check_object_interval: 0.5
  object_stopped_location_threshold: 0.01
  object_stopped_rotation_threshold: 1.0
  substeps_per_frame: 10
  solver_iters: 10

The grid provides deterministic starting cells inside the bin, while physics produces the final resting poses. Instance bounds must fit the grid capacity: this grid has six cells. The complete example spec also defines light_randomizer.lights, with randomized area and spot lights positioned above the selected container.

Choose the output format

Add the split configuration so the dataset comes out training-ready:

yaml

output:
  shard_name_template: "shard_{date}_{uuid}"
  max_size_gb: 50
  dataset_format: yolo      # None, coco, or yolo
  train_val_tes_ratio: [0.7, 0.2, 0.1]
  stratify: True
  seed: 42

With dataset_format: yolo, merging also writes an Ultralytics YOLO-seg dataset with a data.yaml. Use coco for RF-DETR and other COCO trainers, or None to keep only the merged COCO file and split it later with DatasetConverter.

max_size_gb is a safety cap: generation stops after the first shard that pushes the dataset past it.

Run and inspect a modified spec

bash

python generate_bin_picking_dataset.py \
  --spec_file my_bracket_spec.yaml --no-preview

To inspect the result later in FiftyOne, install the optional viewer extra, set DATASET_DIR at the top of examples/view_dataset.py to the dataset root, and run:

bash

python -m pip install -e ".[viewer]"
python examples/view_dataset.py

The script detects the layout, prints a per-class distribution table across the splits, and opens the FiftyOne app. Check the table before training: a class that appears in the spec but barely appears in the table usually means its instance bounds or the grid capacity are too tight.

Summary

You have:

Run a complete, sharded dataset from a spec YAML.
Read the shard and merged output layout, and understood which file to train on.
Retargeted the spec to a different part and chosen the output dataset format.

Next Steps

Work through the full spec YAML reference to tune placement, camera, and physics.
Tune the randomizer tree interactively before committing to a long run, in Tuning a Randomizer Tree in Blender.
Browse the published synthetic datasets generated with this pipeline.

Illusion - Generating a Dataset with a Worker ​

Background ​

Quickstart ​

Create configs/quickstart_bin_picking.yaml ​

Create generate_bin_picking_dataset.py ​

Generate and merge the example dataset ​

Walkthrough ​

Command options ​

Generate and merge from Python ​

Read the output layout ​

Register models and categories ​

Configure randomization and physics ​

Choose the output format ​

Run and inspect a modified spec ​

Summary ​

Next Steps ​

Illusion - Generating a Dataset with a Worker

Background

Quickstart

Create `configs/quickstart_bin_picking.yaml`

Create `generate_bin_picking_dataset.py`

Generate and merge the example dataset

Walkthrough

Command options

Generate and merge from Python

Read the output layout

Register models and categories

Configure randomization and physics

Choose the output format

Run and inspect a modified spec

Summary

Next Steps