Illusion - Flying Things

Goal: Generate a labeled COCO instance-segmentation dataset of parts scattered in mid-air.

Level: Beginner

Time: ~10 minutes

Background

In this tutorial you will build a "flying things" scene: two mechanical parts are scattered in mid-air, with no physics simulation, against randomized industrial and studio backgrounds. This is the simplest useful randomizer tree – every axis of variation is present, but nothing has to settle or rest on anything.

You will register models on a Context, add one randomizer node per axis of variation, and render five scenes with two camera views each.

This tutorial uses the assets bundled with telekinesis-illusion, so no downloads are needed. Complete Install telekinesis-illusion first.

The finished script is also available as examples/quickstart_flying_things.py in the repository.

1. Create the Context and Register Models

Create a file named quickstart_flying_things.py:

"""Generate a "flying things"-type dataset."""

import numpy as np

from telekinesis.illusion.core.synthetic_data_generator import (
    SyntheticDataGenerator,
)
from telekinesis.illusion.core.context import Context
from telekinesis.illusion.types.object import Object
from telekinesis.illusion.sampler.camera_pose_sampler import shell_sampler
from telekinesis.illusion.randomizer.randomizer import Randomizer
from telekinesis.illusion.randomizer.randomizer_node import (
    ObjectPoseRandomizer,
    ObjectInstanceRandomizer,
    BackgroundRandomizer,
    MaterialRandomizer,
    CameraPoseRandomizer,
)
from telekinesis.illusion.writer.writer import CocoWriter
from telekinesis.illusion.viewer.shard_viewer import view_coco
from telekinesis.illusion.utils.assets import resolve_asset_dir


def main():
    # Create the context
    context = Context()

    assets_dir = resolve_asset_dir()

    # Add models to the context
    model_1_path = str(
        assets_dir / "models" / "mechanical_parts" / "gearwheel_1.glb"
    )
    context.add_model(
        model_1_path,
        object_name="part_1",
        min_number_instances=1,
        max_number_instances=3,
    )

    model_2_path = str(
        assets_dir / "models" / "mechanical_parts" / "pipe_1.glb"
    )
    context.add_model(
        model_2_path,
        object_name="part_2",
        min_number_instances=1,
        max_number_instances=1,
    )

max_number_instances=3 pre-creates three linked duplicates of the gearwheel. Instance randomization then decides how many of them are visible in each scene, without reloading geometry.

2. Randomize How Many Objects Appear

Add the first randomizer node to the tree:

    # Create the randomizer
    randomizer = Randomizer()

    # Add obect instance randomizer
    object_instance_randomizer = ObjectInstanceRandomizer(
        target_objects=["part_1", "part_2"],
        min_num_total_objects=2,
        max_num_total_objects=4,
    )

    randomizer.add_randomizer(
        randomizer_node=object_instance_randomizer,
        node_name="instance_randomizer_objects",
    )

Each scene now shows between two and four objects, drawn from the two models. Instance randomization runs first, because every later node only acts on the objects that are actually visible.

3. Randomize Where They Are

The pose sampling function defines the distribution; the node handles collision checking:

    # Add object pose randomizer
    def sample_pose(obj: Object):
        """
        Randomly samples and applies a 6-DoF pose to an object.

        The object's location is sampled uniformly within an axis-aligned box
        centered around the origin.
        """
        obj.set_location(np.random.uniform((-0.1, -0.1, 0.1), (0.1, 0.1, 0.1)))
        obj.set_rotation(np.random.uniform((-180, -180, -180), (180, 180, 180)))

    object_pose_randomizer = ObjectPoseRandomizer(
        pose_sampling_function=sample_pose, target_objects=["part_1", "part_2"]
    )

    randomizer.add_randomizer(
        randomizer_node=object_pose_randomizer, node_name="pose_randomizer"
    )

Locations are sampled uniformly inside a 20 cm box, and orientations across the full rotation range. If an object cannot be placed collision-free, it is hidden for that scene and left out of the annotations.

4. Randomize Appearance

Two nodes cover appearance: the material of the parts, and the background HDRI that also lights the scene.

    # Add matrial randomizer
    material_randomizer = MaterialRandomizer(
        target_objects=["part_1", "part_2"], types=["metal"], context=context
    )

    randomizer.add_randomizer(
        randomizer_node=material_randomizer, node_name="material_randomizer"
    )

    # Add background randomizer
    background_randomizer = BackgroundRandomizer(
        categories=["indoor/industrial", "indoor/studio"]
    )

    randomizer.add_randomizer(
        randomizer_node=background_randomizer, node_name="background_randomizer"
    )

5. Randomize the Camera

    # Add camera pose randomizer
    camera_pose_randomizer = CameraPoseRandomizer(
        pose_sampling_function=shell_sampler,
        number_of_views=2,
        radius_min=0.5,
        radius_max=0.7,
    )

    randomizer.add_randomizer(
        randomizer_node=camera_pose_randomizer,
        node_name="camera_pose_randomizer",
    )

shell_sampler places the camera on a spherical shell 50 to 70 cm around the scene. With number_of_views=2, every randomized scene is rendered from two viewpoints.

6. Generate and View

    # Create the writer
    writer = CocoWriter()

    # Create the data generator with context, randomizer and writer
    data_generator = SyntheticDataGenerator(
        context=context, randomizer=randomizer, writer=writer
    )

    # Generate data
    data_generator.generate(num_images=5, save_blender_scene=False)

    # View data
    view_coco(writer.get_output_dir())


if __name__ == "__main__":
    main()

Run it:

python quickstart_flying_things.py

INFO

The first run can take a few minutes while the render kernels compile. Loading render kernels (may take a few minutes the first time) in the terminal is expected.

The dataset is written to ./output/synthetic_data_<timestamp>/, containing the rendered PNGs and a coco_annotations.json with RLE instance masks. The viewer opens automatically – use ←/→ to page through frames, and b, l, and m to toggle boxes, labels, and masks.

Summary

You have:

• Registered two models on a Context, with instance pools sized for randomization.
• Built a randomizer tree covering instance count, pose, material, background, and camera.
• Rendered a labeled COCO instance-segmentation dataset and inspected it.

Next Steps

• Widen the pose sampling box or the camera radius, and see how much the dataset changes.
• Add a bin and let physics settle the parts inside it, in Parts in a Bin.
• Move the whole configuration into a spec YAML with Generating a Dataset with a Worker.