Telekinesis Agentic Skill Library

Appearance

Illusion - Parts in a Bin ​

Goal: Generate a physically simulated bin-picking dataset.

Level: Beginner

Time: ~15 minutes

Background ​

The examples/quickstart_parts_in_bin.py script adds a static concave bin and physics-active parts to the flying-things pipeline. Objects begin above the container, settle under physics, and are then rendered with randomized materials, background, lights, and cameras.

This tutorial uses telekinesis-illusion 0.2.6.

Complete Install telekinesis-illusion, activate that environment, and run from the telekinesis-illusion repository root.

Quickstart ​

Create quickstart_parts_in_bin.py ​

Create the file in the telekinesis-illusion repository root:

python
"""Generate a bin-picking dataset."""

import argparse

import numpy as np

from telekinesis.illusion.core.engine import (
    BlenderEngine,
    PhysicsConfig,
    RenderConfig,
)
from telekinesis.illusion.core.scene import Scene
from telekinesis.illusion.core.synthetic_data_generator import (
    SyntheticDataGenerator,
)
from telekinesis.illusion.randomizer.randomizer import Randomizer
from telekinesis.illusion.randomizer.randomizer_node import (
    BackgroundRandomizer,
    CameraPoseRandomizer,
    LightPoseRandomizer,
    LightRandomizer,
    MaterialRandomizer,
    ObjectInstanceRandomizer,
    ObjectPoseRandomizer,
)
from telekinesis.illusion.sampler.camera_pose_sampler import (
    shell_sampler,
    volume_sampler,
)
from telekinesis.illusion.types.camera import CameraConfig
from telekinesis.illusion.types.distribution import uniform
from telekinesis.illusion.types.light import Light
from telekinesis.illusion.types.model_asset import ModelAsset
from telekinesis.illusion.types.object import Object
from telekinesis.illusion.utils.assets import resolve_asset_dir
from telekinesis.illusion.writer.writer import CocoWriter


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument(
        "--preview",
        action=argparse.BooleanOptionalAction,
        default=True,
        help="Open the generated dataset in the interactive viewer.",
    )
    parser.add_argument("--num-images", type=int, default=5)
    parser.add_argument("--output-dir", type=str, default=None)
    parser.add_argument("--resolution", type=int, default=720)
    parser.add_argument("--samples", type=int, default=None)
    parser.add_argument("--device", default="CUDA")
    parser.add_argument("--seed", type=int, default=None)
    args = parser.parse_args()

    if args.seed is not None:
        import random

        random.seed(args.seed)
        np.random.seed(args.seed)

    scene = Scene(
        camera_config=CameraConfig(
            image_width=args.resolution,
            image_height=args.resolution,
        )
    )
    assets_dir = resolve_asset_dir()

    scene.add(Light("softbox", "AREA", power=80.0, size=0.4))
    scene.add(Light("spot", "SPOT", power=20.0, radius=0.03))
    scene.add(
        ModelAsset(
            assets_dir / "models/mechanical_parts/gearwheel_1.glb",
            object_name="part_1",
            uv_mapping="cube",
            active_in_simulation=True,
        )
    )
    scene.add(
        ModelAsset(
            assets_dir / "models/mechanical_parts/pipe_fixture_1.glb",
            object_name="part_2",
            uv_mapping="cube",
            shading="SMOOTH",
            active_in_simulation=True,
        )
    )
    scene.add(
        ModelAsset(
            assets_dir / "models/bins/plastic_bin_2.glb",
            object_name="crate_2",
            uv_mapping="cube",
            collision_shape="MESH",
        )
    )

    randomizer = Randomizer()
    randomizer.add_randomizer(
        ObjectInstanceRandomizer(
            counts={"part_1": (1, 3), "part_2": (0, 1)},
            min_num_total_objects=1,
            max_num_total_objects=4,
        ),
        "instance_randomizer_objects",
    )
    randomizer.add_randomizer(
        ObjectInstanceRandomizer(
            counts={"crate_2": (1, 1)},
            min_num_total_objects=1,
            max_num_total_objects=1,
        ),
        "instance_randomizer_containers",
    )

    def sample_pose(obj: Object) -> None:
        obj.set_location(
            np.random.uniform((-0.15, -0.15, 0.15), (0.15, 0.15, 0.15))
        )
        obj.set_rotation(
            np.random.uniform((-180, -180, -180), (180, 180, 180))
        )

    randomizer.add_randomizer(
        ObjectPoseRandomizer(
            pose_sampling_function=sample_pose,
            target_objects=["part_1", "part_2"],
        ),
        "pose_randomizer",
    )
    randomizer.add_randomizer(
        MaterialRandomizer(
            target_objects=["part_1", "part_2"],
            types=["metal"],
        ),
        "material_randomizer_parts",
    )
    randomizer.add_randomizer(
        MaterialRandomizer(
            target_objects=["crate_2"],
            types=["plastic"],
        ),
        "material_randomizer_crate",
    )
    randomizer.add_randomizer(
        BackgroundRandomizer(
            categories=["indoor/industrial", "indoor/studio"]
        ),
        "background_randomizer",
    )
    randomizer.add_randomizer(
        LightRandomizer(
            ["softbox"],
            color=uniform((0.85, 0.85, 0.85), (1.0, 1.0, 1.0)),
            power=uniform(40.0, 100.0),
            shape=["SQUARE", "RECTANGLE", "DISK", "ELLIPSE"],
            size=uniform(0.3, 0.6),
            size_y=uniform(0.2, 0.4),
        ),
        "area_light_properties",
    )
    randomizer.add_randomizer(
        LightRandomizer(
            ["spot"],
            power=uniform(10.0, 30.0),
            radius=uniform(0.02, 0.05),
            spot_size=uniform(np.deg2rad(45), np.deg2rad(75)),
            spot_blend=uniform(0.3, 0.7),
        ),
        "spot_light_properties",
    )
    randomizer.add_randomizer(
        LightPoseRandomizer(
            shell_sampler,
            target_lights=["softbox", "spot"],
            center=["crate_2"],
            min_distance=0.3,
            radius_min=1.2,
            radius_max=1.6,
            elevation_min=50.0,
            elevation_max=80.0,
        ),
        "light_poses",
    )
    randomizer.add_randomizer(
        CameraPoseRandomizer(volume_sampler, number_of_views=2),
        "camera_pose_randomizer",
    )

    writer = CocoWriter(
        output_dir=args.output_dir,
        include_camera_metadata=True,
    )
    generator = SyntheticDataGenerator(
        scene=scene,
        randomizer=randomizer,
        writer=writer,
        engine=BlenderEngine(
            RenderConfig(device=args.device, samples=args.samples),
            PhysicsConfig(active=True),
        ),
    )
    generator.prepare()
    scene.get_objects_by_name("crate_2").set_location(
        np.array([0.0, 0.0, -0.05])
    )
    generator.generate(num_images=args.num_images)

    if args.preview:
        from telekinesis.illusion.viewer.shard_viewer import view_coco

        view_coco(writer.get_output_dir())


if __name__ == "__main__":
    main()

Generate a small dataset ​

bash
python quickstart_parts_in_bin.py --no-preview --device CPU --num-images 5 --resolution 128 --samples 8 --seed 17 --output-dir output/parts-in-bin

Walkthrough ​

Command options ​

  • --num-images 5 writes exactly five camera views.
  • --resolution 128 --samples 8 keeps the initial render small; the images will be noisy.
  • --device CPU runs without a CUDA-capable GPU.
  • --seed 17 makes Python and NumPy random choices repeatable.
  • --output-dir output/parts-in-bin selects the dataset directory.
  • --no-preview skips the interactive preview; use --preview to open it after generation.

Register the parts and container ​

python
scene = Scene(
    camera_config=CameraConfig(
        image_width=args.resolution,
        image_height=args.resolution,
    )
)
assets_dir = resolve_asset_dir()

scene.add(Light("softbox", "AREA", power=80.0, size=0.4))
scene.add(Light("spot", "SPOT", power=20.0, radius=0.03))
scene.add(
    ModelAsset(
        assets_dir / "models/mechanical_parts/gearwheel_1.glb",
        object_name="part_1",
        uv_mapping="cube",
        active_in_simulation=True,
    )
)
scene.add(
    ModelAsset(
        assets_dir / "models/mechanical_parts/pipe_fixture_1.glb",
        object_name="part_2",
        uv_mapping="cube",
        shading="SMOOTH",
        active_in_simulation=True,
    )
)
scene.add(
    ModelAsset(
        assets_dir / "models/bins/plastic_bin_2.glb",
        object_name="crate_2",
        uv_mapping="cube",
        collision_shape="MESH",
    )
)
  • The two parts are active rigid bodies, so physics moves them.
  • The bin is static and uses a concave MESH collider. A convex hull would cover the opening and prevent parts from falling inside.
  • Object names connect these assets to later randomizer nodes.

Select instances and starting poses ​

python
randomizer = Randomizer()
randomizer.add_randomizer(
    ObjectInstanceRandomizer(
        counts={"part_1": (1, 3), "part_2": (0, 1)},
        min_num_total_objects=1,
        max_num_total_objects=4,
    ),
    "instance_randomizer_objects",
)
randomizer.add_randomizer(
    ObjectInstanceRandomizer(
        counts={"crate_2": (1, 1)},
        min_num_total_objects=1,
        max_num_total_objects=1,
    ),
    "instance_randomizer_containers",
)

def sample_pose(obj: Object) -> None:
    obj.set_location(
        np.random.uniform((-0.15, -0.15, 0.15), (0.15, 0.15, 0.15))
    )
    obj.set_rotation(
        np.random.uniform((-180, -180, -180), (180, 180, 180))
    )

randomizer.add_randomizer(
    ObjectPoseRandomizer(
        pose_sampling_function=sample_pose,
        target_objects=["part_1", "part_2"],
    ),
    "pose_randomizer",
)
  • One to four parts are selected independently from the fixed container.
  • Starting locations are sampled above the bin, and physics determines the final resting poses.

Randomize appearance, illumination, and camera pose ​

python
randomizer.add_randomizer(
    MaterialRandomizer(["part_1", "part_2"], types=["metal"]),
    "material_randomizer_parts",
)
randomizer.add_randomizer(
    MaterialRandomizer(["crate_2"], types=["plastic"]),
    "material_randomizer_crate",
)
randomizer.add_randomizer(
    BackgroundRandomizer(
        categories=["indoor/industrial", "indoor/studio"]
    ),
    "background_randomizer",
)
randomizer.add_randomizer(
    LightRandomizer(
        ["softbox"],
        power=uniform(40.0, 100.0),
        size=uniform(0.3, 0.6),
    ),
    "area_light_properties",
)
randomizer.add_randomizer(
    LightRandomizer(
        ["spot"],
        power=uniform(10.0, 30.0),
        spot_size=uniform(np.deg2rad(45), np.deg2rad(75)),
        spot_blend=uniform(0.3, 0.7),
    ),
    "spot_light_properties",
)
randomizer.add_randomizer(
    LightPoseRandomizer(
        shell_sampler,
        target_lights=["softbox", "spot"],
        center=["crate_2"],
        radius_min=1.2,
        radius_max=1.6,
        elevation_min=50.0,
        elevation_max=80.0,
    ),
    "light_poses",
)
randomizer.add_randomizer(
    CameraPoseRandomizer(volume_sampler, number_of_views=2),
    "camera_pose_randomizer",
)
  • Parts receive metal materials, while the bin receives plastic materials.
  • The HDRI and light properties vary, and the lights remain above the container.
  • volume_sampler produces two camera views of each settled scene.

Enable physics and generate ​

python
writer = CocoWriter(
    output_dir=args.output_dir,
    include_camera_metadata=True,
)
generator = SyntheticDataGenerator(
    scene=scene,
    randomizer=randomizer,
    writer=writer,
    engine=BlenderEngine(
        RenderConfig(device=args.device, samples=args.samples),
        PhysicsConfig(active=True),
    ),
)
generator.prepare()
scene.get_objects_by_name("crate_2").set_location(
    np.array([0.0, 0.0, -0.05])
)
generator.generate(num_images=args.num_images)
  • prepare() imports the registered models before the fixed bin transform is applied.
  • The bin is moved five centimetres below the origin, then physics settles the active parts inside it.
  • The writer saves images, COCO annotations, and camera metadata. Physics makes this example slower than Flying Things.

Result ​

text
output/parts-in-bin/
  coco_annotations.json
  images/
    000000.jpg
    ...

A successful run reports five generated images. Open them and confirm that parts rest inside the bin in varied poses. Check coco_annotations.json for five image records and visible per-part annotations with RLE masks, boxes, and areas.

If parts pass through the container, confirm that the bin still uses collision_shape="MESH" and that sampled part locations begin above its opening. If parts are still moving, adjust the PhysicsConfig time ranges in the example.

Run with --preview to open the built-in COCO preview. Use a new output directory for a separate dataset because the writer appends to existing output.

Next Steps ​