Illusion - Parts in a Bin

Goal: Generate a physically simulated bin-picking dataset.

Level: Beginner

Time: ~15 minutes

Background

Bin picking is the canonical robotics perception task: parts lie in a container in arbitrary, overlapping poses, and the model has to separate them. Sampling poses in mid-air is not enough – the parts have to rest on each other the way they would in reality.

In this tutorial you will add a bin to the scene, mark the parts as physically active, and let a physics simulation settle them before each render. If you have not built a randomizer tree before, start with Flying Things.

The finished script is also available as examples/quickstart_parts_in_bin.py in the repository.

1. Register the Parts and the Bin

Create a file named quickstart_parts_in_bin.py:

"""Generate a bin-picking dataset."""

import numpy as np

from telekinesis.illusion.core.synthetic_data_generator import (
    SyntheticDataGenerator,
)
from telekinesis.illusion.core.context import Context
from telekinesis.illusion.types.object import Object
from telekinesis.illusion.sampler.camera_pose_sampler import volume_sampler
from telekinesis.illusion.randomizer.randomizer import Randomizer
from telekinesis.illusion.randomizer.randomizer_node import (
    ObjectPoseRandomizer,
    ObjectInstanceRandomizer,
    BackgroundRandomizer,
    MaterialRandomizer,
    CameraPoseRandomizer,
)
from telekinesis.illusion.writer.writer import CocoWriter
from telekinesis.illusion.viewer.shard_viewer import view_coco
from telekinesis.illusion.utils.assets import resolve_asset_dir


def main():
    # Create the context
    context = Context()

    assets_dir = resolve_asset_dir()

    # Add models to the context
    model_1_path = str(
        assets_dir / "models" / "mechanical_parts" / "gearwheel_1.glb"
    )
    context.add_model(
        model_1_path,
        object_name="part_1",
        min_number_instances=1,
        max_number_instances=3,
        active_in_simulation=True,
    )

    model_2_path = str(
        assets_dir / "models" / "mechanical_parts" / "pipe_fixture_1.glb"
    )
    context.add_model(
        model_2_path,
        object_name="part_2",
        min_number_instances=0,
        max_number_instances=1,
        active_in_simulation=True,
    )

    model_3_path = str(assets_dir / "models" / "bins" / "plastic_bin_2.glb")
    context.add_model(
        model_3_path,
        object_name="crate_2",
        min_number_instances=1,
        max_number_instances=1,
        collision_shape="MESH",
    )

    context.get_objects_by_name("crate_2").set_location(
        np.array([0.0, 0.0, -0.05])
    )

Two settings do the work here:

• active_in_simulation=True on the parts makes them fall and collide.
• collision_shape="MESH" on the bin gives it a concave collider, so parts land inside it. The default CONVEX_HULL would fill the opening and the parts would rest on a lid that is not there.

The bin itself stays out of the simulation – it is static geometry – and is moved slightly below the origin so the parts drop into it.

2. Randomize Instances

The parts and the container get separate instance randomizers, because the container count is fixed at exactly one:

    # Create the randomizer
    randomizer = Randomizer()

    # Add object instance ranodmizer
    object_instance_randomizer = ObjectInstanceRandomizer(
        target_objects=["part_1", "part_2"],
        min_num_total_objects=1,
        max_num_total_objects=4,
    )

    randomizer.add_randomizer(
        randomizer_node=object_instance_randomizer,
        node_name="instance_randomizer_objects",
    )

    # Add container instance ranodmizer
    container_instance_randomizer = ObjectInstanceRandomizer(
        target_objects=["crate_2"],
        min_num_total_objects=1,
        max_num_total_objects=1,
    )

    randomizer.add_randomizer(
        randomizer_node=container_instance_randomizer,
        node_name="instance_randomizer_containers",
    )

3. Drop the Parts

Poses are sampled above the bin. Physics takes them from there, so the sampled pose only has to be a plausible starting point:

    # Add object pose randomizer
    def sample_pose(obj: Object):
        """
        Randomly samples and applies a 6-DoF pose to an object.

        The object's location is sampled uniformly within an axis-aligned box
        centered around the origin.
        """
        obj.set_location(
            np.random.uniform((-0.15, -0.15, 0.15), (0.15, 0.15, 0.15))
        )
        obj.set_rotation(np.random.uniform((-180, -180, -180), (180, 180, 180)))

    object_pose_randomizer = ObjectPoseRandomizer(
        pose_sampling_function=sample_pose, target_objects=["part_1", "part_2"]
    )

    randomizer.add_randomizer(
        randomizer_node=object_pose_randomizer, node_name="pose_randomizer"
    )

4. Randomize Appearance

Parts and bin are made of different things, so they get one material randomizer each:

    # Add matrial randomizer
    material_randomizer = MaterialRandomizer(
        target_objects=["part_1", "part_2"], types=["metal"], context=context
    )

    randomizer.add_randomizer(
        randomizer_node=material_randomizer,
        node_name="material_randomizer_parts",
    )

    material_randomizer = MaterialRandomizer(
        target_objects=["crate_2"], types=["plastic"], context=context
    )

    randomizer.add_randomizer(
        randomizer_node=material_randomizer,
        node_name="material_randomizer_crate",
    )

    # Add background randomizer
    background_randomizer = BackgroundRandomizer(
        categories=["indoor/industrial", "indoor/studio"]
    )

    randomizer.add_randomizer(
        randomizer_node=background_randomizer, node_name="background_randomizer"
    )

5. Randomize the Camera

volume_sampler samples a camera position in a volume and points it at the scene, which suits a bin viewed from above better than a shell around the parts:

    # Add camera pose randomizer
    camera_pose_randomizer = CameraPoseRandomizer(
        pose_sampling_function=volume_sampler, number_of_views=2
    )

    randomizer.add_randomizer(
        randomizer_node=camera_pose_randomizer,
        node_name="camera_pose_randomizer",
    )

6. Generate with Physics

    # Create the writer
    writer = CocoWriter()

    # Create the data generator with context, randomizer and writer
    data_generator = SyntheticDataGenerator(
        context=context, randomizer=randomizer, writer=writer
    )

    # Generate data
    data_generator.generate(
        num_images=5, simulate_physics=True, save_blender_scene=False
    )

    # View data
    view_coco(writer.get_output_dir())


if __name__ == "__main__":
    main()

Run it:

python quickstart_parts_in_bin.py

simulate_physics=True settles the parts before each render. The simulation runs until the objects come to rest, within the configured time bounds, so scenes with more parts take longer.

INFO

If parts pass through the bin or end up outside it, check that the bin uses collision_shape="MESH" and that the sampled locations start above its opening.

Summary

You have:

• Registered physically active parts and a static bin with a concave collider.
• Sampled starting poses above the bin and let physics produce the final layout.
• Rendered a bin-picking COCO instance-segmentation dataset with per-part masks.

Next Steps

• Raise max_number_instances and max_num_total_objects for denser, more occluded bins.
• Tune the simulation time ranges on generate() if parts are still moving when the render starts.
• Move to a spec-driven run that shards, merges, and splits the dataset in Generating a Dataset with a Worker.