Bin Picking with SAM

SUMMARY

Identify and pick individual items from a bin where objects are randomly arranged, touching, or partially occluded — conditions where a bounding box alone can't guarantee a pick point belongs to a single object. segment_image_using_sam extracts an instance-level mask per box, separating overlapping items so the pick point stays on one object and clear of its neighbors.

Raw Sensor Input

Raw sensor input showing randomly arranged parts in a bin.

Segmentation and Boxes

Segmented image showing object masks and bounding boxes for the detected part.

Code

from telekinesis import cornea

# `image` is a datatypes.Image (or np.ndarray) of the bin.
# `bboxes` is one [x1, y1, x2, y2] box per item — typically from an
# object detector or a user-drawn region around each part in the bin.
segmentation_results = cornea.segment_image_using_sam(
    image=image,
    bboxes=bboxes,
    mask_threshold=0.5,
)

# Each result carries a bounding box, a mask-quality score, and an
# encoded segmentation mask for that item — use the mask's interior,
# away from its edges, to choose a safe pick point.
for result in segmentation_results:
    box = result.bbox
    score = result.score
    mask = result.segmentation