Depalletizing with SAM

SUMMARY

Remove boxes from a pallet one at a time without collisions, even when boxes touch, overlap slightly, or carry tape and printed graphics that confuse a simple bounding box. segment_image_using_sam extracts a pixel-accurate mask per box, exposing the flat, interior region that's safe for suction picking.

Raw Sensor Input

Raw sensor input showing a pallet stacked with boxes.

Segmentation and Boxes

Segmented image with masks and bounding boxes for detected box on the pallet.

Code

from telekinesis import cornea

# `image` is a datatypes.Image (or np.ndarray) of the pallet.
# `bboxes` is one [x1, y1, x2, y2] box per box on the pallet — typically
# from an object detector or a user-drawn region around each box.
segmentation_results = cornea.segment_image_using_sam(
    image=image,
    bboxes=bboxes,
    mask_threshold=0.5,
)

# Each result carries a bounding box, a mask-quality score, and an
# encoded segmentation mask for that box — use the mask's interior,
# away from edges, corners, and tape seams, to choose a safe suction
# pick point.
for result in segmentation_results:
    box = result.bbox
    score = result.score
    mask = result.segmentation