Segment Image Using SAM

SUMMARY

Segment Image Using SAM segments objects inside given bounding boxes using SAM (Segment Anything Model).

For each box in bboxes, SAM predicts a precise segmentation mask of the object inside it — a deep-learning alternative to segment_image_using_grab_cut (a classical color-model algorithm) that generally produces cleaner masks, especially on complex textures/backgrounds, at the cost of needing a model inference call. A common source for bboxes is an object detector, e.g. retina.detect_objects_using_yolox/retina.detect_objects_using_rfdetr.

Use this Skill when you want to segment precise per-object masks from bounding box prompts, especially on complex textures or backgrounds where classical color-model segmentation struggles.

The Skill

from telekinesis import cornea

segmentation_results = cornea.segment_image_using_sam(
    image=image,
    bboxes=[[40, 70, 330, 414]],
    mask_threshold=0.5,
)

API Reference

Full parameter and return type documentation for segment_image_using_sam.

View Reference →

Example

Input Image

Original image for SAM segmentation

Output Image

SAM's predicted segmentation mask for the boxed object

The Code

"""
Demonstrates segmentation using SAM (Segment Anything Model).
"""

from loguru import logger
import rerun as rr

from telekinesis import cornea, datatypes

def segment_image_using_sam_example():
    """Segments an image using the Segment Anything Model (SAM)."""
    # ===================== Load Image ==========================================
    image_url = "https://assets.telekinesis.ai/examples/v1/images/pedestrians.jpg"
    image = datatypes.Image.from_url(url=image_url)

    # ===================== Run Skill ==========================================
    bboxes = [[40, 70, 330, 414]]
    segmentation_results = cornea.segment_image_using_sam(
        image=image, bboxes=bboxes, mask_threshold=0.5
    )

    # ===================== Log ================================================
    logger.success(f"Segmented {image} using SAM.")
    logger.success(f"Results: {segmentation_results}")
    logger.info(f"Number of segmented objects: {len(segmentation_results)}")

    # ===================== Visualization  (Optional) ======================
    rr.init("segment_image_using_sam_example", spawn=True)
    datatypes.visualize(image, segmentation_results, entity_path="/Image/overlayed_segmentations")


if __name__ == "__main__":
    segment_image_using_sam_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/segmentation/segment_image_using_sam.py

Parameter Configuration

Key	Type	Default	Description
image	datatypes.Image | np.ndarray	required	Input image to segment, shape (H, W, 3)
bboxes	datatypes.Boxes2D | list[list[int]]	required	One box per object to segment, each [x1, y1, x2, y2] (top-left and bottom-right corners, in pixel coordinates)
mask_threshold	datatypes.Float | float	0.5	Probability threshold in [0.0, 1.0] used to binarize SAM's per-pixel mask probability map into the final 0/1 mask

Returns

Type	Description
datatypes.COCOObjectDetectionResults	A batch with one entry per box in bboxes, in the same order, each carrying SAM's predicted segmentation mask for that box. Use len(...) for the count, or index with results[i] to get a single COCOObjectDetectionResult with .bbox, .score (the mask's predicted quality), and .segmentation (the predicted mask, as an encoded COCO RLE).

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	image is not in the shape (H, W, 3)
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Cornea service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Cornea service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Cornea service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

segment_image_using_sam has one tunable parameter.

mask_threshold

• Controls: How much predicted mask probability a pixel needs before it's counted as foreground, when binarizing SAM's per-pixel probability map into the final 0/1 mask.
• Units: Probability, 0.0–1.0
• Default: 0.5
• Increase → stricter, smaller mask
• Decrease → looser, larger mask
• Typical range: 0.3–0.7

bboxes itself isn't tunable, but its quality directly caps the result: SAM only searches for a mask inside each box, so a tight, accurately-placed box around the object produces a much better mask than a loose or mis-placed one.

TIP

Best practice: Feed bboxes from an upstream object detector (e.g. retina.detect_objects_using_yolox/retina.detect_objects_using_rfdetr) when one is available, rather than hand-drawn boxes, for consistent results across a pipeline.

Where to Use the Skill

Common pipelines include:

• Detector-to-mask segmentation – Turning bounding boxes from an object detector into precise per-object masks
• Interactive segmentation – Segmenting an object from a user-drawn or user-selected box
• Multi-object segmentation – Segmenting several objects in one call by passing multiple boxes
• Robotic manipulation – Producing precise object masks for grasp planning or pose estimation

Alternative Skills

Skill	vs. Segment Image Using SAM
segment_image_using_grab_cut	GrabCut is a classical color-model algorithm; SAM is a deep model. Use GrabCut for a lighter-weight, box-guided cutout; use SAM on harder scenes (complex textures/backgrounds) where GrabCut's color-distribution model struggles.
segment_image_foreground_using_birefnet	BiRefNet needs no bounding box — it automatically finds the most salient object. Use SAM to segment specific objects via boxes; use BiRefNet for automatic, prompt-free foreground extraction.

When Not to Use the Skill

Do not use Segment Image Using SAM when:

• You don't have bounding boxes — SAM needs one box per object; use segment_image_foreground_using_birefnet for automatic, box-free segmentation, or an object detector (e.g. retina.detect_objects_using_yolox) to get boxes first
• Speed is critical or no GPU is available — SAM is a deep model and can be slow on CPU; consider segment_image_using_grab_cut for a lighter-weight classical alternative
• The box is loose or mis-placed — a poorly fitted box limits how well SAM can isolate the intended object

TIP

If SAM's mask bleeds outside the object or misses part of it, check bboxes before adjusting mask_threshold — a loose or mis-placed box caps mask quality regardless of the probability threshold.