Track Masks Using SAM3

SUMMARY

Sam3Tracker propagates seeded object masks through successive frames with a recurrent SAM3 ONNX bundle, preserving their object IDs. It uses the same seed() / execute() / correct() session interface as CUTIE, so the two are interchangeable in mask-tracking code.

Installation

Install the base package using the Trackers setup guide. SAM3 runs through ONNX Runtime on CPU or CUDA; no PyTorch installation is needed for inference.

Bundle required

There is no default SAM3 download and no SAM3 exporter. You must supply a compatible SAM3 ONNX bundle directory, or a URL to a bundle ZIP. An ordinary SAM3 checkpoint or an image-only ONNX graph is not sufficient.

A compatible bundle contains:

File	Contents
manifest.json	Format telekinesis-sam3-onnx, version 1, positive integer height, width, and objects, and a nonnegative state_count
seed.onnx	Inputs uint8 image (1, H, W, 3) and binary uint8 masks (1, O, H, W); outputs mask_logits (1, O, H, W), object_scores (1, O), and state_0 ... state_N
step.onnx	Inputs image followed by the recurrent states; outputs mask_logits, object_scores, and matching next_state_* tensors

The graphs own preprocessing and the recurrent memory update. Graph input and output names are checked against this contract on construction.

The Skill

labels, alive_fraction = tracker.execute(image_rgb)

Seed the tracker before calling execute(). Frames must match the bundle's fixed dimensions, so this example resizes them to the manifest's size:

import numpy as np
from PIL import Image
from telekinesis.trackers import Sam3Tracker

tracker = Sam3Tracker("bundles/sam3/default", device="cpu")
size = (tracker.config["width"], tracker.config["height"])

first_rgb = np.asarray(Image.open("frame_000.png").convert("RGB").resize(size))
next_rgb = np.asarray(Image.open("frame_001.png").convert("RGB").resize(size))
# A grayscale label image: 0 = background, 1 = object one, 2 = object two, ...
seed_labels = np.asarray(
    Image.open("labels.png").convert("L").resize(size, Image.Resampling.NEAREST)
)

tracker.seed(seed_labels, first_rgb)
labels, alive_fraction = tracker.execute(next_rgb)

if labels is not None:
    Image.fromarray(labels).save("tracked_labels.png")
print(f"Objects with nonempty masks: {alive_fraction:.0%}")

Resize label images with nearest-neighbor sampling so object IDs are not blended. For click-assisted seed masks, see RITM.

Constructor Parameters

Parameter	Default	Description
model_dir	Required	Local SAM3 bundle directory
model_url	None	Optional bundle ZIP URL; downloaded into model_dir when the bundle there is missing or incomplete
score_threshold	0.4	Value in [0, 1]; objects whose object_scores fall below it are suppressed for that frame
mask_threshold	0.0	Finite logit threshold; pixels whose best mask logit is at or below it become background
threads	1	Positive ONNX Runtime intra-operation thread count
device	"auto"	"auto", "cpu", "cuda", or "cuda:N"

All parameters after model_dir are keyword-only. tracker.config holds the parsed manifest, and tracker.device reports the selected device.

Session Methods

Method	Behavior
seed(labels, image_rgb, obj_ids=None)	Start a new session and return its label image. All nonzero IDs in the seed mask are tracked; if given, obj_ids must list exactly those IDs
execute(image_rgb)	Process the next frame and return (labels, alive_fraction)
correct(image_rgb, labels)	Re-prompt the current objects with a corrected mask and restart the recurrent state from it. Labels must use existing object IDs
reset_session()	Clear the tracking state

labels is a uint8 array (H, W) at the bundle's resolution, or None from execute() when no foreground remains. Among objects that pass score_threshold, each pixel takes the ID with the highest mask logit, unless that logit is at or below mask_threshold. alive_fraction is the fraction of seeded objects with nonempty output masks, not a confidence score.

Inspect is_seeded, object_ids, and last_labels for session state.

Bundle Dimensions

SAM3 bundles are fixed-shape. Every frame, seed mask, and correction must be exactly config["height"] by config["width"]; the tracker does not resize inputs. At most config["objects"] objects can be seeded, and never more than 255. Seeding fewer objects than the bundle supports is allowed.

Common Errors

• Calling execute() or correct() before seeding raises RuntimeError.
• Frames whose size differs from the bundle dimensions raise ValueError.
• More seeded objects than the bundle supports, corrections containing new object IDs, an unsupported manifest, or graphs that do not match the bundle contract raise ValueError.
• A missing or incomplete bundle without model_url raises FileNotFoundError.

For video and camera workflows, see Video and Live Examples.