Trackers: Visual Tracking Skills
SUMMARY
Telekinesis Trackers follows objects and points across video frames. Initialize a mask, select query points, or supply an initial object pose on one frame, then process subsequent frames with a tracker that retains state between calls. RITM creates the initial object masks from clicks.
Choose a Tracker
| Component | Use it for | Initialization | Runtime |
|---|---|---|---|
| CUTIE | Propagating multiple object masks | An integer label image | ONNX Runtime, CPU or CUDA |
| SAM3 | Propagating object masks with a fixed-shape bundle | An integer label image and a local bundle | ONNX Runtime, CPU or CUDA |
| TAPIR | Online or offline point tracking | Pixel coordinates | ONNX Runtime, CPU or CUDA |
| TAPNext++ | Causal point tracking | Pixel coordinates and a local model | ONNX Runtime or PyTorch, CPU or CUDA |
| McByte | Assigning persistent IDs to detections | Per-frame bounding boxes and local bundles | ONNX Runtime, CPU only; mcbyte extra |
| M3T | Tracking 6-DoF object poses in aligned RGB-D frames | Mesh, camera intrinsics, and one initial pose per instance | Native CPU tracking with OpenGL; m3t extra |
| RITM | Creating an object's seed mask | Positive and optional negative clicks | ONNX Runtime; examples use CPU |
CUTIE and SAM3 produce a label image for each frame and share the same session interface. TAPIR and TAPNext++ produce point coordinates and visibility information. McByte returns your detections with persistent tracker IDs; it includes no detector. M3T returns mesh-to-camera pose transforms and requires an initial pose from another source. RITM operates on individual images and does not track across frames.
Installation
Use Python 3.11 or newer in an isolated environment:
python -m pip install telekinesis-trackersThe distribution is telekinesis-trackers; the Python import is telekinesis.trackers. The base package includes CPU ONNX Runtime, NumPy, Pillow, and requests, which is enough for CUTIE, SAM3, TAPIR, RITM, and TAPNext++ ONNX bundles. Other components need an optional extra:
| Extra | Adds | Needed for |
|---|---|---|
gpu | ONNX Runtime GPU and CuPy | CUDA execution of ONNX trackers; see GPU setup |
mcbyte | Upstream trackers[mask] and supervision | McByte |
m3t | Native telekinesis-m3t extension | M3T; requires trackers 0.1.6+, supported native wheels, and OpenGL |
tapnextpp | einops only | TAPNext++ checkpoint backend, together with separately installed PyTorch and tapnet |
cpu is an empty compatibility alias for the base installation. Install an extra with, for example, python -m pip install "telekinesis-trackers[mcbyte]".
Check the import without downloading or loading a model:
from telekinesis import trackers
print(trackers.CutieTracker)NVIDIA GPU Setup for ONNX
After installing the base package, replace CPU ONNX Runtime with its GPU distribution and install the CUDA libraries and CuPy:
python -m pip uninstall -y onnxruntime onnxruntime-gpu
python -m pip install "onnxruntime-gpu[cuda,cudnn]>=1.21,<1.27" "cupy-cuda12x[ctk]>=14,<15"This setup requires a CUDA 12-compatible NVIDIA driver. The runtime upper bound keeps this installation on CUDA 12. CuPy is used by CUTIE's CUDA memory attention and resizing.
ONNX Runtime package replacement
The CPU and GPU distributions share the same Python module directory. Installing the package's [gpu] extra alone installs both distributions; use the replacement commands above afterward. Because telekinesis-trackers declares the CPU distribution as a dependency, pip check reports it as missing after replacement. Upgrading or reinstalling the tracker package may restore it, requiring the replacement again.
Use device="cpu", device="cuda", or device="cuda:N" to select a device. For ONNX trackers, device="auto" selects CUDA when ONNX Runtime advertises that provider, otherwise CPU. CUTIE and TAPIR check that their sessions retain the selected CUDA provider. This does not guarantee that every graph operator executes on GPU. McByte always runs on CPU, so this setup does not affect it.
Model Loading
CUTIE, RITM, and TAPIR download default bundles on first construction and reuse them from ~/.cache/telekinesis/trackers/. The asset origin is https://assets.telekinesis.ai/trackers/.
| Component | Cache directory below the tracker cache |
|---|---|
| CUTIE | cutie/cutie-dynamic-spatial |
| RITM | ritm/480x864-20 |
| TAPIR online | tapir-online/causal-2 |
| TAPIR offline | tapir-offline/offline-2-2frames |
Pass model_dir to use an existing local bundle instead. Bundles contain a manifest and the model graphs; a single arbitrary .onnx file is insufficient. For use without network access, supply a complete local bundle or populate the default cache beforehand.
TAPNext++ requires a local ONNX bundle directory or checkpoint path, passed as its first constructor argument. It loads the model on load() or seed().
SAM3 and McByte have no default download: pass local bundle directories. They also accept a bundle ZIP URL (model_url, or sam_model_url and cutie_model_url), which is downloaded into the given directory when the bundle there is missing or incomplete.
Inputs and Results
M3T generates models from your mesh and caches them under ~/.cache/telekinesis-m3t; see M3T model settings. It does not use the ONNX bundle cache above.
All image inputs are NumPy uint8 RGB arrays shaped (H, W, 3). Convert OpenCV's BGR frames before calling a tracker. Frames must keep the same dimensions within an online tracking session.
| Input | Format |
|---|---|
| CUTIE and SAM3 seed labels | Integer array (H, W), with 0 for background and object IDs 1 through 255 |
| Query points and RITM clicks | (N, 2) pixel coordinates in (x, y) order, inside the input image |
| TAPIR offline video | uint8 RGB array (T, H, W, 3) |
| McByte detections | supervision.Detections with (x1, y1, x2, y2) pixel boxes |
| M3T depth | (H, W) depth in metres, aligned to the colour image |
| M3T poses | One (4, 4) mesh-to-colour-camera transform per instance; translations in metres |
PointTracks
TAPIR and TAPNext++ return PointTracks, with coordinates expressed in the original input image's pixels:
| Field | Shape | Meaning |
|---|---|---|
xy | (T, N, 2) | float32 positions in (x, y) order |
visibility | (T, N) | float32 visibility signal |
occlusion | (T, N) | float32 occlusion signal |
visible | (T, N) | Boolean visibility decisions |
frame_indices | (T,) | Integer frame indices, starting from zero for a new session |
Online calls return one frame (T = 1). Coordinates may leave the image bounds when a point moves out of view. TAPIR combines occlusion and tracking uncertainty to calculate visibility. TAPNext++ exposes thresholded visibility: its visibility values are 0 or 1, and occlusion is their complement.
For TAPIR and TAPNext++, call reset_session() before starting an unrelated sequence, or reseed with a new frame and selection. Resetting TAPNext++ clears tracking state but keeps the model loaded; unload() releases it.
M3T Poses
M3T uses initialize(rgb, depth_m, poses) followed by step(rgb, depth_m, iteration). Each step returns (count, 4, 4) mesh-to-camera transforms in initialization order. To recover after tracking loss, obtain fresh poses and call initialize() again. See the M3T guide for calibration, mesh units, and optional texture tracking.
Tracking Skills
tracker.execute()TrackersTrack Masks Using CUTIE
Seed multiple object masks and propagate their labels through RGB video frames with CUTIE on CPU or CUDA.
tracker.execute()TrackersTrack Masks Using SAM3
Propagate seeded object masks through RGB frames with a fixed-shape, recurrent SAM3 ONNX bundle on CPU or CUDA.
tracker.step()TrackersTrack Points Using TAPIR
Track selected pixel locations online or through an offline clip with TAPIR, returning coordinates, visibility, and occlusion.
tracker.step()TrackersTrack Points Using TAPNext++
Run causal TAPNext++ point tracking from a local ONNX bundle or PyTorch checkpoint, with explicit model and session lifecycle control.
tracker.update()TrackersTrack Detections Using McByte
Assign persistent IDs to per-frame object detections with McByte, using local SAM and CUTIE ONNX models for mask-guided association on CPU.
Track Object Poses Using M3T
Track known objects in aligned RGB-D frames with M3T, using a mesh, an initial pose, and region, depth, and optional texture modalities.
initializer.predict()TrackersInitialize Masks Using RITM
Create and refine an object mask from positive and negative clicks, then use it to seed CUTIE tracking.
For command-line workflows, see Video and Live Examples.