Track Points Using TAPNext++

SUMMARY

TapNextPPTracker tracks points causally: seed a frame with pixel coordinates, then process each subsequent frame with step(). Supply either an exported ONNX bundle directory or a local PyTorch checkpoint.

ONNX Backend

Install the base tracker package using the setup guide. ONNX inference does not require PyTorch or tapnet. Supply a complete TAPNext++ bundle, including its manifest and graphs:

from telekinesis.trackers import TapNextPPTracker

tracker = TapNextPPTracker("bundles/tapnextpp/512-dynamic", device="cpu")

The first parameter is named checkpoint, but accepts either a bundle directory or a checkpoint file. There is no default model download in this constructor.

To create a bundle from a trackers source checkout, install its export requirements in a separate export environment and run:

python -m pip install -r scripts/requirements-export.txt
python scripts/prepare_bundles.py --models tapnextpp --points 2 --smoke-test

The exporter downloads the official checkpoint if needed. --points 2 sets the trace example; current dynamic bundles accept different positive point counts across sessions. Export and its smoke test can require substantial CPU time and memory.

Checkpoint Backend

Checkpoint inference requires a compatible PyTorch and torchvision installation, the einops dependency, a separately installed tapnet checkout, and local weights. Install the PyTorch build appropriate for your CPU or CUDA environment first, then:

python -m pip install "telekinesis-trackers[tapnextpp]"
python -m pip install -e "../tapnet"

The editable installation assumes a compatible tapnet checkout at ../tapnet. The tapnextpp extra installs einops only; it does not install PyTorch, torchvision, tapnet, CUDA, or model weights. ONNX Runtime GPU setup is not needed solely for checkpoint inference.

from telekinesis.trackers import TapNextPPTracker

tracker = TapNextPPTracker("tapnextpp_512.ckpt", device="cuda")

The checkpoint model loads lazily on load() or seed(). With device="auto", this backend uses CUDA when PyTorch exposes it, otherwise CPU.

The Skill

tracks = tracker.step(image_rgb)

This example uses an existing ONNX bundle and two local frames:

import numpy as np
from PIL import Image
from telekinesis.trackers import TapNextPPTracker

first_rgb = np.asarray(Image.open("frame_000.png").convert("RGB"))
next_rgb = np.asarray(Image.open("frame_001.png").convert("RGB"))
height, width = first_rgb.shape[:2]
points = np.array([
    [0.25 * (width - 1), 0.5 * (height - 1)],
    [0.75 * (width - 1), 0.5 * (height - 1)],
], dtype=np.float32)

tracker = TapNextPPTracker("bundles/tapnextpp/512-dynamic", device="cpu")
try:
    first_tracks = tracker.seed(first_rgb, points)
    tracks = tracker.step(next_rgb)
    print(tracks.xy[0], tracks.visible[0])
finally:
    tracker.unload()

Replace the sample coordinates with points on the features of interest. Results use the shared PointTracks format. Visibility is binary: visibility contains 0 or 1, and occlusion is its complement, rather than an independent confidence estimate.

Constructor Parameters

Parameter	Default	Description
checkpoint	Required	Local ONNX bundle directory or checkpoint file
input_resolution	512	Checkpoint input resolution; integer multiple of 8, at least 8. ONNX loading takes the resolution from its manifest
device	"auto"	"auto", "cpu", "cuda", or "cuda:N"
half_precision	False	Checkpoint backend only; requires CUDA
compile_model	False	Checkpoint backend only; enables compilation through its wrapper
threads	1	Positive thread count for ONNX Runtime

Match input_resolution to the checkpoint. half_precision and compile_model are rejected when loading an ONNX bundle.

Model and Session Lifecycle

Method or property	Behavior
load()	Load the selected backend without starting a session; repeated calls keep the loaded model
is_loaded	Whether the model is loaded
seed(image_rgb, query_points)	Load if necessary, clear old session state, and return predictions for the seed frame
step(image_rgb)	Advance the session by one frame
last_tracks	Most recent result, or None after reset
reset_session()	Clear recurrent state and numbering while retaining the loaded model
unload()	Release model and session state; also clear cached CUDA memory for a PyTorch CUDA backend

Current dynamic ONNX bundles and the checkpoint backend accept any positive point count, fixed for the duration of a session. Reseed to add or remove points. Older fixed-count bundles retain their exported count. Input frame dimensions must also remain constant within a session.

The model's processing resolution remains fixed even with dynamic point counts. The ONNX implementation stores recurrent state in NumPy between calls, including when its graphs execute on CUDA. The checkpoint backend keeps recurrent state on the selected device.

See Video and Live Examples for command-line use.