Track Object Poses Using M3T
SUMMARY
M3TTracker follows the 6-DoF pose of a known rigid object in successive RGB-D frames. Supply its mesh and an initial mesh-to-camera pose; the tracker refines that pose using region and depth information. An optional texture modality adds image feature matching. It can track multiple instances of the same mesh.
Installation
The integration requires telekinesis-trackers 0.1.6 or newer. Version 0.1.6 pins the native extension to telekinesis-m3t 0.1.1, which includes custom modality settings and texture bindings:
python -m pip install "telekinesis-trackers[m3t]==0.1.6"If 0.1.6 is not yet available from your package index, use a trackers checkout containing the M3T integration and run python -m pip install -e ".[m3t]" from that checkout. Trackers 0.1.5 does not contain the integration.
Native 0.1.1 wheels support CPython 3.11–3.13 on Windows x86-64, Linux x86-64 (glibc 2.28+), and macOS ARM64 (macOS 15+). Tracking needs OpenGL 3.3 and a working graphics context. Headless Linux needs a virtual display such as Xvfb. The ONNX [gpu] extra and CUDA are not required for M3T.
Import from the M3T submodule:
from telekinesis.trackers.m3t import M3TTrackerThis checks the import, not OpenGL initialization or tracking.
Inputs and Coordinates
| Input | Format |
|---|---|
mesh_path | Path to a known mesh, for example part.obj |
mesh_scale | Metres per mesh coordinate unit: 1.0 for metres, 0.001 for millimetres |
K | Colour camera intrinsics, a (3, 3) NumPy array |
size | Colour frame dimensions in (width, height) order |
rgb | uint8 RGB array (height, width, 3) |
depth_m | Depth in metres, shaped (height, width), aligned to the colour camera |
| Initial poses | One (4, 4) transform per instance, mapping mesh coordinates in metres into the colour camera frame |
Use the colour camera calibration for the supplied image resolution. Align depth to the colour image before passing it to the tracker; the wrapper does not register separate camera views. Convert OpenCV BGR images to RGB. For raw depth, multiply by the camera's actual depth scale to obtain metres.
Pose translation is always in metres. mesh_scale scales the mesh geometry, not the translation in an initial pose. M3T does not detect an object's initial pose: obtain it from a pose estimator such as FoundationPose or another initialization method.
The Skill
The following example assumes you have calibrated K, aligned RGB-D frames, and a measured initial_pose:
from telekinesis.trackers.m3t import M3TTracker
height, width = first_rgb.shape[:2]
tracker = M3TTracker(
"part.obj",
mesh_scale=1.0,
K=K,
size=(width, height),
)
tracker.initialize(first_rgb, first_depth_m, [initial_pose])
# rgbd_frames yields subsequent (RGB image, aligned depth in metres) pairs.
for iteration, (rgb, depth_m) in enumerate(rgbd_frames, start=1):
poses = tracker.step(rgb, depth_m, iteration)
mesh_to_camera = poses[0] # A 4x4 transform; translation is in metres.step() returns a NumPy array shaped (count, 4, 4). Pose order matches the initial pose order. Keep frame dimensions and calibration fixed throughout the session.
For multiple instances, set count=N on construction and pass exactly N initial poses to initialize(). All instances share one mesh and its precomputed models.
Custom Settings and Texture Tracking
Pass native modality parameter names in region_settings and depth_settings. Set texture_settings={} to enable texture tracking with native defaults, or provide texture parameters:
tracker = M3TTracker(
"part.obj",
mesh_scale=1.0,
K=K,
size=(width, height),
region_settings={"n_lines_max": 100},
depth_settings={
"n_points_max": 100,
"standard_deviations": [0.03, 0.01, 0.005],
},
texture_settings={"orb_n_features": 200},
)These values illustrate configuration, not a recommended calibration for every camera. Depth standard deviations are in metres. Omitted settings retain native defaults. texture_settings=None disables the texture modality. Unknown parameter names raise ValueError.
Constructor Parameters
| Parameter | Default | Description |
|---|---|---|
mesh_path, mesh_scale, K, size | Required | Mesh, units, calibration, and frame dimensions described above |
count | 1 | Number of instances of the same mesh; use a positive integer |
cache | None | Model cache root; defaults to ~/.cache/telekinesis-m3t |
divides | 3 | Model-view subdivision level |
image_size | 1000 | Resolution used to generate model views, not camera input size |
model_points | 200 | Stored contour/surface points per model view |
rotation_regularization | 1000.0 | Rotation regularization passed to the native optimizer |
translation_regularization | 30000.0 | Translation regularization passed to the native optimizer |
region_settings, depth_settings | None | Native parameter dictionaries for region and depth modalities |
texture_settings | None | Enable and configure texture tracking when a dictionary is supplied |
Models are generated from the mesh on construction and cached in a subdirectory keyed by mesh contents, scale, and model-generation settings. First construction can take longer; later instances reuse matching cached models. M3T does not download an ONNX bundle.
Methods and Recovery
| Method | Behavior |
|---|---|
initialize(rgb, depth_m, poses) | Set the starting poses and initialize tracking; one pose per instance is required |
step(rgb, depth_m, iteration) | Process the next frame and return (count, 4, 4) mesh-to-camera transforms |
set_images(rgb, depth_m) | Update native camera images without running a tracking step; normally called internally |
The wrapper converts RGB to native BGR and metric depth to uint16 millimetres. Invalid, non-positive, or out-of-range depth values become zero.
Native setup or tracking failures raise RuntimeError. step() also rejects non-finite poses and poses with non-positive camera Z translation. These checks do not detect every tracking loss; a plausible pose can still drift. If tracking is lost, obtain fresh initial poses and call initialize() again. There is no reset_session() method for M3T. Construct a new tracker when changing the mesh, calibration, frame dimensions, or instance count.