Skip to content

Track Object Poses Using M3T ​

SUMMARY

M3TTracker follows the 6-DoF pose of a known rigid object in successive RGB-D frames. Supply its mesh and an initial mesh-to-camera pose; the tracker refines that pose using region and depth information. An optional texture modality adds image feature matching. It can track multiple instances of the same mesh.

Installation ​

The integration requires telekinesis-trackers 0.1.6 or newer. Version 0.1.6 pins the native extension to telekinesis-m3t 0.1.1, which includes custom modality settings and texture bindings:

sh
python -m pip install "telekinesis-trackers[m3t]==0.1.6"

If 0.1.6 is not yet available from your package index, use a trackers checkout containing the M3T integration and run python -m pip install -e ".[m3t]" from that checkout. Trackers 0.1.5 does not contain the integration.

Native 0.1.1 wheels support CPython 3.11–3.13 on Windows x86-64, Linux x86-64 (glibc 2.28+), and macOS ARM64 (macOS 15+). Tracking needs OpenGL 3.3 and a working graphics context. Headless Linux needs a virtual display such as Xvfb. The ONNX [gpu] extra and CUDA are not required for M3T.

Import from the M3T submodule:

python
from telekinesis.trackers.m3t import M3TTracker

This checks the import, not OpenGL initialization or tracking.

Inputs and Coordinates ​

InputFormat
mesh_pathPath to a known mesh, for example part.obj
mesh_scaleMetres per mesh coordinate unit: 1.0 for metres, 0.001 for millimetres
KColour camera intrinsics, a (3, 3) NumPy array
sizeColour frame dimensions in (width, height) order
rgbuint8 RGB array (height, width, 3)
depth_mDepth in metres, shaped (height, width), aligned to the colour camera
Initial posesOne (4, 4) transform per instance, mapping mesh coordinates in metres into the colour camera frame

Use the colour camera calibration for the supplied image resolution. Align depth to the colour image before passing it to the tracker; the wrapper does not register separate camera views. Convert OpenCV BGR images to RGB. For raw depth, multiply by the camera's actual depth scale to obtain metres.

Pose translation is always in metres. mesh_scale scales the mesh geometry, not the translation in an initial pose. M3T does not detect an object's initial pose: obtain it from a pose estimator such as FoundationPose or another initialization method.

The Skill ​

The following example assumes you have calibrated K, aligned RGB-D frames, and a measured initial_pose:

python
from telekinesis.trackers.m3t import M3TTracker

height, width = first_rgb.shape[:2]
tracker = M3TTracker(
    "part.obj",
    mesh_scale=1.0,
    K=K,
    size=(width, height),
)
tracker.initialize(first_rgb, first_depth_m, [initial_pose])

# rgbd_frames yields subsequent (RGB image, aligned depth in metres) pairs.
for iteration, (rgb, depth_m) in enumerate(rgbd_frames, start=1):
    poses = tracker.step(rgb, depth_m, iteration)
    mesh_to_camera = poses[0]  # A 4x4 transform; translation is in metres.

step() returns a NumPy array shaped (count, 4, 4). Pose order matches the initial pose order. Keep frame dimensions and calibration fixed throughout the session.

For multiple instances, set count=N on construction and pass exactly N initial poses to initialize(). All instances share one mesh and its precomputed models.

Custom Settings and Texture Tracking ​

Pass native modality parameter names in region_settings and depth_settings. Set texture_settings={} to enable texture tracking with native defaults, or provide texture parameters:

python
tracker = M3TTracker(
    "part.obj",
    mesh_scale=1.0,
    K=K,
    size=(width, height),
    region_settings={"n_lines_max": 100},
    depth_settings={
        "n_points_max": 100,
        "standard_deviations": [0.03, 0.01, 0.005],
    },
    texture_settings={"orb_n_features": 200},
)

These values illustrate configuration, not a recommended calibration for every camera. Depth standard deviations are in metres. Omitted settings retain native defaults. texture_settings=None disables the texture modality. Unknown parameter names raise ValueError.

Constructor Parameters ​

ParameterDefaultDescription
mesh_path, mesh_scale, K, sizeRequiredMesh, units, calibration, and frame dimensions described above
count1Number of instances of the same mesh; use a positive integer
cacheNoneModel cache root; defaults to ~/.cache/telekinesis-m3t
divides3Model-view subdivision level
image_size1000Resolution used to generate model views, not camera input size
model_points200Stored contour/surface points per model view
rotation_regularization1000.0Rotation regularization passed to the native optimizer
translation_regularization30000.0Translation regularization passed to the native optimizer
region_settings, depth_settingsNoneNative parameter dictionaries for region and depth modalities
texture_settingsNoneEnable and configure texture tracking when a dictionary is supplied

Models are generated from the mesh on construction and cached in a subdirectory keyed by mesh contents, scale, and model-generation settings. First construction can take longer; later instances reuse matching cached models. M3T does not download an ONNX bundle.

Methods and Recovery ​

MethodBehavior
initialize(rgb, depth_m, poses)Set the starting poses and initialize tracking; one pose per instance is required
step(rgb, depth_m, iteration)Process the next frame and return (count, 4, 4) mesh-to-camera transforms
set_images(rgb, depth_m)Update native camera images without running a tracking step; normally called internally

The wrapper converts RGB to native BGR and metric depth to uint16 millimetres. Invalid, non-positive, or out-of-range depth values become zero.

Native setup or tracking failures raise RuntimeError. step() also rejects non-finite poses and poses with non-positive camera Z translation. These checks do not detect every tracking loss; a plausible pose can still drift. If tracking is lost, obtain fresh initial poses and call initialize() again. There is no reset_session() method for M3T. Construct a new tracker when changing the mesh, calibration, frame dimensions, or instance count.