Telekinesis Agentic Skill Library

Appearance

Estimate Planar Pose ​

SUMMARY

Estimate Planar Pose estimates the 3D pose of a flat object from its mask and an aligned depth image.

It combines 2D PCA on mask (for position and in-plane orientation) with a pinhole back-projection of the mask's centroid through depth_image (for depth) to produce a datatypes.Pose3D: the translation is the mask's centroid back-projected into the camera frame using camera_calibration, and the yaw is the mask's principal-axis angle -- roll and pitch are always 0. It's meant for objects that lie flat and only vary in position and in-plane rotation, such as a part on a conveyor.

Use this Skill when you want to recover a flat object's 3D position and in-plane rotation from a mask and a depth image.

The Skill ​

python
from telekinesis import pupil

pose_3d = pupil.estimate_planar_pose(
    mask=mask,
    depth_image=depth_image,
    camera_calibration=camera_calibration,
)
API Reference
Full parameter and return type documentation for estimate_planar_pose.
View Reference →

The Code ​

python
"""Demonstrates estimate_planar_poses operation."""

import numpy as np
from loguru import logger
import rerun as rr

from telekinesis import pupil, datatypes


def estimate_planar_poses_example():
    """Estimates 3D planar poses from a mask and depth image."""
    # ===================== Create Parameters ==========================================
    mask = datatypes.SegmentationImage.from_url(
        "https://assets.telekinesis.ai/examples/v1/images/permanent_marker_pen.png"
    )
    # Depth image saved as mm
    depth_image_mm = datatypes.DepthImage.from_url(
        "https://assets.telekinesis.ai/examples/v1/depth_images/permanent_marker_pen_depth.png"
    )
    depth_image_m = depth_image_mm.depth.astype(np.float32) / 1000.0

    # Camera_intrinsics: [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
    camera_intrinsics = np.array(
        [
            [435.46856689, 0.0, 420.72525024],
            [0.0, 434.52377319, 244.55136108],
            [0.0, 0.0, 1.0],
        ],
        dtype=np.float64,
    )
    # Distortion coefficients: [k1, k2, p1, p2, k3]
    distortion_parameters = np.array(
        [-0.05338378, 0.0578178, 0.00045024, 0.00018894, -0.018924], dtype=np.float64
    )
    camera_calibration = datatypes.CameraCalibration(
        height=mask.height,
        width=mask.width,
        distortion_model="plumb_bob",
        distortion_parameters=distortion_parameters,
        intrinsic_matrix=camera_intrinsics,
    )

    # ===================== Run Skill ==========================================
    camera_object_pose = pupil.estimate_planar_pose(
        mask=mask, depth_image=depth_image_m, camera_calibration=camera_calibration
    )

    # ===================== Log ================================================
    logger.success(f"Estimated planar poses using mask of shape {mask.shape}")
    logger.success(f"Result: {camera_object_pose}")

    # ===================== Visualization  (Optional) ======================
    rr.init("estimate_planar_poses_example", spawn=True)
    datatypes.visualize(mask, entity_path="/1-mask")
    datatypes.visualize(depth_image_mm, entity_path="/2-depthimage")
    datatypes.visualize(camera_object_pose, entity_path="/3-estimated3dpose")


if __name__ == "__main__":
    estimate_planar_poses_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

bash
cd telekinesis-examples
python examples/image_processing/estimate_planar_pose.py

Parameter Configuration ​

ParameterTypeDefaultDescription
maskdatatypes.SegmentationImage | np.ndarrayrequiredThe region of interest, shape (H, W), with non-zero values marking the object -- either a binary mask or a multi-region segmentation. Must share depth_image's (H, W)
depth_imagedatatypes.DepthImage | np.ndarrayrequiredThe depth image aligned with mask, shape (H, W). Must share mask's (H, W)
camera_calibrationdatatypes.CameraCalibration | dictrequiredThe camera's intrinsic calibration, providing the intrinsic matrix and lens distortion parameters used to back-project mask's centroid through depth_image. A dict is coerced via CameraCalibration's constructor keyword arguments (width, height, distortion_model, distortion_parameters, intrinsic_matrix)

Returns ​

TypeDescription
datatypes.Pose3DThe object's pose in the camera frame, [x, y, z, roll, pitch, yaw]: x/y/z is the back-projected centroid position and yaw is the mask's principal-axis angle in degrees; roll/pitch are always 0. Access the raw (6,) array via .data

Raises ​

ExceptionCondition
TypeErrorA parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueErrormask has no non-zero pixels, or mask and depth_image don't share the same width and height
ConfigurationErrorThe TELEKINESIS_API_KEY environment variable is not set
SerializationErrorThe request input failed to serialize, the response was not returned as an Arrow stream, or the response failed to deserialize
RequestTimeoutErrorThe request to the Pupil service timed out
TransportErrorA network failure occurred before a response was received
ClientErrorThe Pupil service rejected the request due to invalid or malformed input (HTTP 400/422), an unrecognized endpoint (HTTP 404), or another unexpected 4xx response
AuthenticationErrorThe API key was rejected as invalid or expired (HTTP 401)
AuthenticationServiceErrorThe authentication service returned an invalid response, was temporarily unavailable, or timed out (HTTP 502/503/504)
ServerErrorThe Pupil service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters ​

Like calculate_mask_pca/project_pixel_to_camera_point, these are alignment and calibration inputs rather than tunable knobs — there is no "better" value, only the value that matches your camera, mask, and depth reading.

mask

  • Controls: Which pixels are treated as the flat object -- its centroid becomes the pose's x/y/z (after back-projection), and its principal axis becomes yaw.
  • Practical guidance: A clean, single-region mask produces a stable centroid and orientation; a mask with stray foreground pixels or multiple disconnected objects pulls both toward a combined, less meaningful result. Must contain at least one non-zero pixel.

depth_image

  • Controls: The distance along the optical axis at the mask's centroid, which determines how far out along the projection ray x/y/z lands.
  • Practical guidance: Must be pixel-aligned with mask (same (H, W)) and in the same length units as camera_calibration's intrinsics (e.g. meters). Only the depth around the centroid actually drives the result, but the image must still match mask's size.

camera_calibration

  • Controls: The intrinsic matrix and distortion model/coefficients used to back-project the centroid pixel into a 3D camera-space point.
  • Practical guidance: Obtain from a calibration procedure for the camera that captured depth_image. Pass distortion_parameters as all zeros (matching distortion_model's parameter count) for an ideal, undistorted pinhole model.

TIP

Best practice: This Skill assumes the object lies flat relative to the camera (roll/pitch are always 0) -- if the object can be tilted out of plane, x/y/z/yaw are still computed, but won't capture the true 3D orientation. Confirm the flat-object assumption holds before relying on this Skill for grasp planning.

Where to Use the Skill ​

Common pipelines include:

  • Conveyor picking – Estimate the position and in-plane rotation of a flat part on a conveyor before computing a pick pose
  • Bin picking (flat objects) – Localize thin or flat parts (e.g. sheet metal, labels, gaskets) that only vary in position and yaw
  • Quality inspection – Check that a flat part's position and orientation fall within an expected tolerance
  • Vision-guided alignment – Feed the estimated pose into a robot motion Skill to align a tool or gripper with a flat object

Alternative Skills ​

Skillvs. Estimate Planar Pose
calculate_mask_pcaReturns the mask's 2D centroid, eigenvectors/eigenvalues, and principal angle directly, without back-projecting to 3D or producing a Pose3D. Use it if you only need the 2D result or want to do the back-projection yourself.
project_pixel_to_camera_pointBack-projects an arbitrary pixel + depth to a 3D point, without computing a centroid, orientation, or full pose.
calculate_mask_centroidReturns only the mask's 2D centroid position, without orientation or 3D back-projection.

When Not to Use the Skill ​

Do not use Estimate Planar Pose when:

  • The object isn't flat relative to the camera (roll/pitch are always 0; use a full 3D pose-estimation approach if the object can tilt out of plane)
  • mask may be empty (an all-zero mask raises ValueError; verify the upstream segmentation/thresholding step produced a non-empty mask)
  • mask and depth_image have different sizes (raises ValueError; resize or crop them to match first)
  • You only need the 2D centroid/orientation, not a 3D pose (use calculate_mask_pca instead, which skips the back-projection)
  • The mask contains multiple disconnected objects (the centroid and principal axis are computed over all non-zero pixels combined, not per object; segment and mask each object separately first)