Estimate Planar Pose

SUMMARY

Estimate Planar Pose estimates the 3D pose of a flat object from its mask and an aligned depth image.

It combines 2D PCA on mask (for position and in-plane orientation) with a pinhole back-projection of the mask's centroid through depth_image (for depth) to produce a datatypes.Pose3D: the translation is the mask's centroid back-projected into the camera frame using camera_calibration, and the yaw is the mask's principal-axis angle -- roll and pitch are always 0. It's meant for objects that lie flat and only vary in position and in-plane rotation, such as a part on a conveyor.

Use this Skill when you want to recover a flat object's 3D position and in-plane rotation from a mask and a depth image.

The Skill

from telekinesis import pupil

pose_3d = pupil.estimate_planar_pose(
    mask=mask,
    depth_image=depth_image,
    camera_calibration=camera_calibration,
)

API Reference

Full parameter and return type documentation for estimate_planar_pose.

View Reference →

The Code

"""Demonstrates estimating a flat object's 3D planar pose from a mask and depth image."""

import numpy as np
from loguru import logger
import rerun as rr

from telekinesis import pupil, datatypes, cornea


def estimate_planar_pose_example():
    """Estimates the 3D planar pose of a flat object from a mask and depth image."""
    # ===================== Load Image ==========================================
    image_url = "https://assets.telekinesis.ai/examples/v1/images/can_vertical_6_mask.png"
    image = datatypes.Image.from_url(image_url)
    mask = cornea.segment_image_using_otsu_threshold(image=image)

    # ===================== Create Parameters ==========================================
    # Depth image: aligned with mask, shape (H, W). Here a flat plane 1.5m from the camera.
    depth_image = datatypes.DepthImage(np.full(mask.shape, 1.5, dtype=np.float32))

    # Camera calibration: intrinsics + distortion model used to back-project the centroid
    camera_calibration = datatypes.CameraCalibration(
        width=640,
        height=480,
        distortion_model="plumb_bob",
        distortion_parameters=[0.0, 0.0, 0.0, 0.0, 0.0],
        intrinsic_matrix=[500.0, 0.0, 320.0, 0.0, 500.0, 240.0, 0.0, 0.0, 1.0],
    )

    # ===================== Run Skill ==========================================
    pose_3d = pupil.estimate_planar_pose(
        mask=mask,
        depth_image=depth_image,
        camera_calibration=camera_calibration,
    )

    # ===================== Log ================================================
    logger.success(f"Estimated planar pose from {image}")
    logger.success(f"Result: {pose_3d}")
    x, y, z, roll, pitch, yaw = pose_3d.data
    logger.info(
        f"position=({x:.3f}, {y:.3f}, {z:.3f}), yaw={yaw:.2f} deg (roll={roll}, pitch={pitch})"
    )

    # ===================== Visualization  (Optional) ======================
    rr.init("estimate_planar_pose_example", spawn=True)
    datatypes.visualize(image, entity_path="1-Image")
    datatypes.visualize(mask, entity_path="2-Mask")
    datatypes.visualize(pose_3d, entity_path="3-Planar Pose", label="Planar Pose")

if __name__ == "__main__":
    estimate_planar_pose_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/image_processing/estimate_planar_pose.py

Parameter Configuration

Key	Type	Default	Description
mask	datatypes.SegmentationImage | np.ndarray	required	The region of interest, shape (H, W), with non-zero values marking the object -- either a binary mask or a multi-region segmentation. Must share depth_image's (H, W)
depth_image	datatypes.DepthImage | np.ndarray	required	The depth image aligned with mask, shape (H, W). Must share mask's (H, W)
camera_calibration	datatypes.CameraCalibration | dict	required	The camera's intrinsic calibration, providing the intrinsic matrix and lens distortion parameters used to back-project mask's centroid through depth_image. A dict is coerced via CameraCalibration's constructor keyword arguments (width, height, distortion_model, distortion_parameters, intrinsic_matrix)

Returns

Type	Description
datatypes.Pose3D	The object's pose in the camera frame, [x, y, z, roll, pitch, yaw]: x/y/z is the back-projected centroid position and yaw is the mask's principal-axis angle in degrees; roll/pitch are always 0. Access the raw (6,) array via .data

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	mask has no non-zero pixels, or mask and depth_image don't share the same width and height
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Pupil service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Pupil service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Pupil service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

Like calculate_mask_pca/project_pixel_to_camera_point, these are alignment and calibration inputs rather than tunable knobs — there is no "better" value, only the value that matches your camera, mask, and depth reading.

mask

• Controls: Which pixels are treated as the flat object -- its centroid becomes the pose's x/y/z (after back-projection), and its principal axis becomes yaw.
• Practical guidance: A clean, single-region mask produces a stable centroid and orientation; a mask with stray foreground pixels or multiple disconnected objects pulls both toward a combined, less meaningful result. Must contain at least one non-zero pixel.

depth_image

• Controls: The distance along the optical axis at the mask's centroid, which determines how far out along the projection ray x/y/z lands.
• Practical guidance: Must be pixel-aligned with mask (same (H, W)) and in the same length units as camera_calibration's intrinsics (e.g. meters). Only the depth around the centroid actually drives the result, but the image must still match mask's size.

camera_calibration

• Controls: The intrinsic matrix and distortion model/coefficients used to back-project the centroid pixel into a 3D camera-space point.
• Practical guidance: Obtain from a calibration procedure for the camera that captured depth_image. Pass distortion_parameters as all zeros (matching distortion_model's parameter count) for an ideal, undistorted pinhole model.

TIP

Best practice: This Skill assumes the object lies flat relative to the camera (roll/pitch are always 0) -- if the object can be tilted out of plane, x/y/z/yaw are still computed, but won't capture the true 3D orientation. Confirm the flat-object assumption holds before relying on this Skill for grasp planning.

Where to Use the Skill

Common pipelines include:

• Conveyor picking – Estimate the position and in-plane rotation of a flat part on a conveyor before computing a pick pose
• Bin picking (flat objects) – Localize thin or flat parts (e.g. sheet metal, labels, gaskets) that only vary in position and yaw
• Quality inspection – Check that a flat part's position and orientation fall within an expected tolerance
• Vision-guided alignment – Feed the estimated pose into a robot motion Skill to align a tool or gripper with a flat object

Alternative Skills

Skill	vs. Estimate Planar Pose
calculate_mask_pca	Returns the mask's 2D centroid, eigenvectors/eigenvalues, and principal angle directly, without back-projecting to 3D or producing a Pose3D. Use it if you only need the 2D result or want to do the back-projection yourself.
project_pixel_to_camera_point	Back-projects an arbitrary pixel + depth to a 3D point, without computing a centroid, orientation, or full pose.
calculate_mask_centroid	Returns only the mask's 2D centroid position, without orientation or 3D back-projection.

When Not to Use the Skill

Do not use Estimate Planar Pose when:

• The object isn't flat relative to the camera (roll/pitch are always 0; use a full 3D pose-estimation approach if the object can tilt out of plane)
• mask may be empty (an all-zero mask raises ValueError; verify the upstream segmentation/thresholding step produced a non-empty mask)
• mask and depth_image have different sizes (raises ValueError; resize or crop them to match first)
• You only need the 2D centroid/orientation, not a 3D pose (use calculate_mask_pca instead, which skips the back-projection)
• The mask contains multiple disconnected objects (the centroid and principal axis are computed over all non-zero pixels combined, not per object; segment and mask each object separately first)