Convert Depth Image to Point Cloud

SUMMARY

Convert Depth Image to Point Cloud back-projects a depth image into a 3D point cloud using a pinhole camera model.

It converts a datatypes.DepthImage into a datatypes.PointCloud in the camera frame: for each pixel (u, v) holding a depth value Z, it computes the 3D point X = (u - cx) * Z / fx, Y = (v - cy) * Z / fy using the camera's intrinsic_matrix. This is the whole-image counterpart to project_pixel_to_camera_point — use this Skill when you have a full depth image to convert, rather than a single pixel.

Use this Skill when you want to turn a depth image from an RGB-D or depth sensor into a 3D point cloud for downstream point-cloud processing, registration, or pose estimation.

The Skill

from telekinesis import vitreous

point_cloud = vitreous.convert_depth_image_to_point_cloud(
    depth_image=depth_image,
    intrinsic_matrix=intrinsic_matrix,
)

API Reference

Full parameter and return type documentation for convert_depth_image_to_point_cloud.

View Reference →

Data Transfer Notice

There is no longer a fixed limit of 1 million points per request. However, very large datasets may result in slower data transfer and processing times. We are continuously optimizing performance as part of our beta program, with ongoing improvements to enhance speed and reliability.

The Code

"""
Demonstrates back-projecting a depth image into a 3D point cloud using a pinhole camera model.
"""

import numpy as np
from loguru import logger
import rerun as rr

from telekinesis import vitreous, datatypes


def convert_depth_image_to_point_cloud_example():
    """
    Back-projects a depth image into a 3D point cloud using a pinhole camera model.

    For each pixel (u, v) holding a depth value Z, computes the 3D point
    X = (u - cx) * Z / fx, Y = (v - cy) * Z / fy.
    """
    # ===================== Load Data ==========================================
    depth_image = datatypes.DepthImage(
        np.full((480, 640), 1.5, dtype=np.float32)  # a flat wall 1.5m away
    )
    intrinsic_matrix = np.array(
        [
            [600.0, 0.0, 320.0],
            [0.0, 600.0, 240.0],
            [0.0, 0.0, 1.0],
        ],
        dtype=np.float32,
    )

    # ===================== Run Skill ==========================================
    point_cloud = vitreous.convert_depth_image_to_point_cloud(
        depth_image=depth_image,
        intrinsic_matrix=intrinsic_matrix,
    )

    # ===================== Log ================================================
    logger.success(f"Converted {depth_image} to a point cloud")
    logger.success(f"Results: {point_cloud}")
    logger.info(f"Point cloud has {len(point_cloud.positions)} points")

    # ===================== Visualization  (Optional) ===========================
    rr.init("convert_depth_image_to_point_cloud_example", spawn=True)
    datatypes.visualize(depth_image, entity_path="/1-depth_image")
    datatypes.visualize(point_cloud, entity_path="/2-point_cloud")


if __name__ == "__main__":
    convert_depth_image_to_point_cloud_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/point_cloud/convert_depth_image_to_point_cloud.py

Parameter Configuration

Key	Type	Default	Description
depth_image	datatypes.DepthImage | np.ndarray	required	The depth image, shape (H, W), where each pixel holds a depth/distance value (in whatever length unit your sensor and intrinsic_matrix agree on, e.g. millimeters or meters).
intrinsic_matrix	datatypes.Mat3x3 | np.ndarray | list[list[float]]	required	The camera's 3x3 intrinsic matrix [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], where fx/fy are the focal lengths in pixels and cx/cy are the principal point in pixel coordinates.

Returns

Type	Description
datatypes.PointCloud	A point cloud in the camera frame, with one 3D position per depth_image pixel. Use .positions for the (H*W, 3) array or len(...) for the point count.

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Vitreous service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Vitreous service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Vitreous service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

convert_depth_image_to_point_cloud takes only depth_image and intrinsic_matrix — both required, with no optional knobs to tune. The result is a direct geometric back-projection, so its accuracy depends entirely on how well these two inputs match reality:

• depth_image units must match intrinsic_matrix's implicit units. fx/fy/cx/cy are in pixels, but the scale of the resulting X/Y/Z positions follows whatever length unit your depth values are in (millimeters, meters, etc.) — mixing a millimeter depth image with intrinsics calibrated for meters (or vice versa) will silently produce a point cloud scaled by 1000x.
• intrinsic_matrix must be calibrated for the same resolution as depth_image. Intrinsics calibrated at one resolution (e.g. 1280x720) don't transfer directly to a depth image captured or resized to a different resolution (e.g. 640x480) without rescaling fx/fy/cx/cy accordingly.

If the resulting point cloud looks stretched, squashed, or offset from where you expect, double-check these two things before assuming the Skill itself is wrong.

Where to Use the Skill

Common pipelines include:

• RGB-D sensor ingestion – converting a raw depth frame from a depth camera or RGB-D sensor into a point cloud when the sensor/driver doesn't already provide one
• Depth-based perception pipelines – feeding the resulting point cloud into downstream Skills such as filter_point_cloud_using_pass_through_filter, segment_point_cloud_using_plane, or registration Skills
• Synthetic depth-to-point-cloud testing – converting a synthetically generated or simulated depth image into a point cloud for testing point-cloud Skills

Alternative Skills

Skill	vs. Convert Depth Image to Point Cloud
project_pixel_to_camera_point	Back-projects a single pixel + depth value to a 3D camera-space point, instead of an entire depth image at once. Use it when you only need one point (e.g. a detected keypoint), not a full point cloud.
convert_mesh_to_point_cloud	Also produces a datatypes.PointCloud, but samples it from a datatypes.Mesh3D's surface instead of back-projecting a depth image.

When Not to Use the Skill

Do not use Convert Depth Image to Point Cloud when:

• Your sensor or driver already outputs a point cloud – this Skill is only needed when you have raw depth data and no point cloud yet
• You only need a single pixel's 3D position – use project_pixel_to_camera_point instead of converting the whole image
• You don't have accurate camera intrinsics for the depth image – the back-projection is only as accurate as intrinsic_matrix; without a proper calibration, the resulting geometry will be distorted

TIP

If you also have a registered color image, visualize the point cloud alongside it to sanity-check the back-projection before feeding it into a filtering or registration pipeline — an incorrect intrinsic_matrix is usually obvious once you can see the resulting geometry.