Project Pixel To World Point

SUMMARY

Project Pixel To World Point projects a pixel and depth to a 3D point in world coordinates.

It behaves like project_pixel_to_camera_point, but additionally applies world_T_camera — the 4x4 homogeneous transform from camera frame to world frame — to express the back-projected point in world coordinates rather than camera coordinates. Use this when you need the point in a robot/world frame (e.g. for motion planning) rather than relative to the camera.

Use this Skill when you want to back-project a pixel plus a depth reading directly into a world/robot-frame 3D point.

The Skill

from telekinesis import pupil
import numpy as np

# [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
camera_intrinsics = np.array(
    [[500.0, 0, 320.0], [0, 500.0, 240.0], [0, 0, 1.0]], dtype=np.float64
)
# [k1, k2, p1, p2, k3]
distortion_coefficients = np.array([0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64)
# [u, v] in pixel coordinates
pixel = np.array([320.0, 240.0], dtype=np.float64)
depth = 1.0

world_T_camera = np.eye(4, dtype=np.float64)
world_T_camera[2, 3] = 1.0

world_point = pupil.project_pixel_to_world_point(
    camera_intrinsics=camera_intrinsics,
    distortion_coefficients=distortion_coefficients,
    pixel=pixel,
    depth=depth,
    world_T_camera=world_T_camera,
)

API Reference

Full parameter and return type documentation for project_pixel_to_world_point.

View Reference →

Example

Back-projecting pixel [320.0, 240.0] at depth = 1.0 through the same intrinsics as above, with the camera positioned at world_T_camera[2, 3] = 1.0 (one unit up along the world z-axis, identity rotation), yields the world-frame point directly below/along the camera's optical axis at that depth. There is no before/after image for this Skill: verify a projection by comparing the returned (x, y, z) against a known world-frame landmark, or by visualizing it alongside a robot/world coordinate frame in Rerun.

The Code

"""Demonstrates projecting a pixel and depth to a 3D world point."""

import numpy as np
from loguru import logger
import rerun as rr

from telekinesis import pupil, datatypes


def project_pixel_to_world_point_example():
    """Projects a pixel and depth to a 3D world point."""
    # ===================== Create Parameters ==========================================
    # Pixel: [u, v] in pixel coordinates
    pixel = np.array([320.0, 240.0], dtype=np.float64)

    # Depth: scalar value representing the distance from the camera to the point
    depth = 1.0

    # World-to-camera transformation matrix (4x4)
    world_T_camera = np.eye(4, dtype=np.float64)
    world_T_camera[2, 3] = 1.0
    
    # Camera_intrinsics: [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
    camera_intrinsics = np.array(
        [[500.0, 0, 320.0],
         [0, 500.0, 240.0],
         [0, 0, 1.0]],
        dtype=np.float64,
    )

    # Distortion coefficients: [k1, k2, p1, p2, k3]
    distortion_coefficients = np.array(
        [0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64
    )

    # ===================== Run Skill ==========================================
    world_point = pupil.project_pixel_to_world_point(
        pixel=pixel,
        depth=depth,
        world_T_camera=world_T_camera,
        camera_intrinsics=camera_intrinsics,
        distortion_coefficients=distortion_coefficients,
    )

    # ===================== Log ================================================
    logger.success(f"Projected pixel to world point using {pixel} and depth {depth}")
    logger.success(f"Result: {world_point}")

    # ===================== Visualization  (Optional) ======================
    rr.init("project_pixel_to_world_point_example", spawn=True)
    datatypes.visualize(world_point, entity_path="1-World Point")

if __name__ == "__main__":
    project_pixel_to_world_point_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/image_processing/project_pixel_to_world_point.py

Parameter Configuration

Key	Type	Default	Description
pixel	datatypes.Point2D | np.ndarray | list	required	Pixel coordinates [x, y], shape (2,)
depth	datatypes.Float | float | int	required	The depth at pixel, in the same units as camera_intrinsics
world_T_camera	datatypes.Mat4x4 | np.ndarray | list	required	The 4x4 homogeneous transform from camera frame to world frame, shape (4, 4)
camera_intrinsics	datatypes.Mat3x3 | np.ndarray | list	required	3x3 camera intrinsic matrix [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], shape (3, 3)
distortion_coefficients	datatypes.Array | np.ndarray | list	required	Lens distortion coefficients [k1, k2, p1, p2, k3]. Pass all zeros for an undistorted/ideal pinhole model

Returns

Type	Description
datatypes.Point3D	The back-projected point [x, y, z], in world coordinates. Access the raw (3,) array via .data

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	distortion_coefficients is not shape (5,), or contains NaN/inf
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Pupil service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Pupil service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Pupil service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

These are camera-geometry and measurement inputs rather than tunable knobs in the usual sense — there is no "better" or "worse" value, only the value that matches your camera, pixel, depth reading, and camera pose.

pixel

• Controls: The image location, in pixel coordinates, to back-project.
• Practical guidance: Typically comes from a detector (e.g. an object detection or keypoint Skill) rather than being hand-picked.

depth

• Controls: The distance along the optical axis at pixel.
• Practical guidance: Usually read from a depth camera or stereo/depth-estimation pipeline at the same pixel. Must be valid and positive.

world_T_camera

• Controls: The camera's pose (position and orientation) in the world frame, applied after back-projecting into camera coordinates.
• Practical guidance: Obtain from camera pose estimation, robot forward kinematics, or extrinsic calibration. Must be a valid 4x4 homogeneous transform (rotation + translation).

camera_intrinsics

• Controls: The focal lengths (fx, fy) and principal point (cx, cy) of the camera.
• Practical guidance: Obtain from a camera calibration procedure; must match the units used for depth.

distortion_coefficients

• Controls: Lens distortion correction applied during back-projection.
• Default: [0.0, 0.0, 0.0, 0.0, 0.0] (no distortion, ideal pinhole model)
• Practical guidance: Use the 5-element [k1, k2, p1, p2, k3] vector from the same calibration that produced camera_intrinsics. Must be exactly shape (5,) with no NaN/inf values.

TIP

Best practice: Keep world_T_camera current for the exact moment pixel/depth were captured — if the camera is mounted on a moving robot or gimbal, a stale pose silently shifts the resulting world point.

Where to Use the Skill

Common pipelines include:

• Robot picking – Convert a 2D pick point plus a depth reading directly into a 3D grasp pose in the robot's world frame
• 3D mapping – Build world-frame point clouds from RGB-D data across multiple camera poses
• Object localization – Determine an object's position in world/robot coordinates from a single camera detection
• Multi-camera fusion – Bring detections from cameras with different poses into one shared world frame

Alternative Skills

Skill	vs. Project Pixel To World Point
project_pixel_to_camera_point	Same back-projection, but stops at camera coordinates — use when a world/robot frame isn't needed.
project_world_point_to_pixel	The inverse operation: a 3D world-space point projected forward to a pixel.
project_camera_point_to_pixel	Forward projection from camera coordinates, no world_T_camera involved.

When Not to Use the Skill

Do not use Project Pixel To World Point when:

• Camera-frame coordinates are sufficient (use project_pixel_to_camera_point and skip the extra world_T_camera dependency)
• You have a world-frame point and need a pixel (use project_world_point_to_pixel, the inverse operation)
• world_T_camera is unknown or stale (estimate/calibrate the current camera pose first; an outdated transform silently shifts the result)
• depth is invalid, missing, or non-positive at the pixel (verify the depth reading before projecting)