Project Pixel To Camera Point

SUMMARY

Project Pixel To Camera Point projects a pixel and depth to a 3D point in camera coordinates.

It is the inverse of project_camera_point_to_pixel: given a pixel [u, v], the depth measured at that pixel, camera_intrinsics, and distortion_coefficients, it back-projects to the 3D point [x, y, z] in camera coordinates that produced it. Use project_pixel_to_world_point instead if you need the result in world coordinates rather than camera coordinates.

Use this Skill when you want to back-project a detected pixel plus a depth reading into a 3D camera-space point.

The Skill

from telekinesis import pupil
import numpy as np

# [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
camera_intrinsics = np.array(
    [[500.0, 0, 320.0], [0, 500.0, 240.0], [0, 0, 1.0]], dtype=np.float64
)
# [k1, k2, p1, p2, k3]
distortion_coefficients = np.array([0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64)
# [u, v] in pixel coordinates
pixel = np.array([320.0, 240.0], dtype=np.float64)
depth = 1.0

camera_point = pupil.project_pixel_to_camera_point(
    camera_intrinsics=camera_intrinsics,
    distortion_coefficients=distortion_coefficients,
    pixel=pixel,
    depth=depth,
)

API Reference

Full parameter and return type documentation for project_pixel_to_camera_point.

View Reference →

Example

Back-Projected Camera Point

Rerun visualization of the 3D point back-projected from pixel (320, 240) at depth 1.0, in camera coordinates.

The Code

"""Demonstrates projecting a pixel and depth to a 3D camera point."""

import numpy as np
from loguru import logger
import rerun as rr

from telekinesis import pupil, datatypes


def project_pixel_to_camera_point_example():
    """Projects a pixel and depth to a 3D camera point."""
    # ===================== Create Parameters ==========================================
    # Pixel: [u, v] in pixel coordinates
    pixel = np.array([320.0, 240.0], dtype=np.float64)

    # Depth: scalar value representing the distance from the camera to the point
    depth = 1.0

    # Camera_intrinsics: [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
    camera_intrinsics = np.array(
        [[500.0, 0, 320.0], 
         [0, 500.0, 240.0], 
         [0, 0, 1.0]],
        dtype=np.float64,
    )

    # Distortion coefficients: [k1, k2, p1, p2, k3]
    distortion_coefficients = np.array(
        [0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64
    )

    # ===================== Run Skill ==========================================
    # Returns 4x4 transformation matrix representing the camera-to-point transformation
    camera_point = pupil.project_pixel_to_camera_point(
        pixel=pixel,
        depth=depth,
        camera_intrinsics=camera_intrinsics,
        distortion_coefficients=distortion_coefficients,
    )

    # ===================== Log ================================================
    logger.success(f"Projected pixel to camera point using {pixel} and depth {depth}")
    logger.success(f"Result: {camera_point}")

    # ===================== Visualization  (Optional) ======================
    rr.init("project_pixel_to_camera_point_example", spawn=True)
    datatypes.visualize(camera_point, entity_path="1-Camera Point")

if __name__ == "__main__":
    project_pixel_to_camera_point_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/image_processing/project_pixel_to_camera_point.py

Parameter Configuration

Key	Type	Default	Description
pixel	datatypes.Point2D | np.ndarray | list	required	Pixel coordinates [x, y], shape (2,)
depth	datatypes.Float | float | int	required	The depth at pixel, in the same units as camera_intrinsics (e.g. meters if camera_intrinsics is in meter-based units)
camera_intrinsics	datatypes.Mat3x3 | np.ndarray | list	required	3x3 camera intrinsic matrix [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], shape (3, 3)
distortion_coefficients	datatypes.Array | np.ndarray | list	required	Lens distortion coefficients [k1, k2, p1, p2, k3]. Pass all zeros for an undistorted/ideal pinhole model
theta	datatypes.Float | float | int | None	None	Optional additional angle parameter for the projection model. When omitted, the request sent to the server defaults it to 0.0

Returns

Type	Description
datatypes.Point3D	The back-projected point [x, y, z], in camera coordinates. Access the raw (3,) array via .data

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	distortion_coefficients is not shape (5,), or contains NaN/inf
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Pupil service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Pupil service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Pupil service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

These are camera-geometry and measurement inputs rather than tunable knobs in the usual sense — there is no "better" or "worse" value, only the value that matches your camera, pixel, and depth reading.

pixel

• Controls: The image location, in pixel coordinates, to back-project.
• Practical guidance: Typically comes from a detector (e.g. an object detection or keypoint Skill) rather than being hand-picked.

depth

• Controls: The distance along the optical axis at pixel, which determines how far out along the projection ray the 3D point lands.
• Practical guidance: Usually read from a depth camera or stereo/depth-estimation pipeline at the same pixel. Must be valid and positive — a zero, negative, or missing depth reading produces a meaningless 3D point.

camera_intrinsics

• Controls: The focal lengths (fx, fy) and principal point (cx, cy) of the camera.
• Practical guidance: Obtain from a camera calibration procedure; must match the units used for depth.

distortion_coefficients

• Controls: Lens distortion correction applied during back-projection.
• Default: [0.0, 0.0, 0.0, 0.0, 0.0] (no distortion, ideal pinhole model)
• Practical guidance: Use the 5-element [k1, k2, p1, p2, k3] vector from the same calibration that produced camera_intrinsics. Must be exactly shape (5,) with no NaN/inf values.

theta

• Controls: An additional angle parameter for the projection model.
• Default: None (sent to the server as 0.0)
• Practical guidance: Leave unset unless your projection model specifically requires a non-zero angle term.

TIP

Best practice: Pair pixel and depth from the same sensor frame/timestamp — projecting a pixel from one frame against a depth reading from another (e.g. after camera or object motion) silently produces an incorrect 3D point.

Where to Use the Skill

Common pipelines include:

• RGB-D processing – Convert a depth image, pixel-by-pixel, into a 3D point cloud
• Robot picking – Convert a detected 2D pick point plus a depth reading into a 3D grasp target
• 3D reconstruction – Unproject detected feature pixels into 3D for structure estimation
• Sensor fusion – Bring 2D detections and depth sensor data into a common 3D camera frame

Alternative Skills

Skill	vs. Project Pixel To Camera Point
project_camera_point_to_pixel	The inverse operation: a 3D camera-space point projected forward to a pixel.
project_pixel_to_world_point	Same back-projection, but additionally applies world_T_camera to return the point in world coordinates.
project_world_point_to_pixel	Forward projection from world coordinates instead of camera coordinates.

When Not to Use the Skill

Do not use Project Pixel To Camera Point when:

• You need the result in world/robot coordinates (use project_pixel_to_world_point, which additionally applies world_T_camera)
• You already have a 3D camera point and need a pixel (use project_camera_point_to_pixel, the inverse operation)
• depth is invalid, missing, or non-positive at the pixel (verify the depth reading before projecting; garbage in produces a garbage 3D point)
• You don't have a calibrated camera_intrinsics/distortion_coefficients pair (calibrate the camera first; guessed values produce silently wrong 3D points)