Project World Point To Pixel

SUMMARY

Project World Point To Pixel projects a 3D point in world coordinates to pixel coordinates.

It behaves like project_camera_point_to_pixel, but the input point is given in world coordinates rather than camera coordinates — world_T_camera is used to first bring point into the camera frame before applying the forward pinhole projection. Use project_camera_point_to_pixel directly if the point is already in camera coordinates.

Use this Skill when you want to convert a 3D point in world/robot coordinates into a pixel location.

The Skill

from telekinesis import pupil
import numpy as np

# [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
camera_intrinsics = np.array(
    [[500.0, 0, 320.0], [0, 500.0, 240.0], [0, 0, 1.0]], dtype=np.float64
)
# [k1, k2, p1, p2, k3]
distortion_coefficients = np.array([0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64)
# [x, y, z] in world coordinates
point = np.array([0.0, 0.0, 1.0], dtype=np.float64)

world_T_camera = np.eye(4, dtype=np.float64)
world_T_camera[2, 3] = 1.0

pixel = pupil.project_world_point_to_pixel(
    camera_intrinsics=camera_intrinsics,
    distortion_coefficients=distortion_coefficients,
    point=point,
    world_T_camera=world_T_camera,
)

API Reference

Full parameter and return type documentation for project_world_point_to_pixel.

View Reference →

Example

Projecting the world-frame point [0.0, 0.0, 1.0] with the camera posed at world_T_camera[2, 3] = 1.0 (one unit up along the world z-axis, identity rotation) brings the point to [0, 0, 0] in the camera frame before projection — a degenerate case on the camera origin. In practice, pick a point/world_T_camera pair that places the point in front of the camera (positive z in camera coordinates). There is no before/after image for this Skill: verify a projection by drawing a marker at the returned (u, v) on the corresponding image, or by comparing against a known world-to-pixel correspondence.

The Code

"""Demonstrates projecting a 3D world point to pixel coordinates."""

import numpy as np
from loguru import logger
import rerun as rr

from telekinesis import pupil, datatypes


def project_world_point_to_pixel_example():
    """Projects a 3D world point to pixel coordinates."""
    # ===================== Create Parameters ==========================================
    # Point: [x, y, z] in world coordinates
    point = np.array([0.0, 0.0, 1.0], dtype=np.float64)

    # World-to-camera transformation matrix (4x4)
    world_T_camera = np.eye(4, dtype=np.float64)
    world_T_camera[2, 3] = 1.0

    # Camera_intrinsics: [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
    camera_intrinsics = np.array(
        [[500.0, 0, 320.0], [0, 500.0, 240.0], [0, 0, 1.0]],
        dtype=np.float64,
    )
    # Distortion coefficients: [k1, k2, p1, p2, k3]
    distortion_coefficients = np.array(
        [0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64
    )

    # ===================== Run Skill ==========================================
    pixel = pupil.project_world_point_to_pixel(
        point=point,
        world_T_camera=world_T_camera,
        camera_intrinsics=camera_intrinsics,
        distortion_coefficients=distortion_coefficients,
    )

    # ===================== Log ================================================
    logger.success(f"Projected world point to pixel using {point}")
    logger.success(f"Result: {pixel}")

    # ===================== Visualization  (Optional) ======================
    rr.init("project_world_point_to_pixel_example", spawn=True)
    datatypes.visualize(pixel, entity_path="1-Pixel")

if __name__ == "__main__":
    project_world_point_to_pixel_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/image_processing/project_world_point_to_pixel.py

Parameter Configuration

Key	Type	Default	Description
point	datatypes.Point3D | np.ndarray | list	required	The 3D point [x, y, z] in world coordinates, shape (3,)
world_T_camera	datatypes.Mat4x4 | np.ndarray | list	required	The 4x4 homogeneous transform from camera frame to world frame, shape (4, 4)
camera_intrinsics	datatypes.Mat3x3 | np.ndarray | list	required	3x3 camera intrinsic matrix [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], shape (3, 3)
distortion_coefficients	datatypes.Array | np.ndarray | list	required	Lens distortion coefficients [k1, k2, p1, p2, k3]. Pass all zeros for an undistorted/ideal pinhole model

Returns

Type	Description
datatypes.Point2D	The projected pixel coordinates [x, y]. Access the raw (2,) array via .data

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	distortion_coefficients is not shape (5,), or contains NaN/inf
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Pupil service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Pupil service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Pupil service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

These are camera-geometry inputs rather than tunable knobs in the usual sense — there is no "better" or "worse" value, only the value that matches your camera, its pose, and the point you want projected.

point

• Controls: The 3D location, in world coordinates, to project.
• Practical guidance: After being transformed into camera coordinates via world_T_camera, the resulting point must have z > 0 to lie in front of the camera and project meaningfully.

world_T_camera

• Controls: The camera's pose (position and orientation) in the world frame, inverted internally to bring point into camera coordinates.
• Practical guidance: Obtain from camera pose estimation, robot forward kinematics, or extrinsic calibration. Must be a valid 4x4 homogeneous transform (rotation + translation).

camera_intrinsics

• Controls: The focal lengths (fx, fy) and principal point (cx, cy) of the camera.
• Practical guidance: Obtain from a camera calibration procedure.

distortion_coefficients

• Controls: Lens distortion correction applied during projection.
• Default: [0.0, 0.0, 0.0, 0.0, 0.0] (no distortion, ideal pinhole model)
• Practical guidance: Use the 5-element [k1, k2, p1, p2, k3] vector from the same calibration that produced camera_intrinsics. Must be exactly shape (5,) with no NaN/inf values.

TIP

Best practice: Keep world_T_camera current for the exact moment you want to visualize point at — if the camera is mounted on a moving robot or gimbal, a stale pose silently projects to the wrong pixel.

Where to Use the Skill

Common pipelines include:

• Robot/scene visualization – Overlay a robot pose, waypoint, or object position on the camera image
• 3D annotation – Draw world-frame landmarks or planned trajectories on images
• Verification – Check whether a world-frame 3D estimate (e.g. from SLAM or a pose graph) projects to the expected image location
• AR/overlay – Render world-frame content aligned with a live camera feed

Alternative Skills

Skill	vs. Project World Point To Pixel
project_camera_point_to_pixel	Use when the point is already in camera coordinates and no world_T_camera is needed.
project_pixel_to_world_point	The inverse operation: pixel + depth back-projected to a world-frame point.
project_pixel_to_camera_point	Inverse into camera coordinates only, without the world_T_camera step.

When Not to Use the Skill

Do not use Project World Point To Pixel when:

• The point is already in camera coordinates (use project_camera_point_to_pixel directly and skip the extra world_T_camera step)
• You have a pixel and need a 3D world point (use project_pixel_to_world_point, the inverse operation)
• world_T_camera is unknown or stale (estimate/calibrate the current camera pose first; an outdated transform silently produces the wrong pixel)
• The point may fall behind the camera after the transform (verify z > 0 in camera coordinates before relying on the projected pixel)