Telekinesis Agentic Skill Library

Appearance

Project World Point to Pixel ​

SUMMARY

Project World Point to Pixel projects a 3D point in world coordinates to pixel coordinates.

Transforms the world point to camera frame using the camera pose (world_T_camera), then projects to 2D image coordinates. Essential for overlaying world-frame 3D data on images and for robotics visualization.

Use this Skill when you want to convert 3D world point to 2D pixel.

The Skill ​

python
from telekinesis import pupil
import numpy as np

pixel = pupil.project_world_point_to_pixel(
    camera_intrinsics=camera_intrinsics,
    distortion_coefficients=distortion_coefficients,
    point=point,
    world_T_camera=world_T_camera,
)
positions = pixel.to_numpy().reshape(-1, 2)

API Reference

Example ​

Projects 3D point (0, 0, 1.0) in world coordinates to pixel coordinates. The camera pose (world_T_camera) defines the camera position in the world frame. The result can be drawn on the image.

The Code ​

python
from telekinesis import pupil
import numpy as np
from loguru import logger

camera_intrinsics = np.array(
    [[500.0, 0, 320.0], [0, 500.0, 240.0], [0, 0, 1.0]],
    dtype=np.float64,
)
distortion_coefficients = np.array([0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64)
point = np.array([0.0, 0.0, 1.0], dtype=np.float64)
world_T_camera = np.eye(4, dtype=np.float64)
world_T_camera[2, 3] = 1.0

pixel = pupil.project_world_point_to_pixel(
    camera_intrinsics=camera_intrinsics,
    distortion_coefficients=distortion_coefficients,
    point=point,
    world_T_camera=world_T_camera,
)

positions = pixel.to_numpy().reshape(-1, 2)
logger.success("Projected world point to pixel. Pixel: {}", positions)

The Explanation of the Code ​

The code begins by importing the necessary modules: pupil for camera projection operations, numpy for numerical operations, and loguru for logging.

python
from telekinesis import pupil
import numpy as np
from loguru import logger

Next, camera intrinsics, distortion coefficients, the 3D world point, and camera pose are configured. The point is (x, y, z) in world coordinates; world_T_camera is the 4x4 transform from camera to world frame.

python
camera_intrinsics = np.array(
    [[500.0, 0, 320.0], [0, 500.0, 240.0], [0, 0, 1.0]],
    dtype=np.float64,
)
distortion_coefficients = np.array([0.0, 0.0, 0.0, 0.0, 0.0], dtype=np.float64)
point = np.array([0.0, 0.0, 1.0], dtype=np.float64)
world_T_camera = np.eye(4, dtype=np.float64)
world_T_camera[2, 3] = 1.0

The main operation uses the project_world_point_to_pixel Skill from the pupil module. This Skill transforms the world point to camera frame using the camera pose, then projects to 2D image coordinates using intrinsics and distortion.

python
pixel = pupil.project_world_point_to_pixel(
    camera_intrinsics=camera_intrinsics,
    distortion_coefficients=distortion_coefficients,
    point=point,
    world_T_camera=world_T_camera,
)

Finally, the projected pixel coordinates are converted to a NumPy array using to_numpy() for further processing, visualization, or downstream tasks.

python
positions = pixel.to_numpy().reshape(-1, 2)
logger.success(f"Projected pixel: {positions}")

This operation is particularly useful in robotics and vision pipelines for 3D overlay on images, robot visualization, and world-to-pixel conversion, where converting 3D world points to 2D pixels is required.

Running the Example ​

Runnable examples are available in the Telekinesis examples repository. Follow the README in that repository to set up the environment. Once set up, you can run this specific example with:

bash
cd telekinesis-examples
python examples/pupil_examples.py --example project_world_point_to_pixel

How to Tune the Parameters ​

The project_world_point_to_pixel Skill has no tunable parameters in the traditional sense. It requires camera calibration data, a 3D world point, and camera pose:

camera_intrinsics (no default—required): 3x3 matrix with fx, fy, cx, cy. Obtain from camera calibration.

distortion_coefficients (default: np.array([0.0, 0.0, 0.0, 0.0, 0.0])): Lens distortion coefficients. Use zeros for undistorted models.

point (no default—required): 3D point (x, y, z) in world coordinates. Ensure point is in front of camera after transform.

world_T_camera (no default—required): 4x4 transform from camera to world frame. Obtain from camera pose estimation or calibration.

Where to Use the Skill in a Pipeline ​

Project World Point to Pixel is commonly used in the following pipelines:

  • Robot visualization - Overlay robot/object poses on camera image
  • 3D annotation - Draw world-frame 3D data on images
  • Verification - Check if 3D estimates project correctly
  • AR/overlay - Overlay world-frame content

Related skills to build such a pipeline:

  • project_pixel_to_world_point: Inverse operation
  • project_camera_point_to_pixel: When point is already in camera frame
  • apply_transform_to_point_cloud: Transform point clouds (Vitreous)

Alternative Skills ​

Skillvs. Project World Point to Pixel
project_camera_point_to_pixelUse when the point is already in camera coordinates.
project_pixel_to_world_pointInverse: pixel + depth → world point.

When Not to Use the Skill ​

Do not use Project World Point to Pixel when:

  • Point is in camera frame (Use project_camera_point_to_pixel)
  • You need 3D from pixel (Use project_pixel_to_world_point)
  • world_T_camera is unknown (Calibrate or estimate camera pose first)