Filter Point Cloud Using Mask

SUMMARY

Filter Point Cloud Using Mask keeps only the points whose corresponding pixel in a 2D mask is True.

It only works on an organized (structured) point cloud — one whose points are still in the same row-major (H, W) pixel order they came from a depth camera in, such as straight out of convert_depth_image_to_point_cloud, before any filtering or downsampling reorders or drops points. Each mask pixel (row, col) is matched to the point at that same position, and only points behind a True pixel survive — this is how a 2D image-based mask (e.g. from cornea.segment_image_foreground_using_birefnet, pupil.calculate_mask_centroid, or a hand-drawn ROI) gets combined with its corresponding depth data.

Use this Skill when you want to apply a 2D image mask directly to its corresponding organized point cloud.

The Skill

from telekinesis import vitreous
import numpy as np

mask = np.array([[True, False, True], [False, True, False]])

filtered_point_cloud = vitreous.filter_point_cloud_using_mask(
    point_cloud=point_cloud,
    mask=mask,
)

API Reference

Full parameter and return type documentation for filter_point_cloud_using_mask.

View Reference →

Data Transfer Notice

There is no longer a fixed limit of 1 million points per request. However, very large datasets may result in slower data transfer and processing times. We are continuously optimizing performance as part of our beta program, with ongoing improvements to enhance speed and reliability.

Example

Raw Sensor Input

Unprocessed point cloud captured directly from the sensor.

Boolean Mask

Boolean Mask for selecting a subset of points, i.e. the cans.

Masked Output

Masked point cloud showing only the points selected by the boolean mask. The remaining thin lines can be removed in a post-processing step, for example using statistical outlier removal.

The Code

"""
Demonstrates filtering a structured point cloud using a 2D binary mask, keeping only points where the mask is True.
"""

from loguru import logger
import rerun as rr
import rerun.blueprint as rrb

from telekinesis import vitreous, datatypes


def filter_point_cloud_using_mask_example():
    """
    Filters a structured point cloud using a 2D binary mask.

    Applies a 2D image mask to an organized point cloud, keeping only points
    where the corresponding pixel is True.
    """
    # ===================== Load Data ==========================================
    point_cloud_url = (
        "https://assets.telekinesis.ai/examples/v1/point_clouds/can_vertical_6_raw.ply"
    )
    point_cloud = datatypes.PointCloud.from_url(url=point_cloud_url, use_cache=True)

    mask_url = (
        "https://assets.telekinesis.ai/examples/v1/images/can_vertical_6_mask.png"
    )
    mask = datatypes.Image.from_url(url=mask_url).to_binary_mask(threshold=127)

    # ===================== Run Skill ==========================================
    result_point_cloud = vitreous.filter_point_cloud_using_mask(
        point_cloud=point_cloud,
        mask=mask,
    )

    # ===================== Log ================================================
    logger.success(f"Filtered {point_cloud} using mask")
    logger.success(f"Results: {result_point_cloud}")
    logger.info(
        f"Result point cloud positions shape: {result_point_cloud.positions.shape}"
    )
    logger.info(
        f"Result point cloud has normals shape: "
        f"{result_point_cloud.normals.shape if result_point_cloud.has_normals else None}"
    )
    logger.info(
        f"Result point cloud has colors shape: "
        f"{result_point_cloud.colors.shape if result_point_cloud.has_colors else None}"
    )

    # ===================== Visualization  (Optional) ===========================
    rr.init("filter_point_cloud_using_mask_example", spawn=True)
    rr.send_blueprint(
        rrb.Blueprint(
            rrb.Horizontal(
                rrb.Spatial3DView(
                    name="Input Point Cloud",
                    origin="/1-input_point_cloud",
                ),
                rrb.Spatial2DView(
                    name="Binary Mask",
                    origin="/2-binary_mask",
                ),
                rrb.Spatial3DView(
                    name="Masked Point Cloud",
                    origin="/3-masked_point_cloud",
                ),
            )
        )
    )
    datatypes.visualize(point_cloud, entity_path="/1-input_point_cloud")
    datatypes.visualize(mask, entity_path="/2-binary_mask")
    datatypes.visualize(result_point_cloud, entity_path="/3-masked_point_cloud")


if __name__ == "__main__":
    filter_point_cloud_using_mask_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/point_cloud/filter_point_cloud_using_mask.py

Parameter Configuration

point_cloud must be organized/structured for mask to align correctly — see the precondition in the summary above.

Key	Type	Default	Description
point_cloud	datatypes.PointCloud	required	The organized/structured point cloud to filter — its points must still correspond 1:1 to mask's pixel grid, in the same row-major order
mask	datatypes.Image | np.ndarray | list	required	The 2D mask selecting which points to keep, shape (H, W) matching point_cloud's pixel grid (or (H, W, 1), squeezed automatically). Any non-zero/True value marks a point to keep. Build one from a grayscale mask image with datatypes.Image.to_binary_mask(threshold=...)

Returns

Type	Description
datatypes.PointCloud	A point cloud containing only the points behind a True mask pixel. Use .positions for the surviving (N, 3) position array and len(...) for the point count.

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	mask doesn't reduce to a 2D (H, W) array (after squeezing a trailing single-channel axis, e.g. (H, W, 1))
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, or the response failed to deserialize
RequestTimeoutError	The request to the Vitreous service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Vitreous service rejected the request due to invalid input, invalid data, or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired
AuthenticationServiceError	The authentication service was unavailable
ServerError	The Vitreous service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

filter_point_cloud_using_mask has no numeric threshold to tune — mask is an input defining which points to keep, not a knob with a range to sweep. What you control is the mask itself:

mask

• Controls: Which points are kept — a point survives only if the mask pixel at its (row, col) position is non-zero/True.
• Must match point_cloud's pixel grid shape, (H, W) (a trailing single-channel axis, (H, W, 1), is squeezed automatically).
• Build it from a grayscale mask image with datatypes.Image.to_binary_mask(threshold=...), or from any other 2D detection/segmentation output — e.g. cornea.segment_image_foreground_using_birefnet — that shares the point cloud's pixel grid.
• A tighter mask (smaller True region) keeps fewer points; a looser mask keeps more.

TIP

Load the point cloud with duplicate/infinite/NaN-point removal disabled (e.g. remove_duplicated_points=False, remove_infinite_points=False, remove_nan_points=False) so its point count still matches the mask's pixel grid — those cleanup steps reorder or drop points and would break the row-major correspondence this Skill depends on.

Where to Use the Skill

Common pipelines include:

• Detection-guided isolation – Combining a 2D object mask (e.g. from a segmentation or detection skill) with its source depth frame
• Interactive ROI selection – Keeping only the points behind a hand-drawn or user-selected region of an image
• Multi-object separation – Isolating one object's points from a scene using its per-object mask, one mask at a time
• Preprocessing straight off a depth sensor – Filtering a freshly-converted organized point cloud before any reordering step

Alternative Skills

Skill	vs. Filter Point Cloud Using Mask
segment_point_cloud_using_color	Segments directly by each point's own RGB color instead of a separately-computed 2D mask, and works on any point cloud with per-point colors, organized or not. Use color segmentation when no mask exists yet; use mask filtering when a 2D mask is already available.
filter_point_cloud_using_bounding_box	Filters by a 3D spatial box instead of a 2D pixel-aligned mask, and works on any point cloud, organized or not. Use it when the region of interest is a 3D volume rather than a 2D image region.

When Not to Use the Skill

Do not use Filter Point Cloud Using Mask when:

• The point cloud is unorganized/unstructured – this Skill requires the point cloud's points to still be in the mask's row-major (H, W) pixel order; filtering or downsampling beforehand breaks that correspondence
• mask doesn't reduce to a 2D (H, W) array – a ValueError is raised in that case
• You want to filter by 3D spatial criteria instead of a 2D mask – use filter_point_cloud_using_bounding_box, filter_point_cloud_using_oriented_bounding_box, or a plane-proximity Skill instead
• The point cloud has per-point color and no mask exists yet – segment_point_cloud_using_color segments directly by color without needing a separate 2D mask