Filter Point Cloud Using Mask

SUMMARY

Filter Point Cloud Using Mask keeps only the points whose corresponding pixel in a 2D mask is True.

It only works on an organized (structured) point cloud — one whose points are still in the same row-major (H, W) pixel order they came from a depth camera in, such as straight out of convert_depth_image_to_point_cloud, before any filtering or downsampling reorders or drops points. Each mask pixel (row, col) is matched to the point at that same position, and only points behind a True pixel survive — this is how a 2D image-based mask (e.g. from cornea.segment_image_foreground_using_birefnet, pupil.calculate_mask_centroid, or a hand-drawn ROI) gets combined with its corresponding depth data.

Use this Skill when you want to apply a 2D image mask directly to its corresponding organized point cloud.

The Skill

from telekinesis import vitreous
import numpy as np

mask = np.array([[1, 0, 1], [0, 1, 0]], dtype=np.uint8)

filtered_point_cloud = vitreous.filter_point_cloud_using_mask(
    point_cloud=point_cloud,
    mask=mask,
)

API Reference

Full parameter and return type documentation for filter_point_cloud_using_mask.

View Reference →

Data Transfer Notice

There is no longer a fixed limit of 1 million points per request. However, very large datasets may result in slower data transfer and processing times. We are continuously optimizing performance as part of our beta program, with ongoing improvements to enhance speed and reliability.

Example

Raw Sensor Input

Unprocessed point cloud captured directly from the sensor.

Boolean Mask

Boolean Mask for selecting a subset of points, i.e. the cans.

Masked Output

Masked point cloud showing only the points selected by the boolean mask. The remaining thin lines can be removed in a post-processing step, for example using statistical outlier removal.

The Code

"""
Demonstrates filtering a structured point cloud using a 2D binary mask, keeping only points where the mask is True.
"""

from loguru import logger
import rerun as rr
import rerun.blueprint as rrb

from telekinesis import vitreous, datatypes, cornea


def filter_point_cloud_using_mask_example():
    """
    Filters a structured point cloud using a 2D binary mask.

    Applies a 2D image mask to an organized point cloud, keeping only points
    where the corresponding pixel is True.
    """
    # ===================== Load Data ==========================================
    point_cloud_url = (
        "https://assets.telekinesis.ai/examples/v1/point_clouds/can_vertical_6_raw.ply"
    )
    point_cloud = datatypes.PointCloud.from_url(url=point_cloud_url, use_cache=True)

    mask_url = (
        "https://assets.telekinesis.ai/examples/v1/images/can_vertical_6_mask.png"
    )
    mask_image = datatypes.Image.from_url(url=mask_url)
    mask = cornea.segment_image_using_threshold(image=mask_image, min_value=127)

    # ===================== Run Skill ==========================================
    result_point_cloud = vitreous.filter_point_cloud_using_mask(
        point_cloud=point_cloud,
        mask=mask,
    )

    # ===================== Log ================================================
    logger.success(f"Filtered {point_cloud} using mask")
    logger.success(f"Results: {result_point_cloud}")
    logger.info(
        f"Result point cloud positions shape: {result_point_cloud.positions.shape}"
    )
    logger.info(
        f"Result point cloud has normals shape: "
        f"{result_point_cloud.normals.shape if result_point_cloud.has_normals else None}"
    )
    logger.info(
        f"Result point cloud has colors shape: "
        f"{result_point_cloud.colors.shape if result_point_cloud.has_colors else None}"
    )

    # ===================== Visualization  (Optional) ===========================
    rr.init("filter_point_cloud_using_mask_example", spawn=True)
    rr.send_blueprint(
        rrb.Blueprint(
            rrb.Horizontal(
                rrb.Spatial3DView(
                    name="Input Point Cloud",
                    origin="/1-input_point_cloud",
                ),
                rrb.Spatial2DView(
                    name="Binary Mask",
                    origin="/2-binary_mask",
                ),
                rrb.Spatial3DView(
                    name="Masked Point Cloud",
                    origin="/3-masked_point_cloud",
                ),
            )
        )
    )
    datatypes.visualize(point_cloud, entity_path="/1-input_point_cloud")
    datatypes.visualize(mask, entity_path="/2-binary_mask")
    datatypes.visualize(result_point_cloud, entity_path="/3-masked_point_cloud")


if __name__ == "__main__":
    filter_point_cloud_using_mask_example()

Runnable examples are available in the Telekinesis examples repository.

Follow the README in that repository to set up the environment, run this specific example with:

cd telekinesis-examples
python examples/point_cloud/filter_point_cloud_using_mask.py

Parameter Configuration

point_cloud must be organized/structured for mask to align correctly — see the precondition in the summary above. Points at the camera origin [0, 0, 0] or with non-finite coordinates are dropped even if their mask pixel is True; if masking leaves no valid points, the request fails instead of returning an empty point cloud.

Parameter	Type	Default	Description
point_cloud	datatypes.PointCloud	required	The organized/structured point cloud to filter — its points must still correspond 1:1 to mask's pixel grid, in the same row-major order
mask	datatypes.SegmentationImage | np.ndarray | list	required	The 2D mask selecting which points to keep, shape (H, W) matching point_cloud's pixel grid (or (H, W, 1), squeezed automatically). Any non-zero/True value marks a point to keep. A raw array must use one of SegmentationImage's supported integer dtypes (e.g. uint8) — boolean and floating-point arrays are rejected. Build one from a grayscale mask image with cornea.segment_image_using_threshold

Returns

Type	Description
datatypes.PointCloud	A point cloud containing only the points behind a True mask pixel. Use .positions for the surviving (N, 3) position array and len(...) for the point count.

Raises

Exception	Condition
TypeError	A parameter's value does not match its expected type (see the Parameter Configuration table above)
ValueError	mask doesn't reduce to a 2D (H, W) array (after squeezing a trailing single-channel axis, e.g. (H, W, 1)), or point_cloud's point count doesn't match mask's height * width
ConfigurationError	The TELEKINESIS_API_KEY environment variable is not set
SerializationError	The request input failed to serialize, the response was not returned as an Arrow stream, or the response failed to deserialize
RequestTimeoutError	The request to the Vitreous service timed out
TransportError	A network failure occurred before a response was received
ClientError	The Vitreous service rejected the request due to invalid or malformed input (HTTP 400/422), an unrecognized endpoint (HTTP 404), or another unexpected 4xx response
AuthenticationError	The API key was rejected as invalid or expired (HTTP 401)
AuthenticationServiceError	The authentication service returned an invalid response, was temporarily unavailable, or timed out (HTTP 502/503/504)
ServerError	The Vitreous service returned a 5xx or otherwise unexpected error response

How to Tune the Parameters

filter_point_cloud_using_mask has no numeric threshold to tune — mask is an input defining which points to keep, not a knob with a range to sweep. What you control is the mask itself:

mask

• Controls: Which points are kept — a point survives only if the mask pixel at its (row, col) position is non-zero/True.
• Must match point_cloud's pixel grid shape, (H, W) (a trailing single-channel axis, (H, W, 1), is squeezed automatically).
• Must use a supported integer dtype (e.g. uint8) when passed as a raw array — a boolean or floating-point array is rejected.
• Build it from a grayscale mask image with cornea.segment_image_using_threshold, or from any other 2D detection/segmentation output — e.g. cornea.segment_image_foreground_using_birefnet — that shares the point cloud's pixel grid.
• A tighter mask (smaller True region) keeps fewer points; a looser mask keeps more — but note that masking down to zero remaining points causes the request to fail rather than returning an empty point cloud.

TIP

Load the point cloud with duplicate/infinite/NaN-point removal disabled (e.g. remove_duplicated_points=False, remove_infinite_points=False, remove_nan_points=False) so its point count still matches the mask's pixel grid — those cleanup steps reorder or drop points and would break the row-major correspondence this Skill depends on. Even with that disabled, points at the camera origin [0, 0, 0] or with non-finite coordinates are still dropped internally, regardless of their mask value.

Where to Use the Skill

Common pipelines include:

• Detection-guided isolation – Combining a 2D object mask (e.g. from a segmentation or detection skill) with its source depth frame
• Interactive ROI selection – Keeping only the points behind a hand-drawn or user-selected region of an image
• Multi-object separation – Isolating one object's points from a scene using its per-object mask, one mask at a time
• Preprocessing straight off a depth sensor – Filtering a freshly-converted organized point cloud before any reordering step

Alternative Skills

Skill	vs. Filter Point Cloud Using Mask
segment_point_cloud_using_color	Segments directly by each point's own RGB color instead of a separately-computed 2D mask, and works on any point cloud with per-point colors, organized or not. Use color segmentation when no mask exists yet; use mask filtering when a 2D mask is already available.
filter_point_cloud_using_bounding_box	Filters by a 3D spatial box instead of a 2D pixel-aligned mask, and works on any point cloud, organized or not. Use it when the region of interest is a 3D volume rather than a 2D image region.

When Not to Use the Skill

Do not use Filter Point Cloud Using Mask when:

• The point cloud is unorganized/unstructured – this Skill requires the point cloud's points to still be in the mask's row-major (H, W) pixel order; filtering or downsampling beforehand breaks that correspondence
• mask doesn't reduce to a 2D (H, W) array – a ValueError is raised in that case
• You want to filter by 3D spatial criteria instead of a 2D mask – use filter_point_cloud_using_bounding_box, filter_point_cloud_using_oriented_bounding_box, or a plane-proximity Skill instead
• The point cloud has per-point color and no mask exists yet – segment_point_cloud_using_color segments directly by color without needing a separate 2D mask