Ground Region Segmentation From RGB Image

SUMMARY

Segment ground or floor from one RGB image for navigation, obstacle mapping, or scene understanding. Uses SAM with a bounding box over the ground; outputs a ground mask and box, with Rerun visualization.

Overview

Mobile robots, AGVs, and scene understanding often require segmenting the walkable or drivable surface (ground/floor) from a single RGB frame to separate free space from obstacles. This example shows how to segment the ground region in an RGB image using a bounding box prompt: you provide the image and an ROI over the ground, and the pipeline returns the ground mask and bounding box for navigation, mapping, or analytics.

Inputs

Single RGB image (e.g. warehouse, indoor, or outdoor scene)
Bounding box over the ground/floor region as [x_min, y_min, x_max, y_max]

Required Telekinesis Skills

Cornea — Segment Image Using SAM for instance segmentation from a bounding box prompt

Optional: Rerun for visualization.

Use Cases

This pipeline segments ground or floor regions in RGB images using SAM with a bounding box prompt.

Typical applications include:

Navigation — Identify walkable or drivable surface for path planning.
Obstacle mapping — Separate ground from obstacles for occupancy or cost maps.
Scene understanding — Use ground mask for layout analysis or semantic labeling.
AGV / mobile robots — Segment floor to plan safe lanes or detect non-ground obstacles.

Input-Output

Raw Sensor Input
Ground Segmentation Input

Raw image of a warehouse or indoor scene.

Segmentation and Boxes
Ground Segmentation Output

Segmented image with ground mask and bounding box.

The Pipeline

The pipeline loads an RGB image, defines a bounding box over the ground region, runs SAM for instance segmentation, then extracts the ground mask and bounding box and visualizes with Rerun.

text

Load RGB Image
        ↓
Define ROI (Bounding Box Prompt)
        ↓
Segment Image Using SAM
        ↓
Postprocess Masks
        ↓
Extract Bounding Boxes
        ↓
Visualize with Rerun

Segment Image Using SAM — Instance segmentation from a bounding box prompt; outputs ground mask and bbox.

The Code

The script loads an image, defines a bounding box over the ground region, runs SAM, extracts the mask and bounding box from the annotations, and visualizes with Rerun. Image path and ROI are set at the top; the pipeline runs in the main block with no function arguments.

python

# Load image
image_path = DATA_DIR / "images/warehouse_ground.jpg"
image = io.load_image(image_path)
logger.info(f"Loaded image shape: {image.to_numpy().shape}")

# Define a bounding box: (x_min, y_min, x_max, y_max)
bounding_box = [3, 294, 794, 499]

# Segment using SAM
result = cornea.segment_image_using_sam(
    image=image,
    bboxes=[bounding_box],
)
annotations = result.to_list()

# Rerun visualization
rr.init("ground_segmentation_using_sam", spawn=False)
try:
    rr.connect()
except Exception as e:
    rr.spawn()

rr.send_blueprint(
    rrb.Blueprint(
            rrb.Horizontal(
                rrb.Spatial2DView(name="Input", origin="input"),
                rrb.Spatial2DView(name="Bboxes & Segments", origin="segmented"),
            ),
        rrb.SelectionPanel(),
        rrb.TimePanel(),
    ),
    make_active=True,
)

image = image.to_numpy()
rr.log("input/image", rr.Image(image=image))
rr.log("segmented/image", rr.Image(image=image))

h, w = image.shape[:2]
segmentation_img = np.zeros((h, w), dtype=np.uint16)
ann_bboxes = []
class_ids = []

for idx, ann in enumerate(annotations):
    label = idx + 1
    mask_i = np.zeros((h, w), dtype=np.uint8)
    if "mask" in ann and isinstance(ann["mask"], np.ndarray):
        m = ann["mask"]
        if m.dtype.kind in ("f", "b"):
            mask_i = (m > 0.5).astype(np.uint8)
        else:
            mask_i = (m > 0).astype(np.uint8)
    elif "segmentation" in ann and ann["segmentation"]:
        seg = ann["segmentation"]
        if isinstance(seg, dict):
            mask_dec = mask_utils.decode(seg)
            if mask_dec.ndim == 3:
                mask_dec = mask_dec[:, :, 0]
            mask_i = (mask_dec > 0).astype(np.uint8)
        elif isinstance(seg, list) and len(seg) > 0:
            temp = np.zeros((h, w), dtype=np.uint8)
            polys = seg if isinstance(seg[0], list) else [seg]
            for poly in polys:
                pts = np.array(poly).reshape(-1, 2).astype(np.int32)
                cv2.fillPoly(temp, [pts], 1)
            mask_i = (temp > 0).astype(np.uint8)
    if mask_i.sum() == 0:
        continue
    segmentation_img[mask_i > 0] = label
    bbox = ann.get("bbox", None)
    if bbox is None:
        continue
    ann_bboxes.append(list(bbox))
    class_ids.append(label)

rr.log("segmented/masks", rr.SegmentationImage(segmentation_img))
if ann_bboxes:
    rr.log(
        "segmented/boxes",
        rr.Boxes2D(
            array=np.asarray(ann_bboxes, dtype=np.float32),
            array_format=rr.Box2DFormat.XYWH,
            class_ids=np.asarray(class_ids, dtype=np.int32),
        ),
    )

Ground Region Segmentation From RGB Image ​

Overview ​

Use Cases ​

Input-Output ​