Telekinesis Agentic Skill Library

Appearance

Ground Segmentation with SAM ​

SUMMARY

This section demonstrates how to use the Cornea module for ground (floor) segmentation in industrial environments.

Ground segmentation is the task of identifying drivable floor surfaces so mobile robots, forklifts, and AMRs can navigate safely in warehouses and factories. Real-world floors contain markings, shadows, reflections, pallets, and debris that make rule-based or color-based methods unreliable.

segment_image_using_sam is used to robustly isolate the true ground surface, enabling navigation and safety logic to operate on drivable space, not visual noise.

Details on the skill, code, and practical usage are provided below.

Raw Sensor Input
Ground Segmentation Input
Raw sensor input showing a warehouse or factory floor.
Segmentation and Mask
Ground Segmentation Output
Segmented image showing the ground mask for drivable area estimation.

The Skill ​

For mobile robots and forklifts, the critical question is not what objects are present, but where the robot can safely drive.

Bounding boxes and object detectors identify items such as pallets or racks, but they do not define continuous, traversable ground.

segment_image_using_sam provides a pixel-accurate ground mask that separates the floor from all non-ground regions, even in the presence of shadows, markings, reflections, and clutter.

This ground mask becomes a reliable input for downstream navigation and safety systems.

This segmentation mask enables industrial applications such as:

  1. Forklift and AMR navigation
  2. Drivable area estimation
  3. Obstacle detection by ground subtraction
  4. Safety zone enforcement

See below for a code example demonstrating how to load a raw sensory input, segments the ground using SAM, and access the resulting annotations for further processing or visualization.

The Code ​

python
from telekinesis import cornea
from datatypes import io

# 1. Load a bin image
image = io.load_image(filepath="warehouse_ground.jpg")

# 2.Define bounding boxes for regions of interest
bounding_boxes = [[x_min, y_min, x_max, y_max], ...]

# 3. Segment objects using SAM
result = cornea.segment_image_using_sam(image=image, bbox=bounding_boxes)

# 4. Access COCO-style annotations for visualization and processing
annotations = result["annotation"].to_list()

This code demonstrates how to load the warehouse ground image from the sensor, segment the ground using segment_image_using_sam, and access the resulting masks and bounding boxes for further processing or visualization.

Going Further: From Ground Segmentation to Navigation ​

Once the ground surface is segmented, the following steps enable safe and reliable robot navigation:

  1. Drivable area estimation: Use the ground mask to define where the robot or forklift can safely move.

  2. Obstacle identification: Treat all non-ground regions as obstacles, including pallets, racks, people, and debris.

  3. Safety zone enforcement: Apply distance and clearance rules on the ground mask to enforce slow-down or stop zones.

  4. Path planning constraints: Constrain navigation and motion planning to valid ground regions only.

  5. Execution and continuous update: Recompute the ground mask as the scene changes to maintain safe operation.

Key takeaway ​

Ground segmentation converts a visually complex scene into explicit drivable space for navigation and safety logic.

Other Typical Applications ​

  • Random bin picking
  • Automated sorting
  • Inventory management
  • Quality inspection
  • Palletizing and depalletizing
  • Conveyor tracking
  • segment_image_using_sam
  • segment_using_rgb
  • segment_using_hsv

Running the Example ​

Runnable examples are available in the Telekinesis examples repository. Follow the README in that repository to set up the environment. Once set up, you can run a similar example with:

bash
cd telekinesis-examples
python examples/cornea_examples.py --example segment_image_using_sam