Forklift Segmentation with SAM

SUMMARY

This section demonstrates how to use the Cornea module for forklift segmentation in industrial environments such as warehouses and factories.

Forklift segmentation is the task of identifying and isolating forklifts in images or video frames, even in cluttered or dynamic scenes. Accurate segmentation enables advanced safety, navigation, and analytics applications where simple bounding boxes are not sufficient.

segment_image_using_sam is used to extract precise masks for each forklift, supporting downstream tasks such as dynamic obstacle avoidance, safety zone enforcement, and fleet monitoring.

Details on the skill, code, and practical usage are provided below.

Raw Sensor Input

Raw sensor input of a warehouse or factory.

Segmentation and Mask

Segmented image with mask and bounding box for the detected forklift.

The Skill

For safety, navigation, and analytics, it is critical to accurately isolate each forklift, even in the presence of occlusions, shadows, or overlapping objects.

Bounding boxes alone are insufficient for precise safety logic, fleet monitoring, or real-time obstacle avoidance. segment_image_using_sam provides pixel-accurate masks for each forklift, enabling:

1. Dynamic obstacle avoidance for robots and AGVs
2. Safety zone enforcement
3. Fleet monitoring and tracking
4. Movement and flow analysis

See below for a code example demonstrating how to load an image, segment forklifts using SAM, and access the resulting annotations for further processing or visualization.

The Code

from telekinesis import cornea
from datatypes import io

# 1. Load an image with forklifts
image = io.load_image(filepath="forklift_scene.jpg")

# 2. Define bounding boxes for regions of interest (optional)
bounding_boxes = [[x_min, y_min, x_max, y_max], ...]

# 3. Segment objects using SAM
result = cornea.segment_image_using_sam(image=image, bbox=bounding_boxes)

# 4. Access COCO-style annotations for visualization and processing
annotations = result["annotation"].to_list()

This code demonstrates how to load an image, segment forklifts using segment_image_using_sam, and access the resulting masks and bounding boxes for further processing or visualization.

Going Further: Navigation, Safety, and Fleet Management

Once forklifts are segmented, you can directly use these masks to enhance navigation, safety, and fleet management in warehouses and factories:

1. Dynamic obstacle avoidance: Treat segmented forklifts as dynamic obstacles in the robot's path. Update navigation and path planning in real time to avoid collisions and maintain safe distances.
2. Safety zone enforcement: Monitor restricted or hazardous areas and trigger slow-downs, stops, or alerts if a forklift enters a danger zone.
3. Fleet monitoring and tracking: Track the number, location, and movement of forklifts for process optimization and safety compliance.
4. Anomaly and intrusion detection: Detect unusual behavior or unauthorized forklift operation by analyzing segmented tracks.
5. Continuous monitoring: Update segmentation in real time to maintain situational awareness in dynamic, high-traffic environments.

Key takeaway

Forklift segmentation transforms a visually complex scene into explicit, actionable regions for navigation, safety, and fleet analytics.

Other Typical Applications in Industrial Robotics

• Random bin picking
• Automated sorting
• Inventory management
• Quality inspection
• Palletizing and depalletizing
• Conveyor tracking
• Ground segmentation

Related Skills

• segment_image_using_sam
• segment_using_rgb
• segment_using_hsv

Running the Example

Runnable examples are available in the Telekinesis examples repository. Follow the README in that repository to set up the environment. Once set up, you can run a similar example with:

cd telekinesis-examples
python examples/cornea_examples.py --example segment_image_using_sam