Train a SAM3-LoRA Model
SUMMARY
Fine-tune SAM3 for text-prompted instance segmentation with compact LoRA adapters. Configure prompts and adapter targets with SAM3LoRAConfig, prepare a mask-annotated COCO dataset, and run or resume training with the shared Iris Trainer.
Python Version Support
Iris training supports Python 3.11 and 3.12 only. Create an environment with one of these versions before installing dependencies.
Requirements
| Requirement | Value |
|---|---|
| Python | 3.11 or 3.12 |
| Platform | Linux, macOS, or Windows |
| Dataset | COCO annotations with an instance mask for every object |
| Base weights | Hugging Face access or a local SAM3 checkpoint |
| GPU | CUDA GPU strongly recommended |
Configure Hugging Face Access
When load_from_hf=True, Iris downloads the base model from the gated facebook/sam3 repository. Complete these steps with the same Hugging Face user account:
- Sign in and request access to
facebook/sam3. Accept the repository terms and wait for approval if required. - After access is granted, create a token with Read access, or fine-grained read access to
facebook/sam3. - Set the token as
HF_TOKENin the terminal used for training.
export HF_TOKEN="hf_your_token"$env:HF_TOKEN = "hf_your_token"Verify that the variable is available without printing the token:
import os
assert os.environ.get("HF_TOKEN"), "HF_TOKEN is not configured"Keep Tokens Private
Gated access is granted to an individual Hugging Face user, so the token must belong to the account whose SAM3 access request was approved. Do not add the token to source files, logs, or version control. These commands set it only for the current terminal session.
Quick Installation
Install PyTorch for your CUDA version, then install Iris with the SAM3-LoRA dependencies:
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
python -m pip install "telekinesis-ai[iris-sam3-lora]"PyTorch Installation
For CPU, macOS, or another CUDA version, select the appropriate command in the PyTorch installation guide.
Training Workflow
The example below derives prompts from COCO categories, exposes LoRA and Trainer settings as command-line arguments, configures optimization and augmentation, and optionally resumes from a checkpoint.
Create train_sam3_lora.py
"""Fine-tune SAM3 with LoRA on a COCO dataset."""
import argparse
from pathlib import Path
import torch
from loguru import logger
from telekinesis.iris.dataset import COCODataset, prepare_coco_datasets
from telekinesis.iris.logger import TensorBoardMetricLogger
from telekinesis.iris.trainer import Precision, Trainer
def parse_args() -> argparse.Namespace:
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--dataset-dir", type=Path, required=True)
parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--run-name", default="sam3_lora")
parser.add_argument("--sam3-checkpoint", type=Path)
parser.add_argument("--load-from-hf", action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--rank", type=int, default=16)
parser.add_argument("--alpha", type=float, default=32.0)
parser.add_argument("--dropout", type=float, default=0.1)
parser.add_argument("--resolution", type=int, default=1008)
parser.add_argument("--num-negative-prompts", type=int, default=3)
parser.add_argument("--epochs", type=int, default=500)
parser.add_argument("--batch-size", type=int, default=1)
parser.add_argument("--gradient-accumulation-steps", type=int, default=32)
parser.add_argument("--num-workers", type=int, default=2)
parser.add_argument("--learning-rate", type=float, default=5e-5)
parser.add_argument("--weight-decay", type=float, default=0.01)
parser.add_argument("--warmup-steps", type=int, default=200)
parser.add_argument("--gradient-clip-norm", type=float, default=0.1)
parser.add_argument("--validation-split", type=float, default=0.3)
parser.add_argument("--evaluation-interval", type=int, default=1)
parser.add_argument("--checkpoint-interval", type=int, default=50)
parser.add_argument("--seed", type=int, default=42)
parser.add_argument("--device", help="For example: cuda, cuda:0, or cpu.")
parser.add_argument(
"--precision",
choices=[precision.value for precision in Precision],
default=Precision.AUTO.value,
)
parser.add_argument("--resume-from", type=Path)
return parser.parse_args()
def train_sam3_lora(args: argparse.Namespace) -> None:
"""Configure and train SAM3 LoRA."""
from telekinesis.iris.models import SAM3LoRA, SAM3LoRAConfig
metadata_dir = args.dataset_dir / "train"
if not metadata_dir.is_dir():
metadata_dir = args.dataset_dir
categories = COCODataset(metadata_dir).categories
torch.manual_seed(args.seed)
config = SAM3LoRAConfig(
class_names=categories,
sam3_checkpoint_path=args.sam3_checkpoint,
load_from_hf=args.load_from_hf,
rank=args.rank,
alpha=args.alpha,
dropout=args.dropout,
apply_to_vision_encoder=True,
apply_to_text_encoder=False,
apply_to_geometry_encoder=True,
apply_to_detr_encoder=True,
apply_to_detr_decoder=True,
apply_to_mask_decoder=True,
resolution=args.resolution,
num_negative_prompts=args.num_negative_prompts,
learning_rate=args.learning_rate,
weight_decay=args.weight_decay,
scheduler="cosine",
scheduler_interval="step",
warmup_steps=args.warmup_steps,
augmentation=[
{"name": "random_scale_crop", "scale_range": (0.8, 1.2)},
{"name": "horizontal_flip", "probability": 0.5},
{
"name": "photometric_distort",
"probability": 0.8,
"brightness": 0.35,
"contrast": 0.35,
"saturation": 0.3,
"hue": 0.03,
"grayscale_probability": 0.05,
},
{"name": "gamma", "probability": 0.4, "gamma_range": (0.7, 1.4)},
{
"name": "gaussian_blur",
"probability": 0.2,
"sigma_range": (0.2, 1.2),
},
{"name": "sharpness", "probability": 0.2, "factor_range": (0.5, 1.5)},
{"name": "gaussian_noise", "probability": 0.3, "std_range": (0.005, 0.03)},
{
"name": "jpeg_compression",
"probability": 0.3,
"quality_range": (60, 100),
},
],
)
model = SAM3LoRA(config=config)
training_dataset, validation_dataset = prepare_coco_datasets(
dataset_dir=args.dataset_dir,
validation_split=args.validation_split,
seed=args.seed,
train_transforms=model.train_transforms,
val_transforms=model.val_transforms,
include_masks=model.requires_masks,
)
device = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
logger.info(f"Training SAM3 LoRA on {len(training_dataset)} images using {device}.")
metric_logger = TensorBoardMetricLogger(args.output_dir / "tensorboard")
model_trainer = Trainer(
model=model,
run_name=args.run_name,
training_dataset=training_dataset,
validation_dataset=validation_dataset,
output_dir=args.output_dir,
epochs=args.epochs,
batch_size=args.batch_size,
device=device,
num_workers=args.num_workers,
gradient_clip_norm=args.gradient_clip_norm,
precision=args.precision,
gradient_accumulation_steps=args.gradient_accumulation_steps,
evaluation_interval=args.evaluation_interval,
checkpoint_interval=args.checkpoint_interval,
seed=args.seed,
metric_logger=metric_logger,
)
if args.resume_from is None:
model_trainer.train()
else:
model_trainer.resume(args.resume_from)
if __name__ == "__main__":
train_sam3_lora(parse_args())Launch Training
Start a run with the default values from parse_args():
python train_sam3_lora.py \
--dataset-dir path/to/dataset \
--output-dir results/sam3-loraOverride the most important model and Trainer settings as needed:
python train_sam3_lora.py \
--dataset-dir path/to/dataset \
--output-dir results/sam3-lora \
--run-name custom-sam3-lora \
--rank 16 \
--alpha 32 \
--epochs 500 \
--batch-size 1 \
--gradient-accumulation-steps 32 \
--learning-rate 5e-5 \
--device cuda \
--precision autoTo use a local SAM3 base checkpoint instead of Hugging Face, add --sam3-checkpoint path/to/sam3.pt --no-load-from-hf.
Resume Training
Resume from a Trainer checkpoint while keeping the model configuration compatible:
python train_sam3_lora.py \
--dataset-dir path/to/dataset \
--output-dir results/sam3-lora \
--resume-from results/sam3-lora/latest.ptCommand-Line Arguments
The training script accepts the following options:
| Argument | Default | Description |
|---|---|---|
--dataset-dir | required | COCO dataset root or split directory |
--output-dir | required | Training outputs, logs, and checkpoints |
--run-name | "sam3_lora" | Name stored in run and export metadata |
--sam3-checkpoint | None | Optional local SAM3 base checkpoint |
--load-from-hf / --no-load-from-hf | enabled | Enable or disable base-weight loading from Hugging Face |
--rank | 16 | LoRA rank |
--alpha | 32.0 | LoRA scaling factor |
--dropout | 0.1 | LoRA dropout probability |
--resolution | 1008 | Square training resolution |
--num-negative-prompts | 3 | Negative prompts sampled per image |
--epochs | 500 | Total training epochs, including restored epochs |
--batch-size | 1 | Samples per micro-batch |
--gradient-accumulation-steps | 32 | Micro-batches accumulated per optimizer update |
--num-workers | 2 | DataLoader worker processes |
--learning-rate | 5e-5 | AdamW learning rate |
--weight-decay | 0.01 | AdamW weight decay |
--warmup-steps | 200 | Scheduler warmup steps |
--gradient-clip-norm | 0.1 | Maximum gradient norm |
--validation-split | 0.3 | Fallback validation fraction when no validation split exists |
--evaluation-interval | 1 | Epochs between validation runs |
--checkpoint-interval | 50 | Epochs between numbered checkpoints |
--seed | 42 | Training and dataset-split seed |
--device | automatic | Device such as cuda, cuda:0, or cpu |
--precision | "auto" | "auto", "fp32", "fp16", or "bf16" |
--resume-from | None | Trainer checkpoint from which to continue |
Training Guidance
See Best Practices: Trainer for guidance on tuning batch size, learning rate, epochs, gradient clipping, workers, evaluation, and checkpoints.
Segmentation Masks and Prompts
Every annotated object must contain a non-empty COCO polygon or RLE segmentation. Use stable, descriptive category names because Iris uses them as the training prompts and stores them in the deployment bundle.
Run the Example
A runnable version is available in the Telekinesis examples repository:
cd telekinesis-examples
python examples/iris/train_sam3_lora.py \
--dataset-dir path/to/dataset \
--output-dir results/sam3-loraAdd --resume-from results/sam3-lora/latest.pt to continue a previous run.
Checkpoints and Metrics
The training script saves the following outputs to the configured output_dir:
| Output | Description |
|---|---|
latest.pt | Updated after every completed epoch |
best.pt | Updated when validation mask mAP improves |
epoch_NNNN.pt | Written according to checkpoint_interval |
| TensorBoard logs | Training loss, learning rate, validation box/mask metrics, and previews |
SAM3-LoRA checkpoints store the compact LoRA state plus the metadata required to reconstruct the model and its prompts.
Model Reference
SAM3LoRAConfig
SAM3LoRAConfig combines base-model loading, prompts, LoRA targets, preprocessing, and optimization settings.
from telekinesis.iris.models import SAM3LoRA, SAM3LoRAConfig
config = SAM3LoRAConfig(
class_names={1: "part", 2: "tool"},
load_from_hf=True,
rank=16,
alpha=32.0,
dropout=0.1,
resolution=1008,
num_negative_prompts=3,
learning_rate=5e-5,
weight_decay=0.01,
scheduler="cosine",
scheduler_interval="step",
warmup_steps=200,
)
model = SAM3LoRA(config)| Option | Default | Description |
|---|---|---|
class_names | None | Category-ID-to-prompt mapping or ordered prompt sequence; required by SAM3LoRA |
sam3_checkpoint_path | None | Optional local SAM3 base checkpoint |
lora_weights_path | None | Optional LoRA weights loaded after model construction |
load_from_hf | True | Loads the base SAM3 checkpoint from Hugging Face |
rank | 8 | Positive LoRA decomposition rank |
alpha | 16.0 | Positive LoRA scaling factor |
dropout | 0.1 | LoRA dropout probability in [0, 1) |
target_modules | standard projection names | Module-name suffixes eligible for LoRA injection |
apply_to_vision_encoder | True | Adapts the vision encoder |
apply_to_text_encoder | True | Adapts the text encoder |
apply_to_geometry_encoder | False | Adapts the geometry encoder |
apply_to_detr_encoder | True | Adapts the DETR encoder |
apply_to_detr_decoder | True | Adapts the DETR decoder |
apply_to_mask_decoder | False | Adapts the mask decoder |
resolution | 1008 | Positive square training resolution |
num_negative_prompts | 2 | Negative category prompts sampled per image |
optimizer | "adamw" | Optimizer built by the model adapter |
optimizer_options | {} | Additional optimizer keyword arguments |
learning_rate | 1e-4 | Optimizer learning rate |
weight_decay | 0.01 | Optimizer weight decay |
scheduler | None | Optional "step", "cosine", or "plateau" scheduler |
scheduler_options | {} | Scheduler-specific keyword arguments |
scheduler_interval | "epoch" | "step" or "epoch" |
scheduler_monitor | None | Metric used by a monitored scheduler |
warmup_steps | 0 | Linear warmup optimizer steps |
augmentation | () | Preset name or declarative target-aware pipeline |
SAM3LoRA Properties
| Property | Type | Description |
|---|---|---|
config | SAM3LoRAConfig | Public unified configuration |
model_config | SAM3LoRAConfig | Model-construction configuration |
adapter_config | SAM3LoRAConfig | Configuration used by the training adapter |
model | nn.Module | Underlying SAM3 LoRA PyTorch model |
adapter | SAM3LoRATrainingAdapter | Cached prompted-training adapter |
descriptor | ModelDescriptor | SAM3-LoRA family and instance-segmentation task metadata |
train_transforms | target-aware callable | Configured shared augmentation pipeline; SAM3 preprocessing remains adapter-owned |
val_transforms | None | Preprocessing is handled by the adapter |
requires_masks | bool | Always True |
Trainer Reference
See Configuration: Trainer Reference for all Trainer constructor options, defaults, and methods.
SAM3 Best Practices
Prompts and Categories
- Use stable, descriptive COCO category names because they become the text prompts used during training.
- Keep category IDs and names consistent between training and validation datasets.
- Avoid renaming or reordering prompts when resuming a checkpoint.
- Inspect prompt-specific validation results when classes are visually similar or semantically ambiguous.
Base Checkpoint Loading
With the default load_from_hf=True, Iris downloads the SAM3 base checkpoint from Hugging Face. Use a local checkpoint when training must run offline or when a specific base checkpoint must be reproduced:
config = SAM3LoRAConfig(
class_names=categories,
sam3_checkpoint_path="checkpoints/sam3.pt",
load_from_hf=False,
)Checkpoint Loading
sam3_checkpoint_path is required when load_from_hf=False. Use lora_weights_path only when initializing the model with existing LoRA parameters.
LoRA Capacity
rank, alpha, and dropout control LoRA capacity and regularization:
- Increase
rankwhen the default adapters do not have enough capacity. Higher ranks add trainable parameters and consume more memory. - Tune
alphawithrank; a common starting relationship isalphaequal to one or two times the rank. - Increase
dropoutwhen the adapted model overfits. Reduce it when training is underfitting or unstable because too much signal is being dropped. - Change one setting at a time and compare validation mask metrics.
config = SAM3LoRAConfig(
class_names=categories,
rank=16,
alpha=32.0,
dropout=0.1,
)LoRA Target Selection
Enable only the model components that need adaptation. Adapting more components increases capacity, memory use, and training time.
config = SAM3LoRAConfig(
class_names=categories,
rank=16,
alpha=32.0,
apply_to_vision_encoder=True,
apply_to_text_encoder=False,
apply_to_geometry_encoder=True,
apply_to_detr_encoder=True,
apply_to_detr_decoder=True,
apply_to_mask_decoder=True,
)Start with the defaults, then expand the targets only when validation results show that more adaptation is needed. At least one enabled component must contain a name matching target_modules.
The default target_modules are q_proj, k_proj, v_proj, out_proj, qkv, proj, fc1, fc2, c_fc, c_proj, linear1, and linear2. Override this tuple only when the target SAM3 implementation uses different module names or you intentionally want a narrower adapter set.
Resolution and Negative Prompts
- Keep
resolution=1008initially. Higher resolutions increase memory and compute, while lower resolutions may lose small-object detail. - Reduce the Trainer
batch_sizebefore reducing resolution when GPU memory is limited. - Increase
num_negative_promptswhen the model confuses absent categories with foreground objects. - Too many negative prompts increase work per image and may dilute positive supervision; validate each change.
Existing LoRA Weights and Resume
- Use
lora_weights_pathto initialize a new model from existing LoRA parameters. - Use
Trainer.resume()when continuing an Iris training run because it also restores optimizer, scheduler, scaler, history, and epoch state. - Keep prompts, resolution, LoRA rank, enabled targets, and target module names compatible with the saved weights.
- Preserve
best.ptfor deployment selection andlatest.ptfor continuing interrupted training.