Skip to content

Train a SAM3-LoRA Model ​

SUMMARY

Fine-tune SAM3 for text-prompted instance segmentation with compact LoRA adapters. Configure prompts and adapter targets with SAM3LoRAConfig, prepare a mask-annotated COCO dataset, and run or resume training with the shared Iris Trainer.

Python Version Support

Iris training supports Python 3.11 and 3.12 only. Create an environment with one of these versions before installing dependencies.

Requirements ​

RequirementValue
Python3.11 or 3.12
PlatformLinux, macOS, or Windows
DatasetCOCO annotations with an instance mask for every object
Base weightsHugging Face access or a local SAM3 checkpoint
GPUCUDA GPU strongly recommended

Configure Hugging Face Access ​

When load_from_hf=True, Iris downloads the base model from the gated facebook/sam3 repository. Complete these steps with the same Hugging Face user account:

  1. Sign in and request access to facebook/sam3. Accept the repository terms and wait for approval if required.
  2. After access is granted, create a token with Read access, or fine-grained read access to facebook/sam3.
  3. Set the token as HF_TOKEN in the terminal used for training.
bash
export HF_TOKEN="hf_your_token"
powershell
$env:HF_TOKEN = "hf_your_token"

Verify that the variable is available without printing the token:

python
import os

assert os.environ.get("HF_TOKEN"), "HF_TOKEN is not configured"

Keep Tokens Private

Gated access is granted to an individual Hugging Face user, so the token must belong to the account whose SAM3 access request was approved. Do not add the token to source files, logs, or version control. These commands set it only for the current terminal session.

Quick Installation ​

Install PyTorch for your CUDA version, then install Iris with the SAM3-LoRA dependencies:

bash
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
python -m pip install "telekinesis-ai[iris-sam3-lora]"

PyTorch Installation

For CPU, macOS, or another CUDA version, select the appropriate command in the PyTorch installation guide.

Training Workflow ​

The example below derives prompts from COCO categories, exposes LoRA and Trainer settings as command-line arguments, configures optimization and augmentation, and optionally resumes from a checkpoint.

Create train_sam3_lora.py ​

python
"""Fine-tune SAM3 with LoRA on a COCO dataset."""

import argparse
from pathlib import Path

import torch
from loguru import logger

from telekinesis.iris.dataset import COCODataset, prepare_coco_datasets
from telekinesis.iris.logger import TensorBoardMetricLogger
from telekinesis.iris.trainer import Precision, Trainer


def parse_args() -> argparse.Namespace:
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--dataset-dir", type=Path, required=True)
    parser.add_argument("--output-dir", type=Path, required=True)
    parser.add_argument("--run-name", default="sam3_lora")
    parser.add_argument("--sam3-checkpoint", type=Path)
    parser.add_argument("--load-from-hf", action=argparse.BooleanOptionalAction, default=True)
    parser.add_argument("--rank", type=int, default=16)
    parser.add_argument("--alpha", type=float, default=32.0)
    parser.add_argument("--dropout", type=float, default=0.1)
    parser.add_argument("--resolution", type=int, default=1008)
    parser.add_argument("--num-negative-prompts", type=int, default=3)
    parser.add_argument("--epochs", type=int, default=500)
    parser.add_argument("--batch-size", type=int, default=1)
    parser.add_argument("--gradient-accumulation-steps", type=int, default=32)
    parser.add_argument("--num-workers", type=int, default=2)
    parser.add_argument("--learning-rate", type=float, default=5e-5)
    parser.add_argument("--weight-decay", type=float, default=0.01)
    parser.add_argument("--warmup-steps", type=int, default=200)
    parser.add_argument("--gradient-clip-norm", type=float, default=0.1)
    parser.add_argument("--validation-split", type=float, default=0.3)
    parser.add_argument("--evaluation-interval", type=int, default=1)
    parser.add_argument("--checkpoint-interval", type=int, default=50)
    parser.add_argument("--seed", type=int, default=42)
    parser.add_argument("--device", help="For example: cuda, cuda:0, or cpu.")
    parser.add_argument(
        "--precision",
        choices=[precision.value for precision in Precision],
        default=Precision.AUTO.value,
    )
    parser.add_argument("--resume-from", type=Path)
    return parser.parse_args()


def train_sam3_lora(args: argparse.Namespace) -> None:
    """Configure and train SAM3 LoRA."""
    from telekinesis.iris.models import SAM3LoRA, SAM3LoRAConfig

    metadata_dir = args.dataset_dir / "train"
    if not metadata_dir.is_dir():
        metadata_dir = args.dataset_dir
    categories = COCODataset(metadata_dir).categories
    torch.manual_seed(args.seed)

    config = SAM3LoRAConfig(
        class_names=categories,
        sam3_checkpoint_path=args.sam3_checkpoint,
        load_from_hf=args.load_from_hf,
        rank=args.rank,
        alpha=args.alpha,
        dropout=args.dropout,
        apply_to_vision_encoder=True,
        apply_to_text_encoder=False,
        apply_to_geometry_encoder=True,
        apply_to_detr_encoder=True,
        apply_to_detr_decoder=True,
        apply_to_mask_decoder=True,
        resolution=args.resolution,
        num_negative_prompts=args.num_negative_prompts,
        learning_rate=args.learning_rate,
        weight_decay=args.weight_decay,
        scheduler="cosine",
        scheduler_interval="step",
        warmup_steps=args.warmup_steps,
        augmentation=[
            {"name": "random_scale_crop", "scale_range": (0.8, 1.2)},
            {"name": "horizontal_flip", "probability": 0.5},
            {
                "name": "photometric_distort",
                "probability": 0.8,
                "brightness": 0.35,
                "contrast": 0.35,
                "saturation": 0.3,
                "hue": 0.03,
                "grayscale_probability": 0.05,
            },
            {"name": "gamma", "probability": 0.4, "gamma_range": (0.7, 1.4)},
            {
                "name": "gaussian_blur",
                "probability": 0.2,
                "sigma_range": (0.2, 1.2),
            },
            {"name": "sharpness", "probability": 0.2, "factor_range": (0.5, 1.5)},
            {"name": "gaussian_noise", "probability": 0.3, "std_range": (0.005, 0.03)},
            {
                "name": "jpeg_compression",
                "probability": 0.3,
                "quality_range": (60, 100),
            },
        ],
    )
    model = SAM3LoRA(config=config)
    training_dataset, validation_dataset = prepare_coco_datasets(
        dataset_dir=args.dataset_dir,
        validation_split=args.validation_split,
        seed=args.seed,
        train_transforms=model.train_transforms,
        val_transforms=model.val_transforms,
        include_masks=model.requires_masks,
    )
    device = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
    logger.info(f"Training SAM3 LoRA on {len(training_dataset)} images using {device}.")
    metric_logger = TensorBoardMetricLogger(args.output_dir / "tensorboard")
    model_trainer = Trainer(
        model=model,
        run_name=args.run_name,
        training_dataset=training_dataset,
        validation_dataset=validation_dataset,
        output_dir=args.output_dir,
        epochs=args.epochs,
        batch_size=args.batch_size,
        device=device,
        num_workers=args.num_workers,
        gradient_clip_norm=args.gradient_clip_norm,
        precision=args.precision,
        gradient_accumulation_steps=args.gradient_accumulation_steps,
        evaluation_interval=args.evaluation_interval,
        checkpoint_interval=args.checkpoint_interval,
        seed=args.seed,
        metric_logger=metric_logger,
    )
    if args.resume_from is None:
        model_trainer.train()
    else:
        model_trainer.resume(args.resume_from)


if __name__ == "__main__":
    train_sam3_lora(parse_args())

Launch Training ​

Start a run with the default values from parse_args():

bash
python train_sam3_lora.py \
  --dataset-dir path/to/dataset \
  --output-dir results/sam3-lora

Override the most important model and Trainer settings as needed:

bash
python train_sam3_lora.py \
  --dataset-dir path/to/dataset \
  --output-dir results/sam3-lora \
  --run-name custom-sam3-lora \
  --rank 16 \
  --alpha 32 \
  --epochs 500 \
  --batch-size 1 \
  --gradient-accumulation-steps 32 \
  --learning-rate 5e-5 \
  --device cuda \
  --precision auto

To use a local SAM3 base checkpoint instead of Hugging Face, add --sam3-checkpoint path/to/sam3.pt --no-load-from-hf.

Resume Training ​

Resume from a Trainer checkpoint while keeping the model configuration compatible:

bash
python train_sam3_lora.py \
  --dataset-dir path/to/dataset \
  --output-dir results/sam3-lora \
  --resume-from results/sam3-lora/latest.pt

Command-Line Arguments ​

The training script accepts the following options:

ArgumentDefaultDescription
--dataset-dirrequiredCOCO dataset root or split directory
--output-dirrequiredTraining outputs, logs, and checkpoints
--run-name"sam3_lora"Name stored in run and export metadata
--sam3-checkpointNoneOptional local SAM3 base checkpoint
--load-from-hf / --no-load-from-hfenabledEnable or disable base-weight loading from Hugging Face
--rank16LoRA rank
--alpha32.0LoRA scaling factor
--dropout0.1LoRA dropout probability
--resolution1008Square training resolution
--num-negative-prompts3Negative prompts sampled per image
--epochs500Total training epochs, including restored epochs
--batch-size1Samples per micro-batch
--gradient-accumulation-steps32Micro-batches accumulated per optimizer update
--num-workers2DataLoader worker processes
--learning-rate5e-5AdamW learning rate
--weight-decay0.01AdamW weight decay
--warmup-steps200Scheduler warmup steps
--gradient-clip-norm0.1Maximum gradient norm
--validation-split0.3Fallback validation fraction when no validation split exists
--evaluation-interval1Epochs between validation runs
--checkpoint-interval50Epochs between numbered checkpoints
--seed42Training and dataset-split seed
--deviceautomaticDevice such as cuda, cuda:0, or cpu
--precision"auto""auto", "fp32", "fp16", or "bf16"
--resume-fromNoneTrainer checkpoint from which to continue

Training Guidance

See Best Practices: Trainer for guidance on tuning batch size, learning rate, epochs, gradient clipping, workers, evaluation, and checkpoints.

Segmentation Masks and Prompts

Every annotated object must contain a non-empty COCO polygon or RLE segmentation. Use stable, descriptive category names because Iris uses them as the training prompts and stores them in the deployment bundle.

Run the Example ​

A runnable version is available in the Telekinesis examples repository:

bash
cd telekinesis-examples
python examples/iris/train_sam3_lora.py \
  --dataset-dir path/to/dataset \
  --output-dir results/sam3-lora

Add --resume-from results/sam3-lora/latest.pt to continue a previous run.

Checkpoints and Metrics ​

The training script saves the following outputs to the configured output_dir:

OutputDescription
latest.ptUpdated after every completed epoch
best.ptUpdated when validation mask mAP improves
epoch_NNNN.ptWritten according to checkpoint_interval
TensorBoard logsTraining loss, learning rate, validation box/mask metrics, and previews

SAM3-LoRA checkpoints store the compact LoRA state plus the metadata required to reconstruct the model and its prompts.

Model Reference ​

SAM3LoRAConfig ​

SAM3LoRAConfig combines base-model loading, prompts, LoRA targets, preprocessing, and optimization settings.

python
from telekinesis.iris.models import SAM3LoRA, SAM3LoRAConfig

config = SAM3LoRAConfig(
    class_names={1: "part", 2: "tool"},
    load_from_hf=True,
    rank=16,
    alpha=32.0,
    dropout=0.1,
    resolution=1008,
    num_negative_prompts=3,
    learning_rate=5e-5,
    weight_decay=0.01,
    scheduler="cosine",
    scheduler_interval="step",
    warmup_steps=200,
)
model = SAM3LoRA(config)
OptionDefaultDescription
class_namesNoneCategory-ID-to-prompt mapping or ordered prompt sequence; required by SAM3LoRA
sam3_checkpoint_pathNoneOptional local SAM3 base checkpoint
lora_weights_pathNoneOptional LoRA weights loaded after model construction
load_from_hfTrueLoads the base SAM3 checkpoint from Hugging Face
rank8Positive LoRA decomposition rank
alpha16.0Positive LoRA scaling factor
dropout0.1LoRA dropout probability in [0, 1)
target_modulesstandard projection namesModule-name suffixes eligible for LoRA injection
apply_to_vision_encoderTrueAdapts the vision encoder
apply_to_text_encoderTrueAdapts the text encoder
apply_to_geometry_encoderFalseAdapts the geometry encoder
apply_to_detr_encoderTrueAdapts the DETR encoder
apply_to_detr_decoderTrueAdapts the DETR decoder
apply_to_mask_decoderFalseAdapts the mask decoder
resolution1008Positive square training resolution
num_negative_prompts2Negative category prompts sampled per image
optimizer"adamw"Optimizer built by the model adapter
optimizer_options{}Additional optimizer keyword arguments
learning_rate1e-4Optimizer learning rate
weight_decay0.01Optimizer weight decay
schedulerNoneOptional "step", "cosine", or "plateau" scheduler
scheduler_options{}Scheduler-specific keyword arguments
scheduler_interval"epoch""step" or "epoch"
scheduler_monitorNoneMetric used by a monitored scheduler
warmup_steps0Linear warmup optimizer steps
augmentation()Preset name or declarative target-aware pipeline

SAM3LoRA Properties ​

PropertyTypeDescription
configSAM3LoRAConfigPublic unified configuration
model_configSAM3LoRAConfigModel-construction configuration
adapter_configSAM3LoRAConfigConfiguration used by the training adapter
modelnn.ModuleUnderlying SAM3 LoRA PyTorch model
adapterSAM3LoRATrainingAdapterCached prompted-training adapter
descriptorModelDescriptorSAM3-LoRA family and instance-segmentation task metadata
train_transformstarget-aware callableConfigured shared augmentation pipeline; SAM3 preprocessing remains adapter-owned
val_transformsNonePreprocessing is handled by the adapter
requires_masksboolAlways True

Trainer Reference

See Configuration: Trainer Reference for all Trainer constructor options, defaults, and methods.

SAM3 Best Practices ​

Prompts and Categories ​

  • Use stable, descriptive COCO category names because they become the text prompts used during training.
  • Keep category IDs and names consistent between training and validation datasets.
  • Avoid renaming or reordering prompts when resuming a checkpoint.
  • Inspect prompt-specific validation results when classes are visually similar or semantically ambiguous.

Base Checkpoint Loading ​

With the default load_from_hf=True, Iris downloads the SAM3 base checkpoint from Hugging Face. Use a local checkpoint when training must run offline or when a specific base checkpoint must be reproduced:

python
config = SAM3LoRAConfig(
    class_names=categories,
    sam3_checkpoint_path="checkpoints/sam3.pt",
    load_from_hf=False,
)

Checkpoint Loading

sam3_checkpoint_path is required when load_from_hf=False. Use lora_weights_path only when initializing the model with existing LoRA parameters.

LoRA Capacity ​

rank, alpha, and dropout control LoRA capacity and regularization:

  • Increase rank when the default adapters do not have enough capacity. Higher ranks add trainable parameters and consume more memory.
  • Tune alpha with rank; a common starting relationship is alpha equal to one or two times the rank.
  • Increase dropout when the adapted model overfits. Reduce it when training is underfitting or unstable because too much signal is being dropped.
  • Change one setting at a time and compare validation mask metrics.
python
config = SAM3LoRAConfig(
    class_names=categories,
    rank=16,
    alpha=32.0,
    dropout=0.1,
)

LoRA Target Selection ​

Enable only the model components that need adaptation. Adapting more components increases capacity, memory use, and training time.

python
config = SAM3LoRAConfig(
    class_names=categories,
    rank=16,
    alpha=32.0,
    apply_to_vision_encoder=True,
    apply_to_text_encoder=False,
    apply_to_geometry_encoder=True,
    apply_to_detr_encoder=True,
    apply_to_detr_decoder=True,
    apply_to_mask_decoder=True,
)

Start with the defaults, then expand the targets only when validation results show that more adaptation is needed. At least one enabled component must contain a name matching target_modules.

The default target_modules are q_proj, k_proj, v_proj, out_proj, qkv, proj, fc1, fc2, c_fc, c_proj, linear1, and linear2. Override this tuple only when the target SAM3 implementation uses different module names or you intentionally want a narrower adapter set.

Resolution and Negative Prompts ​

  • Keep resolution=1008 initially. Higher resolutions increase memory and compute, while lower resolutions may lose small-object detail.
  • Reduce the Trainer batch_size before reducing resolution when GPU memory is limited.
  • Increase num_negative_prompts when the model confuses absent categories with foreground objects.
  • Too many negative prompts increase work per image and may dilute positive supervision; validate each change.

Existing LoRA Weights and Resume ​

  • Use lora_weights_path to initialize a new model from existing LoRA parameters.
  • Use Trainer.resume() when continuing an Iris training run because it also restores optimizer, scheduler, scaler, history, and epoch state.
  • Keep prompts, resolution, LoRA rank, enabled targets, and target module names compatible with the saved weights.
  • Preserve best.pt for deployment selection and latest.pt for continuing interrupted training.

Next Steps ​