Skip to content

feat: allow point-prompt initialization by loading backbone#447

Open
aniketsrinivasan wants to merge 1 commit intofacebookresearch:mainfrom
aniketsrinivasan:feat/sam3-point-init
Open

feat: allow point-prompt initialization by loading backbone#447
aniketsrinivasan wants to merge 1 commit intofacebookresearch:mainfrom
aniketsrinivasan:feat/sam3-point-init

Conversation

@aniketsrinivasan
Copy link

PR: Enable SAM3 image point-prompting in sam3_facebook

Summary

  • Fixes missing wiring for SAM1-style interactive point prompting by ensuring the interactive tracker is constructed with a visual backbone when enable_inst_interactivity=True.

What changed

  • model_builder.py
    • Build the interactive tracker with with_backbone=True (so SAM3InteractiveImagePredictor.set_image(*) can call forward_image and compute embeddings).
    • Extend checkpoint loading to also map detector.backbone.* weights into inst_interactive_predictor.model.backbone.* when instance interactivity is enabled.

Why

  • In the upstream sam3 repo, point prompting works because the interactive predictor wraps a tracker that includes a backbone and receives appropriate weights. sam3_facebook previously built the tracker without a backbone, preventing feature initialization needed for point prompts.

Notes

  • No behavior changes to non-interactive inference paths.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant