Getting very low confidence score for certain obvious text prompts

[rajeshgangireddy]

Hi, I have been trying out different things with EfficientSAM3 and most of the times it works great. Thanks for the great work.

However, I have noticed that when using distilled image encoder + text encoder, there are sometimes very low confidence score masks for simple prompt and images.

Using the sam3/efficientsam3_examples/efficientsam3_image_predictor_example.ipynb notebook,
with the provided example image and prompt ("a shoe") , the results are quite good.

But with a different image of a dog and text prompt "dog", I had to lower the confidence threshold by a lot to get any meaningful masks.

dog_image ="dog6/01.jpg"
image = Image.open(dog_image)
width, height = image.size
processor = Sam3Processor(model, confidence_threshold=0.02)
inference_state = processor.set_image(image)

processor.reset_all_prompts(inference_state)
inference_state = processor.set_text_prompt(state=inference_state, prompt="dog")

img0 = Image.open(dog_image)
plot_results(img0, inference_state)

After taking a closer look, I see that this is mostly due to the presence_logit_dec being too low, even for simple cases like this.

Prompt: 'dog'
Presence score (single value): 0.0320
Top-5 classification probs: [0.7695, 0.4902, 0.3438, 0.332, 0.2734]
Top-5 final probs (class × presence): [0.0247, 0.0156, 0.011, 0.0106, 0.0087]

Question/Discussion :

1. Is this expected and is an effect of distillation or the dataset used for distillation itself?
2. Perhaps EfficientSam3 need a less aggressive function to be applied on presence_logit_dec. Currently it is sigmoid which leads to presence probability being pushed to either extremes.
   Have you already considered it experiments around this ?

[SimonZeng7108]

Hi, thanks for raising this! We've observed the same phenomenon and appreciate the deep dive analysis.

Regarding the root cause:

This is primarily an effect of our distillation strategy rather than the dataset. The presence_token_head is a 3-layer MLP trained on teacher feature distributions, which amplifies any distribution mismatch non-linearly. Our Stage 1 image encoder distillation uses MSE + cosine similarity losses, which match feature direction but don't strictly enforce matching the teacher's magnitude/distribution statistics.

Regarding sigmoid sensitivity:

Yes, this is a valid observation. Sigmoid has problematic properties for miscalibrated logits—when the student's presence logits are systematically shifted more negative (e.g., teacher: -5.89 → student: -11.05), sigmoid crushes them to near-zero (0.003% vs 0.27%). This makes the model appear to have much lower confidence even when the object is present.

Our planned remedies for Stage 3 fine-tuning:

1. Distribution normalisation - Add explicit constraints to align student encoder output statistics (mean, variance, L2 norm) to match the teacher encoder distribution, in addition to the current MSE/cosine losses.

2. Decoder fine-tuning - Unfreeze and fine-tune the presence_token_head (and potentially the full decoder) to better adapt to the student encoder's feature characteristics, similar to the phase 2 training in EdgeSAM.

[SimonZeng7108]

@rajeshgangireddy Hey, our latest SAM3-LiteText weights are better tuned and can give a quite accurate segmentation results with the original sam3 image encoder, feel free to give it a try!

[christopheelzeinaty]

@SimonZeng7108 Hey, very nice work, I would love to use the efficientSam3 weights for npu devices, but the weights are huge is that normal? its like we are using the sam3 encoder as is.

thanks for your help!

[SimonZeng7108]

@SimonZeng7108 Hey, very nice work, I would love to use the efficientSam3 weights for npu devices, but the weights are huge is that normal? its like we are using the sam3 encoder as is.

thanks for your help!

Hi, thanks for your interest. For NPU optimisation, I suppose it requires certain model architectures support, I am not sure if the ViT based encoder is friendly, but repvit should be quite flexible for edge devices. The weights are large as the precision is quite high. you could qunatise them to FP8 or even INT4, which will reduce the size quite a bit.