Skip to content

Fix Grad-CAM heatmap generation#84

Closed
NateBioTech wants to merge 1 commit intoKatherLab:mainfrom
NateBioTech:STMAP-heatmapdebug
Closed

Fix Grad-CAM heatmap generation#84
NateBioTech wants to merge 1 commit intoKatherLab:mainfrom
NateBioTech:STMAP-heatmapdebug

Conversation

@NateBioTech
Copy link
Copy Markdown

Added attention weights handling in src/stamp/modeling/vision_transformer.py

@EzicStar EzicStar requested review from EzicStar and s1787956 July 3, 2025 12:01
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused. Where are the attention weights used in the gradcam function?
In general, there are two different approaches for Transformer heatmaps:

  1. gradient back to input (our standard nowadays)

  2. attention activation (attention weights) visualization

  3. has the disadvantage that parts of the model are kinda neglected (MLP parts). And there are multiple heads and layers that have individual activations).

So I am unsure if we are not mixing these things with this approach. Also returning the weights will affect the performance of the ViT.

@EzicStar EzicStar closed this Aug 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants