Question: DropLoss Implementation of Mask2Former

[ChristophReich1996]

Hi @frank-xwang,

I noticed the implementation of the DropLoss of Mask2Former differs from the implementation used in Cascade Mask R-CNN. However, the VideoCutLER paper does not share many details of this implementation. Could you please explain the DropLoss implementation for Mask2Former?

Thanks for the help!
Christoph

[ChristophReich1996]

Hi @frank-xwang again,

From what I see, the threshold (0.01) is only employed in the matching see:

CutLER/videocutler/mask2former/modeling/matcher.py

Lines 32 to 35 in c47b5df

	hw = inputs.shape[1]
	drop_weights = torch.einsum("nc,mc->nm", inputs.ge(0.0).float(), targets).max(-1)[0] / hw
	drop_weights = drop_weights.ge(thresh).float().detach()[:,None]
	loss = drop_weights * loss

and

CutLER/videocutler/mask2former/modeling/matcher.py

Lines 71 to 73 in c47b5df

	drop_weights = torch.einsum("nc,mc->nm", inputs.ge(0.0).float(), targets).max(-1)[0] / hw
	drop_weights = drop_weights.ge(thresh).float().detach()[:,None]
	loss = drop_weights * loss

However, the threshold is not considered in the actual loss computation. For me, applying the drop threshold seems to make no sense. For me, the matching should be untouched, and then object masks that are not matched and are not overlapping with any pseudo mask should be ignored. Am I missing something here?🤔