assign_second_stage is performced only once

[eternaldolphin]

the second stage match is performed with decoder's input 'init reference point', instead of after each layer like other detrs
Have you ever tried to perform label assign after each decoder layer?

[jozhang97]

IoU assignment uses box inputs, DETR uses box outputs. Using box outputs does not work for IoU assignment.

You could get a new box input after each decoding layer, that we have not tried.