Could you point where to insert this CWD module in the net?

[RobinhoodKi]

Excellent Work! I'm quite interested about your work. I've a question about where to do CWD in the network. Do you mean only do CWD in the last output channel? Like for CityScapes, the output channel will be 19 ,each one will be responsible for one class and only do Channel-wise softmax on these 19 channels separately and then compute loss based on KL divergency. Or do you mean we need to insert this CWD module in the middle part of the network too? Can you share more details about it? Thanks!

[drilistbox]

Excellent Work! I'm quite interested about your work. I've a question about where to do CWD in the network. Do you mean only do CWD in the last output channel? Like for CityScapes, the output channel will be 19 ,each one will be responsible for one class and only do Channel-wise softmax on these 19 channels separately and then compute loss based on KL divergency. Or do you mean we need to insert this CWD module in the middle part of the network too? Can you share more details about it? Thanks!

Thank you for your approval. Only do CWD in the last output channel is OK. As for "Or do you mean we need to insert this CWD module in the middle part of the network too?", We only do CWD on the output of decoder (such as, PSPhead and ASPPhead).

[happycoding1996]

@drilistbox But the decoders of the teacher and student have different output channels (for the output of the pyramid pooling modules, 1024 for PSPNet-18 and 4096 for PSPNet-50), how do you solve this issue? By applying a projector on teacher or student or both to make them have the same channel number?

As the training code is not available in this repo, could you provide more details (e.g., the structure of the projector) ?

[drilistbox]

@drilistbox But the decoders of the teacher and student have different output channels (for the output of the pyramid pooling modules, 1024 for PSPNet-18 and 4096 for PSPNet-50), how do you solve this issue? By applying a projector on teacher or student or both to make them have the same channel number?

As the training code is not available in this repo, could you provide more details (e.g., the structure of the projector) ?

We solve this issue by applying a projector(conv-bn-relu) on the student, maybe on teacher or both also worked, which we have not tried.

[happycoding1996]

Thank you~

[irfanICMLL]

The projector should be put on the student (min|T-F(S)|). We guess if adding on the teacher net may not work well as the learning target is changeable (min|F(T)-S|).

[ofishhaha]

@drilistbox But the decoders of the teacher and student have different output channels (for the output of the pyramid pooling modules, 1024 for PSPNet-18 and 4096 for PSPNet-50), how do you solve this issue? By applying a projector on teacher or student or both to make them have the same channel number?
As the training code is not available in this repo, could you provide more details (e.g., the structure of the projector) ?

We solve this issue by applying a projector(conv-bn-relu) on the student, maybe on teacher or both also worked, which we have not tried.

Hi, Thank you for your efforts. There is a question that do you apply the projector first and then apply softmax for channel wise feature distillation? In my implementation, the cw loss decrease really slow. And the performance also not good. I use the pspnet-r101 as teacher and pspnet-r18 as student. the channel of teacher is 512 and for student is 128.