the length loss should devide nsentences?

[PanXiebit]

Mask-Predict/fairseq/criterions/label_smoothed_length_cross_entropy.py

Line 63 in 11c7535

loss = (1. - self.eps) * nll_loss + eps_i * smooth_loss + length_loss

Mask-Predict/fairseq/criterions/label_smoothed_length_cross_entropy.py

Line 73 in 11c7535

'loss': sum(log.get('loss', 0) for log in logging_outputs) / sample_size / math.log(2),

the total loss is sumed by nll_loss, smooth_loss and length loss. When compute the mean loss, nll_loss, smooth_loss should devide ntokens, and length loss should devide nsentences. However, in the source code, both of them are devided by ntokens?

[jungokasai]

That's right. It doesn't seem clear from the paper, but indeed they divide the length loss by the number of tokens. And I found that was actually very crucial; if you divide it by the number of sentences instead, the length loss over-dominates the total loss and leads to a training failure. The master fairseq implementation takes a slightly different approach: https://github.com/pytorch/fairseq/blob/master/fairseq/models/nat/cmlm_transformer.py#L147 They divide the length loss by the number of tokens and then multiply by "length_loss_factor"=0.1. Either way, we have to discount the length loss.