Alternative Losses

[ClashLuke]

Currently, we're using only the softmax classification/cross-entropy loss to create a language-modeling loss for next-token prediction. However, other works such as T-Few showed that adding alternative losses for external benefits such as length explicit penalties during training can help downstream-task performance. Additionally, other works like DCL and InfoLOOB demonstrated that changing the fundamental structure of the loss from softmax classification to something different can help speed up convergence. That's why a similar approach could be beneficial for us.
In this issue, we'll explore whether InfoLOOB's classification loss for the language-modeling objective helps or if we should change the entire objective.

[ClashLuke]

PolyLoss vs CrossEntropy

[ClashLuke]

PolyLoss (green) performs quite a bit worse than CrossEntropy (bisque):

We could still try InfoLOOB (DCL) as it appeared promising before:

However, after reaching a loss of -100, InfoLOOB ran into NaN, which halted the training. Nothing like this has happened with CrossEntropy, which is why CrossEntropy achieved a better final model even though the initial convergence was slower.