-
Language modeling is the problem of predicting the next token in a sequence given the previous N tokens.
-
Built both encoder only and decoder only transformer for unsupervised learning on the Penn Treebank (text-only) dataset.
- Compute the perplexity based on each sentence in the test set.
- Perplexity is a metric that quantifies the model's uncertainty or surprise when predicting the next word in a sequence.