Sequence score includes scores for lang id token

[Enkidu93]

In machine.py, we do something like this, so that only scores for non-special tokens are bubbled up:

        for item in zipped:
            output_ids, scores, sequence_score, attentions = cast(
                Tuple[torch.Tensor, torch.Tensor, Optional[float], Optional[torch.Tensor]], item
            )
            output_tokens: List[str] = []
            output_indices: List[int] = []
            for i, output_id in enumerate(output_ids):
                id = cast(int, output_id.item())
                if id not in all_special_ids:
                    output_tokens.append(self.tokenizer.convert_ids_to_tokens(id))
                    output_indices.append(i)

            scores = scores[output_indices]

In silnlp, we do something similar downstream in hugging_face_config.py:translate().

However, we grab the sequence_scores directly from the model outputs and these sequence scores seem to include the BOS token score which is close to 0. This will presumably bias the sequence score slightly so that shorter output sequences would have a score closer to zero.

We should confirm that these special token scores are being included in the sequence score and then update silnlp accordingly if they are (or maybe even consider submitting an issue in transformers).

[Enkidu93]

@mshannon-sil and I discussed this in this morning's research meeting. Sounds like he'll take a look and confirm whether this is the case. In the meantime, I'll leave the sequence scores alone in machine.py for consistency.

[mshannon-sil]

Looking at the results from silnlp, it doesn't appear that the sequence score is using the BOS token as part of its calculation. For a test case of 3 John verse 1, I got the following values:

Sequence score: -1.0486180782318115
Averaging token log probabilities without the BOS token: -1.0486106382949012
Averaging token log probabilities with the BOS token: -1.0009465183724056
(Just to compare) averaging token log probabilities without the BOS token or lang_id token: -1.1010411702096463

So it does seem that averaging the token log probabilities without the BOS token is the closest to the huggingface sequence score, though once you get a few decimal places down it's not exact. I did need to make sure I excluded any pad tokens from the calculation as well. I've also done the equivalent test on a few other verses at the confidence level rather than the log prob level by taking the geometric mean of the exponentiated token scores, and I got similar results.

Can you go into more detail on what has been leading you to believe that the sequence score in machine.py is using the BOS token in its sequence score calculation?

[Enkidu93]

Thanks, Matthew! If I use the debugger and debug test_hugging_face_nmt_engine.py:test_translate_n_batch_beam() and I put a break point in the _TranslationPipeline's _forward method. I see

transition_scores = tensor([[  0.0000, -11.4344, -11.4029, -11.4053, -11.3460, -11.3949, -11.3492,
         -11.4104, -11.3859],
        [  0.0000, -11.4344, -11.4029, -11.4053, -11.3460, -11.3949, -11.3492,
         -11.4174, -11.3988]])

and

sequences_scores = tensor([-10.1254, -10.1277])

And the sequence score ought to be sum(scores)/len(scores) since we have a default length penalty of 1.0. Notice that the values in sequences_scores are exactly (to 5 decimal places) equal to the arithmetic mean of the corresponding transition score tensor.

For context, the output tokens are:

[
['</s>', '__es__', 'skaber', 'skaber', '▁Dollar', '▁Dollar', '▁፤', '▁፤', '▁gerekir', '▁gerekir'],
['</s>', '__es__', 'skaber', 'skaber', '▁Dollar', '▁Dollar', '▁፤', '▁፤', '▁፤', '▁gerekir'],
]

These values are coming straight from transformers so I assumed that this would be a problem in both repos. Maybe it's model/config-dependent or maybe I'm missing something obvious 🤔. Let me know what you discover! Thank you for looking into this.

[mshannon-sil]

Thanks for sharing the output. I think I see the confusion. The 0 score here corresponds to the lang id token, not the BOS token. Both the BOS token and the lang id token have a log probability of 0 i.e. a probability of 1. You can confirm this by seeing that the transition scores tensor has 9 values in the first sequence, whereas the output token list has 10 values since it includes the BOS token.

It may be worth doing some experiments at a later date to check if we can increase our chrf3/confidence correlation by omitting the lang id log probability from the sequence score calculation.

[Enkidu93]

Thanks for sharing the output. I think I see the confusion. The 0 score here corresponds to the lang id token, not the BOS token. Both the BOS token and the lang id token have a log probability of 0 i.e. a probability of 1. You can confirm this by seeing that the transition scores tensor has 9 values in the first sequence, whereas the output token list has 10 values since it includes the BOS token.

It may be worth doing some experiments at a later date to check if we can increase our chrf3/confidence correlation by omitting the lang id log probability from the sequence score calculation.

Oh, yes, my mistake. If you look at the raw outputs.scores, it includes a value for the BOS token and the language id token (which are both oddly -inf 🤔), but the transition scores that come out of the compute_transition_scores seem to remove the BOS score then and leave the language id token with a log probability of 0. I'll update the issue title.

Regardless of whether it is for the BOS or the language id token, this 0 will inconsistently impact the average across different output lengths. It's probably not that consequential to the correlations 🤷, but it almost seems like a bug given that I don't think we supply a token-level confidence for the language id token; you know what I'm saying? In any case, I'll leave it as-is in machine.py for the time-being for consistency's sake.

[ddaspit]

The sequence probability is usually computed by multiplying the conditional probabilities of each token. This means that probability will decrease the longer the sequence is. When applying the length penalty to normalize the beam score, HF uses the full sequence length out of simplicity. I agree that it makes sense to recompute the sequence score based on the length without the language id. Having said that, the impact is small, so we probably don't need to worry about it. At some point in the future, I could see us computing our own score with the token probabilities as input. When we do that, we should use the real length when normalizing.