Should dynamic masking also ignore ['PAD']

[ccchang0111]

Here is a set of tokens that should not be masked during dynamic masking.

electra/pretrain/pretrain_helpers.py

Line 121 in 7911132

ignore_ids = [vocab["[SEP]"], vocab["[CLS]"], vocab["[MASK]"]]

But should we also avoid masking all those ['PAD'] at the end of a sentence (if the sentence is shorter than max_seq_length and if there is no second sentence segment)?

I understand ['PAD'] itself has token_id = 0, but I do not see this being used to prevent masking in downstream steps. If we do not ignore it, this will affect the probability calculation here

electra/pretrain/pretrain_helpers.py

Lines 167 to 170 in 7911132

	# Get a probability of masking each position in the sequence
	candidate_mask_float = tf.cast(candidates_mask, tf.float32)
	sample_prob = (proposal_distribution * candidate_mask_float)
	sample_prob /= tf.reduce_sum(sample_prob, axis=-1, keepdims=True)

Also, we will be trying to predict 'PAD' that is outside a sequence, which is a bit unintuitive.

Maybe I am missing something here. Thanks again for putting up such a great work!

[clarkkev]

Good question! We don't need to add PAD to the ignore ids because of this line:

electra/pretrain/pretrain_helpers.py

Line 125 in 7911132

candidates_mask &= tf.cast(inputs.input_mask, tf.bool)

the input_mask is 1 for non-padding tokens and 0 for padding. But I do agree it would have been clearer in the code if I just had added PAD to the ignore_ids instead.

[ccchang0111]

ah that's true.. it was already taken care of during data generation process. Thanks for the response! I actually have a PR here that adds [PAD] into the ignore. But feel free to merge or drop it #60