Skip to content

After the weight refactor, some tie weight might not have gone as expected. If the lm head was removed, then we are kinda fucked when using from_pretrained. #42362

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

After the weight refactor, some tie weight might not have gone as expected. If the lm head was removed, then we are kinda fucked when using from_pretrained. #42362

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions src/transformers/modeling_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2247,7 +2247,7 @@ def __init__(self):
)
source_params = sorted(filter(lambda x: re.search(source_name, x), top_level_params.keys()))
target_params = sorted(filter(lambda x: re.search(target_name, x), top_level_params.keys()))
if not len(source_params) > 0 or len(target_params) % len(source_params) != 0:
if (len(source_params) > 0 and len(target_params) !=0) and len(target_params) % len(source_params) != 0 :
raise ValueError(
f"There is an issue with your definition of `tie_weights_keys` for {source_name}:{target_name}. We found {source_params} to tie into {target_params}"
)
Expand All @@ -2264,14 +2264,14 @@ def __init__(self):
self._adjust_bias(parent, top_level_params[source_n])
if missing_keys and source_is_there: # test_model_weights_reload_no_missing_tied_weights
missing_keys.discard(target_n)
else:
elif len(source_params) > 0:
target_is_not_there = missing_keys and re.search(
target_name, "\n".join(missing_keys), flags=re.MULTILINE
)
raise ValueError(
"There is a problem in the way you tie your keys or the way they were saved.\n"
f"source_is_there={source_is_there}, target_is_there={not target_is_not_there}, missing_keys={missing_keys},"
"tie_word_embeddings/tie_encoder_decoder={(self.config.tie_word_embeddings or self.config.tie_encoder_decoder)}"
f" source_is_there={source_is_there}, target_is_there={not target_is_not_there}, missing_keys={missing_keys},"
f" tie_word_embeddings/tie_encoder_decoder = {(self.config.tie_word_embeddings or self.config.tie_encoder_decoder)}"
)

def _adjust_bias(self, output_embeddings, input_embeddings):
Expand Down
Loading