Remove unnecessary tied weights for Seamless M4T? #42377

ebezzam · 2025-11-25T07:21:44Z

What does this PR do?

The tied weights refactoring from #41580 may have introduced an error for the Seamless M4T model? Opening this PR to not lose track.

Perhaps #42362 will address?

Modeling tests before fix (11 errors)

FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithSpeechInputTest::test_model_from_pretrained - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_model_from_pretrained - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_embeddings_untied_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_tokens_embeddings_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_speech_to_speech_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_speech_to_text_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_text_to_speech_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_text_to_text_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_eng_text - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_rus_speech - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_swh_text - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
========== 11 failed, 79 passed, 193 skipped, 5 warnings in 116.67s (0:01:56) ====

Modeling tests after fix (7 errors)

FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_embeddings_untied_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_tokens_embeddings_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_speech_to_speech_model - AssertionError: 86400 != 86080
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_text_to_speech_model - AssertionError: 90560 != 86720
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_eng_text - AssertionError: Lists differ: [2, 10051, 8980, 8212, 949, 1270, 4311, 1123, 5918, 2333] != [2, 10051, 9253, 9253, 9253, 9253, 3765, 3765, 3765, 3765]
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_rus_speech - AssertionError: Lists differ: [2, 10067, 5729, 4798, 9631, 8378, 4446, 2393, 6901, 5983] != [2, 10067, 5560, 5560, 5560, 5560, 5560, 5560, 5560, 5560]
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_swh_text - AssertionError: Lists differ: [2, 10071, 5729, 9995, 3089, 7546, 1204, 1721, 2532, 4340] != [2, 10071, 2679, 2679, 2679, 2679, 2679, 2679, 9178, 9178]
=============== 7 failed, 89 passed, 187 skipped, 5 warnings in 171.26s (0:02:51) =========

Essentially such errors are resolved:

ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_model.lm_head.weight (which is present).

But integration tests are failing which is probably better for another PR?

cc: @vasqu, @ArthurZucker

ebezzam · 2025-11-25T07:23:01Z

run-slow: seamless_m4t

github-actions · 2025-11-25T07:24:10Z

This comment contains run-slow, running the specified jobs:

models: ["models/seamless_m4t"]
quantizations: []

HuggingFaceDocBuilderDev · 2025-11-25T07:31:02Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2025-11-25T07:41:10Z

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

ebezzam · 2025-11-25T07:57:34Z

Integration tests are actually failing: https://github.com/huggingface/transformers/actions/runs/19661588575/job/56309175315#step:14:21132

but as mentioned above, this is the case before the above mentioned fix

github-actions · 2025-11-25T17:34:25Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: seamless_m4t

molbap · 2025-11-25T21:54:26Z

Hi! likely linked to #42385 as well

ArthurZucker

Ty for having a look! this one is a bit bigger would be happy if we have a more general solution

ArthurZucker · 2025-11-26T10:29:03Z

src/transformers/models/seamless_m4t/modeling_seamless_m4t.py

+    def tie_weights(self, missing_keys: Optional[set[str]] = None, recompute_mapping: bool = True):
+        """We need to overload here to handle the wrong key saved in some main checkpoints."""
+        if self.config.tie_word_embeddings:
+            # Some model checkpoints like "facebook/hf-seamless-m4t-medium"'s embedding weight is in decoder.embed_tokens,
+            # need check here
+            if missing_keys is not None:
+                if "lm_head.weight" in missing_keys and "model.decoder.embed_tokens.weight" not in missing_keys:
+                    self.lm_head.weight = self.decoder.embed_tokens.weight
+                    missing_keys.discard("lm_head.weight")
+
+        # needs to be done after, otherwise it raises an Error because the correct weights are not present
+        super().tie_weights(missing_keys=missing_keys, recompute_mapping=recompute_mapping)


yep this makes sense, tho I think the explicit mapping can work in revers basically. #42362.
It affects more models!

vasqu · 2025-11-26T13:55:56Z

Converting to draft for now and essentially closing it when we have the reverse mapping PR landing. Seems like more models are affected so it's not worth to create exceptions everywhere!

Remove unnecessary tied weights?

1e17ea4

ebezzam requested a review from vasqu November 25, 2025 07:21

ebezzam added bug Audio labels Nov 25, 2025

ebezzam mentioned this pull request Nov 25, 2025

Fix processor usage + add chat_template support to TTS pipeline, and shift common chat template logic to base class. #42326

Merged

fix

ed8b0a6

ArthurZucker reviewed Nov 26, 2025

View reviewed changes

vasqu marked this pull request as draft November 26, 2025 13:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove unnecessary tied weights for Seamless M4T? #42377

Remove unnecessary tied weights for Seamless M4T? #42377

ebezzam commented Nov 25, 2025

Uh oh!

ebezzam commented Nov 25, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Nov 25, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

ebezzam commented Nov 25, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

molbap commented Nov 25, 2025

Uh oh!

ArthurZucker left a comment

Uh oh!

ArthurZucker Nov 26, 2025

Uh oh!

vasqu commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Remove unnecessary tied weights for Seamless M4T? #42377

Are you sure you want to change the base?

Remove unnecessary tied weights for Seamless M4T? #42377

Conversation

ebezzam commented Nov 25, 2025

What does this PR do?

Uh oh!

ebezzam commented Nov 25, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Nov 25, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

CI Results

Uh oh!

ebezzam commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

molbap commented Nov 25, 2025

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurZucker Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

vasqu commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ebezzam commented Nov 25, 2025 •

edited

Loading