Error(s) in loading state_dict for exBertEmbeddings

[ken19980727]

Environment info

I'm tying to using fine-tune my own dataset in exBert with examples/pytorch/language-modeling/run_mlm.py.
Also successful export the following files under the path of exbert-mlm:

vocab.txt
train_result.json
pytorch_model.bin
config.json

This is the function I use

from transformers import exBertModel, exBertTokenizer

model = exBertModel.from_pretrained("exbert-mlm/")

But an error occurred :

RuntimeError                              Traceback (most recent call last)
<ipython-input-2-33cf1285cf06> in <module>
      1 from transformers import exBertModel, exBertTokenizer
      2 
----> 3 model = exBertModel.from_pretrained("exbert-mlm/")
      4 # tokenizer = exBertTokenizer.from_pretrained()；

~\AppData\Roaming\Python\Python38\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1171         else:
   1172             with no_init_weights(_enable=_fast_init):
-> 1173                 model = cls(config, *model_args, **model_kwargs)
   1174 
   1175         if from_tf:

~\AppData\Roaming\Python\Python38\site-packages\transformers\models\exbert\modeling_exbert.py in __init__(self, config)
    644 
    645             base_model_weight = BertModel.from_pretrained(self.config.base_model)
--> 646             self.embeddings.load_state_dict(base_model_weight.embeddings.state_dict(), strict=False)
    647             self.encoder.load_state_dict(base_model_weight.encoder.state_dict(), strict=False)
    648 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in load_state_dict(self, state_dict, strict)
   1049 
   1050         if len(error_msgs) > 0:
-> 1051             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   1052                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1053         return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for exBertEmbeddings:
	size mismatch for word_embeddings.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([55707, 768]).

• transformers : 4.7.0.dev0
• Python version: 3.7
• PyTorch version (GPU):1.7.1+cu11.0 with RTX3090

why would a mismatch show up if I'm just trying to load from pretrained local files?

[ken19980727]

Supplement

This is my config.json
{ "architectures": [ "exBertForMaskedLM" ], "attention_probs_dropout_prob": 0.1, "base_model": "bert-base-cased", "bos_token_id": 0, "directionality": "bidi", "eos_token_id": 2, "ex_hidden_size": 252, "ex_intermediate_size": 1024, "ex_num_attention_heads": 12, "ex_num_hidden_layers": 12, "ex_vocab_size": 26711, "freeze_base_model": true, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "model_type": "exbert", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 1, "pooler_fc_size": 768, "pooler_num_attention_heads": 12, "pooler_num_fc_layers": 3, "pooler_size_per_head": 128, "pooler_type": "first_token_transform", "transformers_version": "4.7.0.dev0", "type_vocab_size": 2, "use_cache": true, "vocab_size": 55707 }

And yes. I used the bert-base-cased as my base_model

[ken19980727]

Hi Mahesh Deshwal: Sorry for the late reply Yes, I can implement their code but see lots of problems. Seems like we want to do the same thing. My idea is to use the original bert (i.e. multilingual bert) then combine the another embedding which is pre-trained on my dataset. You-Qian Mahesh Deshwal ***@***.***> 於 2022年4月12日 週二 下午7:25寫道：

…

Are you using the original code that was used for exBERT? I actually tried using their code but lots of errors occurred specially GPU OOM and they don't have the requirements.txt. Also, there's no support on github for their code right now. *I want to train and save a generic model on my domain specific vocab to use its embedding or fine tune later for specific tasks* — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKOXZ4MUMJVGIR5FMNLLUYLVEVMSZANCNFSM5IAUJCWQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>