Paper pretraining config

[mojivalipour]

It appears that config/pretrain-alldata-base.json is not your paper pretraining configuration. There is no cls_concat setting in this configuration file, so it uses the default value. As a result, unlike your paper, this configuration uses MLM instead of VMLM. Could you please provide a correct configuration that reproduces your results?

[intersun]

Yep, you can simply change the config setting, please see https://github.com/intersun/LightningDOT/blob/5f2880f69ba87b8701ab89348d70ebb11432578c/dvl/models/bi_encoder.py#L423 for more details. Thanks!