Questions regarding the deviation of experimental results and testing details

[brightest66]

Hello, thank you for contributing the code.

When reproducing your code on an NVIDIA GeForce RTX 4090, the code for obtaining the experimental results on the MOSI dataset is as follows:

dataset="mosi" # Can be in ["mosi", "humor", "sarcasm", "mimic", "visionandtouch", "visionandtouch-bin"]
python main_multibench.py \
      data.data_module.dataset=${dataset} \
      model="comm" \
      trainer.max_epochs=100 \
      optim.lr=1e-3 \
      optim.weight_decay=1e-2 \
      +trainer.devices=[1] \
      mode="test" \
      +ckpt_path="CoMM/mosi/logs/version_0/checkpoints/epoch=99-step=2100.ckpt"

The obtained results are as follows:

As can be seen, this result differs significantly from the result in the paper. Even within an error range of 1.3%, there is a rather large discrepancy. I would like to ask how you obtained the experimental results. Did you select the model from the last epoch for testing or the model that performed optimally on the validation dataset for testing? And which specific test set was chosen? If possible, could you please tell me more details about the testing on other datasets?

Thank you very much for your assistance!

[Duplums]

Hello, we performed a set of 5 runs for each experiment by setting the seed from 42 to 47 and we picked the best checkpoint based on the validation set in each run (which is usually 100 epochs except for smaller-scale datasets for which variance in the performance is large). The test set always follows the official ones from MultiBench (in this case for MOSI).

[brightest66]

Hello, Thank you for your reply! Can you tell me more details about the best checkpoint? Is the evaluation metric based on downstream task accuracy or contrastive loss on validation set?

[Duplums]

Hello, the evaluation metric is based on the downstream task validation accuracy in this case although we also found a good correlation with the contrastive validation loss. Let me know if you need more help.

[brightest66]

Could you please tell me the evaluation metrics for the best checkpoints of other datasets? For some datasets, I used accuracy, and for others, I used contrastive loss. However, the experimental results I reproduced differ significantly from those in the paper.