Skip to content
This repository was archived by the owner on Oct 31, 2023. It is now read-only.
This repository was archived by the owner on Oct 31, 2023. It is now read-only.

Why is the BLEU obtained from the training model provided much higher than the value on paper? #11

@PanXiebit

Description

@PanXiebit

I download the provided trained model, and test on test dataset, but get much higher BLEU than the values in paper.

I use the scripts provided, and don't change anything:

python preprocess.py \
  --source-lang de \
  --target-lang en \
  --trainpref data/wmt14.en-de/train \
  --validpref data/wmt14.en-de/valid \
  --testpref data/wmt14.en-de/test \
  --destdir output/data-bin/wmt14.de-en \
  --srcdict output/maskPredict_de_en/dict.de.txt \
  --tgtdict output/maskPredict_de_en/dict.en.txt

python generate_cmlm.py output/data-bin/wmt14.${src}-${tgt}  \
    --path ${model_dir}/checkpoint_best.pt \
    --task translation_self \
    --remove-bpe True \
    --max-sentences 20 \
    --decoding-iterations ${iteration} \
    --decoding-strategy mask_predict 

I get 34.42 on WMT14 DE->EN, 35.20 on WMT16 EN->RO, 35.62 on WMT RO->EN. These values are much higher than that in origin paper. This is strange, and what happened?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions