Cannot reproduce results using provided trained models

[IsabelFunke]

Hello,
thank you for providing this great repository!

I attempted to reproduce your results by following the steps in the Readme file and using your trained models, provided in the Dropbox (see details below).
Unfortunately, I only achieved the following results:

Mean jaccard: 60.64 +- 16.34
Mean accuracy: 82.35 +-  8.89
Mean precision: 80.64 +-  8.28
Mean recall: 75.20 +- 21.44

Could you maybe please check my steps below and give me a hint of what I might be doing wrong?
Your help is very much appreciated!

Best regards,
Isabel

---

Used python environment:

• python 3.7
• pytorch 1.5.1 (to use a version that is not too far from python 1.0) with cudatoolkit 10.2 and torchvision 0.6.1
• resnest 0.0.6
• numpy 1.19
• scipy 1.7.3
• opencv-python 4.7.0
• pillow 6.1.0

---

Steps

• Data preprocessing: I extracted the video frames from Cholec80 using your script video2frame_cutmargin.py (with a small modification to only store frames where frame_no % 25 == 0).
  Here, I had to remove the lines

  img_result = cv2.cvtColor(img_result, cv2.COLOR_BGR2RGB)
  img_result = PIL.Image.fromarray(img_result)
  print(img_result.mode)
  

  The reason for this was that OpenCV gave me an error when trying to store the PIL image:
  ... (-5:Bad argument) in function 'imwrite' ... img is not a numpy array, neither a scalar
  Could my fix (commenting out lines 79-81) be a problem? The images extracted this way looked fine to me.

• Training: I skipped training the memory bank and the TMRNet model because I wanted to try your trained models.

• Testing:

  • I ran get_paths_labels.py in ./code/eval/python/ to generate the test_paths_labels.pkl
  • Then I ran test_singlenet_phase_non-local_pretrained_2fc_copy_mutiConv6_resnest.py -t 500 -n ./pretrained_model/pretrained_mutiConv_resnest.pth. I set load_LFB to False to generate the memory bank.
    Due to GPU memory limits, I had to reduce the test batch size to 500, but that shouldn't be the problem.
    I modified the code to use ./pretrained_model/pretrained_LFB_model.pth to generate the memory bank features (in the code, this was originally lstm_epoch_12_length_10_opt_0_mulopt_1_flip_1_crop_1_batch_400_train_9989_val_8839.pth).
    I am wondering whether maybe the provided model pretrained_LFB_model.pth is the problem, for example, because it doesn't go well with the pretrained_mutiConv_resnest.pth? Or maybe the provided models were trained on the M2CAI dataset instead of Cholec80?
    Or maybe I used the wrong argument for --crop? I kept it as 1, which was the specified default value.
  • Finally, I ran export_phase_copy.py -n ./pretrained_model/pretrained_mutiConv_resnest_test_8004_crop_1.pkl to generate the files for the M2CAI evaluation script and ran ./code/eval/result/matlab-eval/Main.m in Octave to obtain the evaluation results.

[Yipinggggg]

me too, have you found out the problem now?