Problems of reproducing the results of EOPSN

[HeimingX]

Hi,

Thanks for the interesting paper and open-sourced code.

Recently, I ran the EOPSN method on K20 setting folllowing the given guideline (w/o any editing) and I found the results of unknown things are quite different from the reported one.

Unk	PQ	SQ	RQ
EOPSN reported	11.3	73.8	15.3
EOPSN reproduced	15.6	79.2	19.6

From the table, it seems that the released code achieves a much better improvement than the reported one. However, when I further inspect the predictions of class-wise unkown things, it seems that EOPSN's unkown recognition is dominated by the "car" class and other unkown classes are rarely detected. Moreover, the reproduced results may not support the visualization results in Fig5 since several unkown classes are shown to be detected, e.g., stop sign, keyboard, banana, and toilet. So, could you please release the EOPSN checkpoint which supports the reported results? Thanks a lot.

BTW, I found that the training of EOPSN requires the pre-trained model of Void-Suppression, but the current released codebase only contains the void-train method. I wonder could you please release the void-suppression code for better reproduction? Thanks again.

FYI, the predictions of class-wise unkown things on Void-Suppression method are as follows and the results are identical to the reported ones in the paper

[jd730]

Hi @HeimingX, I apologize for facing the difficulty. However, I am extremely busy these days. I will resolve this as soon as possible.

[jd730]

Hi @HeimingX, I apologize for the delay. I haven't tested yet and I do not think I have time before the upcoming CVPR deadline... However, it seems that the problem comes from sampling object proposals for clustering or clustering itself. You can relax the condition.

Once, I have time, I will also figure out why the problem was caused while publishing the codebase.

[HeimingX]

Hi,
It's been a while since CVPR DDL and I wonder if it is possible for you to release the checkpoint of EOPSN method for further exploration? Many thanks

[jd730]

Hi @HeimingX,
Thank you for reminding me. I just uploaded EOPSN_K10.pth, EOPSN_K05.pth, EOPSN_K20.pth.
I did not run the inference again now, so with the little possibility, I could upload the wrong files. If so, please let me know.

[HeimingX]

Hi @HeimingX, Thank you for reminding me. I just uploaded EOPSN_K10.pth, EOPSN_K05.pth, EOPSN_K20.pth. I did not run the inference again now, so with the little possibility, I could upload the wrong files. If so, please let me know.

Hi, thanks for the releasing of EOPSN checkpoints. The EOPSN_K05.pth, EOPSN_K20.pth can produce indentical results as reported in the paper. But EOPSN_K10.pth raises a bug during the model loading and the error log is as follows:

According to the log, it seems that the released EOPSN_K10.pth does not use the default cluster number(i.e., 128).

[jd730]

Thank you for letting me know. It seems that the checkpoint is for K20 (weight difference is 8 which is 10% of COCO think class). I will do my best to find the right checkpoint.

[HeimingX]

Hi, many thanks for your kind and prompt response!

Recently, I am conducting some analysis to one of the baseline methods (void-train) and I use this repo to reproduce the void-train method and obtain a slightly worse result:

So I'd like to know if it is possible for you to release the checkpoint of void-train baseline for K=20% setting, many many thanks~