Question about using test set to select model?

[kingofapplehead]

Hi,

Thanks for sharing your work and code! I wonder how your final best result is obtained?

I would assume that the results reported in the sota comparison table (TABLE II) are your test set results. And I noticed that that result (89.2 89.7 89.5 78.9) are just the best results in your ablation study in TABLE VIII. So, it seems to me that you relied on the test set to select the best kernel size/fusion to report the best results. Wouldn't this make your results biased and overfit on the test set? I thought that test sets should never be involved in any model selection /hyperparam selection process.

Following this, could you give more details about how the train, val, and test set are used in your model training and hyperparameter tuning? For example, why did you use the val set to select the best learning rate but not other hyperparameters like the kernel size/fusion?

Please excuse me and correct me if I understand your work incorrectly.

Thanks.

[YuemingJin]

Hi,

Thanks for your interest in our work.
We use the 32-40 videos as the validation set to select the learning rate hyperparameter and model in the first training stage, and then we put these videos back to further finetune the network and select the model with the fixed training epoch number as the final model. Details please refer to the implementation details in the paper.

Results with other different hyperparameters that were reported in the paper, are the results of the test videos.
But the best models with these different hyperparameters are selected using the same process as the above.

Thanks.

[kingofapplehead]

Hi Jin,

Thanks for your reply. That makes sense to me. When you add the validation data back for network fine-tuning, how did you decide the value of the fixed epoch to select the final model (since you dont have a validation set now)?

Btw, what algorithm/tool did you use to cut the black margin in the input frames?

Thanks in advance.

[TzuYuOu]

Hi Jin, I also want to ask this question what algorithm/tool did you use to cut the black margin in the input frames?

Thanks in advance.

[YuemingJin]

Hi,

I have uploaded a file called 'video2frame_cutmargin.py', you can refer to the function of change_size().

Thanks