Loss is Nan!

[YangJae96]

Hello. Thank you for your greak work.

I've run the training code with market-cuhk-duke-msmt-cuhk03,
but the RD_loss, PD_loss, Conf_loss shows up as Nan in some epoch. Also, Div_loss is always Nan.

Is this normal? or something going wrong?..

[feifeiobama]

It's not normal. I have never seen this before.

Can you try another run? maybe it is a accidental error. I will look into it if the Nan persists.

[LeeHoang2710]

Hello,
I am currently working on this topic, and I found your work to be really awesome.
I just downloaded the Market-1501 dataset to test it first, but I encountered this error. I have already converted all relevant variables to int32, but I still get the error. Could you please take a look and help me?

Thank you!

[feifeiobama]

You may add the following code before fast_evaluate_rank() in test_p_s.py:

from IPython import embed
embed()

Then print or check each input arg. If they all contain correct value but only with wrong dtype, you can cast them explicitly.

[Victor-ice]

Hello.
I am currently working on this project and I found your source code and read your paper.
I am trying to train on Market1501 dataset but encountered this error. Could you please take a look? I would really appreciate your help.

[feifeiobama]

Hello. I am currently working on this project and I found your source code and read your paper. I am trying to train on Market1501 dataset but encountered this error. Could you please take a look? I would really appreciate your help.

I think you need to set it to 'market', see https://github.com/feifeiobama/PatchKD/blob/main/train_test.py#L145-L146

[Victor-ice]

Thank you for replying. Yes i set it to 'market' but I got this error.

I think it's trying to create an additional path that doesn't exist

[feifeiobama]

Thank you for replying. Yes i set it to 'market' but I got this error.

I think it's trying to create an additional path that doesn't exist

You should set the test_dataset as well, otherwise the pipeline still loads all datasets for testing purposes, see https://github.com/feifeiobama/PatchKD/blob/main/train_test.py#L147-L148.