NAN appears at training

[ZhangZ7112]

Excuse me. When I reproduce the work to train on the provided preprocessed data of BTCV dataset. I didn't make any other changes except 1)using a single GPU (RTX 4090) instead; 2) using pre-trained SAM ViT_B. However, NAN appears at about epoch 4/400.

I insert some code snippets to help find the reason, through which I find the origin of NAN occurs no later than the output['low_res_logits'] .

Could you please offer some hints to further identify and resolve the problem?

[ZhangHaoyang493]

I have encounter the same problem. I have no idea on how to solve it.

[lihe-CV]

I have encounter the same problem, can you solve it?

[MaoFuyou]

can you see your lr?