Nan loss

[LUOBO123LUOBO123]

I change the input resolution to 416*416 when I train custom datasets. When the network is trained for 49 epochs, the print loss is nan.What could be the reason for this?

[SelfSup-MIM]

Hi, there are several tips that may help alleviate the issue.

1. Decrease learning rate.
2. Enlarge the drop path ratio.
3. Decrease clip grad number.
4. Enlarge warm up epochs.
5. Use FP32 for Attention block and layernorm, instead of FP16.
6. Adapt the decay number of AdamW optimizer. The default is betas=(0.9, 0.999) while MAE uses betas=(0.9, 0.95). Maybe it helps improve the stability of training.