Some NCCL operations have failed or timed out #27
Description
has anyone had the same issue as me?
[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=247474, OpType=ALLGATHER, NumelIn=64, NumelOut=256, Timeout(ms)=600000) ran for 600004 milliseconds before timing out.
