程序卡死的问题

[jinge170]

当使用MNE数据集参与训练时，总是在某一个位置卡死

[jinge170]

[rank1]:[E529 19:22:39.619605859 ProcessGroupNCCL.cpp:607] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=169, OpType=ALLREDUCE, NumelIn=2780859, NumelOut=2780859, Timeout(ms)=600000) ran for 600009 milliseconds before timing out.
[rank1]:[E529 19:22:39.619673649 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 1] Exception (either an error or timeout) detected by watchdog at work: 169, last enqueued NCCL work: 172, last completed NCCL work: 168.
[rank1]:[E529 19:22:39.619676864 ProcessGroupNCCL.cpp:1709] [PG 0 (default_pg) Rank 1] Timeout at NCCL work: 169, last enqueued NCCL work: 172, last completed NCCL work: 168.
[rank1]:[E529 19:22:39.619679254 ProcessGroupNCCL.cpp:621] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E529 19:22:39.619681482 ProcessGroupNCCL.cpp:627] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E529 19:22:39.620440568 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=169, OpType=ALLREDUCE, NumelIn=2780859, NumelOut=2780859, Timeout(ms)=600000) ran for 600009 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff669d77f86 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7ff5f7bc88f2 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7ff5f7bcf333 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7ff5f7bd171c in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xdbbf4 (0x7ff6692dbbf4 in /usr/local/share/anaconda3/envs/hwt/bin/../lib/libstdc++.so.6)
frame #5: + 0x94ac3 (0x7ff66e494ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126850 (0x7ff66e526850 in /lib/x86_64-linux-gnu/libc.so.6)
这是报错问题，是因为我的设备问题吗

[TongkunGuan]

[rank1]:[E529 19:22:39.619605859 ProcessGroupNCCL.cpp:607] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=169, OpType=ALLREDUCE, NumelIn=2780859, NumelOut=2780859, Timeout(ms)=600000) ran for 600009 milliseconds before timing out. [rank1]:[E529 19:22:39.619673649 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 1] Exception (either an error or timeout) detected by watchdog at work: 169, last enqueued NCCL work: 172, last completed NCCL work: 168. [rank1]:[E529 19:22:39.619676864 ProcessGroupNCCL.cpp:1709] [PG 0 (default_pg) Rank 1] Timeout at NCCL work: 169, last enqueued NCCL work: 172, last completed NCCL work: 168. [rank1]:[E529 19:22:39.619679254 ProcessGroupNCCL.cpp:621] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [rank1]:[E529 19:22:39.619681482 ProcessGroupNCCL.cpp:627] [Rank 1] To avoid data inconsistency, we are taking the entire process down. [rank1]:[E529 19:22:39.620440568 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=169, OpType=ALLREDUCE, NumelIn=2780859, NumelOut=2780859, Timeout(ms)=600000) ran for 600009 milliseconds before timing out. Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff669d77f86 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libc10.so) frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7ff5f7bc88f2 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7ff5f7bcf333 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7ff5f7bd171c in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #4: + 0xdbbf4 (0x7ff6692dbbf4 in /usr/local/share/anaconda3/envs/hwt/bin/../lib/libstdc++.so.6) frame #5: + 0x94ac3 (0x7ff66e494ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #6: + 0x126850 (0x7ff66e526850 in /lib/x86_64-linux-gnu/libc.so.6) 这是报错问题，是因为我的设备问题吗

我们之前没遇到过这种情况 看上去是看门狗超时 可以减少验证的频率

[jinge170]

[rank1]:[E529 19:22:39.619605859 ProcessGroupNCCL.cpp:607] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=169, OpType=ALLREDUCE, NumelIn=2780859, NumelOut=2780859, Timeout(ms)=600000) ran for 600009 milliseconds before timing out. [rank1]:[E529 19:22:39.619673649 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 1] Exception (either an error or timeout) detected by watchdog at work: 169, last enqueued NCCL work: 172, last completed NCCL work: 168. [rank1]:[E529 19:22:39.619676864 ProcessGroupNCCL.cpp:1709] [PG 0 (default_pg) Rank 1] Timeout at NCCL work: 169, last enqueued NCCL work: 172, last completed NCCL work: 168. [rank1]:[E529 19:22:39.619679254 ProcessGroupNCCL.cpp:621] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [rank1]:[E529 19:22:39.619681482 ProcessGroupNCCL.cpp:627] [Rank 1] To avoid data inconsistency, we are taking the entire process down. [rank1]:[E529 19:22:39.620440568 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=169, OpType=ALLREDUCE, NumelIn=2780859, NumelOut=2780859, Timeout(ms)=600000) ran for 600009 milliseconds before timing out. Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff669d77f86 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libc10.so) frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7ff5f7bc88f2 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7ff5f7bcf333 in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7ff5f7bd171c in /home/libj/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so) frame #4: + 0xdbbf4 (0x7ff6692dbbf4 in /usr/local/share/anaconda3/envs/hwt/bin/../lib/libstdc++.so.6) frame #5: + 0x94ac3 (0x7ff66e494ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #6: + 0x126850 (0x7ff66e526850 in /lib/x86_64-linux-gnu/libc.so.6) 这是报错问题，是因为我的设备问题吗

我们之前没遇到过这种情况 看上去是看门狗超时 可以减少验证的
请问在构建新的训练数据集时对这个图片的尺寸有什么要求吗，我是重新构建了这个数据集然后出现这种程序卡死问题，第一个训练epoch都过不了