I have generated (C,S,R)=(8,3,3) alltoall algorithm using SCCL synthesizer on DGX-1 topology. And I use msccl ncclize <file.json> to generate xml file for msccl runtime. Then, I use master branch of msccl as msccl-rt to run this scheme. Above msccl, a fixed version of nccl-tests for alltoall is used.
My script is just like:
...
NCCL_DEBUG="INFO" \
NCCL_DEBUG_SUBSYS="INIT,ENV" \
MSCCL_XML_FILES="schemes/Alltoall.n8-DGX1-steps3.msccl.xml" \
NCCL_ALGO=MSCCL,RING,TREE \
LD_LIBRARY_PATH=${NCCL_BUILD_PATH}/lib:$LD_LIBRARY_PATH \
srun -N 1 --ntasks-per-node 8 \
nccl-tests/build_msccl/alltoall_perf \
-b32M -e256M -f 2 -g 1
And the results are below:
We only care about out-of-place results, because nccl-tests doesn't support in-place mode for alltoall. We can see that there is a big gap of busBW between size=64MB and 128MB which is extremely strange.
After I read the source codes of msccl, I know that it's because for function ncclAlltoAll in all_to_all.cc of msccl, it use msccl algorithm for size < 128MB , otherwise, it use naive implementation for alltoall.
Thus the question is that, in SCCL paper, the performance data of (8,3,3) are higher than baseline, i.e. naive implementation, In the case of all message sizes. But why in my experiments, (8,3,3) algorithm even shows worse performance than baseline?
In addition, In theoretical analysis under $\alpha,\beta$ p2p communication model, (8,3,3) algorithm should have been better than baseline, at least under the condition of large message size.
Thus, is it due to the poor implementation of cudaKernel for MSCCL algorithm ?
I have generated
(C,S,R)=(8,3,3)alltoall algorithm using SCCL synthesizer on DGX-1 topology. And I usemsccl ncclize <file.json>to generate xml file for msccl runtime. Then, I use master branch of msccl as msccl-rt to run this scheme. Above msccl, a fixed version of nccl-tests for alltoall is used.My script is just like:
And the results are below:
We only care about out-of-place results, because nccl-tests doesn't support in-place mode for alltoall. We can see that there is a big gap of busBW between size=64MB and 128MB which is extremely strange.
After I read the source codes of msccl, I know that it's because for function
ncclAlltoAllinall_to_all.ccof msccl, it use msccl algorithm forsize < 128MB, otherwise, it use naive implementation for alltoall.Thus the question is that, in SCCL paper, the performance data of (8,3,3) are higher than baseline, i.e. naive implementation, In the case of all message sizes. But why in my experiments, (8,3,3) algorithm even shows worse performance than baseline?
$\alpha,\beta$ p2p communication model, (8,3,3) algorithm should have been better than baseline, at least under the condition of large message size.
In addition, In theoretical analysis under
Thus, is it due to the poor implementation of cudaKernel for MSCCL algorithm ?