Synthesize on multiple-machine topology

[DictXiong]

Hi. I'm trying to synthesize some pareto-optimal allreduce algorithms on several DGX-A100 machines, which are connected via a single switch. I didn't find this among the pre-defined topologies, and I also don't know how to write a proper topology file. Could you please help me with this?

For example, there are three DGX-A100 machines. Each of them has 8 A100 GPUs and 4 200Gbps NICs. The total 12 NICs are connected to one switch. What's the best practice to synthesize allreduce algorithms on this topology?

Thanks.

[saeedmaleki]

Sorry for late reply! For synthesizing multi-node algorithm, please use TACCL. However, if you wish to write your algorithm manually, you could start from one of the many multi-node A100 algorithms. For example, this one.