Readings of how comm groups are decided?

[kelayamatoz]

Hi,

Thank you so much for your contribution! This is a super interesting tool and I can definitely see the value.

Out of curiosity, do you have any plans to write up how the partitioning and mapping algorithms are designed? While the codebase works smoothly, it is a little bit dense and kinda hard to follow. Do you plan to write a paper about the trace generator at some point?

Thanks!

[kelayamatoz]

Also, is there a plan to simulate SDPA (flash attention) training? Thank you!

[jonassvedas]

If the communication group file is implemented as per astra-sim input: https://astra-sim.github.io/astra-sim-docs/getting-started/argument-workload-config.html#enable-communicator-groups

Can someone explain why this is the comm_group:
{"1": [0, 1], "2": [0], "3": [1], "4": [0], "5": [1]}

for the following command:
python main.py --output_dir llm_dp2/ --output_name workload.%d.et --comm_group_file comm_group.json --dp 2 --tp 1 --pp 1

4 and 5 seem to be redundant. Also why have a comm group with only one device in it (2,3,4,5)