multi-node multi-GPU training

[Polycoriander]

Hello, I’m training Kinetics-600 on V100 with 17 frames of 128×128, latent size 5×16×16, and a single-GPU batch size of 4. I’m puzzled because using multiple nodes does not speed up training, adding more GPUs does not help, reducing the sub-dataset size does not make it faster, and using gradient accumulation actually makes it slower. Have you encountered this issue?

[buoyancy99]

An increasing number of nodes is effectively increasing batch size - 4 data points per GPU * 32 GPUs will be 128 points per gradient iteration. Bigger batch size leads to more stable gradients, but won't accelerate training much unless you increase the learning rate proportionally!

This has been well-studied by an OpenAI paper - if you double your batch size, consider doubling your lr as well for better speed.

[Polycoriander]

An increasing number of nodes is effectively increasing batch size - 4 data points per GPU * 32 GPUs will be 128 points per gradient iteration. Bigger batch size leads to more stable gradients, but won't accelerate training much unless you increase the learning rate proportionally!

This has been well-studied by an OpenAI paper - if you double your batch size, consider doubling your lr as well for better speed.

Thank you for your reply, but what I meant was the training duration per epoch, and this time has not sped up.