How long you train 800 epochs?

[ngoanpv]

Hi @ZitongYu
I'd like to know how long you train the model with 800 epochs?
In my experiment, I trained 1 epoch (batch size is 8 and 20000 steps) spending 12 hours on a single GPU (P100).
It's so long and I think something that went wrong, any suggestion for me?

[rollovd]

As for me, it takes about 17 hours per 1 epoch (batch size 13 (if more then it will be CUDA out of memory error), 1m images, 2080ti). Pretty slow fitting. Can't understand where is a bottleneck.

[ngoanpv]

It's too slow to fit, as you mention it spend 17h/1 epoch, so to train as authors that train 800 epochs to fit you have to spend 13600h. It is unacceptable

[haiderasad]

@ngoanpv @rollovd what was the size of your model?

[abhirajasp]

Agreed, the model training is quite slow. What is the reason? @ZitongYu

[YaphetS7]

@ngoanpv @rollovd @abhirajasp #64