Running into GPU out-of-memory issue during pretraining. Any suggestions to reduce GPU memory usage?

[sythello]

As title.
By the way, I saw there is an argument empty-cache-freq, would decreasing that help? What would be the side effects of decreasing it?

Any inputs are appreciated. Thanks!