GPU utility extremely low

[czy97]

When I run the callhome recipe using the default config file, the GPU utility is extremely low (less than 10%). Is this normal?

[czy97]

When I run the callhome recipe using the default config file, the GPU utility is extremely low (less than 10%). Is this normal?
Could this be caused by the pit loss calculation? I found that the code calculates pit loss serially.

[nttcslab-sp-admin]

It can be related to the PIT loss calculation, but probably not.
The current setting (conf/train.yaml) assumes that a user uses a relatively old/weak GPU with a small memory size. If you are using a GPU with large memory, you can decrease batchsize_per_gpu (in conf/train.yaml) and simultaneously increase batchsize (in conf/train.yaml) to put the training data as much as you can onto a GPU, while always making sure that batchsize * batchsize_per_gpu remains 1024, e.g. batchsize:512, batchsize_per_gpu:2. By doing so, you should be able to increase the GPU utility.

[czy97]

Thanks for the comment. By the way, can you upload the log of Callhome recipe if possible. I can't reproduce the results you listed and I want to find the reasons.

[czy97]

In addtion, I find that the spk loss and pit loss calculation do influence the training speed a lot. I update the calculation here. You can check it.

[nttcslab-sp-admin]

Thanks for the feedback. We uploaded the log files in https://github.com/nttcslab-sp/EEND-vector-clustering/blob/main/egs/callhome/v1/Log.tar.gz

[czy97]

Sorry for bothering again, I find my reproduction can achieve similar results with yours when the speaker number is small. The results get worse when there is more speakers. Can you give me some advice. Thanks.

 Spk# | Spk2 | Spk3 | Spk4 | Spk5 | Spk6 | Spk_all
Yours | 7.96 | 11.93 | 16.38 | 21.21 | 23.10 | 12.49
Mine | 7.96 | 12.69 | 19.45 | 29.01 | 32.34 | 14.13

[kli017]

hello can you upload the loss of mini_Librispeech?

[nttcslab-sp-admin]

Hi, kli017!
We have an excerpt of validation loss transition for mini_librispeech in https://github.com/nttcslab-sp/EEND-vector-clustering/blob/main/egs/mini_librispeech/v1/RESULT.md.
Is this sufficient for your purpose? Or, you need an entire log for the training? We may need a couple of days to reproduce the log (since we first need to restore the experimental condition we used before).

[kli017]

@nttcslab-sp-admin Hi Thanks for the quick reply! I check the train log in the RESULT.md and found that the Mean Loss of mine is much higher than yours. I trained the model for 10 epochs and the loss just decrease from 0.6628450117613139 to 0.6555626417461194. And hte DER for nspk0 and nspk1 are 48.71 and 52.89 respecitvely. Have you changed any parameters or something in the recipe?

train.log.txt

[nttcslab-sp-admin]

Hi, @kli017!
It looks like you are using 8GPUs. Could you try to use only one GPU, i.e., CUDA_VISIBLE_DEVICES=0 and rerun the recipe? With that much GPUs (that actually changes effective batchsize, etc.), our preset hyper-parameters are simply far off optimal values, I guess.

[nttcslab-sp-admin]

@czy97
We previously suggested to change the batchsize to speed up your training, such that batchsize * batchsize_per_gpu remains 1024 (more strictly, batchsize * batchsize_per_gpu * num_GPUs remains 1024), but it turned out that we cannot reproduce the same/similar results in that way with e.g. batchsize:512, batchsize_per_gpu:2. We sometimes got a very bad result as you did. We are looking into this issue.

In the meanwhile, we found that if you change the chunk_size: 150 to chunk_size: 500 in conf/train.yaml and use 1 GPU for the training, you can speed up the training, and obtain OK-ish result (something like 12.98% for unknown Spk# conditions). But this report is not the final one. We'll get back to you once we find what really is the problem there and the solution for it.

[czy97]

Thanks for the reply. Looking forward to the final solution.

[kli017]

@nttcslab-sp-admin yes I was training with 8GPUs. So current code does not support multi GPU training? I also tried 2 speaker without overlap for 100 epochs. The Mean loss is around 0.46 and DER is 45.19. I ran the inference on dev set and cut the audio according to the rttm and found the reulst really bad, in some result even only got 1 speaker. I checked the EEND hitachi-speech/EEND#4 and they said they never validate on data less than 100hours. So, I was confusing what lead to the problem. multi GPUs or the training set was too small.