Fine-tuning Details and question about erb bands

[ARKEYTECT]

Hi there:
Great work, the opensource models have great results.
Two questions:
1. I see that you decided to replace 32 erb bands with 480 magnitude bands. Is it because erb bands do not work out very well?
2. Regarding fine-tuning, deepfilternet models definitely suffers from the long-silence issue. And I am looking to recreate it for deepfilternet.
My question, do you just extend the augmentation to 30-40 seconds, or implement special implementation for 30-40 seconds for fine-tuning.
And do you have to freeze some of the models, and fine-tune the grus only?

Thanks!

[danielr-ceva]

@ARKEYTECT Hi,

Thanks for the kind words!

1. The 32 ERB bands are quite compressed and didn’t provide enough frequency resolution for effective noise reduction in the 4.8-24 kHz range. To address this, we introduced a “high-resolution” (HR) variant that operates directly on 480 magnitude bands, allowing for much finer spectral detail.

2. The fine-tuning stage is a bit different from the initial training. For each sample, we concatenate speech segments with long non-speech intervals in between (typically ~3-15 seconds), and allow the noise characteristics to change abruptly within the same sample. These “stress” samples help improve robustness and address the long-silence issue.

   We don’t freeze any part of the model during fine-tuning - all parameters are updated.

Hope this helps 🙂

[ARKEYTECT]

@ARKEYTECT Hi,

Thanks for the kind words!

1. The 32 ERB bands are quite compressed and didn’t provide enough frequency resolution for effective noise reduction in the 4.8-24 kHz range. To address this, we introduced a “high-resolution” (HR) variant that operates directly on 480 magnitude bands, allowing for much finer spectral detail.
2. The fine-tuning stage is a bit different from the initial training. For each sample, we concatenate speech segments with long non-speech intervals in between (typically ~3-15 seconds), and allow the noise characteristics to change abruptly within the same sample. These “stress” samples help improve robustness and address the long-silence issue.
   We don’t freeze any part of the model during fine-tuning - all parameters are updated.

Hope this helps 🙂

Really appreciate the details for the reply.
For 1. Since you are using deepfilternet2 architecture, which is in its nature a two-stage model, then the 480 magnitude bins also seem to help 4.8khz and below too.
@danielr-ceva, Additional question on number of the efficacy of dprnn blocks, I compared the subjective listening quality between the following models: 16khz-dprnn-8, 48khz-dprnn-2 and 48khz-dprnn-8. I noticed that first of all 48khz-dprnn variants are always better in quality, even 48khz-dprnn-2 compared to 16khz-dprnn-8. And then 48khz-dprnn-8 is noticeably better than 48khz-dprnn-2. So my questions are: 1. I think fullband is better than wideband not only because of number of parameters but also because of the additional frequency modeling in the dprnn module, would you agree? 2. was 48khz-dprnn-2 and 48khz-dprnn-8 trained with same parameter? If so, it seems scaling up dprnn provides a substantial improvement in quality.
For 2. Thanks! I will try it out!

[danielr-ceva]

@ARKEYTECT Thanks for the detailed follow-up.

ERB vs magnitude

Yes, using all 480 magnitude bins is also beneficial for frequencies below 4.8 kHz. However, this is less critical in that range because the DF coefficients already perform very well there. The main motivation for using the full set of magnitude bins is to provide higher noise-reduction resolution above 4.8 kHz, where DF is not applied.

16 kHz vs. 48 kHz models

It is interesting that you prefer 48khz-dpdfnet-2 over 16khz-dpdfnet-8, since the latter is a much more capable model in terms of DPRNN depth. In my own evaluations, 16khz-dpdfnet-8 performed better than 48khz-dpdfnet-2.

One possible explanation is that you generally prefer fullband audio over wideband audio, which makes sense perceptually. However, if both outputs were downsampled to 16 kHz, I would expect 16khz-dpdfnet-8 to be preferred.

Another factor that may help the 48 kHz variants is the higher frequency resolution in the magnitude branch. It contains more detailed spectral information than the compressed ERB representation, and this may be especially useful when combined with the DPRNN blocks.

1. So yes, I agree with your point: the fullband model may benefit not only from the number of parameters, but also from the additional frequency modeling provided by the DPRNN module.

2. Yes, 48khz-dpdfnet-2 and 48khz-dpdfnet-8 were trained with the same training setup. The DPRNN blocks are very effective, and they are one of the key components that make DPDFNet significantly better than the original DeepFilterNet2.

[ARKEYTECT]

@danielr-ceva Again, thanks for the detailed reply, it's been a good discussion.
I think what really impressed me about the 48khz-dpdfnet-8 compared to either 16khz-dpdfnet-8 and 48khz-dpdfnet-2 is dereverberation. For this, I had to listen to the output files instead of relying on metrics. Since you mentioned all models are using similar training parameters like loss and rt60 settings, then 48khz-dpdfnet-8 somehow flipped a switch and became very good at dereverberation compared to the rest of the models.