[Feature] audio improvements (VAD / automatic microphone threshold / Noise supression)

[Lama-Thematique]

Hello,

I went through many client, and I like the UI of this one very much.

While testing clients, with my friend we found out that many clients doesn't have proper audio processing leading to high amount of noises.

Only Element has a basic noise reduction but it's still not on par with what I would expect to have a good experience.

I think having a client with proper audio channels would elevate the quality of the client a lot.

Also it doesn't seem like element is willing to implement theses feature due to not being lightweight : element-hq/element-call#492

There is many open source VAD (Voice Activity Detection), and noise suppression models existing that could be used for this purpose.
Examples :

• easyeffects, has all of the featureyou would want (VAD, automatic threshold, noise reduction, noise supression, echo cancelation), GPL-v3 https://github.com/wwmm/easyeffects
• ten-vad, apache 2.0, with WASM : https://github.com/TEN-framework/ten-vad
• DeepFilterNet, MIT/apache 2.0 : https://github.com/Rikorose/DeepFilterNet

[degu-Tanya]

Same, would be very nice. So far this is my fav client because of the look.

[Airyzz]

This is something I'd absolutely love to have, the biggest issue really though is the entire call audio stack is handled by the webrtc library, and it doesnt currently provide any api to implement things like this, so this would have to be something handled upstream

[Lama-Thematique]

It might be possible using the class TrackProcessor (with LocalAudioTrack.setProcessor) in the livekit_client.

There is a javascript example to remove background from video stream in here https://github.com/livekit/track-processors-js/blob/main/src/transformers/BackgroundTransformer.ts

[Lama-Thematique]

line 181 in commet/lib/client/matrix/components/voip_room/matrix_livekit_backend.dart :

it accept a : TrackProcessor<AudioProcessorOptions>? processor

Also it seems like it already has some noise removal by default, I didn't realize due to not being exposed in the settings (and I couldn't test directly it on this client myself due to not having E2EE call).

It seems this issue is better to keep on standbye at least until #572 is merged.