LASER3-type SONAR encoder distillation script

[lydianish]

Following the discussion in this closed issue, I would like to request a LASER3-type distillation script of the SONAR text encoder.

[avidale]

Hi @lydianish!
In #8 (comment), I provided an example of distilling a speech encoder from the text one.
Here, you are looking for a way to distill a language-specific text encoder, right?
This is something that we could do as well.

[lydianish]

Hi @avidale, thanks! Let me have a look.
Yes, I am looking for a way to distill a language-specific text encoder. I think my main concern was that I couldn't find any training scripts in Fairseq2 like we had in Fairseq for distilling LASER, but I see here you used Transformers so that explains it!

[avidale]

@lydianish In fact, I used Transformers only for a couple of helper functions, and wrote most of the training loop from scratch in plain Python and Torch. It is a deliberate decision not to force people in to any specific framework.

In the future, when we develop a robust and easy training pipeline for Fairseq2, we would probably publish it. But currently, the Fairseq2 API is still in rapid progress, and may undergo many backward-incompatible changes, so we don't want yet to popularize it.

[lydianish]

@avidale Thanks for the clarification!