Conditional Synthetic dataset Generation through SONAR

[sleepingcat4]

I have read SONAR and it's reconstruction of text through embeddings is quite exciting and I was thinking if it was possible to tweak with the decoder so that it reconstructs embeddings to text in slightly to intermediately different way.

For example:
`I have always loved you' to now reconstruction: 'I have loved you from beginning'.

such conditional text altercation and generation possibilities could be really helpful and useful. if there are any advice or directions available, I would love to explore.

[nitishpandey04]

Changing the model itself for this purpose would mean reducing the accuracy of the Sonar model. You can try out different sampling methods once the model output probability distribution to generate different sequences. You can explore beam search for this - https://huggingface.co/blog/how-to-generate#beam-search

[sleepingcat4]

@nitishpandey04 also can you share a Colab on how to install it? Because it has dependency errors with FAIR and pytorch and default installation does not seem to resolve the issue.

[nitishpandey04]

See this issue for installation - #49

[avidale]

First, a note on installation: when we merge #62 and upgrade the package version to 0.4, installing an appropriate version of fairseq2 (and matching pytorch) would become easier.

Second, on the substance of the question: SONAR indeed can be used for rewriting sentences with minor alterations.

Sampling random variations of sentences with SONAR

We know that if we encode a sentence with SONAR into an embedding vector, and then decode it back to text, the resulting sentence might be slightly different from the original one. But why would one want to generate many different sentences? Potential motivation might be:

• Just having fun
• Exploring the SONAR space and finding which sentences it finds similar
• Augmenting your (maybe multilingual) text data with multiple alterations of the same sentence

Below, I introduce three ways to achieve this.

Default sentence reconstruction

By default, if you encode the sentence into an embedding and then decode it back to text, the result will always be the same (although not always identical to the original sentence). For example:

import torch
from fairseq2.generation import TopPSampler, TopKSampler
from sonar.inference_pipelines.text import TextToEmbeddingModelPipeline, EmbeddingToTextModelPipeline

embedder = TextToEmbeddingModelPipeline(encoder="text_sonar_basic_encoder", tokenizer="text_sonar_basic_encoder", device=torch.device("cuda"))
unembedder = EmbeddingToTextModelPipeline(decoder="text_sonar_basic_decoder", tokenizer="text_sonar_basic_encoder", device=torch.device("cuda"))

embeddings = embedder.predict(["I have always loved you!"] * 5, source_lang="fra_Latn")
for text in unembedder.predict(embeddings, target_lang="eng_Latn"):
    print(text)
# I've always loved you!
# I've always loved you!
# I've always loved you!
# I've always loved you!
# I've always loved you!

Sampling during generation

This is because by default SONAR uses beam search decoding, which is deterministic. However, we can use a random sampling algorithm instead (for example, top-p or top-k sampling). It is activated by adding a sampler argument when calling the decoder. For example:

embeddings = embedder.predict(["I have always loved you!"] * 5, source_lang="fra_Latn")

torch.manual_seed(42) # if you do not set the random seed, the results won't be reproducible
for text in unembedder.predict(embeddings, target_lang="eng_Latn", sampler=TopPSampler(0.99)):
    print(text)
# I've always loved you!
# I've always loved you!
# I always loved you!
# I have always loved you!
# I've always loved you!

Now 2 out of 5 sentences were generated slightly differently. If you want to introduce even more randomness, you can increase the sampling "temperature" (it is one of the many arguments you can set for a SamplingSeq2SeqGenerator in fairseq2; all these arguments are passed to its constructor). The default value equals 0.6 for some weird reason; increasing the temperature increases the randomness (which may lead to generating really weird stuff):

embeddings = embedder.predict(["I have always loved you!"] * 5, source_lang="fra_Latn")

torch.manual_seed(42) # if you do not set the random seed, the results won't be reproducible
for text in unembedder.predict(embeddings, target_lang="eng_Latn", sampler=TopPSampler(0.99), temperature=0.99):
    print(text)
# I'm never a real drunk!" I kolonekent love you too much្ង edwɛkɛnhosaigh mé níka-wéronกับคุณ爱柱
# I've always love you!
# I always loved you !
# I have always loved you!
# I've always loved you kwedu!

Adding noise to the embeddings

SONAR decoder is doing a pretty good job of trying to reconstruct a fluent sentence that might have been encoded as the given vector... unless we interfere in the decoding by randomizing it. An alternative way to introduce randomness would be to leave the decoder alone, and instead make the sentence embeddings random by adding a small random normal noise to them:

embeddings = embedder.predict(["I have always loved you!"] * 5, source_lang="fra_Latn")

torch.manual_seed(42)
for text in unembedder.predict(embeddings + torch.randn_like(embeddings) * 0.01, target_lang="eng_Latn"):
    print(text)
# I 've always loved you .
# I have always loved you.
# I have always loved you!
# I always loved you!
# I've always loved you.

By increasing the amplitude of this random noise, we can make the resulting diverge further from the original sentence... While still usually staying fluent, unlike the sentences generated with sampling under high temperature!

torch.manual_seed(42)
for text in unembedder.predict(embeddings + torch.randn_like(embeddings) * 0.015, target_lang="eng_Latn"):
    print(text)
# Where is it ?
# I've never seen you.
# I've always been attracted to you.
# I love you!
# I have always loved you.

Letting the encoder use random dropout

A big question here is how exactly you should set the amplitude for the random noise. Unfortunately, you have to find the good value for it with trial and error, and most likely, it is going to depend on the type of the sentences that you want to modify: some sentences are more "fragile" than others (in the sense discussed in Section 2.5.2 of the Large Concept Models: Language Modeling in a Sentence Representation Space paper, which I recommend to read).

One way to avoid making the choice of the noise distribution would be to let the encoder compute embeddings in a random way that is "natural" and to it. And this way has been pre-built into it, because, like many other neural networks, SONAR uses random dropout during training. So if we activate the training mode for the encoder, its outputs will become slightly randomized:

torch.manual_seed(42)
embedder.model.train() # put the model into training mode, which turns on the dropout
embeddings = embedder.predict(["I have always loved you!"] * 5, source_lang="fra_Latn")
embedder.model.eval() # turn off the dropout, just to leave the things as they were

for text in unembedder.predict(embeddings, target_lang="eng_Latn"):
    print(text)

# I always loved you! have
# I've always loved you!
# I've always loved you!
# I always have always loved you!
# I have always loved you!

Because this way of randomization is the least intrusive, I hope that it would work well for a large variety of sentences (but I haven't verified this experimentally!).