Why are only the weights of `id_encoder` updated in the second stage of training?

[gaoyixuan111]

This is a very good piece of work. Could you please explain why, after adding Ledit in the second stage of training, the weights of adapter_modules are fixed and only the weights of id_encoder are updated? Why not fix the weights of id_encoder and only update the adapter_modules instead?

[gaoyixuan111]

Is it not possible to use the RCA connection between two attention layers in the W+Adapter? This would perform significantly better than parallel connections, possibly because of the use of the editing loss?
@csyxwei

[csyxwei]

This is a very good piece of work. Could you please explain why, after adding Ledit in the second stage of training, the weights of adapter_modules are fixed and only the weights of id_encoder are updated? Why not fix the weights of id_encoder and only update the adapter_modules instead?

Hi, thanks for your interest in our work. In our earlier experiments, we tried updating only the adapter modules and found that it affected identity preservation. Therefore, we kept it fixed in our later experiments.

[csyxwei]

Is it not possible to use the RCA connection between two attention layers in the W+Adapter? This would perform significantly better than parallel connections, possibly because of the use of the editing loss? @csyxwei

It is absolutely possible to use an RCA connection similar to the W+ adapter to inject ID information. In our experiments, we used the dual cross-attention approach, following the IP Adapter. Feel free to explore other ways to inject ID information.

[gaoyixuan111]

Thank you for your response. I am making extensive modifications based on your article and my research direction. However, I have one point of confusion: in dataset_laion.py, there is a line of code ref_images = ref_images[:3]. Wouldn't this cause a significant waste of resources when the batch size is set to 32?
@csyxwei

[csyxwei]

In our experiments, we found it is acceptable. You can modify it according to your requirements.