Scalability to larger number of Loras?

[kadarsh22]

Hi, Thanks for your valuable contribution.

I wanted to know if you have tried utilising a large collection of Loras rather than the 28 lora adapters to see the impact of scale. Particularly, have you tried using 256 Loras trained on flan2 as done in Arrow routing?

[ParagonLight]

Hi, thank you for raising this topic.

In terms of the research, the MeteoRA currently designed based on LlaMA 2/3 base model. So we did not take the evaluation on Flan2, which contains 256 tasks. From my point of view, if everything works well (e.g., enough GPU memory for fine-tuning the gate), our implementation could be scaled to such a number of LoRA adapters.