Question about parameters of TOAST-Lite?

[laisimiao]

I see the specific implement of TOAST-Lite is this: lora wrap the TOAST and only tune the lora-related parameters. Therefore, does the parameters of TOAST-Lite mentioned in the paper refer to this part. If so, I think training TOAST-Lite should load pretrained TOAST parameters, right or not?

[bfshi]

Yes. TOAST-Lite tunes the lora parameters in the feedback path (decoders), as well as the feature selection part (prompt and top_down_transform). You should load the pre-tuned TOAST parameters before training TOAST-Lite.

[laisimiao]

Thank you for your timely reply! Best wishes!