Result remains unchanged when swapping input audio order

[Acecy237]

Hello, thank you very much for open-sourcing this work. While testing it, I noticed a potential issue: if the two input audio samples are swapped, the final output remains the same. I have tested this multiple times and consistently observed the same behavior.

Sample 1:
[Final Result] B
Average Score of Audio A: 5.9, Average Score of Audio B: 8.7

Sample 2 (inputs swapped):
[Final Result] B
Average Score of Audio A: 5.0, Average Score of Audio B: 8.6

[R-expression]

Same behaviour observed. Model B is consistantly better than Model A, no matter what the models are.

[RMSnow]

Thanks for pointing out this! The position bias is indeed an important issue. Some existing works suppose this might be raised by the training data, or inherent to the left-to-right architecture of causal transformers (even for some stronger models like Claude and GPT, see: https://arxiv.org/abs/2306.05685).

We will take a closer look for this issue. We hope to fix this in the next version of GRM.