plug-and-play question

[YeLyea]

I ran the script directly and the results were as follows:
Accuracy: 58.2087050882302 Hit: 62.1636930048145 Hit1: 10.421976777116964 F1: 21.019276287307328 Precision: 16.0370342136767 Recall: 58.2087050882302
F1 and Precisionare low,How can I get the correct F1?Thank you!

[cmavro]

"plug-and-play" script uses open-source/black-box LLMs, which are not fine-tuned for generating a list of answers for KGQA. As they generate free-form responses, it is not possible to accurately compute the F1 metric (which requires a list of candidate answers to be compared against the list of ground-truth answers).

A workaround would be to prompt the LLM to explicitly generate a list of answers, e..g, within tags, and then parse this response to obtain the list of candidate answers.