Regarding the inference part of LLM

[2066537830]

Thank you for your great work!
According to the prompt in rag-reasoning.sh, the difference between the operation of GNN-RAG and GNn-rag-RA lies in whether --add_rule is added, that is, whether RA is used, as shown in the figure:

Under the KGQA-GNN-RAG result, RA should not be used, but the result file name you provided uses RA, as shown in the following figure. Why is that?

And rags.txt shows that --add_rule is enabled:

Without the -add_rule parameter, the result I reproduced is:
Accuracy: 51.039005222773895 Hit: 66.15479115479116 Hit1: 60.68796068796069 F1: 49.91688643599812 Precision: 56.828675933272876 Recall: 51.039005222773895。
Add the -add_rule parameter, and the reproduction result is:
Accuracy: 73.76651533728501 Hit: 85.68796068796068 Hit1: 80.58968058968058 F1: 71.28257365406769 Precision: 77.18506449319466 Recall: 73.76651533728501。
I hope you can clear up my doubts!

[cmavro]

Hi,

Without the -add_rule parameter, the result I reproduced is:
Accuracy: 51.039005222773895 Hit: 66.15479115479116 Hit1: 60.68796068796069

It seems the performance you report is similar to LLM inference without using KG as context (e.g., ChatGPT results in Table 1 of the paper). Could you verify that you utilize the graph context retrieved as input for LLM inference?

[2066537830]

Thank you so much for your reply ! I think I utilize the graph context retrieved as input for LLM inference. Could you please clear up the following doubts?  I copied the test.info file and test.json file under gnn to resluts/gnn/, to evaluate GNN-RAG by the provided script. I didn't modify any code, and the result I got Accuracy: 51.039005222773895 Hit: 66.15479115479116 Hit1: 60.68796068796069. May I ask if there is anything wrong with my approach? excuse me ? ***@***.*** 2.Under the KGQA-GNN-RAG result, RA should not be used, but the result file name you provided uses RA, as shown in the following figure. Why is that?   3. The same result can be obtained without using RULE_PATH_G1 as when using RULE_PATH_G1 Maybe I have some wrong understanding of the experiment. Thank you for pointing it out to me. Thank you again for your help ！ 原始邮件 发件人：Costas Mavromatis ***@***.***> 发件时间：2025年9月12日 21:52 收件人：cmavro/GNN-RAG ***@***.***> 抄送：Zhou lincheng ***@***.***>, Author ***@***.***> 主题：Re: [cmavro/GNN-RAG] Regarding the inference part of LLM (Issue #23) cmavro left a comment (cmavro/GNN-RAG#23) Hi, Without the -add_rule parameter, the result I reproduced is: Accuracy: 51.039005222773895 Hit: 66.15479115479116 Hit1: 60.68796068796069 It seems the performance you report is similar to LLM inference without using KG as context (e.g., ChatGPT results in Table 1 of the paper). Could you verify that you utilize the graph context retrieved as input for LLM inference? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[DengShuXIan0122]

Hi,

Without the -add_rule parameter, the result I reproduced is:
Accuracy: 51.039005222773895 Hit: 66.15479115479116 Hit1: 60.68796068796069

It seems the performance you report is similar to LLM inference without using KG as context (e.g., ChatGPT results in Table 1 of the paper). Could you verify that you utilize the graph context retrieved as input for LLM inference?

Hello, I also have the same question, could I ask if you solve this problem and how to solve that.I checked and it seems the test.json file I downloaded from HuggingFace rmanluo/RoG-webqsphas 1628 entries, while thetest.infofile output by the GNN model on the test set (question-by-question retrieval/evaluation) has 1628 lines, causing the misalignment. Is this the problem? How should I downloadtest.json? Should I directly use the test.json` file from the dataset?
Thank you!