Documentation for Dataset Preparation

[samveldfwork]

Hello,

Thank you for your great work, I managed to run the code with the provided dataset.
However, I have been exploring the repository as well as the paper and noticed that the method for preparing the dataset is not clearly documented.
While there are several methods and scripts related to data processing, a comprehensive guide or documentation on how to prepare the dataset using our own data for training and evaluation is missing.

Specifically, it would be much appreciated if you could provide detailed instructions on:

1. The format and structure of the input data.
2. Steps to preprocess and prepare the dataset.
3. Any specific requirements or dependencies needed for dataset preparation.
4. Examples of commands or scripts to run for dataset preparation, especially about how to build and embed the subgraphs.

Thank you for your attention to this matter.

[RidhiChhajer]

is the subgraph retrieved every time the query is asked?

[cmavro]

Thanks for your suggestions!

Here are the basic steps:

GNN:

1. Entity Linking: The question entities are linked to the KG.
2. Subgraph Extraction: The KG subgraphs (e.g., 4-hop neighbors) are extracted based on the linked entities.

For WebQSP and CWQ, we follow the algorithm of NSM. You can find their preprocessing steps here.
After executed, the input data file (json format, e.g., test.json) consists of the following fields; question, seed_entities (obtained via entity linking), subgraph tuples (obtained via subgraph extraction -- these are in the format (head id, relation id, tail id) or (head name, relation, tail name)), answer.

Doing the above steps for train and test questions will result into the necessary data files to train your GNN.
We train the ReaRev GNN described here, but you can use different ones.

RAG:

Please, run inference with the GNN as described here. This will generate the candidate answers obtained by the GNN in the right format.

For RAG, we follow RoG (their github code looks deactivated at the moment) as described in the GNN-RAG/llm folder.
The shortest paths obtained by the GNN are verbalized and concatenated at the input to produce the LLM predictions.

Overall:

KG subgraphs are needed for each question for GNN training/evaluation. Then, the shortest paths between question entities and answer candidates are extracted for RAG. In our work, we follow previous works (GraftNet, NSM) and their preprocessing steps to get the KG subgraphs for WebQSP and CWQ from Freebase KG. If you need to test your own data, you should follow similar data preprocessing steps.

I will try to upload code samples in the next weeks, thanks for your patience!

[RidhiChhajer]

Thank you for your reply.
Looking forward to your uploads next week :)

[RidhiChhajer]

@cmavro how to use the MetaQA data? Do I need to change the pipeline or the data format?

[MINJIK01]

Hello, I am curious about the maximum hop of the sub-graph extracted from the KG. The code you referred to is for only the 2-hop sub-graph. How did you get the over 2 hop sub-graph? Did you modify the code or just use a 2-hop sub-graph?

Thank you.

[Zony7]

@cmavro Hi, thanks a lot for sharing the steps about GNN and RAG. We're really looking forward to the code samples you mentioned. Would you mind letting us know when you'll be able to upload them? It would be a great help to our work. Thanks!

[cmavro]

Hi all,

GNN-RAG follows the graph format of previous works ReaRev, NSM, GraftNet.

You can find preprocessing scripts for Freebase here: https://github.com/RichardHGL/WSDM2021_NSM/tree/main/preprocessing/Freebase

This graph format is used in https://github.com/cmavro/GNN-RAG/blob/main/gnn/dataset_load.py.