Discrepancy in training results on WebQSP with BERT

[AiRyunn]

Hi! I was trying to train Pangu with BERT on WebQSP but can't match the results from the paper or the repo's trained model.
I have experimented with different epochs, but the results are inconsistent. The only one closer to the reported F1 is using 6 epochs, but there is still a gap, and the F1 with 7 epochs drops significantly.

Training Method	F1
Retrain (1 epoch)	68.8
Retrain (2 epochs)	72.2
Retrain (3 epochs)	71.4
Retrain (4 epochs)	71.7
Retrain (5 epochs)	69.9
Retrain (6 epochs)	75.8
Retrain (7 epochs)	72.9
Retrain (10 epochs)	73.0
Trained Model from Repo	77.3
Reported in Paper	77.9

Can you help me understand why I'm seeing these differences? Thank you in advance!

[entslscheia]

Could you also paste your training configs (e.g., all hyper parameters) here? The trend seems somewhat unusual. I refactored and cleaned up the code before uploading it to the repository. However, these modifications were minor and shouldn't have impacted performance, especially not so noticeably. I'll retrain the model on WebQSP using the code here and see if there are any issues. Will keep you updated.

btw, I have uploaded my original prediction file on WebQSP using BERT-base here for your reference.

[AiRyunn]

Thank you for providing the prediction file. I was using the config file from the repo (webq_train.jsonnet), and only changed the number of epochs.

[entslscheia]

I do notice a performance discrepancy between the uploaded code and the original code on WebQSP. I'm currently comparing the two codebases to determine if I made any critical changes. Please stay tuned

[entslscheia]

@AiRyunn
Just wanna let you know that I am still looking into this. Have been too occupied with other stuffs for two weeks.
This week I should have more time to investigate it. Based on my current observation, I think it might be related to the use of em_augmentation. I am doing more tests and will post my findings here

[AiRyunn]

Great, thanks for the update. Take your time with the investigation. I've been experimenting with GrailQA and it's working well, but just having problems with WebQSP. Looking forward to your new findings.

[entslscheia]

Hi @AiRyunn,

Now I can confirm that this is related to em_augmentation, which basically means whether to also treat programs that are semantically equivalent to the gold program also as the ground truth. You should specify em_augmentation as False during training. I actually got a F1 of 77.3 with BERT-base, which is pretty close to the original number (77.9). I trained the model for 8 epochs, where 8 is just kinda a casual number picked based on my experience. Earlier, I employed cross-validation to select the optimal number of epochs for the ACL paper. However, given the time constraints, I opted for a more expedient approach this time. Please bear with me.

It's a bit surprising to me that the choice of em_augmentation would lead to such a big difference, and I think I actually used em_augmentation=True for my ACL experiments; intuitively em_augmentation=True should work better than em_augmentation=False. There might be some bug with em_augmentation=True due to some changes after ACL made by me, but I think it should be fine for you to ignore it for now and just use em_augmentation=False.

Number of questions: 1639
Average precision over questions: 0.777
Average recall over questions: 0.817
Average f1 over questions (accuracy): 0.773
F1 of average recall and average precision: 0.797
True accuracy (ratio of questions answered exactly correctly): 0.733

[ri-git]

@AiRyunn @entslscheia how do you train Pangu using webQSP training dataset?
I'm getting error on training using the original webQSP training data file as it doesn't have fields qid, graph_query etc.

[AiRyunn]

@ri-git
Hi, I'm using the training data from this link, which have these fields.

[ri-git]

@AiRyunn thanks for your reply! I'm not able to access the link you shared (Selected user account does not exist in tenant .. and cannot access the application '00000003-0000-0ff1-ce00-000000000000' in that tenant. The account needs to be added as an external user in the tenant first. Please use a different account.)
Could you please share the link with open access?

[AiRyunn]

@ri-git I have updated the link in the previous comment. It should work now.

[ri-git]

@AiRyunn thank you!

[ri-git]

@AiRyunn if possible, could you please share the script you used to generate the graph_query as well?

[AiRyunn]

@ri-git This data was provided by the author and I do not have the script that generated it.

[PrayushiFaldu]

@AiRyunn and @entslscheia can you please share the cache file for webqsp if possible?