Query Refinement Backtranslation

[DelaramRajaei]

This is the issue where I report my progress on the project.

[DelaramRajaei]

@hosseinfani
Initially, I addressed two bugs related to reading and storing the CSV file in the project. To resolve this issue, I replaced the deprecated dataframe.append command with dataframe.concat. Additionally, for reading the CSV file, I examined the format and observed that all entries start with a <top>. Consequently, I implemented the following code to handle this situation:

if '<top>' in line and not is_tag_file:
   is_tag_file = True

If the file type is txt, the tag will be activated.

I also modified the .../qe/main.py file and replaced the .format() function with f-string style.

Subsequently, I incorporated the backtranslation expander exclusively for the French language. You can find the relevant code snippet in the "../qe/expander/backtranslation.py" file. The settings of the backtranslation model and languages are in the "../qe/cmn/param.py" file.

To facilitate result comparison, I have developed the "toy-compare.py" python code, which can be found in the toy file. However, I plan to relocate this file to the "../qe/eval" directory.

There are three available functions that can be used to compare the results:

1. compare_mAP_each_row(): This function compares the mAP between each row, specifically comparing the mAP of the original query with the mAP of the selected column(this column could be a list- for example different languages in backtranslation expander). The results are written to a CSV file.

2. compare_mAP_all_row(): This function calculates the mean of the mAP values for each column and writes the results to a txt file.

3. plot_result(): Although this function is still under development, it is intended to generate plots of the results for a selected dataset, displaying both the original and backtranslation results.

Next, I made further updates to the code to enable it to handle multiple languages and generate queries accordingly. Subsequently, I compared the results using the "toy-compare.py" script.

However, there are still a few remaining bugs in the project:

1. When running the code for multiple languages it will raise an error for the Dutch language for which I have not yet identified the source.

2. After generating the "topics.robust04.bm25.map.all.csv" file, the first query consistently returns a NaN value, and I am currently investigating the cause of this issue.
   

By next Friday, I have outlined the following tasks to be completed:

1. My priority is to address and resolve the existing bugs.
2. I aim to finalize and complete the plot function.
3. I will conduct a thorough comparison of the results obtained and subsequently prepare a comprehensive report on the findings.

[hosseinfani]

@DelaramRajaei
Thank you for the detailed report.

• Please keep this issue page updated with your progress on a 2-3 daily basis.
• Please read our paper on backtranslation for review analysis. It gives you some idea about the roadmap

[DelaramRajaei]

@hosseinfani

The reported bugs have been successfully resolved.

Queries 301 and 672 were returning NaN values. The first bug was identified as an issue in the code and has been rectified. The second bug occurred due to a topic lacking a "qrel" in the ir-dataset for "robust04."
The absence of the "qrel" was mentioned in the accompanying paper. You can refer to the following image for further details:

Additionally, some modifications were made to improve the code. Specifically, the run() function in the main.py file was restructured, and duplicate lines were removed.

Another bug related to the backtranslation feature was identified and resolved. The issue stemmed from the lowercase storage of the model name in the df dataframe in the main.py file's build function. The model name contained the target language, such as 'fra_Latn,' but it was stored in lowercase, causing the bug.

Both bugs are fixed and a pull request has been sent.

[hosseinfani]

@DelaramRajaei
thank you.
pls put a quick comment in the code about the query with no qrels.
Next step will be the help/hirt chart, right?

[DelaramRajaei]

@hosseinfani
I added the comment on the code and push it to my repository. Should I create a new pull request for this comment?

Yes, in the next step I am completeing the plot and report my finding about it.

[hosseinfani]

i don't think so. it automatically accumulate

[DelaramRajaei]

I have updated the code and pushed the new changes.

I fixed the bug of the antique dataset. I changed main.py and abstractqexpander.py . There were some problems in reading and writing new queries in .txt files.

Here are two logs of running the code with backtranslation expander on 5 different languages for robust04 and antique datasets.
log_file_antique.txt
log_file_robust04.txt

[DelaramRajaei]

@hosseinfani
I need the results of other expanders for overall comparison.

[hosseinfani]

@DelaramRajaei
I'm uploading them in the Query Refinement channel at ReQue >> v2.0 >> qe >> output.

[hosseinfani]

@DelaramRajaei
done!

[DelaramRajaei]

@hosseinfani

I have run the program for below datasets and here are the logs of running the code with backtranslation expander on 5 different languages for these datasets.

• robusts04: log_file_robust04.txt
• antique: log_file_antique.txt
• dbpedia: log_file_dbpedia.txt
• gov2: log_file_gov2.txt

Unfortunately, I was unable to download the indexes for the ClueWeb datasets due to their large size.
Could you please share the indexes with me?

I'm currently in the process of drafting the paper and analyzing the results to identify any trends.