Evaluate Abstractive Question-Answering models

[FrancescoCasalegno]

Context

• In Question-Answering: collect example questions + run first analysis with pre-trained QA models #612 we created a first version of our dataset for Question-Answering. The "ground-truth" answers of this dataset were annotated by selecting a span from the context, so the problem was naturally interpreted as "Extractive Question-Answering" task.
• Again, in Question-Answering: collect example questions + run first analysis with pre-trained QA models #612 we compared the performance of different models for Extractive Question-Answering".
• An alternative is "Abstractive Question-Answering", a.k.a. "Generative Question-Answering". An advantage of this approach is that it does not pose any constraint on the contiguity of the answer in the context, but a disadvantage could be that the predicted answer could be non-sense text.

Actions

• Investigate which Abstractive Question-Answering models could be used on our problem.
• Investigate which metrics we should use to evaluate results: typically EM and F1 are used for QA tasks, but since these are generative models maybe BLEU score could be used?
• Evaluate models and analyse the results.