This is a pratical assignmnet for Web Data Processing Systems course.
- Receiving a question and get the answer from a large language model(LLM).
- Extracting clean answer and identifying the correctness of answer given by LLM.
- Extracting entities from question and answer, disambiguateing them and giving out their corresponding wiki links.
You are called to implement a program that receives as input a question (or more in general a text to be completed), which we henceforth call A and returns as output four things:
- The raw text returned by a large language model (that is, what you would get if you query the language model as is). We call it B
- The answer extracted from B. This answer can be of two types: either yes/no or a Wikipedia entity
- The correctness of the extracted answer (correct/incorrect)
- Entities that have been extracted from A and B
[Input (A)]: "is Managua the capital of Nicaragua?"
[Text returned by the language model (B) (llama 2, 70B)]: "Yes, Managua is the capital city of Nicaragua. It is located in the southwestern part of the country and is home to many important government buildings and institutions, including the President's office and the National Assembly. The city has a population of over one million people and is known for its vibrant cultural scene, historic landmarks, and beautiful natural surroundings."
[Extracted answer]: "yes"
[Correctness of the answer]: "correct"
[Entities extracted]:
Nicaraguahttps://en.wikipedia.org/wiki/Nicaragua
Managuahttps://en.wikipedia.org/wiki/Managua
President's officehttps://en.wikipedia.org/wiki/President_of_Nicaragua
National Assemblyhttps://en.wikipedia.org/wiki/National_Assembly_(Nicaragua)
python -r requirements.txt
using -v to show the running information, output is in the test/output.txt if not specified
python starter.py -i question.txt [output.txt] -v