AMIE RxQA: Multiple-choice question benchmark for medication reasoning

This repository contains data and code corresponding to the OpenFDA-derived multiple choice questions from the RxQA benchmark introduced in “Towards Conversational AI for Disease Management”.

[1] Anil Palepu, Valentin Liévin, Wei-Hung Weng, Khaled Saab, David Stutz, Yong Cheng, Kavita Kulkarni, S. Sara Mahdavi, Joëlle Barral, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Ryutaro Tanno, Vivek Natarajan, Adam Rodman, Tao Tu, Alan Karthikesalingam, Mike Schaekermann Towards Conversational AI for Disease Management. ArXiv, abs/2503.06074.

Overview

AMIE (Articulate Medical Intelligence Explorer) is a medical conversational AI system developed by Google for research purposes. In prior work, “Towards Conversational Diagnostic AI” (https://arxiv.org/pdf/2401.05654), we primarily focused on diagnostic reasoning, while in this follow-up work, “Towards Conversational AI for Disease Management” (https://arxiv.org/abs/2503.06074), we focused primarily on management reasoning.

In the paper, we develop a multiple-choice question benchmark, RxQA, to specifically assess medication reasoning. The questions in this benchmark were initially drafted by Gemini through a detailed process described in the paper, before being validated and revised by expert pharmacists. Question generation was grounded in medication labels from two national drug formularies (US: OpenFDA, UK: British National Formulary (BNF)). However, due to licensing restrictions, we are only open-sourcing the 300 OpenFDA-derived questions at this time.

Data

The RxQA questions with our annotations are available in rxqa_openfda.csv and can easily be loaded using Pandas:

input_file = 'rxqa_openfda.csv'
with open(input_file, 'r') as f:
  df = pd.read_csv(f)
df.head()

The CSV file contains the individual questions as rows, with the following columns:

• An index column;
• q_id: a question id;
• question: the RxQA-OpenFDA question;
• A through D: The answer options;
• correct: The ground truth answer
• correct_idx: The ground truth answer index
• pharmacist_rated_difficulty: The difficulty of the question ({Trivial, Easy, Medium, Hard}) according to one of four US-based pharmacists.

Details on RxQA and the generation method can be found in the paper, Appendix A.14 and Appendix A.15.

Limitations

Several limitations of this benchmark are described in the manuscript. Notably, each question was revised by a single pharmacist, disregarding the potential for inter-pharmacist variability. Additionally, the filtering process selects for questions Gemini originally fails to answer correctly, meaning the benchmark may not properly assess medication reasoning for easier, potentially more common questions.

Citing this work

When using any part of this repository, make sure to cite the paper as follows:

@article{palepu2025towards,
  title={Towards Conversational AI for Disease Management},
  author={Anil Palepu and Valentin Liévin and Wei-Hung Weng and Khaled Saab and David Stutz and Yong Cheng and Kavita Kulkarni and S. Sara Mahdavi and Joëlle Barral and Dale R. Webster and Katherine Chou and Avinatan Hassidim and Yossi Matias and James Manyika and Ryutaro Tanno and Vivek Natarajan and Adam Rodman and Tao Tu and Alan Karthikesalingam and Mike Schaekermann},
  journal={arXiv preprint arXiv:2503.06074},
  year={2025}
}

License and disclaimer

All software is licensed under the Apache License, Version 2.0 (Apache 2.0); you may not use this file except in compliance with the Apache 2.0 license. You may obtain a copy of the Apache 2.0 license at: https://www.apache.org/licenses/LICENSE-2.0

The provided annotations are licensed under the Creative Commons Attribution 4.0 International License (CC-BY). You may obtain a copy of the CC-BY license at: https://creativecommons.org/licenses/by/4.0/legalcode

Unless required by applicable law or agreed to in writing, all software and materials distributed here under the Apache 2.0 or CC-BY licenses are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the licenses for the specific language governing permissions and limitations under those licenses.

This is not an official Google product.