Tools for AI-assisted Scientific Ideation

Lingyu Zhang, Mitchell Wang, Boyuan Chen

Duke University, General Robotics Lab

Overview

This repo contains tools for AI-assisted scientific ideation derived from our paper Scientific Judgment Drifts Over Time in AI Ideation. The three core features are: 1) scientific idea generation, 2) evaluation, and 3) literature synthesis.

Project Website | Video | Paper

Setup

conda create -n ideation python==3.12
conda activate ideation
pip install -r requirements.txt

The functionalities are LLM-driven and literature-informed. To try all tools, a OpenAI API key and a Semantic Scholar API key are needed. If you only want to generate ideas without evaluating them, the Semantic Scholar API key is optional. Configure them as environment variables:

export SS_API_KEY="xxxxxxxxx"
export OPENAI_API_KEY="sk-XXXXXXX"

Quick Start

1. Generating Ideas

To generate scientific ideas based on existing papers, prepare a recent paper and a highly cited paper as base papers. Name them as recent.pdf and impactful.pdf. Create a new ideation project folder inside ideas/, then copy the two PDF files into a base subfolder inside the project folder. An example of this is already provided in ideas/example. It should have the following structure:

-ideas
  |-project_name
      |-base
          |-impactful.pdf
          |-recent.pdf

Run python -m generate --directory <project_name> --num_ideas 5. To run the example: python -m generate --directory example.

Generated ideas can be found under ideas/<project_name>/generated/.

2. Evaluating an Idea

Scientific ideas, whether AI-generated or summarized from existing papers, can be evaluated with:

python -m evaluate --idea_path ideas/example/generated/0.txt

The final evaluation results can be found in the same path as the idea file, named as <idea_path>_eval.json.

3. Literature Synthesis

One can synthesize relevant literature for an idea, whether AI-generated or summarized from existing papers. For example:

python -m literature --idea_path ideas/example/generated/0.txt

If the idea is known to be extracted from an existing paper, the --is_existing_idea flag can be added to avoid retrieving the paper itself. Add --verbose to output retrieval details. The retrieved relevant papers and their semantic similarity to the given idea can be found under the same path as the idea file, named as <idea_file_name>_literature/literature.json.

Citation

If you find this repo useful, please cite our paper:

@article{zhang2025scientific,
  title={Scientific judgment drifts over time in AI ideation},
  author={Zhang, Lingyu and Wang, Mitchell and Chen, Boyuan},
  journal={arXiv preprint arXiv:2511.04964},
  year={2025}
}

Acknowledgement

This work is supported by DARPA FoundSci program under award HR00112490372.