Questions of Dr. Chenhao Tan's Talk on "Science in the Age of AI"

[zixuanzzx01]

Science in the Age of AI

As AI becomes increasingly capable of following instructions and conducting analyses, I believe that scientists will increasingly play the role of selector and evaluator. In this talk, I will introduce our recent work in building an ecosystem for the future of AI & Scientific Discovery. I will share our work in AI-enabled research evaluation and hypothesis generation. First, I will present ongoing work that formalizes the evaluation of research outcomes beyond the paper itself and use AI to conduct robust evaluation of research evaluation, with a case study on mechanistic interpretability. Second, rather than treating AI hallucinations as obstacles to eliminate, we leverage data and literature to steer AI creativity toward generating effective hypotheses. I will also introduce HypoBench, a dedicated benchmark for evaluating hypothesis generation, which reveals significant room for potential improvement of current AI models.

Reading List

• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research
• HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation
• Literature Meets Data: A Synergistic Approach to Hypothesis Generation

[BaileyMeche]

How does your framework ensure that integrating literature-based guidance with data-driven updating avoids reinforcing spurious patterns from the data while still improving generalizability?

[dario-marino]

Isn't the AI at this stage better suited for "peer" review and coding rather than writing research from scratch?

[yuzhaojin1]

Do you think AI will mainly accelerate existing scientific paradigms, or can it genuinely help produce paradigm-shifting ideas that human researchers might otherwise miss?

[jstanleyi]

How might integrating literature and data in hypothesis generation change the way social science research is conducted in practice

[ZW-PaulWang]

How does the checklist-based, execution-grounded evaluation avoid penalizing exploratory research, where results may be unstable or not fully generalizable, but still scientifically valuable?

[mikezheng2001]

You mentioned leveraging AI hallucinations to steer creativity toward hypothesis generation. In the context of HypoBench, how do you distinguish between a 'creative but scientifically plausible' hypothesis and one that is simply a 'logical-sounding hallucination' without empirical potential? How do we prevent the AI from optimizing for novelty at the expense of feasibility?

[CallinDai]

In computational social science, where the goal is often to uncover new or alternative understandings of social phenomena (from different dimensions) rather than just confirm existing ones or new discoveries, how should benchmarks like HypoBench be extended to evaluate whether generated hypotheses offer genuinely new theoretical insights, not just sufficient explanations of the data?

[ralphluo11]

Your talk reframes hallucinations as something that can be steered toward hypothesis generation rather than simply eliminated. How should we distinguish between “productive creativity” from AI and misleading speculation, especially in high-stakes scientific fields?

[dorisduowang]

In your execution-grounded evaluation work and in HypoBench, you show that something can sound convincing in language but still fail under real evaluation, so what do you think is the hardest part of building a metric that rewards hypotheses for being truly testable rather than just sounding plausible?

[bkehoechicago]

You say that "rather than treating AI hallucinations as obstacles to eliminate, we leverage data and literature to steer AI creativity toward generating effective hypotheses". How are you going to ensure that the generated hypothesis is itself not assuming incorrect or hallucinated assumptions?

[larissaxcn]

What is one advantage of using AI agents like MechEvalAgent in research evaluation compared to traditional paper-based review systems?

[rrraoyj]

how can we tell whether the patterns discovered by AI models actually reflect meaningful insights, rather than just correlations in the data?

[ben-knell]

How do you think the academic community should distinguish between AI's innovative hypotheses and noise/correlation without real causation moving forward, especially with AI playing an increasingly large role?

[Zhang-snow-shirley]

When leveraging AI hallucinations for hypothesis generation, how do you distinguish between outputs that are genuinely novel and scientifically promising versus those that merely rephrase existing findings in the literature or produce superficially creative but untestable ideas?

[as7391746]

While leveraging AI 'hallucinations' for hypothesis generation fosters creativity, how do we ensure these AI-steered hypotheses remain grounded in physical reality rather than just mimicking linguistically plausible patterns found in existing literature?