Questions for Dr. Yian Yin's Talk on "Scientific Production in the Era of Large Language Models: Early Evidence from Large-scale Preprint Data"

[xqrue-49]

Scientific Production in the Era of Large Language Models: Early Evidence from Large-scale Preprint Data

The rapid adoption of AI across disciplines is reshaping the landscape of scientific production. While both enthusiasm and concern about generative AI in research are rising, systematic empirical evidence on the impact of large language models (LLMs) remains limited. In this talk, I draw on several large-scale analyses to examine how LLM use affects the productivity of individual scientists, reshapes attention to prior work, introduces hallucinated content into the scientific record, and creates new challenges for peer review. Taken together, these findings provide macro-level evidence on the impact of generative AI on science, highlighting the need for institutions, journals, funding agencies, and the broader public to rethink how scientific work should be evaluated in this new era.

Reading List

• Scientific production in the era of large language models

[dario-marino]

Do you have some estimation of the dynamic effect of the llm on past research, namely not only llm on past human research but also llm on past llm to get an idea of the quality degradation

[ZheruiLU]

The paper frames the shift toward citing younger and less-cited works as LLMs helping researchers "overcome cognitive constraints," but couldn't this just as easily reflect LLMs surfacing plausible sounding references that happen to be more obscure precisely because they're less vetted, including possibly hallucinated or low-quality ones? Did the authors check whether the newly discovered citations are actually substantively relevant to the citing papers, rather than just more diverse?

[ZW-PaulWang]

If LLM use both increases scientific productivity and weakens traditional quality signals like writing complexity, how should peer review systems redesign evaluation criteria to reliably distinguish genuine scientific contributions from superficially polished but low quality work?

[carrieeehuang]

I found the reversal in the relationship between writing complexity and publication outcomes especially interesting, and I would love to hear more about whether the authors think reviewers are already adapting to AI polished prose or whether this effect mainly reflects weaker papers being dressed up by LLMs. More broadly, the paper raises an important concern about traditional quality signals breaking down, so I am curious what concrete evaluation reforms the authors think journals and reviewers should prioritize first.

[BaileyMeche]

How does the documented inversion of the relationship between writing complexity and publication success in LLM-assisted manuscripts challenge traditional peer-review heuristics that treat linguistic sophistication as a proxy for scientific quality?

[yifan-lyf]

To what extent does the ability of LLMs affect the influence?

[rrraoyj]

how do you distinguish the llm-assisted writing and llm-generated hallucinations?

[yuzhaojin1]

Do your findings suggest that LLMs increase the quantity of scientific output more than the quality, and if so, how should institutions respond to that imbalance?

[jstanleyi]

If linguistic complexity is no longer a reliable proxy for scientific merit in the era of LLMs, what alternative mechanisms can prevent the proliferation of superficially convincing but low-quality research?

[mikezheng2001]

Your findings show LLMs can introduce hallucinated citations into the scientific record. In fields like finance and economics, where empirical claims often hinge on specific datasets or model specifications, do you see particular risks of LLM-generated content going undetected — and does your data suggest disciplinary variation in hallucination rates?

[MaoYingrong]

Does this measurement only work for early-age LLM detection, since as LLMs advances it's harder and harder to distinguish between AI-generated and human-generated? If so, are there still other approaches to measure LLM usage in large scale in future studies?

[AnqiW02]

Does the LLM-assisted recombination of existing knowledge require us to reexamine culturally divergent conceptions of originality, and if so, whose epistemological frameworks should shape the future of scholarly assessment?

[ralphluo11]

You mentioned that LLMs reshape attention to prior work. How do you distinguish between productive reframing of literature and the risk of reinforcing existing citation biases at scale?

[bkehoechicago]

It looks like you're using the rate at which researchers publish papers a the measure of productivity. How safe of a measure is this, given one of the main fears of the increase in GenAI is an increase in hallucinations in the work?

[larissaxcn]

If LLMs are increasing research productivity but making writing quality less reliable as a signal, how should peer review and academic evaluation adapt to avoid rewarding weak science?