Can Artificial Intelligence Assess Reproducibility? A Proof-of-Concept Study in Ocean Science
2027-01-15 09:15 (China Standard Time)
Session:Session 18 - Oceanography Meets Network Science, Complexity, and AI
Abstract
Reproducibility is fundamental to scientific integrity, yet low levels of reproducibility have been demonstrated in many field of study. In ocean science, reproducibility is complicated by the use of large observational datasets, numerical models, remote sensing products, and complex computational workflows. Evaluating reproducibility at scale requires substantial time and expertise, creating a need for more efficient assessment methods. Recent advances in artificial intelligence (AI), particularly large language models (LLMs), offer new opportunities to automate aspects of reproducibility assessment. This proof-of-concept study investigates whether LLMs can be used to evaluate the computational reproducibility of ocean science publications. We developed a framework in which LLMs analyze published articles and associated reproducibility materials and assign a score using the 10-level Accelerating Computational Reproducibility (ACRe) framework. The ACRe framework evaluates the availability and completeness of data, code, documentation, and other materials required to reproduce published results. A sample of ocean science articles spanning multiple sub-disciplines, including physical oceanography, marine ecology, biogeochemistry, and remote sensing, is assessed. AI-generated scores are compared with human evaluations to determine the extent to which LLMs can reliably identify reproducibility evidence and apply the ACRe scoring framework. Particular attention is given to distinguishing between objective indicators, such as the presence of data repositories and source code, and more nuanced judgments regarding the completeness and usability of reproducibility materials. We hypothesize that LLMs will perform well when identifying explicit reproducibility indicators but less well when assessing whether available materials are sufficient to reproduce published findings. If successful, AI-assisted assessment could provide a scalable approach for screening large numbers of publications. Beyond research assessment, such tools could support the peer-review and publication process by assisting reviewers and editors in evaluating reproducibility, screening submitted manuscripts for compliance with journal requirements, and communicating reproducibility expectations to authors.
Speaker
Author
Important Dates
-
Jan 30
2026
Session / Event proposals open
-
Mar 31
2026
Session / Event proposals close
(Extended to April 7) -
Apr 30
2026
Session acceptance notifications
-
Apr 30
2026
Abstract submissions & registration open
-
Jun 30
2026
Abstract submissions close
(Extended to July 21) -
Aug 31
2026
Abstract acceptance notifications
-
Sep 15
2026
Scientific program released
-
Oct 15
2026
Early bird registration closes
-
Jan 12–15
2027
XMAS 2027
