The Xiamen Symposium on Marine Environmental Sciences

2027 Xiamen Symposium on Marine Environmental Sciences

January 12-15, 2027

Can Artificial Intelligence Assess Reproducibility? A Proof-of-Concept Study in Ocean Science
ID:1618 Oral Presentation

2027-01-15 09:15 (China Standard Time)

Session:Session 18 - Oceanography Meets Network Science, Complexity, and AI

Abstract

Reproducibility is fundamental to scientific integrity, yet low levels of reproducibility have been demonstrated in many field of study. In ocean science, reproducibility is complicated by the use of large observational datasets, numerical models, remote sensing products, and complex computational workflows. Evaluating reproducibility at scale requires substantial time and expertise, creating a need for more efficient assessment methods. Recent advances in artificial intelligence (AI), particularly large language models (LLMs), offer new opportunities to automate aspects of reproducibility assessment. This proof-of-concept study investigates whether LLMs can be used to evaluate the computational reproducibility of ocean science publications. We developed a framework in which LLMs analyze published articles and associated reproducibility materials and assign a score using the 10-level Accelerating Computational Reproducibility (ACRe) framework. The ACRe framework evaluates the availability and completeness of data, code, documentation, and other materials required to reproduce published results. A sample of ocean science articles spanning multiple sub-disciplines, including physical oceanography, marine ecology, biogeochemistry, and remote sensing, is assessed. AI-generated scores are compared with human evaluations to determine the extent to which LLMs can reliably identify reproducibility evidence and apply the ACRe scoring framework. Particular attention is given to distinguishing between objective indicators, such as the presence of data repositories and source code, and more nuanced judgments regarding the completeness and usability of reproducibility materials. We hypothesize that LLMs will perform well when identifying explicit reproducibility indicators but less well when assessing whether available materials are sufficient to reproduce published findings. If successful, AI-assisted assessment could provide a scalable approach for screening large numbers of publications. Beyond research assessment, such tools could support the peer-review and publication process by assisting reviewers and editors in evaluating reproducibility, screening submitted manuscripts for compliance with journal requirements, and communicating reproducibility expectations to authors.

Speaker
Anders Knudby University of Ottawa

Author
Anders Knudby University of Ottawa

    Important Dates

    • Jan 30

      2026

      Session / Event proposals open

    • Mar 31

      2026

      Session / Event proposals close
      (Extended to April 7)

    • Apr 30

      2026

      Session acceptance notifications

    • Apr 30

      2026

      Abstract submissions & registration open

    • Jun 30

      2026

      Abstract submissions close
      (Extended to July 21)

    • Aug 31

      2026

      Abstract acceptance notifications

    • Sep 15

      2026

      Scientific program released

    • Oct 15

      2026

      Early bird registration closes

    • Jan 12–15

      2027

      XMAS 2027