{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T17:34:03Z","timestamp":1793381643621,"version":"4.5.0"},"reference-count":29,"publisher":"CERN","license":[{"start":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T00:00:00Z","timestamp":1788739200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Commission","award":["101094725"],"award-info":[{"award-number":["101094725"]}],"id":[{"id":"https:\/\/ror.org\/00k4n6c32","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Open Res Europe"],"abstract":"<jats:p>Background\n                    Open science practices are increasingly promoted to improve the transparency and reproducibility of biomedical research, and their large-scale monitoring has been identified as a priority by initiatives such as the UNESCO Recommendation on Open Science. Existing automated tools cover only narrow subsets of practices. We developed and validated an automated tool, based on locally run open-weights large language models (LLMs), to extract a core set of 13 open science practices from published medical research articles.\n                \n                \n                    Methods\n                    We built a validation database of 303 research articles published in 2020\u20132021 in 10 major general medical journals, stratified across randomized controlled trials (RCTs), meta-analyses, and other designs. Duplicate manual extraction with expert adjudication served as the reference standard. The pipeline combined document conversion (GROBID), BM25-based chunk retrieval, and classification by three Llama models (3-8B, 3-70B, 3.3-70B) run entirely locally, with and without supplementary materials. Diagnostic performance (sensitivity, specificity, predictive values, F1) was computed against the reference standard.\n                \n                \n                    Results\n                    Usable outputs were obtained for 301\/303 articles. Performance was heterogeneous across criteria rather than across models: practices reported in standardized statements (conflicts of interest, funding, author contributions, data sharing, pre-registration, reporting guidelines) were extracted reliably (F1\u00a0\u2265\u00a00.75 for at least one model), whereas rare or inconsistently reported practices (code sharing, statistical analysis plan, protocol sharing, ORCID, open access status, preprints) remained difficult regardless of model size. Study-type classification exceeded 0.90 on all metrics. Larger models improved several criteria at a 6.6\u20137.5\u00d7 computational cost, and adding supplementary materials did not improve performance, mainly due to conversion failures.\n                \n                \n                    Conclusions\n                    A fully local, open-weights LLM pipeline can reliably monitor a substantial subset of open science practices. Document conversion and layout heterogeneity, rather than language understanding, constitute the main barrier to scaling automated open science monitoring.<\/jats:p>","DOI":"10.12688\/openreseurope.24615.1","type":"journal-article","created":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T11:34:05Z","timestamp":1788780845000},"update-policy":"https:\/\/doi.org\/10.68336\/openreseurope.updatepolicy","source":"Crossref","is-referenced-by-count":0,"title":["Core open science practices in major medical journals: development of an automatized tool based on a Large Language Model"],"prefix":"10.68336","author":[{"given":"Constant","family":"Vinatier","sequence":"first","affiliation":[{"name":"Univ Rennes, CHU Rennes, Inserm, EHESP, Irset (Institut de recherche en sant\u00e9, environnement et travail) \u2013 UMR_S 1085, Rennes, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2262-0974","authenticated-orcid":true,"given":"Guillaume","family":"Freyermuth","sequence":"additional","affiliation":[{"name":"Univ Rennes, IRISA, Inria, CNRS, F-35000 Rennes, France, Rennes, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margaux","family":"Millour","sequence":"additional","affiliation":[{"name":"Univ Rennes, IRISA, Inria, CNRS, F-35000 Rennes, France, Rennes, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gauthieu","family":"Le Bartz Lyan","sequence":"additional","affiliation":[{"name":"Univ Rennes, IRISA, Inria, CNRS, F-35000 Rennes, France, Rennes, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8023-3210","authenticated-orcid":true,"given":"Sarah","family":"Buet","sequence":"additional","affiliation":[{"name":"University of Rennes Faculty of Medicine, Rennes, Brittany, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Perrine","family":"Lunel","sequence":"additional","affiliation":[{"name":"University of Rennes Faculty of Medicine, Rennes, Brittany, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2175-986X","authenticated-orcid":true,"given":"Nicolas","family":"Robillard","sequence":"additional","affiliation":[{"name":"University of Rennes Faculty of Medicine, Rennes, Brittany, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3760-3801","authenticated-orcid":true,"given":"Florian","family":"Naudet","sequence":"additional","affiliation":[{"name":"Univ Rennes, CHU Rennes, Inserm, EHESP, Irset (Institut de recherche en sant\u00e9, environnement et travail) \u2013 UMR_S 1085, Rennes, France"},{"name":"Institut Universitaire de France, Paris, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mathieu","family":"Acher","sequence":"additional","affiliation":[{"name":"Univ Rennes, IRISA, Inria, CNRS, F-35000 Rennes, France, Rennes, France"},{"name":"Institut Universitaire de France, Paris, France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"13904","published-online":{"date-parts":[[2026,9,7]]},"reference":[{"key":"120447","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"120448","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"120449","doi-asserted-by":"publisher","DOI":"10.1126\/science.aac4716"},{"key":"120450","doi-asserted-by":"publisher","DOI":"10.1027\/1864-9335\/a000178"},{"key":"120451","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-026-10251-x"},{"key":"120452","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-016-0021"},{"key":"120453","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000737"},{"key":"120454","doi-asserted-by":"publisher","DOI":"10.52949\/32"},{"key":"120455","doi-asserted-by":"publisher","DOI":"10.5281\/ZENODO.15807480"},{"key":"120456","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3001949"},{"key":"120457","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2411.03417"},{"key":"120458","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3001107"},{"key":"120459","doi-asserted-by":"publisher","DOI":"10.5334\/dsj-2020-042"},{"key":"120460","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2026.32070"},{"key":"120461","doi-asserted-by":"publisher","DOI":"10.17605\/OSF.IO\/T4RXC"},{"key":"120462","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03425-5"},{"key":"120463","unstructured":"17. GitLab: Mr_OASO \u00b7 GitLab. 2026. Reference Source (https:\/\/gitlab.inria.fr\/glebartz\/mr_oaso)"},{"key":"120464","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2016.01.021"},{"key":"120465","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04346-8_62"},{"key":"120466","doi-asserted-by":"publisher","DOI":"10.1145\/2682862.2682863"},{"key":"120467","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-024-02203-8"},{"key":"120468","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-015-0668-z"},{"key":"120469","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2511.16134"},{"key":"120470","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-017-0431-4"},{"key":"120471","doi-asserted-by":"publisher","DOI":"10.1002\/cesm.70036"},{"key":"120472","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2505.09598"},{"key":"120473","doi-asserted-by":"publisher","DOI":"10.1016\/j.joule.2023.09.004"},{"key":"120474","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-018-05968-3"},{"key":"120475","doi-asserted-by":"publisher","DOI":"10.5281\/ZENODO.21704471"}],"container-title":["Open Research Europe"],"original-title":[],"link":[{"URL":"https:\/\/open-research-europe.ec.europa.eu\/articles\/6-326\/v1\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ore.eu\/ore\/article\/download\/1521\/version\/2491\/4979","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ore.eu\/ore\/article\/download\/1521\/version\/2491\/4980","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ore.eu\/ore\/article\/download\/1521\/version\/2491\/4979","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T17:11:44Z","timestamp":1793380304000},"score":1,"resource":{"primary":{"URL":"https:\/\/ore.eu\/ore\/article\/view\/1521\/version\/2491"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,7]]},"references-count":29,"URL":"https:\/\/doi.org\/10.12688\/openreseurope.24615.1","relation":{"has-review":[{"id-type":"doi","id":"10.21956\/openreseurope.26672.r80669","asserted-by":"object"},{"id-type":"doi","id":"10.21956\/openreseurope.26672.r80670","asserted-by":"object"},{"id-type":"doi","id":"10.21956\/openreseurope.26672.r80671","asserted-by":"object"}]},"ISSN":["2732-5121"],"issn-type":[{"value":"2732-5121","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,7]]},"assertion":[{"value":"2026-09-07","name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}},{"value":"This project has received funding from the European Union\u2019s Horizon Europe research and innovation programme under grant agreement No [101094725. (CV, and FN are members of this project.).\n\n                    The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","name":"funding_statement","label":"Funding Statement","group":{"name":"funding_information","label":"Funding Information"}}],"article-number":"326"}}