{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T03:40:57Z","timestamp":1788579657039,"version":"build-2803163510"},"reference-count":30,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T00:00:00Z","timestamp":1770940800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Research Council","award":["951393"],"award-info":[{"award-number":["951393"]}]},{"DOI":"10.13039\/501100004795","name":"Institut Universitaire de France","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004795","id-type":"DOI","asserted-by":"crossref"}]},{"name":"European Research Council","award":["951393"],"award-info":[{"award-number":["951393"]}]},{"name":"National Library of Medicine of the National Institutes of Health","award":["R01LM014079"],"award-info":[{"award-number":["R01LM014079"]}]},{"name":"Wellcome Trust Early-Career Award","award":["306380\/Z\/23\/Z"],"award-info":[{"award-number":["306380\/Z\/23\/Z"]}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["GM144308"],"award-info":[{"award-number":["GM144308"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Chan Zuckerberg Initiative DAF","award":["2022- 250218"],"award-info":[{"award-number":["2022- 250218"]}]},{"DOI":"10.13039\/100000923","name":"Silicon Valley Community Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>\n                    The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/PeterEckmann1\/tool-comparison\" xlink:type=\"simple\">https:\/\/github.com\/PeterEckmann1\/tool-comparison<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1371\/journal.pone.0342225","type":"journal-article","created":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T18:40:34Z","timestamp":1771008034000},"page":"e0342225","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Use as directed? A comparison of software tools intended to check rigor and transparency of published work"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5388-9451","authenticated-orcid":true,"given":"Peter","family":"Eckmann","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian","family":"Barnett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3161-1395","authenticated-orcid":true,"given":"Alexandra","family":"Bannach-Brown","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elisa Pilar","family":"Bascunan Atria","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillaume","family":"Cabanac","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Louise Delwen","family":"Owen Franzen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8369-1350","authenticated-orcid":true,"given":"Ma\u0142gorzata Anna","family":"Gazda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0180-7343","authenticated-orcid":true,"given":"Kaitlyn","family":"Hair","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5702-149X","authenticated-orcid":true,"given":"James","family":"Howison","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Halil","family":"Kilicoglu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cyril","family":"Labbe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"McCann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vladislav","family":"Nachev","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martijn","family":"Roelandse","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maia","family":"Salholz-Hillel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4830-309X","authenticated-orcid":true,"given":"Robert","family":"Schulz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2231-7637","authenticated-orcid":true,"given":"Gerben","family":"ter Riet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Colby","family":"Vorland","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anita","family":"Bandrowski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7490-2600","authenticated-orcid":true,"given":"Tracey","family":"Weissgerber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,2,13]]},"reference":[{"issue":"7604","key":"pone.0342225.ref001","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"M Baker","year":"2016","journal-title":"Nature."},{"issue":"1","key":"pone.0342225.ref002","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1186\/s13104-021-05875-3","article-title":"Reproducibility and research integrity: the role of scientists and institutions","volume":"14","author":"P Diaba-Nuhoho","year":"2021","journal-title":"BMC Res Notes."},{"issue":"9","key":"pone.0342225.ref003","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1038\/s41562-018-0399-z","article-title":"Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015","volume":"2","author":"CF Camerer","year":"2018","journal-title":"Nat Hum Behav."},{"key":"pone.0342225.ref004","article-title":"Challenges for assessing replicability in preclinical cancer biology","volume":"10","author":"TM Errington","year":"2021","journal-title":"Elife."},{"issue":"1","key":"pone.0342225.ref005","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1038\/s44271-023-00003-2","article-title":"The replication crisis has led to positive structural, procedural, and community changes","volume":"1","author":"M Korbmacher","year":"2023","journal-title":"Commun Psychol."},{"issue":"10489","key":"pone.0342225.ref006","doi-asserted-by":"crossref","first-page":"1633","DOI":"10.1016\/S0140-6736(25)00672-5","article-title":"CONSORT 2025 statement: updated guideline for reporting randomised trials","volume":"405","author":"S Hopewell","year":"2025","journal-title":"The Lancet."},{"issue":"9","key":"pone.0342225.ref007","doi-asserted-by":"crossref","first-page":"1769","DOI":"10.1177\/0271678X20943823","article-title":"The ARRIVE guidelines 2.0: updated guidelines for reporting animal research","volume":"40","author":"N Percie du Sert","year":"2020","journal-title":"J Cereb Blood Flow Metab."},{"key":"pone.0342225.ref008","unstructured":"National Institutes of Health. Guidance: Rigor and Reproducibility in Grant Applications. NIH Grants & Funding. 2024. https:\/\/grants.nih.gov\/policy\/reproducibility\/guidance.htm"},{"issue":"5","key":"pone.0342225.ref009","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0197882","article-title":"ARRIVE has not ARRIVEd: Support for the ARRIVE (Animal Research: Reporting of in vivo Experiments) guidelines does not improve the reporting quality of papers in animal welfare, analgesia or anesthesia","volume":"13","author":"V Leung","year":"2018","journal-title":"PLoS One."},{"issue":"5","key":"pone.0342225.ref010","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2018-026589","article-title":"Scoping review on interventions to improve adherence to reporting guidelines in health research","volume":"9","author":"D Blanco","year":"2019","journal-title":"BMJ Open."},{"key":"pone.0342225.ref011","doi-asserted-by":"crossref","first-page":"1276407","DOI":"10.3389\/fninf.2023.1276407","article-title":"The past, present and future of neuroscience data sharing: a perspective on the state of practices and infrastructure for FAIR","volume":"17","author":"ME Martone","year":"2024","journal-title":"Front Neuroinform."},{"key":"pone.0342225.ref012","article-title":"Six factors affecting reproducibility in life science research and how to handle them","author":"P Nature","year":"2019","journal-title":"Nat Rev Methods Primers."},{"issue":"1","key":"pone.0342225.ref013","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1007\/s11192-018-2969-2","article-title":"The ability of different peer review procedures to flag problematic publications","volume":"118","author":"SPJM Horbach","year":"2019","journal-title":"Scientometrics."},{"key":"pone.0342225.ref014","unstructured":"Oransky I. Nearing 5, 000 retractions: a review of 2022 . Retraction Watch. 2022. https:\/\/retractionwatch.com\/2022\/12\/27\/nearing-5000-retractions-a-review-of-2022\/"},{"issue":"1","key":"pone.0342225.ref015","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1186\/s13104-022-06080-6","article-title":"Is the future of peer review automated?","volume":"15","author":"R Schulz","year":"2022","journal-title":"BMC Research Notes."},{"key":"pone.0342225.ref016","unstructured":"van Rossum J. Guest post: The STM integrity hub \u2014 connecting the dots in a dynamic landscape. The Scholarly Kitchen. 2024. https:\/\/scholarlykitchen.sspnet.org\/2024\/05\/23\/guest-post-the-research-integrity-hub-connecting-the-dots-in-a-dynamic-landscape\/"},{"issue":"6","key":"pone.0342225.ref017","doi-asserted-by":"crossref","DOI":"10.2196\/37324","article-title":"Establishing institutional scores with the rigor and transparency index: large-scale analysis of scientific reporting quality","volume":"24","author":"J Menke","year":"2022","journal-title":"J Med Internet Res."},{"key":"pone.0342225.ref018","article-title":"ODDPub\u2013a text-mining algorithm to detect data sharing in biomedical publications","author":"N Riedel","year":"2020","journal-title":"BioRxiv."},{"issue":"3","key":"pone.0342225.ref019","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1002\/jrsm.1533","article-title":"Risk of bias assessment in preclinical literature using natural language processing","volume":"13","author":"Q Wang","year":"2022","journal-title":"Res Synth Methods."},{"key":"pone.0342225.ref020","doi-asserted-by":"crossref","first-page":"103717","DOI":"10.1016\/j.jbi.2021.103717","article-title":"Toward assessing clinical trial publications for reporting transparency","volume":"116","author":"H Kilicoglu","year":"2021","journal-title":"J Biomed Inform."},{"key":"pone.0342225.ref021","article-title":"Reproducibility in machine learning-driven research","author":"H Semmelrock","year":"2023","journal-title":"arXiv preprint"},{"key":"pone.0342225.ref022","doi-asserted-by":"crossref","unstructured":"Ferrari Dacrema M, Cremonesi P, Jannach D. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In: Proceedings of the 13th ACM Conference on Recommender Systems. 2019. p. 101\u20139. https:\/\/doi.org\/10.1145\/3298689.3347058","DOI":"10.1145\/3298689.3347058"},{"issue":"1","key":"pone.0342225.ref023","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1038\/s41591-020-01203-7","article-title":"Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?","volume":"27","author":"T Weissgerber","year":"2021","journal-title":"Nat Med."},{"issue":"7419","key":"pone.0342225.ref024","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/nature11556","article-title":"A call for transparent reporting to optimize the predictive value of preclinical research","volume":"490","author":"SC Landis","year":"2012","journal-title":"Nature."},{"key":"pone.0342225.ref025","unstructured":"eLife. Materials Design Analysis Reporting (MDAR) Checklist for Authors. https:\/\/cdn.elifesciences.org\/articles\/81727\/elife-81727-mdarchecklist1-v3.pdf"},{"key":"pone.0342225.ref026","doi-asserted-by":"crossref","unstructured":"Cleverdon CW. The ASLIB cranfield research project on the comparative efficiency of indexing systems. Aslib Proceedings. 1960;12(12):421\u201331. https:\/\/doi.org\/10.1108\/eb049778","DOI":"10.1108\/eb049778"},{"issue":"11","key":"pone.0342225.ref027","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1145\/1297797.1297822","article-title":"TREC: Continuing information retrieval\u2019s tradition of experimentation","volume":"50","author":"EM Voorhees","year":"2007","journal-title":"Communications of the ACM."},{"issue":"12","key":"pone.0342225.ref028","doi-asserted-by":"crossref","first-page":"1938","DOI":"10.1177\/00220345920710121601","article-title":"Robustness of the t test applied to data distorted from normality by floor effects","volume":"71","author":"LM Sullivan","year":"1992","journal-title":"Journal of Dental Research."},{"key":"pone.0342225.ref029","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples","volume":"13","author":"N Wongpakaran","year":"2013","journal-title":"BMC Med Res Methodol."},{"key":"pone.0342225.ref030","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019. p. 4171\u201386."}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0342225","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T18:40:41Z","timestamp":1771008041000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0342225"}},"subtitle":[],"editor":[{"given":"Jinran","family":"Wu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,2,13]]},"references-count":30,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,2,13]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0342225","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,13]]}}}