{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T03:48:52Z","timestamp":1791431332812,"version":"4.3.3"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T00:00:00Z","timestamp":1774137600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T00:00:00Z","timestamp":1778112000000},"content-version":"vor","delay-in-days":46,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Innovative Health Initative Joint Undertaking","award":["101132847"],"award-info":[{"award-number":["101132847"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Systematic reviews are essential for evidence-based research, yet the traditional screening process is time-consuming and difficult to scale. Human-only screening can introduce inconsistency, while fully automated approaches employing Large Language Models often lack the contextual judgement required for complex decisions. To address this, we introduce a crowd-based screening methodology that integrates human expertise with adaptive machine learning. The methods have been applied in the context of a large EU project where experts from 27 collaborating partners jointly screened 5842 papers across eleven disease topics related to patient-generated health data in a span of 2 days. Post-processing played a central role in ensuring data quality, including topic reallocation, targeted full-text verification, and noisy-label filtering. This Screenathon resulted in 487 records being labeled as relevant and 6,463 records as irrelevant. The number of records screened per participant ranged from 3 to 2496, with a mean of 216.4 records per screener (\n                    <jats:italic>SE<\/jats:italic>\n                    \u2009=\u200995.19). Exploratory analyses using survey results indicated increased trust in AI-assisted systematic reviewing after the event, along with generally positive evaluations of usability. The current Screenathon demonstrates that crowdsourced human\u2013AI collaboration requires thoughtful training and calibration, together with strong post-processing safeguards.\n                  <\/jats:p>","DOI":"10.1038\/s41598-026-45385-5","type":"journal-article","created":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T03:54:41Z","timestamp":1774151681000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Screenathon 2.0: human\u2013AI collaborative screening applied to patient-generated health data"],"prefix":"10.1038","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-9881-7893","authenticated-orcid":false,"given":"Jonas","family":"Bergmann","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9314-1067","authenticated-orcid":false,"given":"Tiago","family":"Azzi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3824-8727","authenticated-orcid":false,"given":"Rutger","family":"Neeleman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3368-2457","authenticated-orcid":false,"given":"Kianush","family":"Monschau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2610-9354","authenticated-orcid":false,"given":"Berke","family":"Yazan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7315-5494","authenticated-orcid":false,"given":"Elena","family":"Jalsovec","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-6664-8833","authenticated-orcid":false,"given":"Emily","family":"Westerbeek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5150-1102","authenticated-orcid":false,"given":"Felix","family":"Weijdema","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4297-0502","authenticated-orcid":false,"given":"Jonathan","family":"de Bruin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2689-6653","authenticated-orcid":false,"given":"Qixiang","family":"Fang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7736-2091","authenticated-orcid":false,"given":"Rens","family":"van de Schoot","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,3,22]]},"reference":[{"key":"45385_CR1","doi-asserted-by":"crossref","unstructured":"Bergmann, J. et al. Materials & Data: Screenathon 2.0: Human\u2013AI Collaborative Screening Applied to Patient-Generated Health Data. (2025). https:\/\/osf.io\/hyuxw","DOI":"10.31234\/osf.io\/e2qfj_v1"},{"key":"45385_CR2","doi-asserted-by":"publisher","unstructured":"Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ n71 (2021). https:\/\/doi.org\/10.1136\/bmj.n71","DOI":"10.1136\/bmj.n71"},{"key":"45385_CR3","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1057\/s41599-021-00903-w","volume":"8","author":"L Bornmann","year":"2021","unstructured":"Bornmann, L., Haunschild, R. & Mutz, R. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanit. Soc. Sci. Commun. 8, 224 (2021).","journal-title":"Humanit. Soc. Sci. Commun."},{"key":"45385_CR4","doi-asserted-by":"publisher","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","volume":"7","author":"R Borah","year":"2017","unstructured":"Borah, R., Brown, A. W., Capers, P. L. & Kaiser, K. A. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 7, e012545 (2017).","journal-title":"BMJ Open."},{"key":"45385_CR5","doi-asserted-by":"publisher","first-page":"e0227742","DOI":"10.1371\/journal.pone.0227742","volume":"15","author":"Z Wang","year":"2020","unstructured":"Wang, Z., Nayfeh, T., Tetzlaff, J., O\u2019Blenis, P. & Murad, M. H. Error rates of human reviewers during abstract screening in systematic reviews. PLOS ONE. 15, e0227742 (2020).","journal-title":"PLOS ONE"},{"key":"45385_CR6","doi-asserted-by":"publisher","unstructured":"Monschau, K. et al. Introducing the screenathon review: crowdsourcing literature screening in large-scale research collaborations. preprint at (2024). https:\/\/doi.org\/10.31234\/osf.io\/vxg6d","DOI":"10.31234\/osf.io\/vxg6d"},{"key":"45385_CR7","doi-asserted-by":"publisher","first-page":"672","DOI":"10.1016\/j.csbj.2024.10.026","volume":"24","author":"F Folkvord","year":"2024","unstructured":"Folkvord, F. et al. Using patient-generated health data more efficient and effectively to facilitate the implementation of value-based healthcare in the EU \u2013 Innovation report. Comput. Struct. Biotechnol. J. 24, 672\u2013678 (2024).","journal-title":"Comput. Struct. Biotechnol. J."},{"key":"45385_CR8","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1186\/s12874-021-01451-2","volume":"21","author":"C Hamel","year":"2021","unstructured":"Hamel, C. et al. Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses. BMC Med. Res. Methodol. 21, 285 (2021).","journal-title":"BMC Med. Res. Methodol."},{"key":"45385_CR9","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","volume":"3","author":"R Van De Schoot","year":"2021","unstructured":"Van De Schoot, R. et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat. Mach. Intell. 3, 125\u2013133 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"45385_CR10","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1186\/s13643-023-02257-7","volume":"12","author":"G Ferdinands","year":"2023","unstructured":"Ferdinands, G. et al. Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. Syst. Rev. 12, 100 (2023).","journal-title":"Syst. Rev."},{"key":"45385_CR11","doi-asserted-by":"publisher","first-page":"2293","DOI":"10.1038\/s41562-024-02024-1","volume":"8","author":"M Vaccaro","year":"2024","unstructured":"Vaccaro, M., Almaatouq, A. & Malone, T. When combinations of humans and AI are useful: a systematic review and meta-analysis. Nat. Hum. Behav. 8, 2293\u20132303 (2024).","journal-title":"Nat. Hum. Behav."},{"key":"45385_CR12","doi-asserted-by":"publisher","first-page":"113559","DOI":"10.1016\/j.isci.2025.113559","volume":"28","author":"M Moens","year":"2025","unstructured":"Moens, M., Nagels, G., Wake, N. & Goudman, L. Artificial intelligence as team member versus manual screening to conduct systematic reviews in medical sciences. iScience 28, 113559 (2025).","journal-title":"iScience"},{"key":"45385_CR13","doi-asserted-by":"publisher","first-page":"2546214","DOI":"10.1080\/20008066.2025.2546214","volume":"16","author":"R Van De Schoot","year":"2025","unstructured":"Van De Schoot, R. et al. The hunt for the last relevant paper: blending the best of humans and AI. Eur. J. Psychotraumatology. 16, 2546214 (2025).","journal-title":"Eur. J. Psychotraumatology"},{"key":"45385_CR14","doi-asserted-by":"publisher","unstructured":"Teijema, J. J. et al. Simulation-based active learning for systematic reviews: a scoping review of literature. Preprint at (2023). https:\/\/doi.org\/10.31234\/osf.io\/67zmt","DOI":"10.31234\/osf.io\/67zmt"},{"key":"45385_CR15","doi-asserted-by":"publisher","first-page":"101318","DOI":"10.1016\/j.patter.2025.101318","volume":"6","author":"J De Bruin","year":"2025","unstructured":"De Bruin, J. et al. ASReview LAB v.2: open-source text screening with multiple agents and a crowd of experts. Patterns 6, 101318 (2025).","journal-title":"Patterns"},{"key":"45385_CR16","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/s13643-024-02472-w","volume":"13","author":"R Neeleman","year":"2024","unstructured":"Neeleman, R., Leenaars, C. H. C., Oud, M., Weijdema, F. & Van De Schoot, R. Addressing the challenges of reconstructing systematic reviews datasets: a case study and a noisy label filter procedure. Syst. Rev. 13, 69 (2024).","journal-title":"Syst. Rev."},{"key":"45385_CR17","doi-asserted-by":"publisher","unstructured":"Researcher-in-the-Loop for Systematic Reviewing of Text Databases. Zenodo, (2020). https:\/\/doi.org\/10.5281\/zenodo.4013207","DOI":"10.5281\/zenodo.4013207"},{"key":"45385_CR18","first-page":"1","volume":"4","author":"PRISMAP Group","year":"2015","unstructured":"Group, P. R. I. S. M. A. P. et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4, 1 (2015).","journal-title":"Syst. Rev."},{"key":"45385_CR19","doi-asserted-by":"crossref","unstructured":"JASP Team. JASP (Version 0.19.1) [Computer software]. JASP - Free and User-Friendly Statistical Software (2024). https:\/\/jasp-stats.org\/","DOI":"10.4324\/9781003025405-1"},{"key":"45385_CR20","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing [Computer Software]. (2025). https:\/\/www.r-project.org\/"},{"key":"45385_CR21","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1186\/s12874-021-01271-4","volume":"21","author":"AH Noel-Storr","year":"2021","unstructured":"Noel-Storr, A. H. et al. Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study. BMC Med. Res. Methodol. 21, 88 (2021).","journal-title":"BMC Med. Res. Methodol."},{"key":"45385_CR22","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1002\/jrsm.1354","volume":"10","author":"JR Polanin","year":"2019","unstructured":"Polanin, J. R., Pigott, T. D., Espelage, D. L. & Grotpeter, J. K. Best practice guidelines for abstract screening large-evidence systematic reviews and meta\u2010analyses. Res. Synth. Methods. 10, 330\u2013342 (2019).","journal-title":"Res. Synth. Methods"},{"key":"45385_CR23","doi-asserted-by":"publisher","first-page":"14952","DOI":"10.1038\/s41598-022-18751-2","volume":"12","author":"C Reverberi","year":"2022","unstructured":"Reverberi, C. et al. Experimental evidence of effective human\u2013AI collaboration in medical decision-making. Sci. Rep. 12, 14952 (2022).","journal-title":"Sci. Rep."},{"key":"45385_CR24","doi-asserted-by":"publisher","first-page":"31150","DOI":"10.1038\/s41598-024-82501-9","volume":"14","author":"J Senoner","year":"2024","unstructured":"Senoner, J. et al. Explainable AI improves task performance in human\u2013AI collaboration. Sci. Rep. 14, 31150 (2024).","journal-title":"Sci. Rep."},{"key":"45385_CR25","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1186\/s13731-025-00563-z","volume":"14","author":"L Peschke","year":"2025","unstructured":"Peschke, L. et al. Mode 3 knowledge production and exchange processes of research and innovation networks in AI-based research environments. J. Innov. Entrep. 14, 116 (2025).","journal-title":"J. Innov. Entrep"},{"key":"45385_CR26","doi-asserted-by":"publisher","first-page":"877","DOI":"10.1007\/s10796-022-10284-3","volume":"24","author":"M V\u00f6ssing","year":"2022","unstructured":"V\u00f6ssing, M. et al. Designing transparency for effective human-AI collaboration. Inf. Syst. Front. 24, 877\u2013895 (2022).","journal-title":"Inf. Syst. Front."},{"key":"45385_CR27","doi-asserted-by":"publisher","unstructured":"Wallace, B. C. et al. ACM,. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. in Proc. 2nd ACM SIGHIT Int. Health Informatics Symp. 819\u2013824 (2012). https:\/\/doi.org\/10.1145\/2110363.2110464","DOI":"10.1145\/2110363.2110464"},{"key":"45385_CR28","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s12874-020-0897-3","volume":"20","author":"H Harrison","year":"2020","unstructured":"Harrison, H. et al. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med. Res. Methodol. 20, 7 (2020).","journal-title":"BMC Med. Res. Methodol."},{"key":"45385_CR29","unstructured":"Rayyan https:\/\/www.rayyan.ai\/"},{"key":"45385_CR30","doi-asserted-by":"crossref","unstructured":"Smela, B. et al. PNS60 use of artificial intelligence with distillersr software in selected systematic literature reviews. Value Health Reg. Issues 22, S92 (2020).","DOI":"10.1016\/j.vhri.2020.07.479"},{"key":"45385_CR31","doi-asserted-by":"publisher","first-page":"256","DOI":"10.1186\/s12874-020-01129-1","volume":"20","author":"C Hamel","year":"2020","unstructured":"Hamel, C. et al. Towards adequate methods for evaluating human\u2013AI collaboration in systematic reviews. BMC Med. Res. Methodol. 20, 256 (2020).","journal-title":"BMC Med. Res. Methodol."},{"key":"45385_CR32","doi-asserted-by":"publisher","first-page":"e70100","DOI":"10.1111\/jep.70100","volume":"31","author":"L Schmidt","year":"2025","unstructured":"Schmidt, L., Cree, I. & Campbell, F. WCT EVI MAP group. Digital tools to support the systematic review process: an introduction. J. Eval Clin. Pract. 31, e70100 (2025).","journal-title":"J. Eval Clin. Pract."},{"key":"45385_CR33","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/s13347-025-00916-2","volume":"38","author":"T Mitchell","year":"2025","unstructured":"Mitchell, T. Trust and transparency in artificial intelligence. Philos. Technol. 38, 87 (2025).","journal-title":"Philos. Technol."},{"key":"45385_CR34","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2025.111738","author":"X Pang","year":"2025","unstructured":"Pang, X. et al. Comparing Artificial Intelligence and manual methods in systematic review processes. J. Clin. Epidemiol. https:\/\/doi.org\/10.1016\/j.jclinepi.2025.111738 (2025).","journal-title":"J. Clin. Epidemiol."},{"key":"45385_CR35","doi-asserted-by":"publisher","first-page":"5435","DOI":"10.1007\/s41060-025-00777-0","volume":"20","author":"JJ Teijema","year":"2025","unstructured":"Teijema, J. J. et al. Large-scale simulation study of active learning models for systematic reviews. Int. J. Data Sci. Anal. 20, 5435\u20135456 (2025).","journal-title":"Int. J. Data Sci. Anal."},{"key":"45385_CR36","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1186\/s12874-025-02583-5","volume":"25","author":"F Trad","year":"2025","unstructured":"Trad, F. et al. Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation. BMC Med. Res. Methodol. 25, 130 (2025).","journal-title":"BMC Med. Res. Methodol."},{"key":"45385_CR37","doi-asserted-by":"crossref","unstructured":"Neeleman, R. C. et al. Updated approach to error rates in systematic review screening: integrating active learning, large language models, and full-text screening data. Preprint at (2026). https:\/\/osf.io\/preprints\/psyarxiv\/r3m2d_v1","DOI":"10.31234\/osf.io\/r3m2d_v1"},{"key":"45385_CR38","first-page":"9780","volume":"33","author":"D Pedreschi","year":"2019","unstructured":"Pedreschi, D. et al. Meaningful explanations of black box AI decision systems. Proc. AAAI Conf. Artif. Intell. 33, 9780\u20139784 (2019).","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"45385_CR39","unstructured":"ASReview Model Selection Guide. https:\/\/asreview.nl\/blog\/asreview-model-selectionguide\/"},{"issue":"9","key":"45385_CR40","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1080\/14737167.2023.2234639","volume":"23","author":"MJ Oude Wolcherink","year":"2023","unstructured":"Oude Wolcherink, M. J., Pouwels, X. G. L. V., van Dijk, S. H. B., Doggen, C. J. M. & Koffijberg, H. Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles? Expert Rev. PharmacoEcon. Outcomes Res. 23 (9), 1049\u20131056. https:\/\/doi.org\/10.1080\/14737167.2023.2234639 (2023).","journal-title":"Expert Rev. PharmacoEcon. Outcomes Res."},{"key":"45385_CR41","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1186\/s13643-02402587-0","volume":"13","author":"F Byrne","year":"2024","unstructured":"Byrne, F., Hofstee, L., Teijema, J., De Bruin, J. & van de Schoot, R. Impact of active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study. Syst. Reviews. 13, 175. https:\/\/doi.org\/10.1186\/s13643-02402587-0 (2024).","journal-title":"Syst. Reviews"},{"key":"45385_CR42","doi-asserted-by":"publisher","unstructured":"Bron, M., van der Heijden, P. G. M., Feelders, A. & Siebes, A. Using Chao\u2019s estimator as. a stopping criterion for technology-assisted review. ACM Trans. Inf. Syst. 43(3), 1\u201351 https:\/\/doi.org\/10.1145\/3724116 (2025).","DOI":"10.1145\/3724116"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-45385-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-45385-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-45385-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T20:39:15Z","timestamp":1781642355000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-45385-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,22]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["45385"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-45385-5","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,22]]},"assertion":[{"value":"8 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 March 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 March 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"The Screenathon 2.0 protocol and data collection for the investigation were accepted by the Ethics Committee of the Faculty of Social and Behavioral Sciences of Utrecht University on 20 May 2025 (file no. 25\u20130241). All participants provided informed consent prior to participating in the study. We confirm that all experiments were performed in accordance with the relevant guidelines and regulations.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval and consent to participate"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Large language models were not used to generate text for the present paper. They were, however, utilized during editing to correct grammar mistakes and improve the writing style. Furthermore, they were used to help write R code to create figures. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Generative AI usage"}}],"article-number":"14487"}}