{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:00:16Z","timestamp":1782846016489,"version":"3.54.5"},"reference-count":53,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n \n Cochrane plain language summaries (PLSs) aim to make systematic review findings more accessible to the general public. However, inconsistencies in how conclusions are presented may impact comprehension and decision-making. Classifying PLSs based on conclusiveness can improve clarity and facilitate informed health decisions.\n \n <\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n \n This study aimed to develop and evaluate deep learning language models for the classification of PLSs according to 3 levels of conclusiveness (conclusive, inconclusive, and unclear) and to compare their performance with a general-purpose large language model (GPT-4o).\n \n <\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We used a publicly available dataset containing 4405 Cochrane PLSs of systematic reviews published until 2019, already classified by humans according to 9 categories of conclusiveness regarding the intervention\u2019s effectiveness or safety. We merged these categories into 3 classes based on the strength of conclusiveness: conclusive, inconclusive, and unclear. For the fine-tuning, we used Scientific Bidirectional Encoder Representations from Transformers (SciBERT), a pretrained language model trained on 1.14 million papers primarily from the health sciences, and Longformer, a transformer model designed specifically to process long documents. The script was developed using the Python programming language and the PyTorch framework. We computed evaluation metrics using the\n scikit-learn<\/jats:italic>\n machine learning library and determined the area under the curve of the receiver operating characteristic (AUCROC) to measure the model performance in balancing sensitivity and specificity. We also analyzed a separate set of 213 PLSs and compared the predictions of our pretrained models with both manual verification and outputs generated by ChatGPT.\n \n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The model based on SciBERT achieved a balanced accuracy of 56.6%. The AUCROC was 0.91 for \u201cconclusive,\u201d 0.67 for \u201cinconclusive,\u201d and 0.75 for \u201cunclear\u201d conclusiveness classes. The Longformer-based model had a balanced accuracy of 60.9%, with AUCROCs of 0.86 for \u201cconclusive,\u201d 0.67 for \u201cinconclusive,\u201d and 0.72 for \u201cunclear\u201d conclusiveness classes. Both models underperformed compared with ChatGPT, which demonstrated higher accuracy (74.2%), better precision and recall, and a higher Cohen \u03ba (0.57).\n \n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Fine-tuning 2 transformer-based language models showed mixed results in classifying Cochrane PLSs by conclusiveness, likely due to semantic overlap and subtle linguistic differences. Despite satisfactory internal test metrics, the fine-tuned models failed to generalize to newly published PLSs, where performance dropped to near-chance levels. These findings suggest that general-purpose large language models like GPT-4o may currently offer more reliable results for practical classification tasks in biomedical applications.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/72657","type":"journal-article","created":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T19:15:58Z","timestamp":1776194158000},"page":"e72657-e72657","source":"Crossref","is-referenced-by-count":0,"title":["Classification of Cochrane Plain Language Summaries by Conclusiveness Using Transformer-Based Models and ChatGPT: Retrospective Observational Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1733-582X","authenticated-orcid":false,"given":"Antonija","family":"Mijatovi\u0107","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2469-2948","authenticated-orcid":false,"given":"Luka","family":"Ursi\u0107","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9509-0210","authenticated-orcid":false,"given":"Nensi","family":"Brali\u0107","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0811-6983","authenticated-orcid":false,"given":"Ru\u017eica","family":"Bandi\u0107","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4936-3523","authenticated-orcid":false,"given":"Barbara","family":"\u0106a\u0107i\u0107","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8719-7277","authenticated-orcid":false,"given":"Ivan","family":"Buljan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6272-0917","authenticated-orcid":false,"given":"Ana","family":"Maru\u0161i\u0107","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,4,14]]},"reference":[{"key":"R1","doi-asserted-by":"crossref","unstructured":"Whiting P Davenport C . Writing a plain language summary. In: Deeks JJ Bossuyt PM Leeflang MM Takwoingi Y , editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. John Wiley & Sons; 2023.","DOI":"10.1002\/14651858.ED000163"},{"key":"R2","unstructured":"Higgins JP Thomas J Chandler J et al. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Collaboration; 2023. URL: https:\/\/www.cochrane.org\/authors\/handbooks-and-manuals\/handbook [Accessed 30-03-2026]"},{"key":"R3","unstructured":"Weiss BD . Health Literacy and Patient Safety: Help Patients Understand: Manual for Clinicians. 2nd ed. American Medical Association; 2007. URL: https:\/\/books.google.be\/books?id=quJaYgEACAAJ [Accessed 30-03-2026]"},{"key":"R4","unstructured":"Sch\u00fcnemann HJ Vist GE Higgins JP Santesso N Deeks JJ Glasziou P et al. Chapter 15: Interpreting results and drawing conclusions. In: Higgins JP Thomas J Chandler J Cumpston M Li T Page MJ Welch VA , editors. Cochrane Handbook for Systematic Reviews of Interventions Version 6.5. Cochrane Collaboration; 2024."},{"key":"R5","unstructured":"Glenton C Nilsen ES Sporst\u00f8l F\u00f8nhus M Goudie S Noonan E . How to write a plain language summary of a Cochrane intervention review (version 10). Cochrane Norway. URL: https:\/\/www.cochrane.no\/how-write-plain-language-summary [Accessed 22-03-2026]"},{"key":"R6","unstructured":"Lasserson T Santesso N Cumpston M Marshall R \u00d3g\u00e1in ON . Incorporating GRADE in Cochrane reviews: feedback from the CEU screening programme. Cochrane; 2016. URL: https:\/\/www.cochrane.org\/sites\/default\/files\/uploads\/PDFs\/MECIR\/Incorporating%20GRADE%20in%20Cochrane%20Reviews.pdf [Accessed 22-03-2026]"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2019.10.014","article-title":"GRADE guidelines 26: informative statements to communicate the findings of systematic reviews of interventions","volume":"119","author":"Santesso","journal-title":"J Clin Epidemiol"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.34171\/mjiri.34.79","article-title":"Research utilization process model: a cyclical, spiral, and developmental process to provide conclusive research knowledge in health professions education","volume":"34","author":"Changiz","journal-title":"Med J Islam Repub Iran"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1016\/j.pec.2015.07.030","article-title":"Effective presentation of health care performance information for consumer decision making: a systematic review","volume":"99","author":"Kurtzman","journal-title":"Patient Educ Couns"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.21037\/amj.2019.04.05","article-title":"Discussion and conclusion","volume":"4","author":"Gray","journal-title":"AME Med J"},{"issue":"3","key":"R11","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph17030880","article-title":"The impact of online health information on patient health behaviours and making decisions concerning health","volume":"17","author":"Bujnowska-Fedak","journal-title":"Int J Environ Res Public Health"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.5729","article-title":"Internet health information seeking and the patient-physician relationship: a systematic review","volume":"19","author":"Tan","journal-title":"J Med Internet Res"},{"issue":"14","key":"R13","doi-asserted-by":"publisher","first-page":"1727","DOI":"10.1001\/archinte.163.14.1727","article-title":"The impact of health information on the internet on the physician-patient relationship: patient perceptions","volume":"163","author":"Murray","journal-title":"Arch Intern Med"},{"issue":"5","key":"R14","first-page":"407","volume":"60","author":"Tonsaker","journal-title":"Can Fam Physician"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2022.107384","article-title":"How online searches fuel health anxiety: investigating the link between health-related searches, health anxiety, and future intention","volume":"136","author":"Peng","journal-title":"Comput Human Behav"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1055\/a-1348-8059","article-title":"Cyberchondria - a new behavioral syndrome? [Article in German]","volume":"71","author":"M\u00fcller","journal-title":"Psychother Psychosom Med Psychol"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1016\/j.psym.2016.02.002","article-title":"Cyberchondria: parsing health anxiety from online behavior","volume":"57","author":"Doherty-Torstrick","journal-title":"Psychosomatics"},{"issue":"9","key":"R18","doi-asserted-by":"publisher","first-page":"10950","DOI":"10.1002\/cam4.5825","article-title":"Linguistic analysis of plain language summaries and corresponding scientific summaries of Cochrane systematic reviews about oncology interventions","volume":"12","author":"\u0160uto","journal-title":"Cancer Med"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-015-1221-x","article-title":"Conclusiveness of the Cochrane Eye and Vision Group Reviews","volume":"8","author":"Mimouni","journal-title":"BMC Res Notes"},{"issue":"3","key":"R20","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1177\/0300060515574922","article-title":"Conclusiveness of the Cochrane reviews in gynaecological cancer: a systematic analysis","volume":"43","author":"Yin","journal-title":"J Int Med Res"},{"issue":"10","key":"R21","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1080\/08035250600580537","article-title":"Conclusiveness of the Cochrane Neonatal Reviews: a systematic analysis","volume":"95","author":"Mandel","journal-title":"Acta Paediatr"},{"issue":"2","key":"R22","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1097\/MEG.0b013e32835a1083","article-title":"Conclusiveness of the Cochrane Reviews in pediatric-gastroenterology: a systematic analysis","volume":"25","author":"Cohen","journal-title":"Eur J Gastroenterol Hepatol"},{"issue":"12","key":"R23","doi-asserted-by":"publisher","first-page":"808","DOI":"10.1097\/EJA.0000000000000691","article-title":"High quality of evidence is uncommon in Cochrane systematic reviews in Anaesthesia, Critical Care and Emergency Medicine","volume":"34","author":"Conway","journal-title":"Eur J Anaesthesiol"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-022-01721-7","article-title":"Conclusiveness, linguistic characteristics and readability of Cochrane plain language summaries of intervention reviews: a cross-sectional study","volume":"22","author":"Bani\u0107","journal-title":"BMC Med Res Methodol"},{"key":"R25","unstructured":"Bird S Klein E Loper E . Natural Language Processing with Python. O\u2019Reilly Media; 2009. ISBN: 9780596516499"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"3713","DOI":"10.1007\/s11042-022-13428-4","article-title":"Natural language processing: state of the art, current trends and challenges","volume":"82","author":"Khurana","journal-title":"Multimed Tools Appl"},{"key":"R27","unstructured":"A complete guide to natural language processing. DeepLearning.AI. URL: https:\/\/www.deeplearning.ai\/resources\/natural-language-processing [Accessed 21-09-2024]"},{"key":"R28","unstructured":"Srinivasan A Sitaram S Ganu T Dandapat S Bali K Choudhury M . Predicting the performance of multilingual NLP models. arXiv. Preprint posted online on Oct 21, 2021. [doi: 10.48550\/arXiv.2110.08875]"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","journal-title":"ACM Trans Comput Healthcare"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Beltagy I Lo K Cohan A . SciBERT: a pretrained language model for scientific text. Presented at: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Nov 3-7, 2019; Hong Kong, China. [doi: 10.18653\/v1\/D19-1371]","DOI":"10.18653\/v1\/D19-1371"},{"key":"R31","unstructured":"Beltagy I Peters ME Cohan A . Longformer: the long-document transformer. arXiv. Preprint posted online on Apr 10, 2020. [doi: 10.48550\/arXiv.2004.05150]"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2023.1220999","article-title":"The association between research design and the perceived treatment effectiveness: a cross-sectional study","volume":"10","author":"Brali\u0107","journal-title":"Front Med (Lausanne)"},{"key":"R33","unstructured":"Devlin J Chang MW Lee K Toutanova K . BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. Preprint posted online on Oct 11, 2018. [doi: 10.48550\/arXiv.1810.04805]"},{"key":"R34","unstructured":"Causevic S . Evolution of large language models\u2014BERT, GPT3, MUM, and PaLM. Medium. 2022. URL: https:\/\/medium.com\/data-science\/self-supervised-transformer-models-bert-gpt3-mum-and-paml-2b5e29ea0c26 [Accessed 23-03-2026]"},{"key":"R35","unstructured":"Fine-tuning. Hugging Face. URL: https:\/\/huggingface.co\/docs\/transformers\/training [Accessed 23-03-2026]"},{"key":"R36","unstructured":"Winastwan R . Text classification with BERT in PyTorch. Towards Data Science; 2021. URL: https:\/\/towardsdatascience.com\/text-classification-with-bert-in-pytorch-887965e5820f [Accessed 21-09-2024]"},{"key":"R37","unstructured":"What is transfer learning? GeeksforGeeks. URL: https:\/\/www.geeksforgeeks.org\/ml-introduction-to-transfer-learning [Accessed 21-09-2024]"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Oztel I Yolcu G Oz C . Performance comparison of transfer learning and training from scratch approaches for deep facial expression recognition. Presented at: 2019 4th International Conference on Computer Science and Engineering (UBMK); Sep 11-15, 2019; Samsun, Turkey. [doi: 10.1109\/UBMK.2019.8907203]","DOI":"10.1109\/UBMK.2019.8907203"},{"key":"R39","unstructured":"Paszke A Gross S Massa F et al. PyTorch: an imperative style, high-performance deep learning library. In: Wallach H Larochelle H Beygelzimer A Alche\u2019-Buc FD Fox E Garnett R , editors. Advances in Neural Information Processing Systems 32. Curran Associates, Inc; 2019:8024-8035."},{"key":"R40","doi-asserted-by":"crossref","unstructured":"Kluyver T Ragan-Kelley B P\u00e9rez F et al. Jupyter notebooks\u2014a publishing format for reproducible computational workflows. In: Loizides F Schmidt B , editors. Positioning and Power in Academic Publishing: Players, Agents and Agendas. IOS Press; 2016:87-90. [doi: 10.3233\/978-1-61499-649-1-87]","DOI":"10.3233\/978-1-61499-649-1-87"},{"key":"R41","unstructured":"Models. Hugging Face. URL: https:\/\/huggingface.co\/models [Accessed 21-09-2024]"},{"key":"R42","unstructured":"AdamW. PyTorch. URL: https:\/\/pytorch.org\/docs\/stable\/generated\/torch.optim.AdamW.html#adamw [Accessed 21-09-2024]"},{"key":"R43","unstructured":"CrossEntropyLoss. PyTorch. URL: https:\/\/pytorch.org\/docs\/stable\/generated\/torch.nn.CrossEntropyLoss.html [Accessed 21-09-2024]"},{"key":"R44","doi-asserted-by":"crossref","unstructured":"Reed R Marks RJ . Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks. MIT Press; 1999. ISBN: 9780262181907","DOI":"10.7551\/mitpress\/4937.001.0001"},{"key":"R45","unstructured":"scikit-learn: machine learning in Python. scikit-learn. URL: https:\/\/scikit-learn.org\/stable [Accessed 21-09-2024]"},{"key":"R46","unstructured":"Readability, linguistic characteristics and conclusiveness of Cochrane plain language summaries of intervention reviews: a cross-sectional study. OSF. URL: https:\/\/osf.io\/qvu3a\/overview [Accessed 30-03-2026]"},{"key":"R47","unstructured":"Code of ethics of the Croatian Science Foundation. Hrvatska Zaklada za Znanost. 2020. URL: https:\/\/hrzz.hr\/wp-content\/uploads\/2020\/08\/HRZZ-Code-of-Ethics.pdf [Accessed 30-03-2026]"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.11.003","article-title":"Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews","volume":"133","author":"Thomas","journal-title":"J Clin Epidemiol"},{"key":"R49","unstructured":"Povey N . ChatGPT: abstractive text summarization. Medium. 2022. URL: https:\/\/medium.com\/@nadirapovey\/chatgpt-text-summarization-44f768222a4c [Accessed 21-09-2024]"},{"key":"R50","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1169595","article-title":"ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations","volume":"6","author":"Dave","journal-title":"Front Artif Intell"},{"issue":"1","key":"R51","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00896-7","article-title":"Evaluating large language models on medical evidence summarization","volume":"6","author":"Tang","journal-title":"NPJ Digit Med"},{"key":"R52","unstructured":"Brown TB Mann B Ryder N Subbiah M Kaplan J Dhariwal P et al. Language models are few-shot learners. arXiv. Preprint posted online on May 28, 2020. [doi: 10.48550\/arXiv.2005.14165]"},{"key":"R53","doi-asserted-by":"publisher","DOI":"10.1177\/00491241251325243","article-title":"Large language models for text classification: from zero-shot learning to instruction-tuning","author":"Davidson","journal-title":"Sociol Methods Res"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e72657","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:03:01Z","timestamp":1782842581000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e72657"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,14]]},"references-count":53,"URL":"https:\/\/doi.org\/10.2196\/72657","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,14]]},"article-number":"v14i1e72657"}}