{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T03:34:21Z","timestamp":1779248061423,"version":"3.51.4"},"reference-count":34,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2024,5,12]],"date-time":"2024-05-12T00:00:00Z","timestamp":1715472000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Systematic reviews (SRs) are a rigorous method for synthesizing empirical evidence to answer specific research questions. However, they are labor-intensive because of their collaborative nature, strict protocols, and typically large number of documents. Large language models (LLMs) and their applications such as gpt-4\/ChatGPT have the potential to reduce the human workload of the SR process while maintaining accuracy. We propose a new hybrid methodology that combines the strengths of LLMs and humans using the ability of LLMs to summarize large bodies of text autonomously and extract key information. This is then used by a researcher to make inclusion\/exclusion decisions quickly. This process replaces the typical manually performed title\/abstract screening, full-text screening, and data extraction steps in an SR while keeping a human in the loop for quality control. We developed a semi-automated LLM-assisted (Gemini-Pro) workflow with a novel innovative prompt development strategy. This involves extracting three categories of information including identifier, verifier, and data field (IVD) from the formatted documents. We present a case study where our hybrid approach reduced errors compared with a human-only SR. The hybrid workflow improved the accuracy of the case study by identifying 6\/390 (1.53%) articles that were misclassified by the human-only process. It also matched the human-only decisions completely regarding the rest of the 384 articles. Given the rapid advances in LLM technology, these results will undoubtedly improve over time.<\/jats:p>","DOI":"10.3390\/fi16050167","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T08:33:03Z","timestamp":1715589183000},"page":"167","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["A Hybrid Semi-Automated Workflow for Systematic and Literature Review Processes with Large Language Model Analysis"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5888-4994","authenticated-orcid":false,"given":"Anjia","family":"Ye","sequence":"first","affiliation":[{"name":"School of Education, University of Tasmania, Launceston, TAS 7248, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0785-5282","authenticated-orcid":false,"given":"Ananda","family":"Maiti","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3221, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9844-3296","authenticated-orcid":false,"given":"Matthew","family":"Schmidt","sequence":"additional","affiliation":[{"name":"School of Health Sciences, University of Tasmania, Sandy Bay, TAS 7005, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8566-7693","authenticated-orcid":false,"given":"Scott J.","family":"Pedersen","sequence":"additional","affiliation":[{"name":"School of Education, University of Tasmania, Launceston, TAS 7248, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,5,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1016\/j.ijsu.2010.02.007","article-title":"Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement","volume":"8","author":"Moher","year":"2010","journal-title":"Int. J. Surg."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"862","DOI":"10.1136\/bmj.309.6958.862","article-title":"Reporting, updating, and correcting systematic reviews of the effects of health care","volume":"309","author":"Chalmers","year":"1994","journal-title":"BMJ"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Higgins, J.P.T., and Green, S. (2008). Cochrane Handbook for Systematic Reviews of Interventions, Wiley.","DOI":"10.1002\/9780470712184"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/2046-4053-3-60","article-title":"Integration of existing systematic reviews into new reviews: Identification of guidance needs","volume":"3","author":"Robinson","year":"2014","journal-title":"Syst. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"103","DOI":"10.4097\/kjae.2018.71.2.103","article-title":"Introduction to systematic review and meta-analysis","volume":"71","author":"Ahn","year":"2018","journal-title":"Korean J. Anesthesiol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"b2700","DOI":"10.1136\/bmj.b2700","article-title":"The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: Explanation and elaboration","volume":"339","author":"Liberati","year":"2009","journal-title":"BMJ"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"100443","DOI":"10.1016\/j.conctc.2019.100443","article-title":"The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials","volume":"16","author":"Michelson","year":"2019","journal-title":"Contemp. Clin. Trials. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., and Hadfield, K. (2023). Can large language models replace humans in the systematic review process? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. arXiv.","DOI":"10.1002\/jrsm.1715"},{"key":"ref_10","unstructured":"Syriani, E., David, I., and Kumar, G. (2023). Assessing the ability of ChatGPT to screen articles for systematic reviews. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"467","DOI":"10.7326\/M18-0850","article-title":"PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation","volume":"169","author":"Tricco","year":"2018","journal-title":"Ann. Intern. Med."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Goodyear-Smith, F.A., van Driel, M.L., Arroll, B., and Del Mar, C. (2012). Analysis of decisions made in meta-analyses of depression screening and the risk of confirmation bias: A case study. BMC Med. Res. Methodol., 12.","DOI":"10.1186\/1471-2288-12-76"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1186\/2046-4053-3-74","article-title":"Systematic review automation technologies","volume":"3","author":"Tsafnat","year":"2014","journal-title":"Syst. Rev."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1097\/XEB.0000000000000055","article-title":"Summarizing systematic reviews: Methodological development, conduct and reporting of an umbrella review approach","volume":"13","author":"Aromataris","year":"2015","journal-title":"Int. J. Evid. Based Healthc."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1044\/cicsd_33_S_21","article-title":"Selecting studies for systemic review: Inclusion and exclusion criteria","volume":"33","author":"Meline","year":"2006","journal-title":"Contemp. Issues Commun. Sci. Disord."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1186\/s13643-019-0942-7","article-title":"Machine learning algorithms for systematic review: Reducing workload in a preclinical review of animal studies and reducing human screening error","volume":"8","author":"Thomas","year":"2019","journal-title":"Syst. Rev."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.eswa.2018.11.021","article-title":"FAST2: An intelligent assistant for finding relevant papers","volume":"120","author":"Yu","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","article-title":"An open source machine learning framework for efficient and transparent systematic reviews","volume":"3","author":"Schram","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: A practical guide to using machine learning tools in research synthesis","volume":"8","author":"Marshall","year":"2019","journal-title":"Syst. Rev."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A.E.E., and Zayed, T. (2023). Harnessing the power of ChatGPT for automating systematic review process: Methodology, case study, limitations, and future directions. Systems, 11.","DOI":"10.3390\/systems11070351"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1186\/s13643-023-02243-z","article-title":"Are ChatGPT and large language models \u201cthe answer\u201d to bringing us closer to systematic review automation?","volume":"12","author":"Qureshi","year":"2023","journal-title":"Syst. Rev."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e48996","DOI":"10.2196\/48996","article-title":"Automated paper screening for clinical reviews using large language models: Data analysis study","volume":"26","author":"Guo","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e072254","DOI":"10.1136\/bmjopen-2023-072254","article-title":"Artificial intelligence in systematic reviews: Promising when appropriately used","volume":"13","author":"Doggen","year":"2023","journal-title":"BMJ Open"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2171","DOI":"10.1007\/s00607-023-01181-x","article-title":"Artificial intelligence to automate the systematic review of scientific literature","volume":"105","author":"Romero","year":"2023","journal-title":"Computing"},{"key":"ref_25","unstructured":"Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., and Le, Q.V. (2021). Finetuned language models are zero-shot learners. arXiv."},{"key":"ref_26","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (arXiv, 2023). LLaMA: Open and efficient foundation language models, arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work?. arXiv.","DOI":"10.18653\/v1\/2022.emnlp-main.759"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chu, X., Ilyas, I.F., Krishnan, S., and Wang, J. (July, January 26). Data cleaning. Proceedings of the 2016 International Conference on Management of Data, New York, NY, USA.","DOI":"10.1145\/2882903.2912574"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: An updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"102962","DOI":"10.1016\/j.apergo.2019.102962","article-title":"Interventions to promote work ability by increasing sedentary workers\u2019 physical activity at workplaces\u2014A scoping review","volume":"82","author":"Lusa","year":"2020","journal-title":"Appl. Ergon."},{"key":"ref_31","unstructured":"Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., and Zhou, D. (2023). Larger language models do in-context learning differently. arXiv."},{"key":"ref_32","unstructured":"Gemini, T., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., and Hauth, A. (2023). Gemini: A family of highly capable multimodal models. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Horsley, T., Dingwall, O., and Sampson, M. (2011). Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst. Rev.","DOI":"10.1002\/14651858.MR000026.pub2"},{"key":"ref_34","unstructured":"(2024, March 19). AMSTAR Checklist. Available online: https:\/\/amstar.ca\/Amstar_Checklist.php."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/5\/167\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:41:08Z","timestamp":1760107268000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/5\/167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,12]]},"references-count":34,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["fi16050167"],"URL":"https:\/\/doi.org\/10.3390\/fi16050167","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,12]]}}}