{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T11:22:20Z","timestamp":1780572140112,"version":"3.54.1"},"reference-count":69,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T00:00:00Z","timestamp":1719446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","award":["UL1TR002377"],"award-info":[{"award-number":["UL1TR002377"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R01-LM011934"],"award-info":[{"award-number":["R01-LM011934"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>The surge in patient portal messages (PPMs) with increasing needs and workloads for efficient PPM triage in healthcare settings has spurred the exploration of AI-driven solutions to streamline the healthcare workflow processes, ensuring timely responses to patients to satisfy their healthcare needs. However, there has been less focus on isolating and understanding patient primary concerns in PPMs\u2014a practice which holds the potential to yield more nuanced insights and enhances the quality of healthcare delivery and patient-centered care.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We propose a fusion framework to leverage pretrained language models (LMs) with different language advantages via a Convolution Neural Network for precise identification of patient primary concerns via multi-class classification. We examined 3 traditional machine learning models, 9 BERT-based language models, 6 fusion models, and 2 ensemble models.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The outcomes of our experimentation underscore the superior performance achieved by BERT-based models in comparison to traditional machine learning models. Remarkably, our fusion model emerges as the top-performing solution, delivering a notably improved accuracy score of 77.67\u2009\u00b1\u20092.74% and an F1 score of 74.37\u2009\u00b1\u20093.70% in macro-average.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>This study highlights the feasibility and effectiveness of multi-class classification for patient primary concern detection and the proposed fusion framework for enhancing primary concern detection.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The use of multi-class classification enhanced by a fusion of multiple pretrained LMs not only improves the accuracy and efficiency of patient primary concern identification in PPMs but also aids in managing the rising volume of PPMs in healthcare, ensuring critical patient communications are addressed promptly and accurately.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae144","type":"journal-article","created":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T10:16:22Z","timestamp":1719483382000},"page":"1714-1724","source":"Crossref","is-referenced-by-count":6,"title":["Automatic uncovering of patient primary concerns in portal messages using a fusion framework of pretrained language models"],"prefix":"10.1093","volume":"31","author":[{"given":"Yang","family":"Ren","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, University of South Carolina , Columbia, SC 29208, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuqi","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN 55905, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jungwei W","family":"Fan","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN 55905, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9148-9531","authenticated-orcid":false,"given":"Aditya","family":"Khurana","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Mayo Clinic, Rochester, MN 55905, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1691-5179","authenticated-orcid":false,"given":"Sunyang","family":"Fu","sequence":"additional","affiliation":[{"name":"Department of Health Data Science and Artificial Intelligence, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dezhi","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Integrated Information Technology, University of South Carolina, Columbia, SC 29208, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2570-3741","authenticated-orcid":false,"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Health Data Science and Artificial Intelligence, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN 55905, United States"},{"name":"Department of Health Data Science and Artificial Intelligence, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,6,27]]},"reference":[{"issue":"6","key":"2024071907522397700_ocae144-B1","doi-asserted-by":"crossref","first-page":"e148","DOI":"10.2196\/jmir.4255","article-title":"Patient portals and patient engagement: a state of the science review","volume":"17","author":"Irizarry","year":"2015","journal-title":"J Med Internet Res"},{"issue":"5","key":"2024071907522397700_ocae144-B2","doi-asserted-by":"crossref","first-page":"e17955","DOI":"10.2196\/17955","article-title":"Effects of a web-based patient portal on patient satisfaction and missed appointment rates: survey study","volume":"22","author":"Graham","year":"2020","journal-title":"J Med Internet Res"},{"issue":"1","key":"2024071907522397700_ocae144-B3","doi-asserted-by":"crossref","first-page":"e17273","DOI":"10.2196\/17273","article-title":"Characterizing patient-clinician communication in secure medical messages: retrospective study","volume":"24","author":"Huang","year":"2022","journal-title":"J Med Internet Res"},{"issue":"4","key":"2024071907522397700_ocae144-B4","doi-asserted-by":"crossref","first-page":"e15038","DOI":"10.2196\/15038","article-title":"Using patient portals to improve patient outcomes: systematic review","volume":"6","author":"Han","year":"2019","journal-title":"JMIR Hum Factors"},{"issue":"3","key":"2024071907522397700_ocae144-B5","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1089\/tmj.2013.0097","article-title":"Impact of patient portal secure messages and electronic visits on adult primary care office visits","volume":"20","author":"North","year":"2014","journal-title":"Telemed e-Health"},{"issue":"2","key":"2024071907522397700_ocae144-B6","doi-asserted-by":"crossref","first-page":"699","DOI":"10.25300\/MISQ\/2020\/14180","article-title":"Patient-provider engagement and its impact on health outcomes: a longitudinal study of patient portal use","volume":"44","author":"Bao","year":"2020","journal-title":"MISQ"},{"issue":"4","key":"2024071907522397700_ocae144-B7","doi-asserted-by":"crossref","first-page":"1432","DOI":"10.1007\/s00464-015-4347-y","article-title":"Rapid growth in surgeons\u2019 use of secure messaging in a patient portal","volume":"30","author":"Shenson","year":"2016","journal-title":"Surg Endosc"},{"key":"2024071907522397700_ocae144-B8","first-page":"1930","article-title":"Adoption of secure messaging in a patient portal across pediatric specialties","volume":"2016","author":"Masterman","year":"2017","journal-title":"AMIA Annu Symp Proc"},{"key":"2024071907522397700_ocae144-B9","author":"Carayon","year":"2015"},{"issue":"7","key":"2024071907522397700_ocae144-B10","doi-asserted-by":"crossref","first-page":"e26770","DOI":"10.2196\/26770","article-title":"Analyzing patient secure messages using a fast health care interoperability resources (FIHR)-based data model: development and topic modeling study","volume":"23","author":"De","year":"2021","journal-title":"J Med Internet Res"},{"issue":"4","key":"2024071907522397700_ocae144-B11","doi-asserted-by":"crossref","first-page":"908","DOI":"10.1111\/jrh.12657","article-title":"Midwest rural-urban disparities in use of patient online services for COVID-19","volume":"38","author":"Huang","year":"2022","journal-title":"J Rural Health"},{"issue":"4","key":"2024071907522397700_ocae144-B12","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1136\/amiajnl-2011-000382","article-title":"Triaging patients at risk of influenza using a patient portal","volume":"19","author":"Trent Rosenbloom","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2024071907522397700_ocae144-B13","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.jbi.2017.08.014","article-title":"Classifying patient portal messages using convolutional neural networks","volume":"74","author":"Sulieman","year":"2017","journal-title":"J Biomed Inform"},{"key":"2024071907522397700_ocae144-B14","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.ijmedinf.2017.06.004","article-title":"A comparison of rule-based and machine learning approaches for classifying patient portal messages","volume":"105","author":"Cronin","year":"2017","journal-title":"Int J Med Inform"},{"key":"2024071907522397700_ocae144-B15","first-page":"1861","article-title":"Automated classification of consumer health information needs in patient portal messages","volume":"2015","author":"Cronin","year":"2015","journal-title":"Am Med Inform Assoc"},{"key":"2024071907522397700_ocae144-B16","author":"Lau","year":"2016"},{"key":"2024071907522397700_ocae144-B17","first-page":"1380","author":"Tafti","year":"2019"},{"key":"2024071907522397700_ocae144-B18","first-page":"176","author":"Ren","year":"2023"},{"key":"2024071907522397700_ocae144-B19","author":"Harzand","year":"2023"},{"issue":"8","key":"2024071907522397700_ocae144-B20","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2024071907522397700_ocae144-B21","doi-asserted-by":"crossref","first-page":"105949","DOI":"10.1016\/j.knosys.2020.105949","article-title":"A novel method for sentiment classification of drug reviews using fusion of deep and machine learning techniques","volume":"198","author":"Basiri","year":"2020","journal-title":"Knowl-Based Syst"},{"key":"2024071907522397700_ocae144-B22","first-page":"98","author":"Ding","year":"2020"},{"key":"2024071907522397700_ocae144-B23","author":"Yin","year":"2017"},{"key":"2024071907522397700_ocae144-B24","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-14596-5","volume-title":"Deep Learning for NLP and Speech Recognition","author":"Kamath","year":"2019"},{"issue":"2","key":"2024071907522397700_ocae144-B25","doi-asserted-by":"crossref","first-page":"272","DOI":"10.15837\/ijccc.2019.2.3375","article-title":"Ensemble sentiment analysis method based on R-CNN and C-RNN with fusion gate","volume":"14","author":"Yang","year":"2019","journal-title":"Int J Comput Commun"},{"key":"2024071907522397700_ocae144-B26","author":"Lai","year":"2015"},{"key":"2024071907522397700_ocae144-B27","author":"Zhou","year":"2015"},{"key":"2024071907522397700_ocae144-B28","doi-asserted-by":"crossref","first-page":"12467","DOI":"10.1109\/ACCESS.2019.2891770","article-title":"A single attention-based combination of CNN and RNN for relation classification","volume":"7","author":"Guo","year":"2019","journal-title":"IEEE Access"},{"key":"2024071907522397700_ocae144-B29","author":"Gonz\u00e1lez-Carvajal","year":"2020"},{"issue":"2","key":"2024071907522397700_ocae144-B30","first-page":"2943","article-title":"BERT-CNN: a deep learning model for detecting emotions from text","volume":"71","author":"Abas","year":"2022","journal-title":"Comput Mater Contin"},{"key":"2024071907522397700_ocae144-B31","first-page":"193","author":"Chen","year":"2023"},{"key":"2024071907522397700_ocae144-B32","first-page":"1629","author":"Zhang","year":"2020"},{"issue":"11","key":"2024071907522397700_ocae144-B33","doi-asserted-by":"crossref","first-page":"237","DOI":"10.3390\/fi11110237","article-title":"Feature fusion text classification model combining CNN and BiGRU with multi-attention mechanism","volume":"11","author":"Zhang","year":"2019","journal-title":"Future Internet"},{"key":"2024071907522397700_ocae144-B34","doi-asserted-by":"crossref","first-page":"1504854","DOI":"10.34133\/2021\/1504854","article-title":"Probing patient messages enhanced by natural language processing: a top-down message corpus analysis","volume":"2021","author":"Mastorakos","year":"2021","journal-title":"Health Data Sci"},{"key":"2024071907522397700_ocae144-B35","first-page":"207","author":"Suthaharan","year":"2016"},{"issue":"2","key":"2024071907522397700_ocae144-B36","first-page":"141","article-title":"Text document categorization using support vector machine","volume":"4","author":"Fatima","year":"2017","journal-title":"Int Res J Eng Technol (IRJET)"},{"issue":"1","key":"2024071907522397700_ocae144-B37","doi-asserted-by":"crossref","first-page":"31","DOI":"10.17849\/insm-47-01-31-39.1","article-title":"Random forest","volume":"47","author":"Rigatti","year":"2017","journal-title":"J Insur Med"},{"key":"2024071907522397700_ocae144-B38","first-page":"758","volume-title":"A Review on Random Forest: An Ensemble Classifier","author":"Parmar","year":"2019"},{"issue":"12","key":"2024071907522397700_ocae144-B39","doi-asserted-by":"crossref","first-page":"6999","DOI":"10.1109\/TNNLS.2021.3084827","article-title":"A survey of convolutional neural networks: analysis, applications, and prospects","volume":"33","author":"Li","year":"2022","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"2024071907522397700_ocae144-B40","doi-asserted-by":"crossref","first-page":"1139","DOI":"10.1016\/j.procs.2020.03.416","article-title":"Sentimental short sentences classification by using CNN deep learning model with fine tuned Word2Vec","volume":"167","author":"Sharma","year":"2020","journal-title":"Procedia Comput Sci"},{"issue":"1","key":"2024071907522397700_ocae144-B41","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1017\/S1351324916000334","article-title":"Word2Vec","volume":"23","author":"Church","year":"2017","journal-title":"Nat Lang Eng"},{"key":"2024071907522397700_ocae144-B42","first-page":"45","author":"Kramer","year":"2016"},{"key":"2024071907522397700_ocae144-B43","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1007\/978-1-4842-2766-4_7","volume-title":"Deep Learning with Python: A Hands-on Introduction","author":"Ketkar","year":"2017"},{"key":"2024071907522397700_ocae144-B44","author":"Devlin","year":"2018"},{"key":"2024071907522397700_ocae144-B45","author":"Liu","year":"2019"},{"key":"2024071907522397700_ocae144-B46","author":"Lan","year":"2019"},{"issue":"1","key":"2024071907522397700_ocae144-B47","doi-asserted-by":"crossref","first-page":"102756","DOI":"10.1016\/j.ipm.2021.102756","article-title":"An ensemble model for classifying idioms and literal texts using BERT and RoBERTa","volume":"59","author":"Briskilal","year":"2022","journal-title":"Inform Process Manage"},{"key":"2024071907522397700_ocae144-B48","author":"Adoma","year":"2020"},{"issue":"9","key":"2024071907522397700_ocae144-B49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","year":"2023","journal-title":"ACM Comput Surv"},{"issue":"4","key":"2024071907522397700_ocae144-B50","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024071907522397700_ocae144-B51","author":"Sch\u00e4fer","year":"2020"},{"key":"2024071907522397700_ocae144-B52","author":"Tian","year":"2021"},{"key":"2024071907522397700_ocae144-B53","author":"Yu","year":"2019"},{"key":"2024071907522397700_ocae144-B54","author":"Alsentzer","year":"2019"},{"key":"2024071907522397700_ocae144-B55","author":"Nguyen","year":"2020"},{"key":"2024071907522397700_ocae144-B56","author":"Zhang","year":"2022"},{"key":"2024071907522397700_ocae144-B57","author":"Zhang","year":"2022"},{"key":"2024071907522397700_ocae144-B58","author":"Singh","year":"2023"},{"issue":"9","key":"2024071907522397700_ocae144-B59","doi-asserted-by":"crossref","first-page":"e26189","DOI":"10.2196\/26189","article-title":"The impact of digital patient portals on health outcomes, system efficiency, and patient attitudes: updated systematic literature review","volume":"23","author":"Carini","year":"2021","journal-title":"J Med Internet Res"},{"key":"2024071907522397700_ocae144-B60","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1016\/j.jss.2020.05.039","article-title":"Automating the classification of complexity of medical decision-making in patient-provider messaging in a patient portal","volume":"255","author":"Sulieman","year":"2020","journal-title":"J Surg Res"},{"key":"2024071907522397700_ocae144-B61","volume-title":"Increasing Efficiency in Addressing Patient Portal Messaging Demands on Healthcare Systems (P6-7.006)","author":"Gordon-Kundu","year":"2023"},{"key":"2024071907522397700_ocae144-B62","author":"Harzand","year":"2023"},{"issue":"3","key":"2024071907522397700_ocae144-B63","doi-asserted-by":"crossref","first-page":"ooad077","DOI":"10.1093\/jamiaopen\/ooad077","article-title":"Patient portal interventions: a scoping review of functionality, automation used, and therapeutic elements of patient portal interventions","volume":"6","author":"Gleason","year":"2023","journal-title":"JAMIA Open"},{"key":"2024071907522397700_ocae144-B64","author":"Araci","year":"2019"},{"key":"2024071907522397700_ocae144-B65","author":"Touvron","year":"2023"},{"key":"2024071907522397700_ocae144-B66","doi-asserted-by":"crossref","first-page":"104770","DOI":"10.1016\/j.ebiom.2023.104770","article-title":"Benchmarking large language models\u2019 performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard","volume":"95","author":"Lim","year":"2023","journal-title":"EBioMedicine"},{"issue":"2023","key":"2024071907522397700_ocae144-B67","first-page":"688","article-title":"Using large language models in psychology","volume":"2","author":"Demszky","year":"2023","journal-title":"Nat Rev Psychol"},{"key":"2024071907522397700_ocae144-B68","author":"Team","year":"2023"},{"key":"2024071907522397700_ocae144-B69","author":"Zhang","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/8\/1714\/58591192\/ocae144.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/8\/1714\/58591192\/ocae144.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T07:52:56Z","timestamp":1721375576000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/8\/1714\/7700019"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,27]]},"references-count":69,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,6,27]]},"published-print":{"date-parts":[[2024,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae144","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,8]]},"published":{"date-parts":[[2024,6,27]]}}}