{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T20:43:35Z","timestamp":1770237815797,"version":"3.49.0"},"reference-count":38,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2024,11,22]],"date-time":"2024-11-22T00:00:00Z","timestamp":1732233600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Evaluation Clinical Practice"],"published-print":{"date-parts":[[2025,2]]},"abstract":"<jats:title>ABSTRACT<\/jats:title><jats:sec><jats:title>Rationale<\/jats:title><jats:p>Limited health literacy (HL) leads to poor health outcomes, psychological stress, and misutilization of medical resources. Although interventions aimed at improving HL may be effective, identifying patients at risk of limited HL in the clinical workflow is challenging. With machine learning (ML) algorithms based on readily available data, healthcare professionals would be enabled to incorporate HL screening without the need for administering in\u2010person HL screening tools.<\/jats:p><\/jats:sec><jats:sec><jats:title>Aims and Objectives<\/jats:title><jats:p>Develop ML algorithms to identify patients at risk for limited HL in spine patients.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Between December 2021 and February 2023, consecutive English\u2010speaking patients over the age of 18 and new to an urban academic outpatient spine clinic were approached for participation in a cross\u2010sectional survey study. HL was assessed using the Newest Vital Sign and the scores were divided into limited (0\u20133) and adequate (4\u20136) HL. Additional patient characteristics were extracted through a sociodemographic survey and electronic health records. Subsequently, feature selection was performed by random forest algorithms with recursive feature selection and five ML models (stochastic gradient boosting, random forest, Bayes point machine, elastic\u2010net penalized logistic regression, support vector machine) were developed to predict limited HL.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Seven hundred and fifty\u2010three patients were included for model development, of whom 259 (34.4%) had limited HL. Variables identified for predicting limited HL were age, Area Deprivation Index\u2010national, Social Vulnerability Index, insurance category, Body Mass Index, race, college education, and employment status. The Elastic\u2010Net Penalized Logistic Regression algorithm achieved the best performance with a c\u2010statistic of 0.766, calibration slope\/intercept of 1.044\/\u22120.037, and Brier score of 0.179.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Elastic\u2010Net Penalized Logistic Regression had the best performance when compared with other ML algorithms with a c\u2010statistic of 0.766, calibration slope\/intercept of 1.044\/\u22120.037, and a Brier score of 0.179. Over one\u2010third of patients presenting to an outpatient spine center were found to have limited HL. While this algorithm is far from being used in clinical practice, ML algorithms offer a potential opportunity for identifying patients at risk for limited HL without administering in\u2010person HL assessments. This could possibly enable screening and early intervention to mitigate the potential negative consequences of limited HL without taxing the existing clinical workflow.<\/jats:p><\/jats:sec>","DOI":"10.1111\/jep.14248","type":"journal-article","created":{"date-parts":[[2024,11,22]],"date-time":"2024-11-22T03:42:42Z","timestamp":1732246962000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Development of Machine Learning Algorithms for Identifying Patients With Limited Health Literacy"],"prefix":"10.1111","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2770-5655","authenticated-orcid":false,"given":"Dylan","family":"Koole","sequence":"first","affiliation":[{"name":"Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA"},{"name":"Department of Orthopaedic Surgery, Leiden University Medical Center Leiden University Leiden The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3854-1470","authenticated-orcid":false,"given":"Oscar","family":"Shen","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4959-6517","authenticated-orcid":false,"given":"Amanda","family":"Lans","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA"},{"name":"Department of Orthopaedic Surgery, University Medical Center Utrecht Utrecht University Utrecht The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8980-5707","authenticated-orcid":false,"given":"Tom M.","family":"de Groot","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA"},{"name":"Department of Orthopaedic Surgery, University Medical Center Groningen University of Groningen Groningen The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8105-6660","authenticated-orcid":false,"given":"J. J.","family":"Verlaan","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, University Medical Center Utrecht Utrecht University Utrecht The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1156-3864","authenticated-orcid":false,"given":"J. H.","family":"Schwab","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2024,11,22]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1525-1497.2006.00540.x"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.spinee.2022.11.001"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6963-13-319"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.spinee.2018.10.012"},{"key":"e_1_2_10_6_1","doi-asserted-by":"publisher","DOI":"10.1111\/acem.13244"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1001\/archinte.167.14.1503"},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-155-2-201107190-00005"},{"key":"e_1_2_10_9_1","doi-asserted-by":"crossref","first-page":"382","DOI":"10.5435\/JAAOS-D-22-01026","article-title":"Health Literacy in Orthopaedics","volume":"31","author":"Lans A.","year":"2023","journal-title":"Journal of the American Academy of Orthopaedic Surgeons"},{"key":"e_1_2_10_10_1","doi-asserted-by":"publisher","DOI":"10.1097\/BRS.0000000000004495"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.spinee.2023.09.016"},{"key":"e_1_2_10_12_1","doi-asserted-by":"publisher","DOI":"10.11124\/JBISRIR-D-18-00023"},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12889-020-08991-0"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1553-7250(14)40008-4"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1097\/JHQ.0000000000000116"},{"key":"e_1_2_10_16_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.g7594"},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.5870"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02640361"},{"key":"e_1_2_10_19_1","doi-asserted-by":"publisher","DOI":"10.1370\/afm.405"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhsa.2015.01.010"},{"key":"e_1_2_10_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.spinee.2022.12.013"},{"key":"e_1_2_10_22_1","unstructured":"U.S. Department of Agriculture and U.S. Department of Health and Human Services. 2020\u20102025 Dietary Guidelines for Americans.2020. Report No.: 9th Edition."},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMp1802313"},{"key":"e_1_2_10_24_1","doi-asserted-by":"publisher","DOI":"10.2202\/1547-7355.1792"},{"key":"e_1_2_10_25_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btr597"},{"key":"e_1_2_10_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-6849-3"},{"key":"e_1_2_10_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"e_1_2_10_28_1","unstructured":"J.Wainer \u201cComparison of 14 Different Families of Classification Algorithms on 115 Binary Datasets \u201d arXiv preprint arXiv:160600930. 2016."},{"key":"e_1_2_10_29_1","doi-asserted-by":"publisher","DOI":"10.1186\/1756-0500-4-299"},{"issue":"1","key":"e_1_2_10_30_1","first-page":"3133","article-title":"Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?","volume":"15","author":"Fern\u00e1ndez\u2010Delgado M.","year":"2014","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_10_31_1","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.106.672402"},{"key":"e_1_2_10_32_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiology.143.1.7063747"},{"key":"e_1_2_10_33_1","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehu207"},{"key":"e_1_2_10_34_1","doi-asserted-by":"publisher","DOI":"10.1097\/EDE.0b013e3181c30fb2"},{"key":"e_1_2_10_35_1","doi-asserted-by":"publisher","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.1177\/0272989X06295361"},{"key":"e_1_2_10_37_1","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2015.65.5654"},{"key":"e_1_2_10_38_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0212488"},{"key":"e_1_2_10_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.arth.2019.02.067"}],"container-title":["Journal of Evaluation in Clinical Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/jep.14248","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,17]],"date-time":"2025-07-17T02:29:51Z","timestamp":1752719391000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/jep.14248"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,22]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,2]]}},"alternative-id":["10.1111\/jep.14248"],"URL":"https:\/\/doi.org\/10.1111\/jep.14248","archive":["Portico"],"relation":{},"ISSN":["1356-1294","1365-2753"],"issn-type":[{"value":"1356-1294","type":"print"},{"value":"1365-2753","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,22]]},"assertion":[{"value":"2024-01-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e14248"}}