{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T15:51:49Z","timestamp":1791474709025,"version":"4.3.4"},"reference-count":45,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,3,7]],"date-time":"2021-03-07T00:00:00Z","timestamp":1615075200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>The most accurate prognostic approach for follicular lymphoma (FL), progression of disease at 24 months (POD24), requires two years\u2019 observation after initiating first-line therapy (L1) to predict outcomes. We applied machine learning to structured electronic health record (EHR) data to predict individual survival at L1 initiation. We grouped 523 observations and 1933 variables from a nationwide cohort of FL patients diagnosed 2006\u20132014 in the Veterans Health Administration into traditionally used prognostic variables (\u201ccurated\u201d), commonly measured labs (\u201clabs\u201d), and International Classification of Diseases diagnostic codes (\u201cICD\u201d) sets. We compared performance of random survival forests (RSF) vs. traditional Cox model using four datasets: curated, curated + labs, curated + ICD, and curated + ICD + labs, also using Cox on curated + POD24. We evaluated variable importance and partial dependence plots with area under the receiver operating characteristic curve (AUC). RSF with curated + labs performed best, with mean AUC 0.73 (95% CI: 0.71\u20130.75). It approximated, but did not surpass, Cox with POD24 (mean AUC 0.74 [95% CI: 0.71\u20130.77]). RSF using EHR data achieved better performance than traditional prognostic variables, setting the foundation for the incorporation of our algorithm into the EHR. It also provides for possible future scenarios in which clinicians could be provided an EHR-based tool which approximates the predictive ability of the most accurate known indicator, using information available 24 months earlier.<\/jats:p>","DOI":"10.3390\/ijerph18052679","type":"journal-article","created":{"date-parts":[[2021,3,7]],"date-time":"2021-03-07T20:20:43Z","timestamp":1615148443000},"page":"2679","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Predicting Survival in Veterans with Follicular Lymphoma Using Structured Electronic Health Record Information and Machine Learning"],"prefix":"10.3390","volume":"18","author":[{"given":"Chunyang","family":"Li","sequence":"first","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vikas","family":"Patil","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2946-9211","authenticated-orcid":false,"given":"Kelli M.","family":"Rasmussen","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christina","family":"Yong","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hsu-Chih","family":"Chien","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Debbie","family":"Morreall","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3981-1217","authenticated-orcid":false,"given":"Jeffrey","family":"Humpherys","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian C.","family":"Sauer","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zachary","family":"Burningham","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmad S.","family":"Halwani","sequence":"additional","affiliation":[{"name":"Veritas, Division of Epidemiology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA"},{"name":"George E. Wahlen Veterans Health Administration, Salt Lake City, UT 84148, USA"},{"name":"Hematology &amp; Hematologic Malignancies, Huntsman Cancer Institute, Salt Lake City, UT 84112, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"69","DOI":"10.3322\/caac.20107","article-title":"Global cancer statistics","volume":"61","author":"Jemal","year":"2011","journal-title":"CA Cancer J. Clin."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1080\/02841860500531682","article-title":"Epidemiology and etiology of non-Hodgkin lymphoma\u2014A review","volume":"45","year":"2006","journal-title":"Acta Oncol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s00277-018-3501-8","article-title":"Burden of illness of follicular lymphoma and marginal zone lymphoma","volume":"98","author":"Monga","year":"2019","journal-title":"Ann. Hematol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1182\/asheducation-2017.1.358","article-title":"Follicular lymphoma: Are we ready for a risk-adapted approach?","volume":"2017","author":"Kahl","year":"2017","journal-title":"Hematol. Am. Soc. Hematol. Educ. Program"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"443","DOI":"10.3322\/caac.21357","article-title":"2016 US lymphoid malignancy statistics by World Health Organization subtypes","volume":"66","author":"Teras","year":"2016","journal-title":"CA Cancer J. Clin."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1002\/ajh.24492","article-title":"Early event status informs subsequent outcome in newly diagnosed follicular lymphoma","volume":"91","author":"Maurer","year":"2016","journal-title":"Am. J. Hematol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.1016\/S0140-6736(12)61763-2","article-title":"Bendamustine plus rituximab versus CHOP plus rituximab as first-line treament for patients with indolent and mantle-cell lymphomas: An open-label, multicentre, randomised, phase 3 non-inferiority trial","volume":"381","author":"Rummel","year":"2013","journal-title":"Lancet"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"984","DOI":"10.1200\/JCO.18.00605","article-title":"First-line treatment of patients with indolent non-Hodgkin lymphoma or mantle-cell lymphoma with bendamustine plus rituximab versus R-CHOP or R-CVP: Results of the BRIGHT 5-year follow-up study","volume":"37","author":"Flinn","year":"2019","journal-title":"J. Clin. Oncol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2094","DOI":"10.1093\/annonc\/mdx189","article-title":"Unmet needs in the first-line treatment of follicular lymphoma","volume":"28","author":"Casulo","year":"2017","journal-title":"Ann. Oncol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e9","DOI":"10.1053\/j.seminoncol.2010.04.001","article-title":"The importance of identifying and validating prognostic factors in oncology","volume":"37","author":"Halabi","year":"2010","journal-title":"Semin. Oncol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.1182\/blood-2003-12-4434","article-title":"Follicular Lymphoma International Prognostic Index","volume":"104","author":"Roy","year":"2004","journal-title":"Blood"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1697","DOI":"10.1093\/annonc\/mdp053","article-title":"Validation, revision and extension of the Follicular Lymphoma International Prognostic Index (FLIPI) in a population-based setting","volume":"20","author":"Steyerberg","year":"2009","journal-title":"Ann. Oncol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1038\/bjc.1982.62","article-title":"A prognostic index in primary breast cancer","volume":"45","author":"Haybittle","year":"1982","journal-title":"Br. J. Cancer"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1136\/gut.51.1.65","article-title":"Identification of objective pathological prognostic determinants and models of prognosis in Dukes\u2018 B colon cancer","volume":"51","author":"Petersen","year":"2002","journal-title":"Gut"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2516","DOI":"10.1200\/JCO.2014.59.7534","article-title":"Early relapse of follicular lymphoma after rituximab plus cyclophosphamide, doxorubicin, vincristine, and prednisone defines patients at high risk for death: An analysis from the National LymphoCare Study","volume":"33","author":"Casulo","year":"2015","journal-title":"J. Clin. Oncol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1216","DOI":"10.1056\/NEJMp1606181","article-title":"Predicting the future-big data, machine learning, and clinical medicine","volume":"375","author":"Obermeyer","year":"2016","journal-title":"NEJM"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1214\/08-AOAS169","article-title":"Random survival forests","volume":"2","author":"Ishwaran","year":"2008","journal-title":"Ann. Appl. Stat."},{"key":"ref_19","first-page":"85","article-title":"A selective review on random survival forests for high dimensional data","volume":"36","author":"Wang","year":"2017","journal-title":"Quant. Biosci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"558","DOI":"10.1002\/sim.7803","article-title":"Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival","volume":"38","author":"Ishwaran","year":"2018","journal-title":"Stat. Med."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1406","DOI":"10.1093\/ije\/dyw145","article-title":"Random Survival Forest in practice: A method for modelling complex metabolomics data in time to event analysis","volume":"45","author":"Dietrich","year":"2016","journal-title":"Int. J. Epidemiol."},{"key":"ref_22","unstructured":"El-Shaarawi, A.H., Piegorsch, W.W., and Zhang, H.H. (2006). Regression trees. Encyclopedia of Environmentrics, Wiley Online Library."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e004007","DOI":"10.1136\/bmjopen-2013-004007","article-title":"Machine-learning prediction of cancer survival: A retrospective study using electronic administrative records and a cancer registry","volume":"4","author":"Gupta","year":"2014","journal-title":"BMJ Open"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e180926","DOI":"10.1001\/jamanetworkopen.2018.0926","article-title":"Development and application of a machine learning approach to assess short-term mortality risk among patients with cancer starting chemotherapy","volume":"1","author":"Elfiky","year":"2018","journal-title":"JAMA Netw. Open"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e1915997","DOI":"10.1001\/jamanetworkopen.2019.15997","article-title":"Machine learning approaches to predict 6-month mortality among patients with cancer","volume":"2","author":"Parikh","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"7537","DOI":"10.1002\/cam4.3420","article-title":"Maintenance rituximab in Veterans with follicular lymphoma","volume":"9","author":"Halwani","year":"2020","journal-title":"Cancer Med."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1002\/sam.11348","article-title":"Random forest missing data algorithms","volume":"10","author":"Tang","year":"2017","journal-title":"Stat. Anal. Data Min."},{"key":"ref_28","unstructured":"Ishwaran, H., Kogalur, U.B., and Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC) (2020, November 10). CRAN R-Project. Available online: https:\/\/cran.r-project.org\/web\/packages\/randomForestSRC\/randomForestSRC.pdf."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/0360-3016(80)90027-9","article-title":"Improved five year survival after combined radiotherapy-chemotherapy for state I-II non-Hodgkin\u2018s lymphoma","volume":"6","author":"Monfardini","year":"1980","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1016\/j.ijrobp.2005.08.010","article-title":"Long-term outcome and mortality trends in early-stage, Grade 1-2 follicular lymphoma treated with radiation therapy","volume":"64","author":"Guadagnolo","year":"2006","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2957","DOI":"10.1182\/blood-2006-04-018218","article-title":"High numbers of tumor-infiltrating FOXP3-positive regulatory T cells are associated with improved overall survival in follicular lymphoma","volume":"108","author":"Carreras","year":"2006","journal-title":"Blood"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1002\/(SICI)1097-0258(19970228)16:4<385::AID-SIM380>3.0.CO;2-3","article-title":"The lasso method for variable selection in the Cox model","volume":"16","author":"Tibshirani","year":"1997","journal-title":"Stat. Med."},{"key":"ref_33","unstructured":"(2020, August 20). Stackexchange. Available online: https:\/\/stats.stackexchange.com\/questions\/36015\/prediction-in-cox-regression."},{"key":"ref_34","unstructured":"Tay, K., Simon, N., Friedman, J., Hastie, T., Tibshirani, R., and Naraimhan, B. (2021, February 11). Regularized Cox Regression. Available online: https:\/\/cran.r-project.org\/web\/packages\/glmnet\/vignettes\/Coxnet.pdf."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhang, C., and Ma, Y. (2012). Random forests. Ensemble Machine Learning, Springer.","DOI":"10.1007\/978-1-4419-9326-7"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Shi, Y., Graves, J.A., Garbett, S.P., Zhou, Z., Marathi, R., Wang, X., Harrell, F.E., Lasko, T.A., Denny, J.C., and Roden, D.M. (2019). A decision-theoretic approach to panel-based, preemptive genotyping. MDM Policy Prac., 4.","DOI":"10.1177\/2381468319864337"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Konerman, M.A., Beste, L.A., Van, T., Liu, B., Zhang, X., Zhu, J., Saini, S.D., Su, G.L., Nallamothu, B.K., and Ioannou, G.N. (2019). Machine learning models to predict disease progression among Veterans with hepatitis C virus. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0208141"},{"key":"ref_38","unstructured":"Gerds, T.A. (2009). Prediction Error Curves for Survival Models, CRAN. R package pec version 1.1.5."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v050.i11","article-title":"Evaluating random forests for survival analysis using prediction error curves","volume":"50","author":"Mogensen","year":"2012","journal-title":"J. Stat. Softw."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1002\/sim.4154","article-title":"On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data","volume":"30","author":"Uno","year":"2011","journal-title":"Stat. Med."},{"key":"ref_41","first-page":"419","article-title":"Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection","volume":"50","author":"Ruopp","year":"2008","journal-title":"Biomed. J."},{"key":"ref_42","unstructured":"Harrell, F.E. (2021, February 11). \u201cwith Contributions from Many Others\u201d. Harrell Miscellaneous. R Package Version 4.1-1. Available online: https:\/\/CRAN.R-project.org\/package=Hmisc."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1080\/10618600.2014.907095","article-title":"Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation","volume":"45","author":"Goldstein","year":"2015","journal-title":"J. Comp. Graph. Stat."},{"key":"ref_45","unstructured":"R Core Team (2021, February 11). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Available online: https:\/\/www.R-project.org."}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/18\/5\/2679\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:34:23Z","timestamp":1760160863000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/18\/5\/2679"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,7]]},"references-count":45,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["ijerph18052679"],"URL":"https:\/\/doi.org\/10.3390\/ijerph18052679","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,7]]}}}