{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T16:35:03Z","timestamp":1792514103017,"version":"4.5.0"},"reference-count":39,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Missing data in electronic health records are highly prevalent and result in analytical concerns such as heterogeneous sources of bias and loss of statistical power. One simple analytic method for addressing missing or unknown covariate values is to treat missingness for a particular variable as a category onto itself, which we refer to as the missing indicator method. For cross-sectional analyses, recent work suggested that there was minimal benefit to the missing indicator method; however, it is unclear how this approach performs in the setting of longitudinal data, in which correlation among clustered repeated measures may be leveraged for potentially improved model performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objectives\">\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>This study aims to conduct a simulation study to evaluate whether the missing indicator method improved model performance and imputation accuracy for longitudinal data mimicking an application of developing a clinical prediction model for falls in older adults based on electronic health record data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We simulated a longitudinal binary outcome using mixed effects logistic regression that emulated a falls assessment at annual follow-up visits. Using multivariate imputation by chained equations, we simulated time-invariant predictors such as sex and medical history, as well as dynamic predictors such as physical function, BMI, and medication use. We induced missing data in predictors under scenarios that had both random (missing at random) and dependent missingness (missing not at random). We evaluated aggregate performance using the area under the receiver operating characteristic curve (AUROC) for models with and with no missing indicators as predictors, as well as complete case analysis, across simulation replicates. We evaluated imputation quality using normalized root-mean-square error for continuous variables and percent falsely classified for categorical variables.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Independent of the mechanism used to simulate missing data (missing at random or missing not at random), overall model performance via AUROC was similar regardless of whether missing indicators were included in the model. The root-mean-square error and percent falsely classified measures were similar for models including missing indicators versus those with no missing indicators. Model performance and imputation quality were similar regardless of whether the outcome was related to missingness. Imputation with or with no missing indicators had similar mean values of AUROC compared with complete case analysis, although complete case analysis had the largest range of values.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The results of this study suggest that the inclusion of missing indicators in longitudinal data modeling neither improves nor worsens overall performance or imputation accuracy. Future research is needed to address whether the inclusion of missing indicators is useful in prediction modeling with longitudinal data in different settings, such as high dimensional data analysis.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/64354","type":"journal-article","created":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T12:01:07Z","timestamp":1741867267000},"page":"e64354-e64354","source":"Crossref","is-referenced-by-count":6,"title":["Imputation and Missing Indicators for Handling Missing Longitudinal Data: Data Simulation Analysis Based on Electronic Health Record Data"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7051-2915","authenticated-orcid":false,"given":"Molly","family":"Ehrig","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0236-9015","authenticated-orcid":false,"given":"Garrett S","family":"Bullock","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7610-7977","authenticated-orcid":false,"given":"Xiaoyan Iris","family":"Leng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4447-6196","authenticated-orcid":false,"given":"Nicholas M","family":"Pajewski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0679-8730","authenticated-orcid":false,"given":"Jaime Lynn","family":"Speiser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"R1","unstructured":"Ehrenstein V Kharrazi H Lehmann H Taylor CO . Obtaining data from electronic health records. In: Tools and Technologies for Registry Interoperability, Registries for Evaluating Patient Outcomes: A User\u2019s Guide. 3rded. Agency for Healthcare Research and Quality; 2019."},{"issue":"3","key":"R2","doi-asserted-by":"publisher","DOI":"10.13063\/2327-9214.1035","article-title":"Strategies for handling missing data in electronic health record derived data","volume":"1","author":"Wells","journal-title":"EGEMS (Wash DC)"},{"issue":"2","key":"R3","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.0184","article-title":"Assessing missing data assumptions in EHR-based studies: a complex and underappreciated task","volume":"4","author":"Haneuse","journal-title":"JAMA Netw Open"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.03.009","article-title":"Strategies for handling missing clinical data for automated surgical site infection detection from the electronic health record","volume":"68","author":"Hu","journal-title":"J Biomed Inform"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","journal-title":"BMJ"},{"issue":"3","key":"R6","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1002\/bjs.9736","article-title":"Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD Statement","volume":"102","author":"Collins","journal-title":"Br J Surg"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1093\/icvts\/ivy102","article-title":"Statistical primer: how to deal with missing data in scientific research?","volume":"27","author":"Papageorgiou","journal-title":"Interact Cardiovasc Thorac Surg"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.3978\/j.issn.2305-5839.2015.12.38","article-title":"Missing data imputation: focusing on single imputation","volume":"4","author":"Zhang","journal-title":"Ann Transl Med"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-021-00516-9","article-title":"A survey on missing data in machine learning","volume":"8","author":"Emmanuel","journal-title":"J Big Data"},{"issue":"18","key":"R10","doi-asserted-by":"publisher","first-page":"1966","DOI":"10.1001\/jama.2015.15281","article-title":"Multiple imputation: a flexible tool for handling missing data","volume":"314","author":"Li","journal-title":"JAMA"},{"issue":"8","key":"R11","doi-asserted-by":"publisher","first-page":"1461","DOI":"10.1177\/09622802231165001","article-title":"Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: a simulation study","volume":"32","author":"Sisk","journal-title":"Stat Methods Med Res"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-022-01554-4","article-title":"Comparing single and multiple imputation strategies for harmonizing substance use data across HIV-related cohort studies","volume":"22","author":"Javanbakht","journal-title":"BMC Med Res Methodol"},{"issue":"11","key":"R13","doi-asserted-by":"publisher","first-page":"1265","DOI":"10.1503\/cmaj.110977","article-title":"Missing covariate data in clinical research: when and when not to use the missing-indicator method for analysis","volume":"184","author":"Groenwold","journal-title":"CMAJ"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Van Ness M Bosschieter TM Halpin-Gregorio R Udell M . The missing indicator method: from low to high dimensions. Presented at: KDD \u201923: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; Aug 6-10, 2023; Long Beach, CA. [doi: 10.1145\/3580305.3599911]","DOI":"10.1145\/3580305.3599911"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.01.009","article-title":"Clinical prediction models: diagnosis versus prognosis","volume":"132","author":"van Smeden","journal-title":"J Clin Epidemiol"},{"issue":"8","key":"R16","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","journal-title":"JAMA Intern Med"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/11605","article-title":"A new insight into missing data in intensive care unit patient profiles: observational study","volume":"7","author":"Sharafoddini","journal-title":"JMIR Med Inform"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2022.08.016","article-title":"Handling missing data in clinical research","volume":"151","author":"Heymans","journal-title":"J Clin Epidemiol"},{"issue":"11","key":"R19","doi-asserted-by":"publisher","first-page":"2074","DOI":"10.1002\/sim.8086","article-title":"Using simulation studies to evaluate statistical methods","volume":"38","author":"Morris","journal-title":"Stat Med"},{"key":"R20","unstructured":"Ehrig M . Missing indicator paper. GitHub. 2024. URL: https:\/\/github.com\/mehrig1\/Missing-Indicator-Paper [Accessed 01-03-2025]"},{"key":"R21","unstructured":"Leisch F Weingessel A Leisch MF . Package bindata. The Comprehensive R Archive Network. 2006. URL: https:\/\/cran.r-project.org\/web\/packages\/bindata\/ [Accessed 01-07-2024]"},{"key":"R22","unstructured":"Ripley B Venables B Bates DM Hornik K Gebhardt A Firth D et al. Package mass. The Comprehensive R Archive Network. 2013. URL: https:\/\/cran.r-project.org\/web\/packages\/MASS\/index.html [Accessed 01-07-2024]"},{"key":"R23","unstructured":"Wickham H Wickham MH . Package tidyverse. The Comprehensive R Archive Network. 2017. URL: https:\/\/cran.r-project.org\/web\/packages\/tidyverse\/index.html [Accessed 01-07-2024]"},{"key":"R24","unstructured":"Bates D Maechler M Bolker B Walker S Christensen RHB Singmann H et al. Package lme4. The Comprehensive R Archive Network. 2015. URL: https:\/\/cran.r-project.org\/web\/packages\/lme4\/index.html [Accessed 01-07-2024]"},{"issue":"13","key":"R25","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v082.i13","article-title":"LmerTest: tests in linear mixed effects models","volume":"2","author":"Kuznetsova","journal-title":"J Stat Softw"},{"key":"R26","unstructured":"Tierney NJ Cook DH . Expanding tidy data principles to facilitate missing data exploration, visualization and assessment of imputations. arXiv. Preprint posted online onSep7, 2018. [doi: 10.48550\/arXiv.1809.02264]"},{"key":"R27","unstructured":"Buuren S Groothuis-Oudshoorn K Robitzsch A Vink G Doove L Jolani S . Package mice. The Comprehensive R Archive Network. 2015. URL: https:\/\/cran.r-project.org\/web\/packages\/mice\/index.html [Accessed 01-07-2024]"},{"key":"R28","unstructured":"Bolker B Robinson D Menne D Gabry J Buerkner P . Package broom.mixed. The Comprehensive R Archive Network. 2019. URL: https:\/\/cran.r-project.org\/web\/packages\/broom.mixed\/index.html [Accessed 01-07-2024]"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-12-77","article-title":"pROC: an open-source package for R and S+ to analyze and compare ROC curves","volume":"12","author":"Robin","journal-title":"BMC Bioinformatics"},{"key":"R30","unstructured":"Signorell A Aho K Alfons A Anderegg N Aragon T Arachchige C et al. Package DescTools. The Comprehensive R Archive Network. 2021. URL: https:\/\/cran.r-project.org\/web\/\/packages\/DescTools\/index.html [Accessed 01-07-2024]"},{"key":"R31","unstructured":"Stekhoven DJ . Package missForest. The Comprehensive R Archive Network. 2013. URL: https:\/\/cran.r-project.org\/web\/\/packages\/\/missForest\/index.html [Accessed 01-07-2024]"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v103.i01","article-title":"Data and model summaries in R","volume":"103","author":"Arel-Bundock","journal-title":"J Stat Softw"},{"key":"R33","unstructured":"L\u00fcdecke D L\u00fcdecke MD . Package sjPlot. The Comprehensive R Archive Network. 2015. URL: https:\/\/cran.r-project.org\/web\/packages\/sjPlot\/index.html [Accessed 01-07-2024]"},{"key":"R34","unstructured":"Auguie B Antonov A Auguie MB . Package gridextra. The Comprehensive R Archive Network. 2017. URL: https:\/\/cran.r-project.org\/web\/packages\/gridExtra\/index.html [Accessed 01-07-2024]"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/10691898.2010.11889587","article-title":"Fun with the R Grid Package","volume":"18","author":"Zhou","journal-title":"J Stat Educ"},{"key":"R36","unstructured":"Fox J Weisberg S Adler D Bates D Baud-Bovy G Ellison S et al. Package car. The Comprehensive R Archive Network. 2012. URL: https:\/\/cran.r-project.org\/web\/packages\/car\/index.html [Accessed 01-07-2024]"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1016\/j.archger.2023.104940","article-title":"Factors associated with falls in older adults: a secondary analysis of a 12-month randomized controlled trial","volume":"108","author":"Hsieh","journal-title":"Arch Gerontol Geriatr"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1093\/bioinformatics\/btr597","article-title":"MissForest--non-parametric missing value imputation for mixed-type data","volume":"28","author":"Stekhoven","journal-title":"Bioinformatics"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-01068-x","article-title":"Multiple imputation with missing indicators as proxies for unmeasured variables: simulation study","volume":"20","author":"Sperrin","journal-title":"BMC Med Res Methodol"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e64354","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:30:30Z","timestamp":1785537030000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e64354"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,13]]},"references-count":39,"URL":"https:\/\/doi.org\/10.2196\/64354","relation":{"is-referenced-by":[{"id-type":"doi","id":"10.1186\/s12877-025-05961-2","asserted-by":"object"}]},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,13]]},"article-number":"v13i3e64354"}}