{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T14:39:58Z","timestamp":1785163198178,"version":"3.55.0"},"reference-count":20,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2018,4,27]],"date-time":"2018-04-27T00:00:00Z","timestamp":1524787200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1251151"],"award-info":[{"award-number":["1251151"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and reproducibility.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Based on existing best practices we propose a 5 step standardized framework for: (1) transparently defining the problem; (2) selecting suitable datasets; (3) constructing variables from the observational data; (4) learning the predictive model; and (5) validating the model performance. We implemented this framework as open-source software utilizing the Observational Medical Outcomes Partnership Common Data Model to enable convenient sharing of models and reproduction of model evaluation across multiple observational datasets. The software implementation contains default covariates and classifiers but the framework enables customization and extension.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>As a proof-of-concept, demonstrating the transparency and ease of model dissemination using the software, we developed prediction models for 21 different outcomes within a target population of people suffering from depression across 4 observational databases. All 84 models are available in an accessible online repository to be implemented by anyone with access to an observational database in the Common Data Model format.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The proof-of-concept study illustrates the framework\u2019s ability to develop reproducible models that can be readily shared and offers the potential to perform extensive external validation of models, and improve their likelihood of clinical uptake. In future work the framework will be applied to perform an \u201call-by-all\u201d prediction analysis to assess the observational data prediction domain across numerous target populations, outcomes and time, and risk settings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocy032","type":"journal-article","created":{"date-parts":[[2018,3,16]],"date-time":"2018-03-16T16:09:22Z","timestamp":1521216562000},"page":"969-975","source":"Crossref","is-referenced-by-count":171,"title":["Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data"],"prefix":"10.1093","volume":"25","author":[{"given":"Jenna M","family":"Reps","sequence":"first","affiliation":[{"name":"Janssen Research and Development, Raritan, NJ, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martijn J","family":"Schuemie","sequence":"additional","affiliation":[{"name":"Janssen Research and Development, Raritan, NJ, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc A","family":"Suchard","sequence":"additional","affiliation":[{"name":"Department of Biomathematics, UCLA School of Medicine, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick B","family":"Ryan","sequence":"additional","affiliation":[{"name":"Janssen Research and Development, Raritan, NJ, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter R","family":"Rijnbeek","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics, Erasmus University Medical Center, Rotterdam,The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,4,27]]},"reference":[{"key":"2020110612224614600_ocy032-B1","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1186\/1741-7015-9-103","article-title":"Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting","volume":"9","author":"Collins","year":"2011","journal-title":"BMC Med"},{"key":"2020110612224614600_ocy032-B2","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.jclinepi.2012.06.020","article-title":"A systematic review finds prediction models for chronic kidney were poorly reported and often developed using inappropriate methods","volume":"66","author":"Collins","year":"2013","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2020110612224614600_ocy032-B3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2288-14-40","article-title":"External validation of multivariable prediction models: a systematic review of methodological conduct and reporting","volume":"14","author":"Collins","year":"2014","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"2020110612224614600_ocy032-B4","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review","volume":"24","author":"Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020110612224614600_ocy032-B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12916-014-0241-z","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement","volume":"13","author":"Collins","year":"2015","journal-title":"BMC Med"},{"key":"2020110612224614600_ocy032-B6","doi-asserted-by":"crossref","first-page":"e5595","DOI":"10.1136\/bmj.e5595","article-title":"Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes","volume":"346","author":"Hemingway","year":"2013","journal-title":"BMJ"},{"issue":"2","key":"2020110612224614600_ocy032-B7","doi-asserted-by":"crossref","first-page":"e1001380","DOI":"10.1371\/journal.pmed.1001380","article-title":"Prognosis Research Strategy (PROGRESS) 2: prognostic factor research","volume":"10","author":"Riley","year":"2013","journal-title":"PLoS Med"},{"issue":"2","key":"2020110612224614600_ocy032-B8","doi-asserted-by":"crossref","first-page":"e1001381","DOI":"10.1371\/journal.pmed.1001381","article-title":"Prognosis Research Strategy (PROGRESS) 3: prognostic model research","volume":"10","author":"Steyerberg","year":"2013","journal-title":"PLoS Med"},{"key":"2020110612224614600_ocy032-B9","doi-asserted-by":"crossref","first-page":"e5793","DOI":"10.1136\/bmj.e5793","article-title":"Prognosis research strategy (PROGRESS) 4: stratified medicine research","volume":"346","author":"Hingorani","year":"2013","journal-title":"BMJ"},{"key":"2020110612224614600_ocy032-B10","first-page":"574","article-title":"Observational health data sciences and informatics (OHDSI): opportunities for observational researchers","volume":"216","author":"Hripcsak","year":"2015","journal-title":"Stud Health Technol Inform"},{"issue":"1","key":"2020110612224614600_ocy032-B11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1136\/amiajnl-2011-000376","article-title":"Validation of a common data model for active safety surveillance research","volume":"19","author":"Overhage","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"27","key":"2020110612224614600_ocy032-B12","doi-asserted-by":"crossref","first-page":"7329","DOI":"10.1073\/pnas.1510502113","article-title":"Characterizing treatment pathways at scale using the OHDSI network","volume":"113","author":"Hripcsak","year":"2016","journal-title":"PNAS"},{"key":"2020110612224614600_ocy032-B13","volume-title":"SqlRender: Rendering Parameterized SQL and Translation to Dialects. R Package Version 1.2.0","author":"Schuemie","year":"2017"},{"key":"2020110612224614600_ocy032-B14","author":"Schuemie","year":"2017"},{"issue":"1","key":"2020110612224614600_ocy032-B15","first-page":"10","article-title":"Massive parallelization of serial inference algorithms for complex generalized linear models","volume":"23","author":"Suchard","year":"2013","journal-title":"ACM Transact Model Comput Simulation"},{"issue":"10","key":"2020110612224614600_ocy032-B16","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1007\/s00392-017-1123-0","article-title":"Clinical scores for outcomes of rhythm control or arrhythmia progression in patients with atrial fibrillation: a systematic review","volume":"106","author":"Deng","year":"2017","journal-title":"Clin Res Cardiol"},{"key":"2020110612224614600_ocy032-B17","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1016\/S0140-6736(09)60443-8","article-title":"Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study","volume":"373.9665","author":"Schnabel","year":"2009","journal-title":"Lancet"},{"key":"2020110612224614600_ocy032-B18","doi-asserted-by":"crossref","first-page":"1903","DOI":"10.1016\/j.ehj.2003.09.002","article-title":"Are the Framingham and PROCAM coronary heart disease risk functions applicable to different European populations? The PRIME Study","volume":"24.21","author":"Empana","year":"2003","journal-title":"Eur Heart J"},{"key":"2020110612224614600_ocy032-B19","doi-asserted-by":"crossref","first-page":"e0174944","DOI":"10.1371\/journal.pone.0174944","article-title":"Can machine-learning improve cardiovascular risk prediction using routine clinical data?","volume":"12.4","author":"Weng","year":"2017","journal-title":"PLoS One"},{"issue":"3","key":"2020110612224614600_ocy032-B20","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1093\/jamia\/ocu023","article-title":"Feasibility and utility of applications of the common data model to multiple, disparate observational health databases","volume":"22","author":"Voss","year":"2015","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/969\/34150768\/ocy032.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/969\/34150768\/ocy032.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T12:24:02Z","timestamp":1604665442000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/8\/969\/4989437"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,4,27]]},"references-count":20,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2018,4,27]]},"published-print":{"date-parts":[[2018,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy032","relation":{"has-review":[{"id-type":"doi","id":"10.3410\/f.733125620.793551678","asserted-by":"object"}]},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,8]]},"published":{"date-parts":[[2018,4,27]]}}}