{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T12:31:21Z","timestamp":1780576281434,"version":"3.54.1"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T00:00:00Z","timestamp":1736467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01 HL167858"],"award-info":[{"award-number":["R01 HL167858"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01 HL169954"],"award-info":[{"award-number":["R01 HL169954"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01 LM006910"],"award-info":[{"award-number":["R01 LM006910"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Propose a framework to empirically evaluate and report validity of findings from observational studies using pre-specified objective diagnostics, increasing trust in real-world evidence (RWE).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>The framework employs objective diagnostic measures to assess the appropriateness of study designs, analytic assumptions, and threats to validity in generating reliable evidence addressing causal questions. Diagnostic evaluations should be interpreted before the unblinding of study results or, alternatively, only unblind results from analyses that pass pre-specified thresholds. We provide a conceptual overview of objective diagnostic measures and demonstrate their impact on the validity of RWE from a large-scale comparative new-user study of various antihypertensive medications. We evaluated expected absolute systematic error (EASE) before and after applying diagnostic thresholds, using a large set of negative control outcomes.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Applying objective diagnostics reduces bias and improves evidence reliability in observational studies. Among 11\u00a0716 analyses (EASE\u2009=\u20090.38), 13.9% met pre-specified diagnostic thresholds which reduced EASE to zero. Objective diagnostics provide a comprehensive and empirical set of tests that increase confidence when passed and raise doubts when failed.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The increasing use of real-world data presents a scientific opportunity; however, the complexity of the evidence generation process poses challenges for understanding study validity and trusting RWE. Deploying objective diagnostics is crucial to reducing bias and improving reliability in RWE generation. Under ideal conditions, multiple study designs pass diagnostics and generate consistent results, deepening understanding of causal relationships. Open-source, standardized programs can facilitate implementation of diagnostic analyses.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Objective diagnostics are a valuable addition to the RWE generation process.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae317","type":"journal-article","created":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T04:38:16Z","timestamp":1736483896000},"page":"518-525","source":"Crossref","is-referenced-by-count":13,"title":["Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5682-3688","authenticated-orcid":false,"given":"Mitchell M","family":"Conover","sequence":"first","affiliation":[{"name":"Coordinating Center, Observational Health Data Science and Informatics , New York City, NY 10032,","place":["United States"]},{"name":"Observational Health Data Analytics, Johnson & Johnson , Titusville, NJ 08560,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick B","family":"Ryan","sequence":"additional","affiliation":[{"name":"Coordinating Center, Observational Health Data Science and Informatics , New York City, NY 10032,","place":["United States"]},{"name":"Observational Health Data Analytics, Johnson & Johnson , Titusville, NJ 08560,","place":["United States"]},{"name":"Department of Biomedical Informatics, Columbia University Medical Center , New York City, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0835-0788","authenticated-orcid":false,"given":"Yong","family":"Chen","sequence":"additional","affiliation":[{"name":"Coordinating Center, Observational Health Data Science and Informatics , New York City, NY 10032,","place":["United States"]},{"name":"Department of Biostatistics, Epidemiology and Informatics (DBEI), The Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9818-479X","authenticated-orcid":false,"given":"Marc A","family":"Suchard","sequence":"additional","affiliation":[{"name":"Coordinating Center, Observational Health Data Science and Informatics , New York City, NY 10032,","place":["United States"]},{"name":"Department of Biostatistics, University of California, Los Angeles , Los Angeles, CA 90095,","place":["United States"]},{"name":"VA Informatics and Computing Infrastructure, US Department of Veterans Affairs , Salt Lake City, UT 20420,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George","family":"Hripcsak","sequence":"additional","affiliation":[{"name":"Coordinating Center, Observational Health Data Science and Informatics , New York City, NY 10032,","place":["United States"]},{"name":"Department of Biomedical Informatics, Columbia University Medical Center , New York City, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martijn J","family":"Schuemie","sequence":"additional","affiliation":[{"name":"Coordinating Center, Observational Health Data Science and Informatics , New York City, NY 10032,","place":["United States"]},{"name":"Observational Health Data Analytics, Johnson & Johnson , Titusville, NJ 08560,","place":["United States"]},{"name":"Department of Biostatistics, University of California, Los Angeles , Los Angeles, CA 90095,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,1,10]]},"reference":[{"key":"2025021811393067700_ocae317-B1","doi-asserted-by":"publisher","first-page":"20170356","DOI":"10.1098\/rsta.2017.0356","article-title":"Improving reproducibility by using high-throughput observational studies with empirical calibration","volume":"376","author":"Schuemie","year":"2018","journal-title":"Philos Trans A Math Phys Eng Sci"},{"key":"2025021811393067700_ocae317-B2","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1001\/jama.2012.8144","article-title":"The importance of potential studies that have not existed and registration of observational data sets","volume":"308","author":"Ioannidis","year":"2012","journal-title":"JAMA"},{"key":"2025021811393067700_ocae317-B3","doi-asserted-by":"publisher","first-page":"546","DOI":"10.1007\/s11606-010-1609-1","article-title":"Healthy user and related biases in observational studies of preventive interventions: a primer for physicians","volume":"26","author":"Shrank","year":"2011","journal-title":"J Gen Intern Med"},{"key":"2025021811393067700_ocae317-B4","doi-asserted-by":"publisher","first-page":"e004939","DOI":"10.1161\/CIRCOUTCOMES.118.004939","article-title":"Exposure misclassification in observational studies: setting new standards","volume":"11","author":"Secemsky","year":"2018","journal-title":"Circ Cardiovasc Qual Outcomes"},{"key":"2025021811393067700_ocae317-B5","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1016\/S0140-6736(02)07451-2","article-title":"Bias and causal associations in observational research","volume":"359","author":"Grimes","year":"2002","journal-title":"Lancet"},{"key":"2025021811393067700_ocae317-B6","year":"2022"},{"key":"2025021811393067700_ocae317-B7","doi-asserted-by":"publisher","first-page":"1845","DOI":"10.1001\/jama.2024.7741","article-title":"Causal inference about the effects of interventions from observational studies in medical journals","volume":"331","author":"Dahabreh","year":"2024","journal-title":"JAMA"},{"key":"2025021811393067700_ocae317-B8","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1001\/jama.2024.8107","article-title":"What does the proposed causal inference framework for observational studies mean for JAMA and the JAMA network journals?","volume":"331","author":"Flanagin","year":"2024","journal-title":"JAMA"},{"key":"2025021811393067700_ocae317-B9","doi-asserted-by":"publisher","first-page":"1331","DOI":"10.1093\/jamia\/ocaa103","article-title":"Principles of large-scale evidence generation and evaluation across a network of databases (LEGEND)","volume":"27","author":"Schuemie","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2025021811393067700_ocae317-B10","doi-asserted-by":"publisher","first-page":"1268","DOI":"10.1093\/jamia\/ocaa124","article-title":"Large-scale evidence generation and evaluation across a network of databases (LEGEND): assessing validity using hypertension as a case study","volume":"27","author":"Schuemie","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2025021811393067700_ocae317-B11","doi-asserted-by":"publisher","first-page":"1816","DOI":"10.1016\/S0140-6736(19)32317-7","article-title":"Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis","volume":"394","author":"Suchard","year":"2019","journal-title":"Lancet"},{"key":"2025021811393067700_ocae317-B12","author":"Blacketer","year":"2023"},{"key":"2025021811393067700_ocae317-B13","doi-asserted-by":"publisher","DOI":"10.1101\/2023.06.28.23291982","article-title":"CohortDiagnostics: phenotype evaluation across a network of observational data sources using population-level characterization","author":"Rao","year":"2023"},{"key":"2025021811393067700_ocae317-B14","doi-asserted-by":"publisher","first-page":"104177","DOI":"10.1016\/j.jbi.2022.104177","article-title":"PheValuator 2.0: methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation","volume":"135","author":"Swerdel","year":"2022","journal-title":"J Biomed Inform"},{"key":"2025021811393067700_ocae317-B15","doi-asserted-by":"publisher","first-page":"e70042","DOI":"10.1002\/pds.70042","article-title":"Standardised and reproducible phenotyping using distributed analytics and tools in the data analysis and real world interrogation network (DARWIN EU)","volume":"33","author":"Dernie","year":"2024","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"2025021811393067700_ocae317-B16","doi-asserted-by":"publisher","first-page":"2618","DOI":"10.1002\/sim.2477","article-title":"Sample sizes for self-controlled case series studies","volume":"25","author":"Musonda","year":"2006","journal-title":"Stat Med"},{"key":"2025021811393067700_ocae317-B17","doi-asserted-by":"publisher","first-page":"499","DOI":"10.2307\/2531021","article-title":"Sample-size formula for the proportional-hazards regression model","volume":"39","author":"Schoenfeld","year":"1983","journal-title":"Biometrics"},{"key":"2025021811393067700_ocae317-B18","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1016\/j.jclinepi.2021.08.028","article-title":"Causal analyses of existing databases: no power calculations required","volume":"144","author":"Hern\u00e1n","year":"2022","journal-title":"J Clin Epidemiol"},{"key":"2025021811393067700_ocae317-B19","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1016\/j.jclinepi.2021.09.024","article-title":"Sample size considerations are needed for the causal analyses of existing databases","volume":"141","author":"Mansournia","year":"2022","journal-title":"J Clin Epidemiol"},{"key":"2025021811393067700_ocae317-B20","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1016\/j.jclinepi.2021.09.026","article-title":"Causal analyses of existing databases: the importance of understanding what can be achieved with your data before analysis (commentary on Hern\u00e1n)","volume":"142","author":"Morris","year":"2022","journal-title":"J Clin Epidemiol"},{"key":"2025021811393067700_ocae317-B21","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.jclinepi.2021.11.039","article-title":"Causal analysis of existing databases: no power calculations required. Responses to Campbell, Morris and Mansournia","volume":"144","author":"Hern\u00e1n","year":"2022","journal-title":"J Clin Epidemiol"},{"key":"2025021811393067700_ocae317-B22","doi-asserted-by":"publisher","first-page":"757","DOI":"10.7326\/0003-4819-127-8_Part_2-199710151-00064","article-title":"Estimating causal effects from large data sets using propensity scores","volume":"127","author":"Rubin","year":"1997","journal-title":"Ann Intern Med"},{"key":"2025021811393067700_ocae317-B23","doi-asserted-by":"publisher","first-page":"1718","DOI":"10.1002\/sim.8866","article-title":"Using propensity scores to estimate effects of treatment initiation decisions: state of the science","volume":"40","author":"Webster-Clark","year":"2021","journal-title":"Stat Med"},{"key":"2025021811393067700_ocae317-B24","doi-asserted-by":"publisher","first-page":"11","DOI":"10.2147\/CER.S40357","article-title":"A tool for assessing the feasibility of comparative effectiveness research","volume":"2013","author":"Walker","year":"2013","journal-title":"CER"},{"key":"2025021811393067700_ocae317-B25","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1093\/aje\/kwp436","article-title":"Invited commentary: positivity in practice","volume":"171","author":"Westreich","year":"2010","journal-title":"Am J Epidemiol"},{"key":"2025021811393067700_ocae317-B26","doi-asserted-by":"publisher","first-page":"1218","DOI":"10.1002\/pds.1674","article-title":"Assessing balance in measured baseline covariates when using many-to-one matching on the propensity-score","volume":"17","author":"Austin","year":"2008","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"2025021811393067700_ocae317-B27","doi-asserted-by":"publisher","first-page":"3083","DOI":"10.1002\/sim.3697","article-title":"Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples","volume":"28","author":"Austin","year":"2009","journal-title":"Stat Med"},{"key":"2025021811393067700_ocae317-B28","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1016\/S0895-4356(00)00321-8","article-title":"Validating recommendations for coronary angiography following acute myocardial infarction in the elderly","volume":"54","author":"Normand","year":"2001","journal-title":"J Clin Epidemiol"},{"key":"2025021811393067700_ocae317-B29","doi-asserted-by":"publisher","first-page":"734","DOI":"10.1002\/sim.2580","article-title":"A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study","volume":"26","author":"Austin","year":"2007","journal-title":"Stat Med"},{"key":"2025021811393067700_ocae317-B30","doi-asserted-by":"publisher","DOI":"10.1101\/2024.04.23.24306230","article-title":"Assessing covariate balance with small sample sizes","author":"Hripcsak","year":"2024"},{"key":"2025021811393067700_ocae317-B31","doi-asserted-by":"publisher","first-page":"478","DOI":"10.3102\/1076998614558486","article-title":"How generalizable is your experiment? An index for comparing experimental samples and populations","volume":"39","author":"Tipton","year":"2014","journal-title":"J Educ Behav Stat"},{"key":"2025021811393067700_ocae317-B32","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1023\/A:1020363010465","article-title":"Using propensity scores to help design observational studies: application to the tobacco litigation","volume":"2","author":"Rubin","year":"2001","journal-title":"Health Serv Outcomes Res Methodol"},{"key":"2025021811393067700_ocae317-B33","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1001\/jama.2012.96867","article-title":"Prespecified falsification end points: can they validate true observational associations?","volume":"309","author":"Prasad","year":"2013","journal-title":"JAMA"},{"key":"2025021811393067700_ocae317-B34","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1097\/EDE.0b013e3181d61eeb","article-title":"Negative controls: a tool for detecting confounding and bias in observational studies","volume":"21","author":"Lipsitch","year":"2010","journal-title":"Epidemiology"},{"key":"2025021811393067700_ocae317-B35","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1214\/16-STS558","article-title":"On negative outcome control of unobserved confounding as a generalization of difference-in-differences","volume":"31","author":"Sofer","year":"2016","journal-title":"Stat Sci"},{"key":"2025021811393067700_ocae317-B36","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1002\/sim.5925","article-title":"Interpreting observational studies: why empirical calibration is needed to correct p-values","volume":"33","author":"Schuemie","year":"2014","journal-title":"Stat Med"},{"key":"2025021811393067700_ocae317-B37","doi-asserted-by":"publisher","first-page":"2571","DOI":"10.1073\/pnas.1708282114","article-title":"Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data","volume":"115","author":"Schuemie","year":"2018","journal-title":"Proc Natl Acad Sci USA"},{"key":"2025021811393067700_ocae317-B38","doi-asserted-by":"publisher","first-page":"2005","DOI":"10.1093\/ije\/dyy120","article-title":"Evaluating large-scale propensity score performance through real-world and synthetic data experiments","volume":"47","author":"Tian","year":"2018","journal-title":"Int J Epidemiol"},{"key":"2025021811393067700_ocae317-B39","doi-asserted-by":"publisher","first-page":"966","DOI":"10.3233\/SHTI231108","article-title":"Health-analytics data to evidence suite (HADES): open-source software for observational research","volume":"310","author":"Schuemie","year":"2024","journal-title":"Stud Health Technol Inform"},{"key":"2025021811393067700_ocae317-B40","doi-asserted-by":"publisher","first-page":"1685","DOI":"10.1002\/sim.6058","article-title":"Metrics for covariate balance in cohort studies of causal effects","volume":"33","author":"Franklin","year":"2014","journal-title":"Stat Med"},{"key":"2025021811393067700_ocae317-B41","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.1007\/s11606-013-2755-z","article-title":"Six persistent research misconceptions","volume":"29","author":"Rothman","year":"2014","journal-title":"J Gen Intern Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/3\/518\/61399098\/ocae317.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/3\/518\/61399098\/ocae317.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,18]],"date-time":"2025-02-18T11:39:45Z","timestamp":1739878785000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/3\/518\/7950905"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,10]]},"references-count":41,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,1,10]]},"published-print":{"date-parts":[[2025,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae317","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,3]]},"published":{"date-parts":[[2025,1,10]]}}}