{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T21:01:49Z","timestamp":1790888509707,"version":"4.1.0"},"reference-count":26,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T00:00:00Z","timestamp":1785801600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100004331","name":"Johnson and Johnson","doi-asserted-by":"publisher","award":["ICD 8593358"],"award-info":[{"award-number":["ICD 8593358"]}],"id":[{"id":"10.13039\/100004331","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000738","name":"Department of Veterans Affairs","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000738","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Janssen Research & Development and Gilead Sciences","id":[{"id":"https:\/\/ror.org\/05wae1a98","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large-scale propensity score (LSPS) models are increasingly used to control confounding in observational studies, but their reliability in small-sample settings is unclear. Small samples can limit a study\u2019s ability to estimate propensity scores accurately and achieve adequate covariate balance, raising concerns about insufficient confounding adjustment. Resulting in small datasets being excluded, despite potentially containing valid information.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We evaluated the balancing and bias reduction performance of LSPS models using six target-comparator pairs. To emulate a distributed data network analysis setting, we partitioned two large real-world data sources into smaller subsets. Within each subset, LSPS models were fit locally and evaluated on covariate balance, treatment effect bias and precision after empirical calibration, using real negative controls and synthetic positive controls. Performance was compared to a global LSPS model trained on the full dataset.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Across most target-comparator pairs, LSPS models substantially reduced bias and, when applying empirical calibration, improved precision. PS matching was more resilient to small sample sizes than PS stratification, achieving adequate balance even at n\u2009=\u2009500, while stratification sometimes failed at n\u2009=\u20094000. A slight bias increase was observed at the smallest sample sizes, though not universally. Standard balance diagnostics consistently failed below n\u2009=\u200920 000, while a recently proposed diagnostic accounting for chance imbalance did not.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LSPS models generally provide reliable bias reduction in small-sample settings, supporting their use in federated analyses. However, standard balance diagnostics may be misleading in small samples, and alternatives should be considered, such as significance checking. When LSPS fails to reduce bias adequately, additional adjustment strategies are required.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag133","type":"journal-article","created":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T11:49:42Z","timestamp":1784116182000},"page":"1992-2001","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating large-scale propensity score adjustment when sample size is small"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-7196-7147","authenticated-orcid":false,"given":"Fleur","family":"Vereijken","sequence":"first","affiliation":[{"name":"Department of Medical Informatics , Erasmus University Medical Center, Rotterdam, 3015 GD,","place":["The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jenna M","family":"Reps","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics , Erasmus University Medical Center, Rotterdam, 3015 GD,","place":["The Netherlands"]},{"name":"Observational Health Data Analytics, Janssen Research and Development , Titusville, NJ, 08560,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9818-479X","authenticated-orcid":false,"given":"Marc A","family":"Suchard","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, University of California , Los Angeles, Los Angeles, CA 90095-1772,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4356-4645","authenticated-orcid":false,"given":"Linying","family":"Zhang","sequence":"additional","affiliation":[{"name":"Data Science and Biostatistics, Washington University School of Medicine , Saint Louis, Missouri, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George","family":"Hripcsak","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0621-1979","authenticated-orcid":false,"given":"Peter R","family":"Rijnbeek","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics , Erasmus University Medical Center, Rotterdam, 3015 GD,","place":["The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7723-417X","authenticated-orcid":false,"given":"Ross D","family":"Williams","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics , Erasmus University Medical Center, Rotterdam, 3015 GD,","place":["The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0817-5361","authenticated-orcid":false,"given":"Martijn J","family":"Schuemie","sequence":"additional","affiliation":[{"name":"Observational Health Data Analytics, Janssen Research and Development , Titusville, NJ, 08560,","place":["United States"]},{"name":"Department of Biostatistics, University of California , Los Angeles, Los Angeles, CA 90095-1772,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,8,4]]},"reference":[{"key":"2026100116371924000_ocag133-B1","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1186\/s12874-022-01590-0","article-title":"Estimation of treatment effects in observational stroke care data: comparison of statistical approaches","volume":"22","author":"Amini","year":"2022","journal-title":"BMC Med Res Methodol"},{"key":"2026100116371924000_ocag133-B2","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1161\/01.CIR.70.5.767","article-title":"Current problems and future challenges in randomized clinical trials","volume":"70","author":"Feinstein","year":"1984","journal-title":"Circulation."},{"key":"2026100116371924000_ocag133-B3","doi-asserted-by":"crossref","first-page":"888","DOI":"10.7326\/0003-4819-123-11-199512010-00015","article-title":"Case-control study, meta-analysis, and bouillabaisse: putting the calcium antagonist scare into context","volume":"123","author":"Messerli","year":"1995","journal-title":"Ann Intern Med"},{"key":"2026100116371924000_ocag133-B4","doi-asserted-by":"crossref","first-page":"1151","DOI":"10.1136\/bmj.315.7116.1151","article-title":"Confounding and indication for treatment in evaluation of drug treatment for hypertension","volume":"315","author":"Grobbee","year":"1997","journal-title":"BMJ"},{"key":"2026100116371924000_ocag133-B5","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1080\/00273171.2011.568786","article-title":"An introduction to propensity score methods for reducing the effects of confounding in observational studies","volume":"46","author":"Austin","year":"2011","journal-title":"Multivariate Behav Res"},{"key":"2026100116371924000_ocag133-B6","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1146\/annurev-publhealth-031811-124610","article-title":"The methods of comparative effectiveness research","volume":"33","author":"Sox","year":"2012","journal-title":"Annu Rev Public Health"},{"key":"2026100116371924000_ocag133-B7","doi-asserted-by":"crossref","first-page":"90590","DOI":"10.5662\/wjm.v14.i1.90590","article-title":"Can propensity score matching replace randomized controlled trials?","volume":"14","author":"Liau","year":"2024","journal-title":"World J Methodol"},{"key":"2026100116371924000_ocag133-B8","doi-asserted-by":"crossref","first-page":"617","DOI":"10.3389\/fpsyg.2020.00617","article-title":"Using meta-analysis and propensity score methods to assess treatment effects toward evidence-based practice in extensive reading","volume":"11","author":"Hamada","year":"2020","journal-title":"Front Psychol"},{"key":"2026100116371924000_ocag133-B9","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1186\/s12874-021-01454-z","article-title":"Oversampling and replacement strategies in propensity score matching: a critical review focused on small sample size in clinical settings","volume":"21","author":"Bottigliengo","year":"2021","journal-title":"BMC Med Res Methodol"},{"key":"2026100116371924000_ocag133-B10","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1186\/1471-2288-12-70","article-title":"Evaluation of the propensity score methods for estimating marginal odds ratios in case of small sample size","volume":"12","author":"Pirracchio","year":"2012","journal-title":"BMC Med Res Methodol"},{"key":"2026100116371924000_ocag133-B11","doi-asserted-by":"crossref","first-page":"2005","DOI":"10.1093\/ije\/dyy120","article-title":"Evaluating large-scale propensity score performance through real-world and synthetic data experiments","volume":"47","author":"Tian","year":"2018","journal-title":"Int J Epidemiol"},{"key":"2026100116371924000_ocag133-B12","doi-asserted-by":"crossref","first-page":"104204","DOI":"10.1016\/j.jbi.2022.104204","article-title":"Adjusting for indirectly measured confounding using large-scale propensity score","volume":"134","author":"Zhang","year":"2022","journal-title":"J Biomed Inform"},{"key":"2026100116371924000_ocag133-B13","doi-asserted-by":"crossref","first-page":"1279","DOI":"10.1007\/s40264-017-0581-7","article-title":"Channeling in the use of nonprescription paracetamol and ibuprofen in an electronic medical records database: evidence and implications","volume":"40","author":"Weinstein","year":"2017","journal-title":"Drug Saf"},{"key":"2026100116371924000_ocag133-B14","doi-asserted-by":"crossref","first-page":"e70212","DOI":"10.1002\/sim.70212","article-title":"Assessing covariate balance with small sample sizes","volume":"44","author":"Hripcsak","year":"2025","journal-title":"Stat Med"},{"key":"2026100116371924000_ocag133-B15","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1002\/sim.5925","article-title":"Interpreting observational studies: why empirical calibration is needed to correct p-values","volume":"33","author":"Schuemie","year":"2014","journal-title":"Stat Med"},{"key":"2026100116371924000_ocag133-B16","doi-asserted-by":"crossref","first-page":"1331","DOI":"10.1093\/jamia\/ocaa103","article-title":"Principles of Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND)","volume":"27","author":"Schuemie","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2026100116371924000_ocag133-B17","first-page":"966","article-title":"Health-Analytics Data to Evidence Suite (HADES): open-source software for observational research","volume":"310","author":"Schuemie","year":"2024","journal-title":"Stud Health Technol Inform."},{"key":"2026100116371924000_ocag133-B18","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1136\/amiajnl-2011-000376","article-title":"Validation of a common data model for active safety surveillance research","volume":"19","author":"Overhage","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2026100116371924000_ocag133-B19","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1093\/jamia\/ocae317","article-title":"Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence","volume":"32","author":"Conover","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"2026100116371924000_ocag133-B20","doi-asserted-by":"crossref","first-page":"1816","DOI":"10.1016\/S0140-6736(19)32317-7","article-title":"Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis","volume":"394","author":"Suchard","year":"2019","journal-title":"Lancet"},{"key":"2026100116371924000_ocag133-B21","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1016\/j.jacc.2024.05.069","article-title":"Comparative effectiveness of second-line antihyperglycemic agents for cardiovascular outcomes: a multinational, federated analysis of LEGEND-T2DM","volume":"84","author":"Khera","year":"2024","journal-title":"J Am Coll Cardiol"},{"key":"2026100116371924000_ocag133-B22","doi-asserted-by":"crossref","first-page":"2571","DOI":"10.1073\/pnas.1708282114","article-title":"Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data","volume":"115","author":"Schuemie","year":"2018","journal-title":"Proc Natl Acad Sci USA"},{"key":"2026100116371924000_ocag133-B23","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1111\/biom.12078","article-title":"Multiple self-controlled case series for large-scale longitudinal observational databases","volume":"69","author":"Simpson","year":"2013","journal-title":"Biometrics."},{"key":"2026100116371924000_ocag133-B24","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1177\/09622802211060518","article-title":"Combining cox regressions across a heterogeneous distributed research network facing small and zero counts","volume":"31","author":"Schuemie","year":"2022","journal-title":"Stat Methods Med Res"},{"key":"2026100116371924000_ocag133-B25","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.147cc28e","article-title":"How confident are we about observational findings in healthcare: a Benchmark study","volume":"2","author":"Schuemie","year":"2020","journal-title":"Harv Data Sci Rev"},{"key":"2026100116371924000_ocag133-B26","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1080\/10543406.2021.1882481","article-title":"Incorporating propensity scores for evidence synthesis under Bayesian framework: review and recommendations for clinical studies","volume":"32","author":"Lin","year":"2022","journal-title":"J Biopharm Stat"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag133\/70479444\/ocag133.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/10\/1992\/70479444\/ocag133.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/10\/1992\/70479444\/ocag133.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T20:37:29Z","timestamp":1790887049000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/10\/1992\/8751035"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,4]]},"references-count":26,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,8,4]]},"published-print":{"date-parts":[[2026,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag133","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,10]]},"published":{"date-parts":[[2026,8,4]]}}}