{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T07:25:12Z","timestamp":1789802712951,"version":"4.0.1"},"reference-count":17,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T00:00:00Z","timestamp":1635120000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T00:00:00Z","timestamp":1635120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Res Methodol"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Statistical inference based on small datasets, commonly found in precision oncology, is subject to low power and high uncertainty. In these settings, drawing strong conclusions about future research utility is difficult when using standard inferential measures. It is therefore important to better quantify the uncertainty associated with both significant and non-significant results based on small sample sizes.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We developed a new method, Bayesian Additional Evidence (BAE), that determines (1) how much additional <jats:italic>supportive<\/jats:italic> evidence is needed for a non-significant result to reach Bayesian posterior credibility, or (2) how much additional <jats:italic>opposing<\/jats:italic> evidence is needed to render a significant result non-credible. Although based in Bayesian analysis, a prior distribution is not needed; instead, the tipping point output is compared to reasonable effect ranges to draw conclusions. We demonstrate our approach in a comparative effectiveness analysis comparing two treatments in a real world biomarker-defined cohort, and provide guidelines for how to apply BAE in practice.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Our initial comparative effectiveness analysis results in a hazard ratio of 0.31 with 95% confidence interval (0.09, 1.1). Applying BAE to this result yields a tipping point of 0.54; thus, an observed hazard ratio of 0.54 or smaller in a replication study would result in posterior credibility for the treatment association. Given that effect sizes in this range are not extreme, and that supportive evidence exists from a similar published study, we conclude that this problem is worthy of further research.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Our proposed method provides a useful framework for interpreting analytic results from small datasets. This can assist researchers in deciding how to interpret and continue their investigations based on an initial analysis that has high uncertainty. Although we illustrated its use in estimating parameters based on time-to-event outcomes, BAE easily applies to any normally-distributed estimator, such as those used for analyzing binary or continuous outcomes.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12874-021-01432-5","type":"journal-article","created":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T02:02:34Z","timestamp":1635127354000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Bayesian additional evidence for decision making under small sample uncertainty"],"prefix":"10.1186","volume":"21","author":[{"given":"Arjun","family":"Sondhi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian","family":"Segal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremy","family":"Snider","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olivier","family":"Humblet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margaret","family":"McCusker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,10,25]]},"reference":[{"key":"1432_CR1","doi-asserted-by":"publisher","unstructured":"Begg CB, Berlin JA. Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society. Series A (Statistics in Society). 1988;151(3):419\u2013463. http:\/\/www.jstor.org\/stable\/2982993. doi: https:\/\/doi.org\/10.2307\/2982993.","DOI":"10.2307\/2982993"},{"key":"1432_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/00031305.2019.1583913","volume":"73","author":"RL Wasserstein","year":"2019","unstructured":"Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond \u201cp\u2009<\u20090.05\u201d. Am Stat. 2019;73:1\u201319. https:\/\/doi.org\/10.1080\/00031305.2019.1583913. https:\/\/doi.org\/10.1080\/00031305.2019.1583913.","journal-title":"Am Stat"},{"issue":"6","key":"1432_CR3","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1177\/1745691614551642","volume":"9","author":"A Gelman","year":"2014","unstructured":"Gelman A, Carlin J. Beyond power calculations: assessing type S (sign) and type M (magnitude) errors. Perspect Psychol Sci. 2014;9(6):641\u201351. https:\/\/doi.org\/10.1177\/1745691614551642.","journal-title":"Perspect Psychol Sci"},{"issue":"2","key":"1432_CR4","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1080\/00031305.2019.1678521","volume":"75","author":"BD Segal","year":"2021","unstructured":"Segal BD. Toward Replicability with confidence intervals for the exceedance probability. Am Stat. 2021;75(2):128\u201338. https:\/\/doi.org\/10.1080\/00031305.2019.1678521.","journal-title":"Am Stat"},{"key":"1432_CR5","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1080\/00031305.2018.1543136","volume":"73","author":"RAJ Matthews","year":"2019","unstructured":"Matthews RAJ. Moving towards the post p\u2009<\u20090.05 era via the analysis of credibility. Am Stat. 2019;73:202\u201312. https:\/\/doi.org\/10.1080\/00031305.2018.1543136.","journal-title":"Am Stat"},{"issue":"4","key":"1432_CR6","doi-asserted-by":"publisher","first-page":"1100","DOI":"10.1214\/aos\/1176345976","volume":"10","author":"PK Andersen","year":"1982","unstructured":"Andersen PK, Gill RD. Cox's regression model for counting processes: a large sample study. Ann Stat. 1982;10(4):1100\u201320 http:\/\/www.jstor.org\/stable\/2240714.","journal-title":"Ann Stat"},{"issue":"14","key":"1432_CR7","doi-asserted-by":"publisher","first-page":"1217","DOI":"10.1200\/JCO.18.01798","volume":"37","author":"F Innocenti","year":"2019","unstructured":"Innocenti F, Ou F, Qu X, et al. Mutational analysis of patients with colorectal cancer in CALGB\/SWOG 80405 identifies new roles of microsatellite instability and tumor mutational burden for patient outcome. JCO. 2019;37(14):1217\u201327. https:\/\/doi.org\/10.1200\/JCO.18.01798.","journal-title":"JCO."},{"key":"1432_CR8","doi-asserted-by":"publisher","unstructured":"Ma X, Long L, Moon S, Adamson BJS, Baxi SS. Comparison of population characteristics in real-world clinical oncology databases in the US: Flatiron Health, SEER, and NPCR. medRxiv. 2020. https:\/\/doi.org\/10.1101\/2020.03.16.20037143.","DOI":"10.1101\/2020.03.16.20037143"},{"key":"1432_CR9","unstructured":"Birnbaum B, Nussbaum N, Seidl-Rathkopf K, et al. Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research. ArXiv. 2020. https:\/\/arxiv.org\/abs\/2001.09765."},{"issue":"6","key":"1432_CR10","doi-asserted-by":"publisher","first-page":"4460","DOI":"10.1111\/1475-6773.12872","volume":"53","author":"MD Curtis","year":"2018","unstructured":"Curtis MD, Griffith SD, Tucker M, et al. Development and validation of a high-quality composite real-world mortality endpoint. Health Serv Res. 2018;53(6):4460\u201376. https:\/\/doi.org\/10.1111\/1475-6773.12872.","journal-title":"Health Serv Res"},{"key":"1432_CR11","doi-asserted-by":"crossref","unstructured":"Upton, G. \u201cFisher's Exact Test.\u201d Journal of the Royal Statistical Society. Series A (Statistics in Society), vol. 155, no. 3, 1992, pp. 395\u2013402. JSTOR, www.jstor.org\/stable\/2982890. Accessed 3 Aug 2021.","DOI":"10.2307\/2982890"},{"issue":"1","key":"1432_CR12","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1093\/biomet\/80.1.27","volume":"80","author":"D Firth","year":"1993","unstructured":"Firth D. Bias reduction of maximum likelihood estimates. Biometrika. March 1993;80(1):27\u201338. https:\/\/doi.org\/10.1093\/biomet\/80.1.27.","journal-title":"Biometrika"},{"issue":"14","key":"1432_CR13","doi-asserted-by":"publisher","first-page":"2099","DOI":"10.1002\/sim.854","volume":"20","author":"DV Mehrotra","year":"2001","unstructured":"Mehrotra DV, Roth AJ. Relative risk estimation and inference using a generalized logrank statistic. Stat Med. 2001;20(14):2099\u2013113. https:\/\/doi.org\/10.1002\/sim.854.","journal-title":"Stat Med"},{"key":"1432_CR14","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1080\/00031305.2018.1537893","volume":"73","author":"JD Blume","year":"2019","unstructured":"Blume JD, Greevy RA, Welty VF, Smith JR, Dupont WD. An introduction to second-generation p-values. Am Stat. 2019;73:157\u201367. https:\/\/doi.org\/10.1080\/00031305.2018.1537893. https:\/\/doi.org\/10.1080\/00031305.2018.1537893.","journal-title":"Am Stat"},{"key":"1432_CR15","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1080\/00031305.2018.1518268","volume":"73","author":"MA Gannon","year":"2019","unstructured":"Gannon MA. de Bragan\u00e7a Pereira, Carlos Alberto, Polpo a. blending bayesian and classical tools to define optimal sample-size-dependent significance levels. Am Stat. 2019;73:213\u201322. https:\/\/doi.org\/10.1080\/00031305.2018.1518268.","journal-title":"Am Stat"},{"key":"1432_CR16","doi-asserted-by":"publisher","unstructured":"Segal, BD. Toward Replicability With Confidence Intervals for the Exceedance Probability, The American Statistician, 75:2, 128\u2013138, DOI: https:\/\/doi.org\/10.1080\/00031305.2019.1678521","DOI":"10.1080\/00031305.2019.1678521"},{"key":"1432_CR17","doi-asserted-by":"publisher","unstructured":"Wasserstein, RL, Schirm, AL, Lazar, NA. Moving to a World Beyond \u201cp<0.05\u201d, The American Statistician, 73:sup1, 1\u201319, DOI: https:\/\/doi.org\/10.1080\/00031305.2019.1583913","DOI":"10.1080\/00031305.2019.1583913"}],"container-title":["BMC Medical Research Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-021-01432-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12874-021-01432-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-021-01432-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T02:04:38Z","timestamp":1635127478000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedresmethodol.biomedcentral.com\/articles\/10.1186\/s12874-021-01432-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,25]]},"references-count":17,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1432"],"URL":"https:\/\/doi.org\/10.1186\/s12874-021-01432-5","relation":{},"ISSN":["1471-2288"],"issn-type":[{"value":"1471-2288","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,25]]},"assertion":[{"value":"7 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Institutional Review Board approval of the study protocol was obtained prior to study conduct from the WCG IRB, and included a waiver of informed consent, in accordance with relevant ethical guidelines (Declaration of Helsinki).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"At the time of the study, AS, BS, JS, OH, MM, report employment in Flatiron Health, Inc., an independent subsidiary of Roche. AS, BS, JS, OH, MM report stock ownership in Roche. BS, JS report equity ownership in Flatiron Health, Inc.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"221"}}