{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:01:45Z","timestamp":1772164905076,"version":"3.50.1"},"reference-count":65,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T00:00:00Z","timestamp":1727654400000},"content-version":"vor","delay-in-days":273,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Department of Health and Human Services\u2019 Office of Research Integrity","award":["ORIIR180041"],"award-info":[{"award-number":["ORIIR180041"]}]},{"name":"Department of Health and Human Services\u2019 Office of Research Integrity","award":["ORIIIR190049"],"award-info":[{"award-number":["ORIIIR190049"]}]},{"name":"Department of Health and Human Services\u2019 Office of Research Integrity","award":["ORIIIR200052"],"award-info":[{"award-number":["ORIIIR200052"]}]},{"name":"Department of Health and Human Services\u2019 Office of Research Integrity","award":["ORIIIR210062"],"award-info":[{"award-number":["ORIIIR210062"]}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Correctness is a key aspiration of the scientific process, yet recent studies suggest that many high-profile findings may be difficult to replicate or require considerable evidence for verification. Proposals to fix these issues typically ask for tighter statistical controls (e.g., stricter p-value thresholds or higher statistical power). However, these approaches often overlook the importance of contemplating research outcomes\u2019 potential costs and benefits. Here, we develop a framework grounded in Bayesian decision theory that seamlessly integrates cost-benefit analysis into evaluating research programs with potentially uncertain results. We derive minimally acceptable prestudy odds and positive predictive values for cost and benefit levels. We show that tolerance to inaccurate results changes dramatically due to uncertainties posed by research. We also show that reducing uncertainties (e.g., by recruiting more subjects) may have limited effects on the expected benefit of continuing specific research programs. We apply our framework to several types of cancer research and their funding. Our analysis shows that highly exploratory research designs are easily justifiable due to their potential benefits, even when probabilistic models suggest otherwise. We discuss how the cost and benefit of research could and should always be part of the toolkit used by scientists, institutions, or funding agencies.<\/jats:p>","DOI":"10.1162\/qss_a_00332","type":"journal-article","created":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T16:30:37Z","timestamp":1727713837000},"page":"1047-1069","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Incorporating costs and benefits to the evaluation of uncertain research results: Applications to cancer research funding"],"prefix":"10.1162","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8076-6660","authenticated-orcid":true,"given":"Han","family":"Zhuang","sequence":"first","affiliation":[{"name":"School of Information Studies, Syracuse University, Syracuse, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7765-1595","authenticated-orcid":true,"given":"Daniel E.","family":"Acuna","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Colorado at Boulder, Boulder, CO, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,11,1]]},"reference":[{"key":"2024122514552574500_bib1","doi-asserted-by":"publisher","first-page":"1553","DOI":"10.2307\/3312947","article-title":"Pricing the priceless: Cost-benefit analysis of environmental protection","volume":"150","author":"Ackerman","year":"2002","journal-title":"University of Pennsylvania Law Review"},{"key":"2024122514552574500_bib2","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1146\/annurev.environ.33.020107.112927","article-title":"Environmental cost-benefit analysis","volume":"33","author":"Atkinson","year":"2008","journal-title":"Annual Review of Environment and Resources"},{"issue":"7391","key":"2024122514552574500_bib3","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1038\/483531a","article-title":"Raise standards for preclinical cancer research","volume":"483","author":"Begley","year":"2012","journal-title":"Nature"},{"issue":"1","key":"2024122514552574500_bib4","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1038\/s41562-017-0189-z","article-title":"Redefine statistical significance","volume":"2","author":"Benjamin","year":"2018","journal-title":"Nature Human Behaviour"},{"key":"2024122514552574500_bib5","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1080\/00031305.2018.1529624","article-title":"The p-value requires context, not a threshold","volume":"73","author":"Betensky","year":"2019","journal-title":"American Statistician"},{"issue":"8","key":"2024122514552574500_bib6","doi-asserted-by":"publisher","first-page":"1384","DOI":"10.1021\/es990673l","article-title":"On measuring economic values for nature","volume":"34","author":"Bockstael","year":"2000","journal-title":"Environmental Science & Technology"},{"issue":"2","key":"2024122514552574500_bib7","doi-asserted-by":"publisher","first-page":"6","DOI":"10.52041\/serj.v11i2.325","article-title":"Students\u2019 perceptions of statistics: An exploration of attitudes, conceptualizations, and content knowledge of statistics","volume":"11","author":"Bond","year":"2012","journal-title":"Statistics Education Research Journal"},{"key":"2024122514552574500_bib8","volume-title":"Biostatistics: An applied introduction for the public health practitioner","author":"Bush","year":"2011"},{"issue":"9","key":"2024122514552574500_bib9","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1038\/s41562-018-0399-z","article-title":"Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015","volume":"2","author":"Camerer","year":"2018","journal-title":"Nature Human Behaviour"},{"key":"2024122514552574500_bib10","volume-title":"United States and Puerto Rico cancer statistics, 1999\u20132016 incidence archive request","author":"Centers for Disease Control and Prevention","year":"2019"},{"key":"2024122514552574500_bib11","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1093\/icc\/9.2.211","article-title":"The explicit economics of knowledge codification and tacitness","volume":"9","author":"Cowan","year":"2000","journal-title":"Industrial and Corporate Change"},{"key":"2024122514552574500_bib12","doi-asserted-by":"publisher","first-page":"161","DOI":"10.7208\/chicago\/9780226903217.003.0006","article-title":"Intensive medical technology and the reduction in disability","volume-title":"Analyses in the economics of aging","author":"Cutler","year":"2005"},{"issue":"6","key":"2024122514552574500_bib13","doi-asserted-by":"publisher","first-page":"1081","DOI":"10.1016\/j.jhealeco.2007.09.003","article-title":"The lifetime costs and benefits of medical technology","volume":"26","author":"Cutler","year":"2007","journal-title":"Journal of Health Economics"},{"issue":"5","key":"2024122514552574500_bib14","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1377\/hlthaff.20.5.11","article-title":"Is technological change in medicine worth it?","volume":"20","author":"Cutler","year":"2001","journal-title":"Health Affairs"},{"key":"2024122514552574500_bib15","doi-asserted-by":"publisher","DOI":"10.3386\/w8556","volume-title":"Changes in the age distribution of mortality over the 20th century","author":"Cutler","year":"2001"},{"key":"2024122514552574500_bib16","volume-title":"NSF budget request to congress","author":"Directorate for Social, Behavioral, and Economic Sciences","year":"2019"},{"issue":"2","key":"2024122514552574500_bib17","doi-asserted-by":"publisher","first-page":"e26","DOI":"10.1371\/journal.pmed.0040026","article-title":"When should potentially false research findings be considered acceptable?","volume":"4","author":"Djulbegovic","year":"2007","journal-title":"PLOS Medicine"},{"issue":"3","key":"2024122514552574500_bib18","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1054\/mehy.1998.0020","article-title":"Acceptable regret in medical decision making","volume":"53","author":"Djulbegovic","year":"1999","journal-title":"Medical Hypotheses"},{"issue":"9","key":"2024122514552574500_bib19","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1016\/0277-9536(92)90128-D","article-title":"Assessing the costs and benefits of medical research: The diabetic retinopathy study","volume":"34","author":"Drummond","year":"1992","journal-title":"Social Science & Medicine"},{"key":"2024122514552574500_bib20","volume-title":"Pattern classification","author":"Duda","year":"2012"},{"issue":"6379","key":"2024122514552574500_bib21","doi-asserted-by":"publisher","first-page":"eaao0185","DOI":"10.1126\/science.aao0185","article-title":"Science of science","volume":"359","author":"Fortunato","year":"2018","journal-title":"Science"},{"issue":"2","key":"2024122514552574500_bib22","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/S0167-6296(99)00033-8","article-title":"The future of health economics","volume":"19","author":"Fuchs","year":"2000","journal-title":"Journal of Health Economics"},{"key":"2024122514552574500_bib23","doi-asserted-by":"publisher","DOI":"10.1177\/2378023118785743","article-title":"How to think scientifically about scientists\u2019 proposals for fixing science","volume":"4","author":"Gelman","year":"2018","journal-title":"Socius"},{"key":"2024122514552574500_bib24","doi-asserted-by":"publisher","DOI":"10.1201\/b16018","volume-title":"Bayesian data analysis","author":"Gelman","year":"2013"},{"issue":"6277","key":"2024122514552574500_bib25","doi-asserted-by":"publisher","first-page":"1037","DOI":"10.1126\/science.aad7243","article-title":"Comment on \u201cEstimating the reproducibility of psychological science\u201d","volume":"351","author":"Gilbert","year":"2016","journal-title":"Science"},{"issue":"2","key":"2024122514552574500_bib26","doi-asserted-by":"publisher","first-page":"e16837","DOI":"10.1371\/journal.pone.0016837","article-title":"NIH disease funding levels and burden of disease","volume":"6","author":"Gillum","year":"2011","journal-title":"PLOS ONE"},{"key":"2024122514552574500_bib27","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1023\/B:SCIE.0000006881.30700.ea","article-title":"Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon","volume":"58","author":"Gl\u00e4nzel","year":"2003","journal-title":"Scientometrics"},{"key":"2024122514552574500_bib28","volume-title":"Introduction to health technology assessment","author":"Goodman","year":"2004"},{"issue":"341","key":"2024122514552574500_bib29","doi-asserted-by":"publisher","first-page":"341ps12","DOI":"10.1126\/scitranslmed.aaf5027","article-title":"What does research reproducibility mean?","volume":"8","author":"Goodman","year":"2016","journal-title":"Science Translational Medicine"},{"issue":"9","key":"2024122514552574500_bib30","doi-asserted-by":"publisher","first-page":"e022131","DOI":"10.1136\/bmjopen-2018-022131","article-title":"Economic returns to medical research funding","volume":"8","author":"Grant","year":"2018","journal-title":"BMJ Open"},{"issue":"2","key":"2024122514552574500_bib31","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1001\/jama.294.2.218","article-title":"Contradicted and initially stronger effects in highly cited clinical research","volume":"294","author":"Ioannidis","year":"2005","journal-title":"JAMA"},{"issue":"8","key":"2024122514552574500_bib32","doi-asserted-by":"publisher","first-page":"e124","DOI":"10.1371\/journal.pmed.0020124","article-title":"Why most published research findings are false","volume":"2","author":"Ioannidis","year":"2005","journal-title":"PLOS Medicine"},{"issue":"24","key":"2024122514552574500_bib33","doi-asserted-by":"publisher","first-page":"7426","DOI":"10.1073\/pnas.1424329112","article-title":"Defining and identifying Sleeping Beauties in science","volume":"112","author":"Ke","year":"2015","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"7","key":"2024122514552574500_bib34","doi-asserted-by":"publisher","first-page":"877","DOI":"10.1097\/ACM.0b013e318257ee6a","article-title":"Perspective: Research misconduct. The search for a remedy","volume":"87","author":"Kornfeld","year":"2012","journal-title":"Academic Medicine"},{"key":"2024122514552574500_bib35","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226458144.001.0001","volume-title":"The structure of scientific revolutions","author":"Kuhn","year":"2012 [1962]"},{"key":"2024122514552574500_bib36","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511521942","volume-title":"Cost-benefit analysis","author":"Layard","year":"1994"},{"issue":"3","key":"2024122514552574500_bib37","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1111\/j.1524-4733.2006.00095.x","article-title":"The return on investment in health care: From 1980 to 2000","volume":"9","author":"Luce","year":"2006","journal-title":"Value in Health"},{"key":"2024122514552574500_bib38","doi-asserted-by":"publisher","DOI":"10.1016\/S0737-1071(04)08002-3","article-title":"The economics of knowledge and learning","volume-title":"Product innovation, interactive learning and economic performance","author":"Lundvall","year":"2004"},{"issue":"4","key":"2024122514552574500_bib39","doi-asserted-by":"publisher","first-page":"357","DOI":"10.2165\/00019053-199915040-00003","article-title":"Recent advances in the methods of cost-benefit analysis in healthcare: Matching the art to the science","volume":"15","author":"McIntosh","year":"1999","journal-title":"PharmacoEconomics"},{"key":"2024122514552574500_bib40","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1080\/00031305.2018.1527253","article-title":"Abandon statistical significance","volume":"73","author":"McShane","year":"2019","journal-title":"American Statistician"},{"issue":"5","key":"2024122514552574500_bib41","doi-asserted-by":"publisher","first-page":"664","DOI":"10.1177\/1745691616649170","article-title":"Optimizing research payoff","volume":"11","author":"Miller","year":"2016","journal-title":"Perspectives on Psychological Science"},{"key":"2024122514552574500_bib42","volume-title":"Cancer statistics","author":"National Cancer Institute","year":"2019"},{"key":"2024122514552574500_bib43","volume-title":"Financial burden of cancer care","author":"National Cancer Institute","year":"2019"},{"key":"2024122514552574500_bib44","volume-title":"NIH-wide strategic plan","author":"National Institutes of Health","year":"2015"},{"key":"2024122514552574500_bib45","volume-title":"Estimates of funding for various research, condition, and disease categories (RCDC)","author":"National Institutes of Health","year":"2019"},{"key":"2024122514552574500_bib46","volume-title":"NIH categorical spending","author":"National Institutes of Health","year":"2022"},{"issue":"4","key":"2024122514552574500_bib47","doi-asserted-by":"publisher","first-page":"283","DOI":"10.18043\/ncm.75.4.283","article-title":"Physical and mental health among cancer survivors: Considerations for long-term care and quality of life","volume":"75","author":"Naughton","year":"2014","journal-title":"North Carolina Medical Journal"},{"issue":"5","key":"2024122514552574500_bib48","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1001\/archinte.161.5.749","article-title":"Cost-benefit analysis of a strategy to vaccinate healthy working adults against influenza","volume":"161","author":"Nichol","year":"2001","journal-title":"Archives of Internal Medicine"},{"key":"2024122514552574500_bib49","doi-asserted-by":"publisher","first-page":"e21451","DOI":"10.7554\/eLife.21451","article-title":"Publication bias and the canonization of false facts","volume":"5","author":"Nissen","year":"2016","journal-title":"eLife"},{"issue":"11","key":"2024122514552574500_bib50","doi-asserted-by":"publisher","first-page":"2600","DOI":"10.1073\/pnas.1708274114","article-title":"The preregistration revolution","volume":"115","author":"Nosek","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"6251","key":"2024122514552574500_bib51","doi-asserted-by":"publisher","first-page":"aac4716","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"issue":"5","key":"2024122514552574500_bib52","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1016\/0048-7333(94)01002-1","article-title":"Toward a new economics of science","volume":"23","author":"Partha","year":"1994","journal-title":"Research Policy"},{"key":"2024122514552574500_bib53","doi-asserted-by":"publisher","DOI":"10.4324\/9780203994627","volume-title":"The logic of scientific discovery","author":"Popper","year":"2005 [1934]"},{"key":"2024122514552574500_bib54","first-page":"5485","article-title":"A step toward quantifying independently reproducible machine learning research","volume-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","author":"Raff","year":"2019"},{"key":"2024122514552574500_bib55","volume-title":"Research methods in practice: Strategies for description and causation","author":"Remler","year":"2014"},{"issue":"7023","key":"2024122514552574500_bib56","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1136\/bmj.312.7023.71","article-title":"Evidence based medicine: What it is and what it isn\u2019t","volume":"312","author":"Sackett","year":"1996","journal-title":"British Medical Journal"},{"issue":"1","key":"2024122514552574500_bib57","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1111\/milq.12005","article-title":"New evidence on the allocation of NIH funds across diseases","volume":"91","author":"Sampat","year":"2013","journal-title":"Milbank Quarterly"},{"key":"2024122514552574500_bib58","volume-title":"Period life tables","author":"Social Security Administration","year":"2017"},{"issue":"5","key":"2024122514552574500_bib59","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1016\/S0301-4215(97)00041-4","article-title":"Limits to the value of external costs","volume":"25","author":"Stirling","year":"1997","journal-title":"Energy Policy"},{"key":"2024122514552574500_bib60","volume-title":"Reinforcement learning: An introduction","author":"Sutton","year":"2018"},{"key":"2024122514552574500_bib61","doi-asserted-by":"publisher","first-page":"e58237","DOI":"10.7554\/eLife.58237","article-title":"Meta-research: Questionable research practices may have little effect on replicability","volume":"9","author":"Ulrich","year":"2020","journal-title":"eLife"},{"issue":"2","key":"2024122514552574500_bib62","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1214\/aoms\/1177730030","article-title":"Statistical decision functions","volume":"20","author":"Wald","year":"1949","journal-title":"Annals of Mathematical Statistics"},{"issue":"5","key":"2024122514552574500_bib63","doi-asserted-by":"publisher","first-page":"397","DOI":"10.1016\/S0002-9343(03)00057-3","article-title":"A cost-benefit analysis of electronic medical records in primary care","volume":"114","author":"Wang","year":"2003","journal-title":"American Journal of Medicine"},{"issue":"8","key":"2024122514552574500_bib64","doi-asserted-by":"publisher","first-page":"703","DOI":"10.1093\/embo-reports\/kvf165","article-title":"The moment of truth for science: The consequences of the \u2018knowledge society\u2019 for society and science","volume":"3","author":"Weingart","year":"2002","journal-title":"EMBO Reports"},{"key":"2024122514552574500_bib65","first-page":"1729","article-title":"Regret minimization in games with incomplete information","volume-title":"Proceedings of the 20th International Conference on Neural Information Processing Systems","author":"Zinkevich","year":"2007"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/5\/4\/1047\/2482654\/qss_a_00332.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/5\/4\/1047\/2482654\/qss_a_00332.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,25]],"date-time":"2024-12-25T09:55:38Z","timestamp":1735120538000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/5\/4\/1047\/124632\/Incorporating-costs-and-benefits-to-the-evaluation"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":65,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,11,1]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00332","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v4\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v5\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v3\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v3\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v5\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v4\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00332\/v4\/review1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}