{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T01:45:11Z","timestamp":1775785511194,"version":"3.50.1"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2021,8,5]],"date-time":"2021-08-05T00:00:00Z","timestamp":1628121600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01 HL144555"],"award-info":[{"award-number":["R01 HL144555"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"name":"STAnford medicine Research data Repository"},{"name":"Stanford Health Care, Stanford Children\u2019s Health, and the University Healthcare Alliance and Packard Children's Health Alliance clinics"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,9,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Using a risk stratification model to guide clinical practice often requires the choice of a cutoff\u2014called the decision threshold\u2014on the model\u2019s output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 pooled cohort equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision making following model deployment.<\/jats:p>","DOI":"10.1093\/jamia\/ocab159","type":"journal-article","created":{"date-parts":[[2021,7,13]],"date-time":"2021-07-13T23:12:49Z","timestamp":1626217969000},"page":"2258-2264","source":"Crossref","is-referenced-by-count":10,"title":["Learning decision thresholds for risk stratification models from aggregate clinician behavior"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7673-6143","authenticated-orcid":false,"given":"Birju S","family":"Patel","sequence":"first","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}]},{"given":"Ethan","family":"Steinberg","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0551-9664","authenticated-orcid":false,"given":"Stephen R","family":"Pfohl","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9385-7158","authenticated-orcid":false,"given":"Nigam H","family":"Shah","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2021,8,5]]},"reference":[{"issue":"3","key":"2021091817523875800_ocab159-B1","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1093\/jamia\/ocaa296","article-title":"Clinician involvement in research on machine learning\u2013based predictive clinical decision support for the hospital setting: a scoping review","volume":"28","author":"Schwartz","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"7","key":"2021091817523875800_ocab159-B2","doi-asserted-by":"crossref","first-page":"e15182","DOI":"10.2196\/15182","article-title":"Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study","volume":"8","author":"Sendak","year":"2020","journal-title":"JMIR Med Inform"},{"key":"2021091817523875800_ocab159-B3","doi-asserted-by":"crossref","DOI":"10.1002\/9781118341544","volume-title":"Medical Decision Making","author":"Sox","year":"2013"},{"issue":"20","key":"2021091817523875800_ocab159-B4","doi-asserted-by":"crossref","first-page":"1109","DOI":"10.1056\/NEJM198005153022003","article-title":"The threshold approach to clinical decision making","volume":"302","author":"Pauker","year":"1980","journal-title":"N Engl J Med"},{"issue":"6","key":"2021091817523875800_ocab159-B5","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1177\/0272989X06295361","article-title":"Decision curve analysis: a novel method for evaluating prediction models","volume":"26","author":"Vickers","year":"2006","journal-title":"Med Decis Making"},{"issue":"1","key":"2021091817523875800_ocab159-B6","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","article-title":"Assessing the performance of prediction models: a framework for traditional and novel measures","volume":"21","author":"Steyerberg","year":"2010","journal-title":"Epidemiology"},{"key":"2021091817523875800_ocab159-B7","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/1741-7015-11-7","article-title":"Support of personalized medicine through risk-stratified treatment recommendations - an environmental scan of clinical practice guidelines","volume":"11","author":"Yu","year":"2013","journal-title":"BMC Med"},{"issue":"1","key":"2021091817523875800_ocab159-B8","first-page":"109","article-title":"The standard of care: legal history and definitions: the bad and good news","volume":"12","author":"Moffett","year":"2011","journal-title":"West J Emerg Med"},{"key":"2021091817523875800_ocab159-B9","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1186\/1472-6947-10-20","article-title":"A new method for determining physician decision thresholds using empiric, uncertain recommendations","volume":"10","author":"Boland","year":"2010","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"2021091817523875800_ocab159-B10","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1136\/ebmed-2014-110140","article-title":"A novel approach to the determination of clinical decision thresholds","volume":"20","author":"Ebell","year":"2015","journal-title":"Evid Based Med"},{"issue":"2","key":"2021091817523875800_ocab159-B11","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1177\/0272989X8300300203","article-title":"Derived thresholds. Determining the diagnostic probabilities at which clinicians initiate testing and treatment","volume":"3","author":"Eisenberg","year":"1983","journal-title":"Med Decis Making"},{"issue":"2","key":"2021091817523875800_ocab159-B12","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1177\/0272989X9801800201","article-title":"An international comparison of physicians' judgments of outcome rates of cardiac procedures and attitudes toward risk, uncertainty, justifiability, and regret","volume":"18","author":"Poses","year":"1998","journal-title":"Med Decis Making"},{"issue":"6 Pt 1","key":"2021091817523875800_ocab159-B13","first-page":"763","article-title":"Comparing aggregate estimates of derived thresholds for clinical decisions","volume":"20","author":"Young","year":"1986","journal-title":"Health Serv Res"},{"issue":"7","key":"2021091817523875800_ocab159-B14","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1377\/hlthaff.2014.0099","article-title":"A \u2018green button\u2019 for using aggregate patient data at the point of care","volume":"33","author":"Longhurst","year":"2014","journal-title":"Health Aff (Millwood)"},{"issue":"5","key":"2021091817523875800_ocab159-B15","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1111\/eci.12421","article-title":"When is rational to order a diagnostic test, or prescribe treatment: the threshold model as an explanation of practice variation","volume":"45","author":"Djulbegovic","year":"2015","journal-title":"Eur J Clin Invest"},{"issue":"3","key":"2021091817523875800_ocab159-B16","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1177\/0272989X9201200306","article-title":"A method to describe physician decision thresholds and its application in examining the diagnosis of coronary artery disease based on exercise treadmill testing","volume":"12","author":"Plasencia","year":"1992","journal-title":"Med Decis Making"},{"key":"2021091817523875800_ocab159-B17","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1016\/j.jbi.2015.12.010","article-title":"An unsupervised learning method to identify reference intervals from a clinical database","volume":"59","author":"Poole","year":"2016","journal-title":"J Biomed Inform"},{"issue":"25 Suppl 2","key":"2021091817523875800_ocab159-B18","first-page":"S49","article-title":"2013 ACC\/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology\/American Heart Association task force on practice guidelines","volume":"129","author":"Goff","year":"2014","journal-title":"Circulation"},{"issue":"25","key":"2021091817523875800_ocab159-B19","first-page":"e1082","article-title":"2018 AHA\/ACC\/AACVPR\/AAPA\/ABC\/ACPM\/ADA\/AGS\/APhA\/ASPC\/NLA\/PCNA guideline on the management of blood cholesterol: a report of the American College of Cardiology\/American Heart Association task force on clinical practice guidelines","volume":"139","author":"Grundy","year":"2019","journal-title":"Circulation"},{"issue":"18","key":"2021091817523875800_ocab159-B20","doi-asserted-by":"crossref","first-page":"2543","DOI":"10.1001\/jama.1982.03320430047030","article-title":"Evaluating the yield of medical tests","volume":"247","author":"Harrell","year":"1982","journal-title":"JAMA"},{"issue":"5","key":"2021091817523875800_ocab159-B21","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1056\/NEJM197507312930505","article-title":"Therapeutic decision making: a cost-benefit analysis","volume":"293","author":"Pauker","year":"1975","journal-title":"N Engl J Med"},{"issue":"6","key":"2021091817523875800_ocab159-B22","doi-asserted-by":"crossref","first-page":"302","DOI":"10.1016\/0300-2977(95)00035-6","article-title":"Clinical decision theory: the threshold concept","volume":"47","author":"Habbema","year":"1995","journal-title":"Neth J Med"},{"issue":"1","key":"2021091817523875800_ocab159-B23","doi-asserted-by":"crossref","first-page":"23","DOI":"10.2307\/1909829","article-title":"Exposition of a new theory on the measurement of risk","volume":"22","author":"Bernoulli","year":"1954","journal-title":"Econometrica"},{"key":"2021091817523875800_ocab159-B24","first-page":"429","volume-title":"Readings in Decision Analysis","author":"Howard","year":"1977"},{"issue":"1","key":"2021091817523875800_ocab159-B25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.7326\/M18-1279","article-title":"Finding the balance between benefits and harms when using statins for primary prevention of cardiovascular disease: a modeling study","volume":"170","author":"Yebyo","year":"2019","journal-title":"Ann Intern Med"},{"issue":"2","key":"2021091817523875800_ocab159-B26","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1001\/jama.2015.6822","article-title":"Cost-effectiveness of 10-year risk thresholds for initiation of statin therapy for primary prevention of cardiovascular disease","volume":"314","author":"Pandya","year":"2015","journal-title":"JAMA"},{"issue":"25","key":"2021091817523875800_ocab159-B27","doi-asserted-by":"crossref","first-page":"3143","DOI":"10.1161\/circ.106.25.3143","article-title":"Third Report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III) final report","volume":"106","year":"2002","journal-title":"Circulation"},{"issue":"3","key":"2021091817523875800_ocab159-B28","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1016\/j.jacl.2015.11.002","article-title":"Provider understanding of the 2013 ACC\/AHA cholesterol guideline","volume":"10","author":"Virani","year":"2016","journal-title":"J Clin Lipidol"},{"issue":"24","key":"2021091817523875800_ocab159-B29","doi-asserted-by":"crossref","first-page":"2405","DOI":"10.1001\/jama.2019.5284","article-title":"The proliferation of reports on clinical scoring systems: issues about uptake and clinical utility","volume":"321","author":"Challener","year":"2019","journal-title":"JAMA"},{"issue":"3","key":"2021091817523875800_ocab159-B30","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1177\/0272989X8600600303","article-title":"Stochastic thresholds","volume":"6","author":"Hartz","year":"1986","journal-title":"Med Decis Making"},{"key":"2021091817523875800_ocab159-B31","first-page":"96","article-title":"Fast threshold tests for detecting discrimination","volume":"84","author":"Pierson","year":"2018","journal-title":"Proc Mach Learn Res"},{"issue":"3","key":"2021091817523875800_ocab159-B32","doi-asserted-by":"crossref","first-page":"1193","DOI":"10.1214\/17-AOAS1058","article-title":"The problem of infra-marginality in outcome tests for discrimination","volume":"11","author":"Simoiu","year":"2017","journal-title":"Ann Appl Stat"},{"key":"2021091817523875800_ocab159-B33","author":"Pierson"},{"key":"2021091817523875800_ocab159-B34","author":"Bakalar"},{"issue":"6464","key":"2021091817523875800_ocab159-B35","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"1","key":"2021091817523875800_ocab159-B36","doi-asserted-by":"crossref","first-page":"20","DOI":"10.7326\/M17-3011","article-title":"Clinical implications of revised pooled cohort equations for estimating atherosclerotic cardiovascular disease risk","volume":"169","author":"Yadlowsky","year":"2018","journal-title":"Ann Intern Med"},{"issue":"23","key":"2021091817523875800_ocab159-B37","doi-asserted-by":"crossref","first-page":"1851","DOI":"10.1001\/jama.1994.03520230061039","article-title":"Error in medicine","volume":"272","author":"Leape","year":"1994","journal-title":"JAMA"},{"issue":"7","key":"2021091817523875800_ocab159-B38","doi-asserted-by":"crossref","first-page":"e15653","DOI":"10.2196\/15653","article-title":"Appropriateness of overridden alerts in computerized physician order entry: systematic review","volume":"8","author":"Poly","year":"2020","journal-title":"JMIR Med Inform"},{"issue":"8","key":"2021091817523875800_ocab159-B39","doi-asserted-by":"crossref","first-page":"1252","DOI":"10.1093\/jamia\/ocaa098","article-title":"The tradeoffs between safety and alert fatigue: Data from a national evaluation of hospital medication-related clinical decision support","volume":"27","author":"Co","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2021091817523875800_ocab159-B40","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1177\/0272989X8800800409","article-title":"Focusing technology assessment using medical decision theory","volume":"8","author":"Phelps","year":"1988","journal-title":"Med Decis Making"},{"issue":"4","key":"2021091817523875800_ocab159-B41","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1007\/s40273-014-0135-0","article-title":"Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature","volume":"32","author":"Marsh","year":"2014","journal-title":"Pharmacoeconomics"},{"key":"2021091817523875800_ocab159-B42","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/s41512-019-0064-7","article-title":"A simple, step-by-step guide to interpreting decision curve analysis","volume":"3","author":"Vickers","year":"2019","journal-title":"Diagn Progn Res"},{"issue":"12","key":"2021091817523875800_ocab159-B43","doi-asserted-by":"crossref","first-page":"1346","DOI":"10.1001\/jamadermatol.2015.3149","article-title":"Detection accuracy of collective intelligence assessments for skin cancer diagnosis","volume":"151","author":"Kurvers","year":"2015","journal-title":"JAMA Dermatol"},{"issue":"3","key":"2021091817523875800_ocab159-B44","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/BF02602334","article-title":"To test or not to test\u2013to treat or not to treat: the decision-threshold approach to patient management","volume":"1","author":"Eraker","year":"1986","journal-title":"J Gen Intern Med"},{"issue":"3","key":"2021091817523875800_ocab159-B45","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/0167-6296(85)90033-5","article-title":"Risk-aversion and physicians' medical decision-making","volume":"4","author":"Eeckhoudt","year":"1985","journal-title":"J Health Econ"},{"issue":"2","key":"2021091817523875800_ocab159-B46","doi-asserted-by":"crossref","first-page":"263","DOI":"10.2307\/1914185","article-title":"Prospect theory: an analysis of decision under risk","volume":"47","author":"Kahneman","year":"1979","journal-title":"Econometrica"},{"issue":"6","key":"2021091817523875800_ocab159-B47","doi-asserted-by":"crossref","first-page":"1057","DOI":"10.1016\/j.jhealeco.2013.08.006","article-title":"Prospect theory in the health domain: a quantitative assessment","volume":"32","author":"Attema","year":"2013","journal-title":"J Health Econ"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/10\/2258\/40408875\/ocab159.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/10\/2258\/40408875\/ocab159.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,18]],"date-time":"2021-09-18T13:54:01Z","timestamp":1631973241000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/10\/2258\/6341111"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,5]]},"references-count":47,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,8,5]]},"published-print":{"date-parts":[[2021,9,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab159","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.02.19.21252069","asserted-by":"object"}]},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,10,1]]},"published":{"date-parts":[[2021,8,5]]}}}