{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T11:40:08Z","timestamp":1750765208207,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":47,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,23]]},"DOI":"10.1145\/3715275.3732216","type":"proceedings-article","created":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T17:01:18Z","timestamp":1750698078000},"page":"3350-3369","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Reconciling Predictive Multiplicity in Practice"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-1157-9082","authenticated-orcid":false,"given":"Tina","family":"Behzad","sequence":"first","affiliation":[{"name":"Computer Science Department, Stony Brook University, Mount Sinai, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5684-4585","authenticated-orcid":false,"given":"S\u00edlvia","family":"Casacuberta","sequence":"additional","affiliation":[{"name":"University of Oxford, oxford, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2386-9126","authenticated-orcid":false,"given":"Emily Ruth","family":"Diana","sequence":"additional","affiliation":[{"name":"CMU Tepper School of Business, pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7598-2905","authenticated-orcid":false,"given":"Alexander Williams","family":"Tolbert","sequence":"additional","affiliation":[{"name":"Emory University, Atlanta, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,6,23]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Douglas Almond Kenneth\u00a0Y Chay and David\u00a0S Lee. 2005. The costs of low birth weight. The Quarterly Journal of Economics 120 3 (2005) 1031\u20131083.","DOI":"10.1162\/003355305774268228"},{"key":"e_1_3_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Susan Athey and Stefan Wager. 2019. Estimating treatment effects with causal forests: An application. Observational studies 5 2 (2019) 37\u201351.","DOI":"10.1353\/obs.2019.0001"},{"key":"e_1_3_3_2_4_2","unstructured":"Barry Becker and Ronny Kohavi. 1996. Adult. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C5XW20."},{"key":"e_1_3_3_2_5_2","doi-asserted-by":"crossref","unstructured":"Emily Black John\u00a0Logan Koepke Pauline\u00a0T Kim Solon Barocas and Mingwei Hsu. 2024. Less discriminatory algorithms. Geo. LJ 113 (2024) 53.","DOI":"10.2139\/ssrn.4590481"},{"key":"e_1_3_3_2_6_2","unstructured":"Emily Black Klas Leino and Matt Fredrikson. 2021. Selective ensembles for consistent predictions. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2111.08230 (2021)."},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533149"},{"key":"e_1_3_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Ren\u00e9e\u00a0Jorgensen Bolinger. 2020. The rational impermissibility of accepting (some) racial generalizations. Synthese 197 6 (2020) 2415\u20132431.","DOI":"10.1007\/s11229-018-1809-5"},{"key":"e_1_3_3_2_9_2","unstructured":"Leo Breiman. 2003. Statistical modeling: The two cultures. Quality control and applied statistics 48 1 (2003) 81\u201382."},{"key":"e_1_3_3_2_10_2","unstructured":"Huigang Chen Totte Harinen Jeong-Yoon Lee Mike Yung and Zhenyu Zhao. 2020. Causalml: Python package for causal machine learning. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2002.11631 (2020)."},{"key":"e_1_3_3_2_11_2","first-page":"247","volume-title":"Methods in comparative effectiveness research","author":"Dahabreh Issa\u00a0J","year":"2017","unstructured":"Issa\u00a0J Dahabreh, Thomas\u00a0A Trikalinos, David\u00a0M Kent, and Christopher\u00a0H Schmid. 2017. Heterogeneity of treatment effects. In Methods in comparative effectiveness research. Chapman and Hall\/CRC, 247\u2013292."},{"key":"e_1_3_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Philip Dawid. 2017. On individual risk. Synthese 194 9 (2017) 3445\u20133474.","DOI":"10.1007\/s11229-015-0953-4"},{"key":"e_1_3_3_2_13_2","unstructured":"Frances Ding Moritz Hardt John Miller and Ludwig Schmidt. 2021. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems 34 (2021) 6478\u20136490."},{"key":"e_1_3_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1201\/9781003484080"},{"key":"e_1_3_3_2_15_2","unstructured":"Jiayun Dong and Cynthia Rudin. 2019. Variable importance clouds: A way to explore variable importance for the set of good models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1901.03209 (2019)."},{"key":"e_1_3_3_2_16_2","unstructured":"Jon Donnelly Srikar Katta Cynthia Rudin and Edward Browne. 2023. The rashomon importance distribution: Getting rid of unstable single model-based variable importance. Advances in Neural Information Processing Systems 36 (2023) 6267\u20136279."},{"key":"e_1_3_3_2_17_2","unstructured":"Ally\u00a0Yalei Du Dung\u00a0Daniel Ngo and Zhiwei\u00a0Steven Wu. 2024. Reconciling Model Multiplicity for Downstream Decision Making. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2405.19667 (2024)."},{"key":"e_1_3_3_2_18_2","unstructured":"Aaron Fisher Cynthia Rudin and Francesca Dominici. 2019. All models are wrong but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research 20 177 (2019) 1\u201381."},{"key":"e_1_3_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Georgi Gardiner. 2018. Evidentialism and moral encroachment. Believing in accordance with the evidence: New essays on evidentialism (2018) 169\u2013195.","DOI":"10.1007\/978-3-319-95993-1_11"},{"key":"e_1_3_3_2_20_2","unstructured":"Parikshit Gopalan Michael\u00a0P Kim and Omer Reingold. 2023. Characterizing notions of omniprediction via multicalibration. arXiv e-prints (2023) arXiv\u20132302."},{"key":"e_1_3_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Ruocheng Guo Lu Cheng Jundong Li P\u00a0Richard Hahn and Huan Liu. 2020. A survey of learning causality with data: Problems and methods. ACM Computing Surveys (CSUR) 53 4 (2020) 1\u201337.","DOI":"10.1145\/3397269"},{"key":"e_1_3_3_2_22_2","first-page":"1939","volume-title":"International Conference on Machine Learning","author":"H\u00e9bert-Johnson Ursula","year":"2018","unstructured":"Ursula H\u00e9bert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. 2018. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning. PMLR, 1939\u20131948."},{"key":"e_1_3_3_2_23_2","unstructured":"Hans Hofmann. 1994. Statlog (German Credit Data). UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C5NC77."},{"key":"e_1_3_3_2_24_2","unstructured":"Hans Hofmann. 1994. Statlog (german credit data) data set. UCI Repository of Machine Learning Databases 53 (1994)."},{"key":"e_1_3_3_2_25_2","unstructured":"Hsiang Hsu and Flavio Calmon. 2022. Rashomon capacity: A metric for predictive multiplicity in classification. Advances in Neural Information Processing Systems 35 (2022) 28988\u201329000."},{"key":"e_1_3_3_2_26_2","doi-asserted-by":"crossref","unstructured":"Edward\u00a0H Kennedy. 2023. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics 17 2 (2023) 3008\u20133049.","DOI":"10.1214\/23-EJS2157"},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"publisher","unstructured":"David\u00a0M Kent and Nilay\u00a0D Shah. 2012. Risk models and patient-centered evidence: should physicians expect one right answer? JAMA 307 15 (April 2012) 1585\u20131586. https:\/\/doi.org\/10.1001\/jama.2012.469","DOI":"10.1001\/jama.2012.469"},{"key":"e_1_3_3_2_28_2","unstructured":"Christoph Kern Michael Kim and Angela Zhou. 2024. Multi-CATE: Multi-Accurate Conditional Average Treatment Effect Estimation Robust to Unknown Covariate Shifts. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2405.18206 (2024)."},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"crossref","unstructured":"Jon Kleinberg and Manish Raghavan. 2021. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences 118 22 (2021) e2018340118.","DOI":"10.1073\/pnas.2018340118"},{"key":"e_1_3_3_2_30_2","doi-asserted-by":"crossref","unstructured":"S\u00f6ren\u00a0R K\u00fcnzel Jasjeet\u00a0S Sekhon Peter\u00a0J Bickel and Bin Yu. 2019. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences 116 10 (2019) 4156\u20134165.","DOI":"10.1073\/pnas.1804597116"},{"key":"e_1_3_3_2_31_2","unstructured":"Julia Larson Julia Angwin Lauren Kirchner and Surya Mattu. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica. https:\/\/www.propublica.org\/article\/how-we-analyzed-the-compas-recidivism-algorithm"},{"key":"e_1_3_3_2_32_2","unstructured":"Benjamin Laufer Manisch Raghavan and Solon Barocas. 2024. Fundamental Limits in the Search for Less Discriminatory Algorithms\u2013and How to Avoid Them. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2412.18138 (2024)."},{"key":"e_1_3_3_2_33_2","unstructured":"William Lincoln and Josef Skrzypek. 1989. Synergy of clustering multiple back propagation networks. Advances in neural information processing systems 2 (1989)."},{"key":"e_1_3_3_2_34_2","unstructured":"Carol Long Hsiang Hsu Wael Alghamdi and Flavio Calmon. 2024. Individual arbitrariness and group fairness. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_3_2_35_2","first-page":"6765","volume-title":"International Conference on Machine Learning","author":"Marx Charles","year":"2020","unstructured":"Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive multiplicity in classification. In International Conference on Machine Learning. PMLR, 6765\u20136774."},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3593988"},{"key":"e_1_3_3_2_37_2","doi-asserted-by":"crossref","unstructured":"Xinkun Nie and Stefan Wager. 2021. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108 2 (2021) 299\u2013319.","DOI":"10.1093\/biomet\/asaa076"},{"key":"e_1_3_3_2_38_2","doi-asserted-by":"crossref","unstructured":"Brian\u00a0A Nosek George Alter George\u00a0C Banks Denny Borsboom Sara\u00a0D Bowman Steven\u00a0J Breckler Stuart Buck Christopher\u00a0D Chambers Gilbert Chin Garret Christensen et\u00a0al. 2015. Promoting an open research culture. Science 348 6242 (2015) 1422\u20131425.","DOI":"10.1126\/science.aab2374"},{"key":"e_1_3_3_2_39_2","unstructured":"F. Pedregosa G. Varoquaux A. Gramfort V. Michel B. Thirion O. Grisel M. Blondel P. Prettenhofer R. Weiss V. Dubourg J. Vanderplas A. Passos D. Cournapeau M. Brucher M. Perrot and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011) 2825\u20132830."},{"key":"e_1_3_3_2_40_2","unstructured":"Michael Redmond. 2009. Communities and Crime. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C53W3X."},{"key":"e_1_3_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3593980"},{"key":"e_1_3_3_2_42_2","doi-asserted-by":"crossref","unstructured":"Ewout\u00a0W Steyerberg Marinus\u00a0JC Eijkemans Eric Boersma and JDF Habbema. 2005. Equally valid models gave divergent predictions for mortality in acute myocardial infarction patients in a comparison of logical regression models. Journal of clinical epidemiology 58 4 (2005) 383\u2013390.","DOI":"10.1016\/j.jclinepi.2004.07.008"},{"key":"e_1_3_3_2_43_2","doi-asserted-by":"crossref","unstructured":"John\u00a0W Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics (1949) 99\u2013114.","DOI":"10.2307\/3001913"},{"key":"e_1_3_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3593998"},{"key":"e_1_3_3_2_45_2","unstructured":"Jamelle Watson-Daniels Flavio du\u00a0Pin Calmon Alexander D\u2019Amour Carol Long David\u00a0C Parkes and Berk Ustun. 2024. Predictive Churn with the Set of Good Models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2402.07745 (2024)."},{"key":"e_1_3_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i9.26227"},{"key":"e_1_3_3_2_47_2","unstructured":"Rui Xin Chudi Zhong Zhi Chen Takuya Takagi Margo Seltzer and Cynthia Rudin. 2022. Exploring the whole rashomon set of sparse decision trees. Advances in neural information processing systems 35 (2022) 14071\u201314084."},{"key":"e_1_3_3_2_48_2","unstructured":"Chudi Zhong Zhi Chen Jiachang Liu Margo Seltzer and Cynthia Rudin. 2024. Exploring and interacting with the set of good sparse generalized additive models. Advances in neural information processing systems 36 (2024)."}],"event":{"name":"FAccT '25: The 2025 ACM Conference on Fairness, Accountability, and Transparency","acronym":"FAccT '25","location":"Athens Greece"},"container-title":["Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715275.3732216","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T11:07:12Z","timestamp":1750763232000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715275.3732216"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":47,"alternative-id":["10.1145\/3715275.3732216","10.1145\/3715275"],"URL":"https:\/\/doi.org\/10.1145\/3715275.3732216","relation":{},"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"2025-06-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}