{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T19:53:56Z","timestamp":1792785236803,"version":"4.5.0"},"reference-count":59,"publisher":"National Academy of Sciences","issue":"39","license":[{"start":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T00:00:00Z","timestamp":1790121600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Stanford University Center for Human-Centered AI","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}],"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2026,9,29]]},"abstract":"<jats:p>\n                    Machine learning is central to modern science, industry, and policy, yet its predictive power often comes at the cost of transparency: We rarely know which input features drive a model\u2019s predictions. Without such understanding, researchers cannot draw reliable conclusions, practitioners cannot ensure fairness or accountability, and policymakers cannot trust or govern model-based decisions. Existing tools for assessing feature influence are limited; most lack statistical guarantees, and many require costly retraining or surrogate modeling, making them impractical for large modern models. We introduce AICO (Add-In COvariates), a broadly applicable framework that turns model interpretability into an efficient statistical exercise. AICO tests whether each feature contributes to predictive performance by masking its information and measuring the resulting change. The method provides exact, finite-sample feature\n                    <jats:inline-formula>\n                      <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"inline\" overflow=\"scroll\">\n                        <mml:mi>P<\/mml:mi>\n                      <\/mml:math>\n                    <\/jats:inline-formula>\n                    -values and CIs for feature importance through a simple, nonasymptotic hypothesis testing procedure. It requires no retraining, surrogate modeling, or distributional assumptions, making it feasible for large-scale algorithms. In both controlled experiments and real applications, from credit scoring to mortgage-behavior prediction, AICO identifies variables that contribute to model behavior, providing a scalable and statistically principled path toward transparent and trustworthy machine learning.\n                  <\/jats:p>","DOI":"10.1073\/pnas.2530045123","type":"journal-article","created":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T16:25:03Z","timestamp":1790180703000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":0,"title":["AICO: Feature significance tests for supervised learning"],"prefix":"10.1073","volume":"123","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5380-2918","authenticated-orcid":false,"given":"Kay","family":"Giesecke","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/03bnmw459","id-type":"ROR","asserted-by":"publisher"}],"name":"Hasso Plattner Institute, University of Potsdam","place":["Potsdam, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Enguerrand","family":"Horel","sequence":"additional","affiliation":[{"name":"Upstart Inc.","place":["San Mateo"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3372-5629","authenticated-orcid":false,"given":"Chartsiri","family":"Jirachotkulthorn","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Institute for Computational and Mathematical Engineering, Stanford University","place":["Stanford"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2026,9,23]]},"reference":[{"key":"e_1_3_4_1_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12340"},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1080\/10618600.2021.1923520"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1307116"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12265"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2019.1660174"},{"key":"e_1_3_4_6_2","unstructured":"E. Epstein A. Sadhwani K. Giesecke A set-sequence model for time series. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2505.11243 (Accessed 27 July 2026)."},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12228"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1900654116"},{"key":"e_1_3_4_9_2","unstructured":"S. M. Lundberg S. I. Lee \u201cA unified approach to interpreting model predictions\u201d in Advances in Neural Information Processing Systems I. Guyon U. von Luxburg S. Bengio H. Wallach R. Fergus S. Vishwanathan R. Garnett Eds. (Lundberg and Lee NeurIPS 2017) pp. 4765\u20134774."},{"key":"e_1_3_4_10_2","doi-asserted-by":"crossref","unstructured":"M. T. Ribeiro S. Singh C. Guestrin \u201cWhy should I trust you?: Explaining the predictions of any classifier\u201d in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining B. Krishnapuram M. Shah A. J. Smola C. C. Aggarwal D. Shen R. Rastogi Eds. (Association for Computing Machinery New York NY 2016) pp. 1135\u20131144.","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_3_4_12_2","first-page":"1","article-title":"All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously","volume":"20","author":"Fisher A.","year":"2019","unstructured":"A. Fisher, C. Rudin, F. Dominici, All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res. 20, 1\u201381 (2019).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_13_2","first-page":"623","article-title":"Feature importance: A closer look at Shapley values and LOCO","volume":"39","author":"Verdinelli I.","year":"2023","unstructured":"I. Verdinelli, L. Wasserman, Feature importance: A closer look at Shapley values and LOCO. Stat. Sci. 39, 623\u2013636 (2023).","journal-title":"Stat. Sci."},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-021-10057-z"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.2307\/2171848"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0266466600164059"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0304-4076(01)00091-4"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203462"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1214\/15-EJS1005"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/72.935080"},{"key":"e_1_3_4_21_2","first-page":"1","article-title":"Significance tests for neural networks","volume":"21","author":"Horel E.","year":"2020","unstructured":"E. Horel, K. Giesecke, Significance tests for neural networks. J. Mach. Learn. Res. 21, 1\u201329 (2020).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2023.105574"},{"key":"e_1_3_4_23_2","first-page":"1","article-title":"Scalable and efficient hypothesis testing with random forests","volume":"23","author":"Coleman T.","year":"2022","unstructured":"T. Coleman, W. Peng, L. Mentch, Scalable and efficient hypothesis testing with random forests. J. Mach. Learn. Res. 23, 1\u201335 (2022).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_24_2","first-page":"1","article-title":"Quantifying uncertainty in random forests via confidence intervals and hypothesis tests","volume":"17","author":"Mentch L.","year":"2016","unstructured":"L. Mentch, G. Hooker, Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. J. Mach. Learn. Res. 17, 1\u201341 (2016).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/asab039"},{"key":"e_1_3_4_26_2","unstructured":"A. Bellot M. v. d. Schaar \u201cConditional independence testing using generative adversarial networks\u201d in Proceedings of the 33rd International Conference on Neural Information Processing Systems H. Wallach H. Larochelle A. Beygelzimer F. d\u2019Alche-Buc E. Fox R. Garnett Eds. (Curran Associates Inc. 2019) pp. 2202\u20132211."},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-021-06030-6"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2205518119"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1214\/24-AOS2447"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1111\/biom.13392"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2021.2003200"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1093\/jrsssb\/qkae115"},{"key":"e_1_3_4_33_2","unstructured":"L. Gan L. Zheng G. I. Allen Model-agnostic confidence intervals for feature importance: A fast and powerful approach using minipatch ensembles. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2206.02088 (Accessed 27 July 2026)."},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1214\/18-AOS1784"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2020.1729163"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/62.2.441"},{"key":"e_1_3_4_37_2","first-page":"358","article-title":"Fuzzy and randomized confidence intervals and p-values","volume":"20","author":"Geyer C. J.","year":"2005","unstructured":"C. J. Geyer, G. D. Meeden, Fuzzy and randomized confidence intervals and p-values. Stat. Sci. 20, 358\u2013366 (2005).","journal-title":"Stat. Sci."},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/asn061"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jspi.2015.06.003"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssc.12529"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1987.10478478"},{"key":"e_1_3_4_43_2","volume-title":"Nonparametrics: Statistical Methods Based on Ranks","author":"Lehmann E.","year":"2006","unstructured":"E. Lehmann, Nonparametrics: Statistical Methods Based on Ranks (Springer, 2006)."},{"key":"e_1_3_4_44_2","unstructured":"N. Hama M. Mase A. B. Owen Model free Shapley values for high dimensional data. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2211.08414 (Accessed 27 July 2026)."},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1080\/02331880412331329854"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611970319"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176347995"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176349256"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-69033-9"},{"key":"e_1_3_4_50_2","first-page":"93","article-title":"An asymptotic UMP sign test for discretised data","volume":"38","author":"Wittkowski K. M.","year":"1989","unstructured":"K. M. Wittkowski, An asymptotic UMP sign test for discretised data. J. R. Stat. Soc. Ser. D Stat. 38, 93\u201396 (1989).","journal-title":"J. R. Stat. Soc. Ser. D Stat."},{"key":"e_1_3_4_51_2","doi-asserted-by":"crossref","unstructured":"T. Chen C. Guestrin \u201cXgboost: A scalable tree boosting system\u201d in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining B. Krishnapuram M. Shah A. J. Smola C. C. Aggarwal D. Shen R. Rastogi Eds. (Association for Computing Machinery New York NY 2016) pp. 785\u2013794.","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_3_4_52_2","doi-asserted-by":"crossref","unstructured":"T. Akiba et al. \u201cA next-generation hyperparameter optimization framework\u201d in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining A. Teredesai V. Kumar Y. Li R. Rosales E. Terzi G. Karypis Eds. (Association for Computing Machinery New York NY 2019) pp. 2623\u20132631.","DOI":"10.1145\/3292500.3330701"},{"key":"e_1_3_4_53_2","first-page":"2546","article-title":"Algorithms for hyper-parameter optimization","volume":"24","author":"Bergstra J.","year":"2011","unstructured":"J. Bergstra, R. Bardenet, Y. Bengio, B. K\u00e9gl, Algorithms for hyper-parameter optimization. Adv. Neural Inf. Process. Syst. 24, 2546\u20132554 (2011).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_3_4_55_2","first-page":"313","article-title":"Deep learning for mortgage risk","volume":"19","author":"Sadhwani A.","year":"2021","unstructured":"A. Sadhwani, K. Giesecke, J. Sirignano, Deep learning for mortgage risk. J. Financ. Econom. 19, 313\u2013368 (2021).","journal-title":"J. Financ. Econom."},{"key":"e_1_3_4_56_2","doi-asserted-by":"crossref","unstructured":"M. Van Ness T. Bosschieter R. Halpin-Gregorio M. Udell \u201cThe missing indicator method: From low to high dimensions\u201d in 29th SIGKDD Conference on Knowledge Discovery and Data Mining A. K. Singh Y. Sun L. Akoglu D. Gunopulos X. Yan R. Kumar F. Ozcan J. Ye Eds. (Association for Computing Machinery New York NY 2023).","DOI":"10.1145\/3580305.3599911"},{"key":"e_1_3_4_57_2","unstructured":"D. Y. Fu et al. \u201cSimple hardware-efficient long convolutions for sequence modeling\u201d in Proceedings of the 40th International Conference on Machine Learning ICML 2023 A. Krause E. Brunskill K. Cho B. Engelhardt S. Sabato J. Scarlett Eds. (Proceedings of Machine Learning Research 2023) vol. 202."},{"key":"e_1_3_4_58_2","unstructured":"C. Jirachotkulthorn AICO. Github https:\/\/github.com\/ji-chartsiri\/AICO. Deposited 8 February 2026."},{"key":"e_1_3_4_59_2","unstructured":"I.-C. Yeh Default of Credit Card Clients [Dataset]. UCI Machine Learning Repository. https:\/\/doi.org\/10.24432\/C55S3H. Accessed 18 April 2024."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2530045123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T16:25:56Z","timestamp":1790180756000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2530045123"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,23]]},"references-count":59,"journal-issue":{"issue":"39","published-print":{"date-parts":[[2026,9,29]]}},"alternative-id":["10.1073\/pnas.2530045123"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2530045123","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,23]]},"assertion":[{"value":"2025-10-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2530045123"}}