{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,28]],"date-time":"2025-09-28T12:46:51Z","timestamp":1759063611626,"version":"3.37.3"},"reference-count":16,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,9,4]],"date-time":"2023-09-04T00:00:00Z","timestamp":1693785600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,4]],"date-time":"2023-09-04T00:00:00Z","timestamp":1693785600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100019827","name":"Meta","doi-asserted-by":"publisher","award":["employee"],"award-info":[{"award-number":["employee"]}],"id":[{"id":"10.13039\/100019827","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Adv Comput Math"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The author\u2019s recent research papers, \u201cCumulative deviation of a subpopulation from the full population\u201d and \u201cA graphical method of cumulative differences between two subpopulations\u201d (both published in volume 8 of Springer\u2019s open-access <jats:italic>Journal of Big Data<\/jats:italic> during 2021), propose graphical methods and summary statistics, without extensively calibrating formal significance tests. The summary metrics and methods can measure the calibration of probabilistic predictions and can assess differences in responses between a subpopulation and the full population while controlling for a covariate or score via conditioning on it. These recently published papers construct significance tests based on the scalar summary statistics, but only sketch how to calibrate the attained significance levels (also known as \u201cP-values\u201d) for the tests. The present article reviews and synthesizes work spanning many decades in order to detail how to calibrate the <jats:italic>P<\/jats:italic>-values. The present paper presents computationally efficient, easily implemented numerical methods for evaluating properly calibrated <jats:italic>P<\/jats:italic>-values, together with rigorous mathematical proofs guaranteeing their accuracy, and illustrates and validates the methods with open-source software and numerical examples.<\/jats:p>","DOI":"10.1007\/s10444-023-10068-6","type":"journal-article","created":{"date-parts":[[2023,9,4]],"date-time":"2023-09-04T07:02:43Z","timestamp":1693810963000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Calibration of P-values for calibration and for deviation of a subpopulation from the full population"],"prefix":"10.1007","volume":"49","author":[{"given":"Mark","family":"Tygert","sequence":"first","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,9,4]]},"reference":[{"key":"10068_CR1","doi-asserted-by":"crossref","unstructured":"Tygert M.: Cumulative deviation of a subpopulation from the full population. J Big Data 8(117), 1\u201360 (2021b). https:\/\/arxiv.org\/abs\/2008.01779","DOI":"10.1186\/s40537-021-00494-y"},{"key":"10068_CR2","doi-asserted-by":"crossref","unstructured":"Tygert M.: A graphical method of cumulative differences between two subpopulations. J Big Data 8(158), 1\u201329 (2021c). https:\/\/arxiv.org\/abs\/2108.02666","DOI":"10.1186\/s40537-021-00540-9"},{"key":"10068_CR3","unstructured":"Kloumann I, Korevaar H, McConnell C, Tygert M, Zhao J.: Cumulative differences between paired samples. Tech. Rep. 2305.11323 (2023). arXiv: https:\/\/arxiv.org\/abs\/2305.11323"},{"key":"10068_CR4","unstructured":"Tygert M.: Controlling for multiple covariates. Tech. Rep. 2112.00672 (2021a). arXiv: https:\/\/arxiv.org\/abs\/2112.00672"},{"key":"10068_CR5","unstructured":"Arrieta-Ibarra I, Gujral P, Tannen J, Tygert M, Xu C.: Metrics of calibration for probabilistic predictions. J Mach Learn Res 23, 1\u201354 (2022). https:\/\/arxiv.org\/abs\/2205.09680"},{"key":"10068_CR6","unstructured":"Lee D, Huang X, Hassani H, Dobriban E (2022) T-Cal: an optimal test for the calibration of predictive models. Tech. Rep. 2203.01850. arXiv"},{"issue":"3","key":"10068_CR7","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1016\/0167-7152(93)90167-H","volume":"17","author":"MA Delgado","year":"1993","unstructured":"Delgado, M.A.: Testing the equality of nonparametric regression curves. Stat Probab Lett 17(3), 199\u2013204 (1993)","journal-title":"Stat Probab Lett"},{"issue":"1","key":"10068_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0378-3758(94)00045-W","volume":"44","author":"J Diebolt","year":"1995","unstructured":"Diebolt, J.: A nonparametric test for the regression function: asymptotic theory. J Stat Plan Inference 44(1), 1\u201317 (1995)","journal-title":"J Stat Plan Inference"},{"issue":"2","key":"10068_CR9","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1214\/aos\/1031833666","volume":"25","author":"W Stute","year":"1997","unstructured":"Stute, W.: Nonparametric model checks for regression. Ann Stat 25(2), 613\u2013641 (1997)","journal-title":"Ann Stat"},{"key":"10068_CR10","first-page":"38","volume":"63","author":"NH Kuiper","year":"1962","unstructured":"Kuiper, N.H.: Tests concerning random points on a circle. Proc Koninklijke Nederlandse Akademie van Wetenschappen Series A 63, 38\u201347 (1962)","journal-title":"Proc Koninklijke Nederlandse Akademie van Wetenschappen Series A"},{"key":"10068_CR11","first-page":"83","volume":"4","author":"AN Kolmogorov","year":"1933","unstructured":"Kolmogorov, A.N.: Sulla determinazione empirica di una legge di distribuzione (On the empirical determination of a distribution function). Giorn Ist Ital Attuar 4, 83\u201391 (1933)","journal-title":"Giorn Ist Ital Attuar"},{"issue":"2","key":"10068_CR12","first-page":"3","volume":"2","author":"N Smirnov","year":"1939","unstructured":"Smirnov, N.: On the estimation of the discrepancy between empirical curves of distribution for two independent samples. Bulletin Math\u00e9matique de l\u2019Universit\u00e9 de Moscou 2(2), 3\u201311 (1939)","journal-title":"Bulletin Math\u00e9matique de l\u2019Universit\u00e9 de Moscou"},{"issue":"3","key":"10068_CR13","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1214\/aoms\/1177729589","volume":"22","author":"W Feller","year":"1951","unstructured":"Feller, W.: The asymptotic distribution of the range of sums of independent random variables. Ann Math Stat 22(3), 427\u2013432 (1951)","journal-title":"Ann Math Stat"},{"issue":"4","key":"10068_CR14","doi-asserted-by":"publisher","first-page":"624","DOI":"10.1214\/aoms\/1177728918","volume":"24","author":"DA Darling","year":"1953","unstructured":"Darling, D.A., Siegert, A.J.F.: The first passage problem for a continuous Markov process. Ann Math Stat 24(4), 624\u2013639 (1953)","journal-title":"Ann Math Stat"},{"issue":"3","key":"10068_CR15","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1090\/S0002-9947-1962-0143257-8","volume":"103","author":"Z Ciesielski","year":"1962","unstructured":"Ciesielski, Z., Taylor, S.J.: First passage times and sojourn times for Brownian motion in space and the exact Hausdorff measure of the sample path. Trans Am Math Soc 103(3), 434\u2013450 (1962)","journal-title":"Trans Am Math Soc"},{"key":"10068_CR16","doi-asserted-by":"crossref","unstructured":"Masoliver J.: Extreme values and the level-crossing problem: an application to the Feller process. Phys Rev E 89(4), 042106 (2014)","DOI":"10.1103\/PhysRevE.89.042106"}],"container-title":["Advances in Computational Mathematics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10444-023-10068-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10444-023-10068-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10444-023-10068-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,23]],"date-time":"2023-10-23T12:04:33Z","timestamp":1698062673000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10444-023-10068-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,4]]},"references-count":16,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["10068"],"URL":"https:\/\/doi.org\/10.1007\/s10444-023-10068-6","relation":{},"ISSN":["1019-7168","1572-9044"],"issn-type":[{"type":"print","value":"1019-7168"},{"type":"electronic","value":"1572-9044"}],"subject":[],"published":{"date-parts":[[2023,9,4]]},"assertion":[{"value":"14 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 July 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Meta Platforms, Inc. employs the author. The author receives a salary and stock from Meta.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"70"}}