{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T15:32:41Z","timestamp":1783438361733,"version":"3.54.6"},"reference-count":45,"publisher":"Wiley","issue":"23-24","license":[{"start":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T00:00:00Z","timestamp":1759968000000},"content-version":"vor","delay-in-days":8,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["BO3139\/7"],"award-info":[{"award-number":["BO3139\/7"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["BO3139\/9\u20101"],"award-info":[{"award-number":["BO3139\/9\u20101"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","award":["MC_UU_00004\/09"],"award-info":[{"award-number":["MC_UU_00004\/09"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Statistics in Medicine"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>Comparison studies in methodological research are intended to compare methods in an evidence\u2010based manner to help data analysts select a suitable method for their application. To provide trustworthy evidence, they must be carefully designed, implemented, and reported, especially given the many decisions made in planning and running. A common challenge in comparison studies is to handle the \u201cfailure\u201d of one or more methods to produce a result for some (real or simulated) data sets, such that their performances cannot be measured in those instances. Despite an increasing emphasis on this topic in recent literature (focusing on non\u2010convergence as a common manifestation), there is little guidance on proper handling and interpretation, and reporting of the chosen approach is often neglected. This paper aims to fill this gap and offers practical guidance on handling method failure in comparison studies. After exploring common handlings across various published comparison studies from classical statistics and predictive modeling, we show that the popular approaches of discarding data sets yielding failure (either for all or the failing methods only) and imputing are inappropriate in most cases. We then recommend a different perspective on method failure\u2014viewing it as the result of a complex interplay of several factors rather than just its manifestation. Building on this, we provide recommendations on more adequate handling of method failure derived from realistic considerations. In particular, we propose considering fallback strategies that directly reflect the behavior of real\u2010world users. Finally, we illustrate our recommendations and the dangers of inadequate handling of method failure through two exemplary comparison studies.<\/jats:p>","DOI":"10.1002\/sim.70257","type":"journal-article","created":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T12:46:44Z","timestamp":1760014004000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Rethinking the Handling of Method Failure in Comparison Studies"],"prefix":"10.1002","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-1982-9260","authenticated-orcid":false,"given":"Milena","family":"W\u00fcnsch","sequence":"first","affiliation":[{"name":"Institute for Medical Information Processing, Biometry, and Epidemiology, Faculty of Medicine LMU Munich  Munich Germany"},{"name":"Munich Center for Machine Learning (MCML)  Munich Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moritz","family":"Herrmann","sequence":"additional","affiliation":[{"name":"Institute for Medical Information Processing, Biometry, and Epidemiology, Faculty of Medicine LMU Munich  Munich Germany"},{"name":"Munich Center for Machine Learning (MCML)  Munich Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elisa","family":"Noltenius","sequence":"additional","affiliation":[{"name":"Department of Statistics LMU Munich  Munich Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mattia","family":"Mohr","sequence":"additional","affiliation":[{"name":"Department of Statistics LMU Munich  Munich Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5850-3610","authenticated-orcid":false,"given":"Tim P.","family":"Morris","sequence":"additional","affiliation":[{"name":"MRC Clinical Trials Unit at UCL University College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2729-0947","authenticated-orcid":false,"given":"Anne\u2010Laure","family":"Boulesteix","sequence":"additional","affiliation":[{"name":"Institute for Medical Information Processing, Biometry, and Epidemiology, Faculty of Medicine LMU Munich  Munich Germany"},{"name":"Munich Center for Machine Learning (MCML)  Munich Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,10,9]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.8086"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.202200091"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1037\/met0000695"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-017-0417-2"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.202200212"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11136-018-1861-0"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-016-0267-3"},{"issue":"101","key":"e_1_2_9_9_1","first-page":"1","article-title":"AMLB: An AutoML Benchmark","volume":"25","author":"Gijsbers P.","year":"2024","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.202300101"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2041-210x.2012.00250.x"},{"key":"e_1_2_9_12_1","doi-asserted-by":"publisher","DOI":"10.1002\/wics.1626"},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prevetmed.2009.10.004"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.6191"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.9932"},{"issue":"1","key":"e_1_2_9_16_1","first-page":"3133","article-title":"Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?","volume":"15","author":"Fern\u00e1ndez\u2010Delgado M.","year":"2014","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbaa167"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00180-013-0420-y"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1441"},{"key":"e_1_2_9_20_1","doi-asserted-by":"crossref","unstructured":"S.Pawel F.Barto\u0161 B. S.Siepe andA.Lohmann \u201cHandling Missingness Failures and Non\u2010Convergence in Simulation Studies: A Review of Current Practices and Recommendations \u201d arXiv preprint arXiv:2409.18527(2024).","DOI":"10.1080\/00031305.2025.2540002"},{"key":"e_1_2_9_21_1","volume-title":"Multiple Imputation for Nonresponse in Surveys","author":"Rubin D. B.","year":"2004"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781119756118"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1214\/09\u2010sts293"},{"key":"e_1_2_9_24_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj\u20102023\u2010076316"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.202200222"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1330"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1093\/ije\/dyad134"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1579"},{"key":"e_1_2_9_29_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-019-1738-8"},{"key":"e_1_2_9_30_1","volume-title":"Empirical Methods for Artificial Intelligence","author":"Cohen P. R.","year":"1995"},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-52250-0"},{"key":"e_1_2_9_32_1","volume-title":"Design and Analysis of Experiments","author":"Montgomery D. C.","year":"2020"},{"key":"e_1_2_9_33_1","doi-asserted-by":"publisher","DOI":"10.1525\/collabra.117094"},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-1809.1955.tb01285.x"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.2307\/2332926"},{"key":"e_1_2_9_36_1","doi-asserted-by":"crossref","unstructured":"T. J.Aragon M. P.Fay D.Wollschlaeger andA.Omidpanah \u201cEpitools: Epidemiology Tools \u201d R package version 0.5\u201310.1(2020) https:\/\/doi.org\/10.32614\/CRAN.package.epitools.","DOI":"10.32614\/CRAN.package.epitools"},{"key":"e_1_2_9_37_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-1809.1955.tb01348.x"},{"key":"e_1_2_9_38_1","doi-asserted-by":"crossref","unstructured":"F.SchaarschmidtandD.Gerhard \u201cpairwiseCI: Confidence Intervals for Two Sample Comparisons \u201d R package version 0.1\u201327(2019) https:\/\/doi.org\/10.32614\/CRAN.package.pairwiseCI.","DOI":"10.32614\/CRAN.package.pairwiseCI"},{"key":"e_1_2_9_39_1","unstructured":"H.Schulz\u2010K\u00fcmpel S.Fischer R.Hornung A. L.Boulesteix T.Nagler andB.Bischl \u201cConstructing Confidence Intervals For'the'generalization Error\u2013a Comprehensive Benchmark Study \u201d arXiv preprint arXiv:2409.18836(2024)."},{"key":"e_1_2_9_40_1","volume-title":"Classification and Regression Trees","author":"Breiman L.","year":"1984"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.5334\/jopd.32"},{"key":"e_1_2_9_42_1","doi-asserted-by":"publisher","DOI":"10.15626\/MP.2020.2556"},{"key":"e_1_2_9_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"e_1_2_9_44_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1024068626366"},{"key":"e_1_2_9_45_1","doi-asserted-by":"publisher","DOI":"10.1093\/aje\/kwx299"},{"key":"e_1_2_9_46_1","doi-asserted-by":"publisher","DOI":"10.1111\/1740-9713.01444"}],"container-title":["Statistics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/sim.70257","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,29]],"date-time":"2025-10-29T06:09:20Z","timestamp":1761718160000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/sim.70257"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10]]},"references-count":45,"journal-issue":{"issue":"23-24","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["10.1002\/sim.70257"],"URL":"https:\/\/doi.org\/10.1002\/sim.70257","archive":["Portico"],"relation":{},"ISSN":["0277-6715","1097-0258"],"issn-type":[{"value":"0277-6715","type":"print"},{"value":"1097-0258","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10]]},"assertion":[{"value":"2024-08-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-28","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70257"}}