{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T15:23:04Z","timestamp":1772810584493,"version":"3.50.1"},"reference-count":25,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,3,24]],"date-time":"2018-03-24T00:00:00Z","timestamp":1521849600000},"content-version":"vor","delay-in-days":365,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000879","name":"Alfred P. Sloan Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000879","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS 1253345"],"award-info":[{"award-number":["CNS 1253345"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2017,3,24]]},"abstract":"<jats:p>Existing approaches to ensuring the validity of inferences drawn from data assume a fixed procedure to be performed, selected before the data are examined. Yet the practice of data analysis is an intrinsically interactive and adaptive process: new analyses and hypotheses are proposed after seeing the results of previous ones, parameters are tuned on the basis of obtained results, and datasets are shared and reused.<\/jats:p>\n                  <jats:p>In this work, we initiate a principled study of how to guarantee the validity of statistical inference in adaptive data analysis. We demonstrate new approaches for addressing the challenges of adaptivity that are based on techniques developed in privacy-preserving data analysis.&lt;!-- END_PAGE_1 --&gt;<\/jats:p>\n                  <jats:p>As an application of our techniques we give a simple and practical method for reusing a holdout (or testing) set to validate the accuracy of hypotheses produced adaptively by a learning algorithm operating on a training set.<\/jats:p>","DOI":"10.1145\/3051088","type":"journal-article","created":{"date-parts":[[2017,3,24]],"date-time":"2017-03-24T10:54:11Z","timestamp":1490352851000},"page":"86-93","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Guilt-free data reuse"],"prefix":"10.1145","volume":"60","author":[{"given":"Cynthia","family":"Dwork","sequence":"first","affiliation":[{"name":"Microsoft Research, Mountain View, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vitaly","family":"Feldman","sequence":"additional","affiliation":[{"name":"IBM Almaden Research Center, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Moritz","family":"Hardt","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Toniann","family":"Pitassi","sequence":"additional","affiliation":[{"name":"University of Toronto, Toronto, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Omer","family":"Reingold","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aaron","family":"Roth","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,3,24]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Aschwanden C. Science isn't broken."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897518.2897566"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065167.1065184"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045226"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760200704"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859921"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866739.1866758"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2746539.2746580"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969442.2969502"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/11681878_14"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1561\/0400000042"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1983.10482729"},{"key":"e_1_2_1_14_1","first-page":"6","article-title":"The statistical crisis in science","volume":"102","author":"Gelman A.","year":"2014","unstructured":"Gelman, A., Loken, E. The statistical crisis in science. Am. Statist. 102, 6 (2014), 460.","journal-title":"Am. Statist."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/FOCS.2010.85"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/FOCS.2014.55"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/293347.293351"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10444-004-7634-z"},{"key":"e_1_2_1_21_1","volume-title":"On the generalization properties of differential privacy. CoRR","author":"Nissim K.","year":"2015","unstructured":"Nissim, K., Stemmer, U. On the generalization properties of differential privacy. CoRR (2015), abs\/1504.05800."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944978"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/2621980"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1953019"},{"key":"e_1_2_1_25_1","volume-title":"COLT","author":"Steinke T.","year":"2015","unstructured":"Steinke, T., Ullman, J. Interactive fingerprinting codes and the hardness of preventing false discovery. In COLT, Paris, France (2015), 1588--1628."}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3051088","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3051088","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3051088","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T09:35:25Z","timestamp":1763458525000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3051088"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,3,24]]},"references-count":25,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017,3,24]]}},"alternative-id":["10.1145\/3051088"],"URL":"https:\/\/doi.org\/10.1145\/3051088","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,3,24]]},"assertion":[{"value":"2017-03-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}