{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T20:25:08Z","timestamp":1793132708300,"version":"4.5.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,2,1]],"date-time":"2019-02-01T00:00:00Z","timestamp":1548979200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Queue"],"published-print":{"date-parts":[[2019,2]]},"abstract":"<jats:p>Flawed scholarship threatens to mislead the public and stymie future research by compromising ML\u2019s intellectual foundations. Indeed, many of these problems have recurred cyclically throughout the history of AI and, more broadly, in scientific research. In 1976, Drew McDermott chastised the AI community for abandoning self-discipline, warning prophetically that \"if we can\u2019t criticize ourselves, someone else will save us the trouble.\" The current strength of machine learning owes to a large body of rigorous research to date, both theoretical and empirical. By promoting clear scientific thinking and communication, our community can sustain the trust and investment it currently enjoys.<\/jats:p>","DOI":"10.1145\/3317287.3328534","type":"journal-article","created":{"date-parts":[[2020,9,3]],"date-time":"2020-09-03T00:23:40Z","timestamp":1599092620000},"page":"45-77","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":106,"title":["Troubling Trends in Machine Learning Scholarship"],"prefix":"10.1145","volume":"17","author":[{"given":"Zachary C.","family":"Lipton","sequence":"first","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"Steinhardt","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,2]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646031"},{"key":"e_1_2_1_2_1","first-page":"437","volume-title":"Neural Networks: Tricks of the Trade, ed. G. Montavon, G. B. Orr, KR M\u00fcller","author":"Bengio Y."},{"key":"e_1_2_1_3_1","unstructured":"Bostrom N. 2017. Superintelligence. Paris: Dunod.  Bostrom N. 2017. Superintelligence. Paris: Dunod."},{"issue":"1","key":"e_1_2_1_4_1","article-title":"Counterfactual reasoning and learning systems: the example of computational advertising","volume":"14","author":"Bottou L.","year":"2013","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.98.150201"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1223"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 18th International Conference on Artificial Intelligence and Statistics.","author":"Choromanska A.","year":"2015"},{"issue":"4","key":"e_1_2_1_8_1","first-page":"35","article-title":"How evaluation guides AI research: the message still counts more than the medium","volume":"9","author":"Cohen P. R.","year":"1988","journal-title":"AI Magazine"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2085"},{"key":"e_1_2_1_10_1","unstructured":"Council of the European Union. 2016. Motion for a European Parliament Resolution with Recommendations to the Commission on Civil Law Rules on Robotics; http:\/\/www.europarl.europa.eu\/sides\/getDoc.do?pubRef=-\/\/EP\/\/NONSGML%2BCOMPARL%2BPE-582.443%2B01%2BDOC%2BPDF%2BV0\/\/EN.  Council of the European Union. 2016. Motion for a European Parliament Resolution with Recommendations to the Commission on Civil Law Rules on Robotics; http:\/\/www.europarl.europa.eu\/sides\/getDoc.do?pubRef=-\/\/EP\/\/NONSGML%2BCOMPARL%2BPE-582.443%2B01%2BDOC%2BPDF%2BV0\/\/EN."},{"key":"e_1_2_1_11_1","unstructured":"Dauphin Y. N. Pascanu R. Gulcehre C. Cho K. Ganguli S. Bengio Y. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in Neural Information Processing Systems 2933?2941.   Dauphin Y. N. Pascanu R. Gulcehre C. Cho K. Ganguli S. Bengio Y. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in Neural Information Processing Systems 2933?2941."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature21056"},{"key":"e_1_2_1_13_1","unstructured":"Gershgorn D. 2017. The data that transformed AI research?and possibly the world. Quartz; https:\/\/qz.com\/1034972\/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world\/.  Gershgorn D. 2017. The data that transformed AI research?and possibly the world. Quartz; https:\/\/qz.com\/1034972\/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world\/."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Goodfellow I. J.","year":"2015"},{"key":"e_1_2_1_15_1","unstructured":"Hazirbas C. Leal-Taix\u00e9 L. Cremers D. 2017. Deep depth from focus. arXiv Preprint arXiv:1704.01085.  Hazirbas C. Leal-Taix\u00e9 L. Cremers D. 2017. Deep depth from focus. arXiv Preprint arXiv:1704.01085."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.123"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 32nd Association for the Advancement of Artificial Intelligence Conference.","author":"Henderson P.","year":"2018"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning 37; http:\/\/proceedings.mlr.press\/v37\/ioffe15","author":"Ioffe S.","year":"2015"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations.","author":"Kingma D. P.","year":"2015"},{"key":"e_1_2_1_20_1","unstructured":"Knuth D. E. Larrabee T. Roberts P. M. 1987. Mathematical writing; http:\/\/jmlr.csail.mit.edu\/reviewing-papers\/knuth_mathematical_writing.pdf.   Knuth D. E. Larrabee T. Roberts P. M. 1987. Mathematical writing; http:\/\/jmlr.csail.mit.edu\/reviewing-papers\/knuth_mathematical_writing.pdf."},{"key":"e_1_2_1_21_1","unstructured":"Langley P. Kibler D. 1991. The experimental study of machine learning; http:\/\/www.isle.org\/~langley\/papers\/mlexp.ps.  Langley P. Kibler D. 1991. The experimental study of machine learning; http:\/\/www.isle.org\/~langley\/papers\/mlexp.ps."},{"key":"e_1_2_1_22_1","volume-title":"International Conference on Machine Learning Workshop on Human Interpretability.","author":"Lipton Z. C.","year":"2016"},{"key":"e_1_2_1_23_1","first-page":"8136","article-title":"Does mitigating ML's impact disparity require treatment disparity","author":"Lipton Z. C.","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 32nd Conference on Neural Information Processing Systems. arXiv Preprint arXiv:1711","author":"Lucic M.","year":"2017"},{"key":"e_1_2_1_25_1","unstructured":"Markoff J. 2014. Researchers announce advance in image-recognition software. New York Times (November 17); https:\/\/www.nytimes.com\/2014\/11\/18\/science\/researchers-announce-breakthrough-in-content-recognition-software.html.  Markoff J. 2014. Researchers announce advance in image-recognition software. New York Times (November 17); https:\/\/www.nytimes.com\/2014\/11\/18\/science\/researchers-announce-breakthrough-in-content-recognition-software.html."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1045339.1045340"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Melis G.","year":"2018"},{"key":"e_1_2_1_28_1","unstructured":"Metz C. 2014. You don't have to be Google to build an artificial brain. Wired (September 26); https:\/\/www.wired.com\/2014\/09\/google-artificial-brain\/.  Metz C. 2014. You don't have to be Google to build an artificial brain. Wired (September 26); https:\/\/www.wired.com\/2014\/09\/google-artificial-brain\/."},{"key":"e_1_2_1_29_1","unstructured":"Minsky M. 2006. The Emotion Machine: Commonsense Thinking Artificial Intelligence and the Future of the Human Mind. New York: Simon & Schuster.   Minsky M. 2006. The Emotion Machine: Commonsense Thinking Artificial Intelligence and the Future of the Human Mind. New York: Simon & Schuster."},{"key":"e_1_2_1_30_1","unstructured":"Mohamed S. Lakshminarayanan B. 2016. Learning in implicit generative models. arXiv Preprint arXiv:1610.03483.  Mohamed S. Lakshminarayanan B. 2016. Learning in implicit generative models. arXiv Preprint arXiv:1610.03483."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.178"},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Nye M. J. 1980. N-rays: an episode in the history and psychology of science. Historical Studies in the Physical Sciences 11(1) 125?56.  Nye M. J. 1980. N-rays: an episode in the history and psychology of science. Historical Studies in the Physical Sciences 11(1) 125?56.","DOI":"10.2307\/27757473"},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349 (6251): aac4716.  Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349 (6251): aac4716.","DOI":"10.1126\/science.aac4716"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.146.3642.347"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Reddi S. J.","year":"2018"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1257\/aer.p20151066"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 32nd Conference on Neural Information Processing Systems; https:\/\/papers.nips.cc\/paper\/7515-how-does-batch-normalization-help-optimization.pdf.","author":"Santurkar S.","year":"2018"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations, Workshop Track.","author":"Sculley D.","year":"2018"},{"issue":"1","key":"e_1_2_1_39_1","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava N.","year":"2014","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning 37","author":"Steinhardt J.","year":"2015"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning 37","author":"Steinhardt J.","year":"2015"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 31st Conference on Neural Information Processing Systems; https:\/\/papers.nips.cc\/paper\/6943-certified-defenses-for-data-poisoning-attacks.pdf.","author":"Steinhardt J.","year":"2017"},{"key":"e_1_2_1_43_1","unstructured":"Stock P. Cisse M. 2017. ConvNets and ImageNet beyond accuracy: explanations bias detection adversarial examples and model criticism. arXiv Preprint arXiv:1711.11443.  Stock P. Cisse M. 2017. ConvNets and ImageNet beyond accuracy: explanations bias detection adversarial examples and model criticism. arXiv Preprint arXiv:1711.11443."},{"key":"e_1_2_1_44_1","volume-title":"International Conference on Learning Representations. arXiv Preprint arXiv:1312","author":"Szegedy C.","year":"2013"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00611"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Zhang C.","year":"2017"}],"container-title":["Queue"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3317287.3328534","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3317287.3328534","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T16:01:17Z","timestamp":1750262477000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3317287.3328534"}},"subtitle":["Some ML papers suffer from flaws that could mislead the public and stymie future research."],"short-title":[],"issued":{"date-parts":[[2019,2]]},"references-count":46,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,2]]}},"alternative-id":["10.1145\/3317287.3328534"],"URL":"https:\/\/doi.org\/10.1145\/3317287.3328534","relation":{},"ISSN":["1542-7730","1542-7749"],"issn-type":[{"value":"1542-7730","type":"print"},{"value":"1542-7749","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,2]]},"assertion":[{"value":"2019-02-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}