{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T18:16:08Z","timestamp":1781979368870,"version":"3.54.5"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2017,7,27]],"date-time":"2017-07-27T00:00:00Z","timestamp":1501113600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Google Research Grant"},{"name":"Simons Investigator Award"},{"name":"Facebook Faculty Research Grant"},{"name":"ARO MURI Grant"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2017,11,30]]},"abstract":"<jats:p>An increasing number of domains are providing us with detailed trace data on human decisions in settings where we can evaluate the quality of these decisions via an algorithm. Motivated by this development, an emerging line of work has begun to consider whether we can characterize and predict the kinds of decisions where people are likely to make errors.<\/jats:p>\n          <jats:p>\n            To investigate what a general framework for human error prediction might look like, we focus on a model system with a rich history in the behavioral sciences: the decisions made by chess players as they select moves in a game. We carry out our analysis at a large scale, employing datasets with several million recorded games, and using\n            <jats:italic>chess tablebases<\/jats:italic>\n            to acquire a form of ground truth for a subset of chess positions that have been completely solved by computers but remain challenging for even the best players in the world.\n          <\/jats:p>\n          <jats:p>We organize our analysis around three categories of features that we argue are present in most settings where the analysis of human error is applicable: the skill of the decision-maker, the time available to make the decision, and the inherent difficulty of the decision. We identify rich structure in all three of these categories of features, and find strong evidence that in our domain, features describing the inherent difficulty of an instance are significantly more powerful than features based on skill or time.<\/jats:p>","DOI":"10.1145\/3046947","type":"journal-article","created":{"date-parts":[[2017,7,28]],"date-time":"2017-07-28T12:19:47Z","timestamp":1501244387000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Assessing Human Error Against a Benchmark of Perfection"],"prefix":"10.1145","volume":"11","author":[{"given":"Ashton","family":"Anderson","sequence":"first","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jon","family":"Kleinberg","sequence":"additional","affiliation":[{"name":"Cornell University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sendhil","family":"Mullainathan","sequence":"additional","affiliation":[{"name":"Harvard University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2017,7,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.97.1.129"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.53.2.244"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogpsych.2007.02.001"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2015.233"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5220\/0005288306020607"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.4159\/harvard.9780674732469"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01359217"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0285(73)90004-2"},{"key":"e_1_2_1_9_1","unstructured":"Adriaan D. De Groot. 1978. Thought and Choice in Chess Vol. 4. Walter de Gruyter.  Adriaan D. De Groot. 1978. Thought and Choice in Chess Vol. 4. Walter de Gruyter."},{"key":"e_1_2_1_10_1","unstructured":"Arpad E. Elo. 1978. The Rating of Chessplayers Past and Present. Arco Pub.  Arpad E. Elo. 1978. The Rating of Chessplayers Past and Present. Arco Pub."},{"key":"e_1_2_1_11_1","unstructured":"Reuben Fine. 1941. Basic Chess Endings Number 3. D. McKay Co.  Reuben Fine. 1941. Basic Chess Endings Number 3. D. McKay Co."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of Advances in Neural Information Processing Systems. 569--576","author":"Herbrich Ralf","year":"2006"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4613-9080-0_10"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/0022-1031(67)90034-0"},{"key":"e_1_2_1_15_1","unstructured":"Barry Kirwan. 1993. Human Reliability Assessment. Wiley Online Library.  Barry Kirwan. 1993. Human Reliability Assessment. Wiley Online Library."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4613-9080-0_2"},{"key":"e_1_2_1_17_1","unstructured":"Himabindu Lakkaraju Jon Kleinberg Jure Leskovec Jens Ludwig and Sendhil Mullainathan. 2016. Human decisions and machine predictions. (2016).  Himabindu Lakkaraju Jon Kleinberg Jure Leskovec Jens Ludwig and Sendhil Mullainathan. 2016. Human decisions and machine predictions. (2016)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974010.21"},{"key":"e_1_2_1_19_1","volume-title":"Accessed","author":"Makhnychev Vladimir","year":"2012"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4613-9080-0_14"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009905800005"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1037\/0012-1649.43.3.687"},{"key":"e_1_2_1_23_1","volume-title":"Accessed","author":"Nalimov Eugene","year":"2005"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327647jcd0501_3"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CIG.2013.6633653"},{"key":"e_1_2_1_26_1","unstructured":"David E. Rumelhart and James L. McClelland. 1985. On Learning the Past Tenses of English Verbs. Technical Report. DTIC Document.  David E. Rumelhart and James L. McClelland. 1985. On Learning the Past Tenses of English Verbs. Technical Report. DTIC Document."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781118131350"},{"key":"e_1_2_1_28_1","unstructured":"Herbert Simon. 1957. Models of man; social and rational. Wiley.  Herbert Simon. 1957. Models of man; social and rational. Wiley."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-1968-0_18"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0010-0277(02)00176-2"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-010-1834-0_8"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3046947","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3046947","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:50:24Z","timestamp":1750218624000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3046947"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,27]]},"references-count":31,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017,11,30]]}},"alternative-id":["10.1145\/3046947"],"URL":"https:\/\/doi.org\/10.1145\/3046947","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,7,27]]},"assertion":[{"value":"2016-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-07-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}