{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T02:03:16Z","timestamp":1777341796808,"version":"3.51.4"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,9,30]],"date-time":"2022-09-30T00:00:00Z","timestamp":1664496000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["CCF-1718549, IIS-2007887, IIS-2007951, and IIS-2143895"],"award-info":[{"award-number":["CCF-1718549, IIS-2007887, IIS-2007951, and IIS-2143895"]}]},{"name":"Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity","award":["2017-17061500006"],"award-info":[{"award-number":["2017-17061500006"]}]},{"name":"Defense Advanced Research Projects Agency (DARPA) and Space and Naval Warfare Systems Center Pacific","award":["N66001-19-C-4014"],"award-info":[{"award-number":["N66001-19-C-4014"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Econ. Comput."],"published-print":{"date-parts":[[2022,9,30]]},"abstract":"<jats:p>\n            Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this article, we extend such scoring rules to settings in which a principal elicits private probabilistic beliefs but only has access to agents\u2019 reports. We name our solution\n            <jats:italic>Surrogate Scoring Rules<\/jats:italic>\n            (SSR). SSR is built on a bias correction step and an error rate estimation procedure for a reference answer defined using agents\u2019 reports. We show that, with a little information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a salient feature of SSR is that they quantify the quality of information despite the lack of ground truth, just as SPSR do for the setting\n            <jats:italic>with<\/jats:italic>\n            ground truth. As a by-product, SSR induce\n            <jats:italic>dominant uniform strategy truthfulness<\/jats:italic>\n            in reporting. Our method is verified both theoretically and empirically using data collected from real human forecasters.\n          <\/jats:p>","DOI":"10.1145\/3565559","type":"journal-article","created":{"date-parts":[[2022,10,8]],"date-time":"2022-10-08T13:05:53Z","timestamp":1665234353000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Surrogate Scoring Rules"],"prefix":"10.1145","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8420-6011","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"first","affiliation":[{"name":"UC Santa Cruz, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4072-9870","authenticated-orcid":false,"given":"Juntao","family":"Wang","sequence":"additional","affiliation":[{"name":"Harvard University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0166-594X","authenticated-orcid":false,"given":"Yiling","family":"Chen","sequence":"additional","affiliation":[{"name":"Harvard University, USA"}]}],"member":"320","published-online":{"date-parts":[[2023,2,15]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0225826"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116829"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2015.2374"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/180139.181176"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488417"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.3905\/jod.1997.407971"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1257\/aer.20181897"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2292894"},{"issue":"2","key":"e_1_3_4_11_2","first-page":"410","article-title":"The value of precision in probability assessment: Evidence from a large-scale geopolitical forecasting tournament","volume":"62","author":"Friedman Jeffrey A.","year":"2018","unstructured":"Jeffrey A. Friedman, Joshua D. Baker, Barbara A. Mellers, Philip E. Tetlock, and Richard Zeckhauser. 2018. The value of precision in probability assessment: Evidence from a large-scale geopolitical forecasting tournament. International Studies Quarterly 62, 2 (2018), 410\u2013422.","journal-title":"International Studies Quarterly"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10050"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1115255"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1198\/016214506000001437"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33011996"},{"key":"e_1_3_4_16_2","doi-asserted-by":"crossref","unstructured":"Suzanne Hoogeveen Alexandra Sarafoglou and Eric-Jan Wagenmakers. 2019. Laypeople can predict which social science studies replicate. (2019).","DOI":"10.31234\/osf.io\/egw9d"},{"key":"e_1_3_4_17_2","article-title":"Hybrid Forecasting Competition","year":"2019","unstructured":"IARPA. 2019. Hybrid Forecasting Competition. Retrieved October 13, 2022 from https:\/\/www.iarpa.gov\/index.php\/research-programs\/hfc?id=661.","journal-title":"https:\/\/www.iarpa.gov\/index.php\/research-programs\/hfc?id=661"},{"key":"e_1_3_4_18_2","unstructured":"Victor Richmond Jose Robert F. Nau and Robert L. Winkler. 2006. Scoring Rules Generalized Entropy and Utility Maximization. (2006). Working Paper Fuqua School of Business Duke University."},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/1250910.1250940"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.5555\/1622716.1622723"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611975994.147"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-54110-4_18"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219166.3219194"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3296670"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i02.5583"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2745844.2745874"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.42.9.654"},{"key":"e_1_3_4_28_2","first-page":"125","volume-title":"International Conference on Machine Learning","author":"Menon Aditya","year":"2015","unstructured":"Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. 2015. Learning from corrupted binary labels via class-probability estimation. In International Conference on Machine Learning. 125\u2013134."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1050.0379"},{"key":"e_1_3_4_30_2","first-page":"1196","volume-title":"Advances in Neural Information Processing Systems","author":"Natarajan Nagarajan","year":"2013","unstructured":"Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep K. Ravikumar, and Ambuj Tewari. 2013. Learning with noisy labels. In Advances in Neural Information Processing Systems. 1196\u20131204."},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1214\/16-EJS1150"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1102081"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature21054"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.5555\/2891460.2891576"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v28i1.8797"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2856102"},{"key":"e_1_3_4_37_2","volume-title":"Proceedings of the 4th Workshop on Social Computing and User Generated Content (SC\u201914)","author":"Riley Blake","year":"2014","unstructured":"Blake Riley. 2014. Minimum truth serums with optional predictions. In Proceedings of the 4th Workshop on Social Computing and User Generated Content (SC\u201914)."},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1971.10482346"},{"key":"e_1_3_4_39_2","article-title":"Learning and strongly truthful multi-task peer prediction: A variational approach","author":"Schoenebeck Grant","year":"2020","unstructured":"Grant Schoenebeck and Fang-Yi Yu. 2020. Learning and strongly truthful multi-task peer prediction: A variational approach. arXiv preprint arXiv:2009.14730 (2020).","journal-title":"arXiv preprint arXiv:2009.14730"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-64946-3_9"},{"key":"e_1_3_4_41_2","volume-title":"AISTATS","author":"Scott Clayton","year":"2015","unstructured":"Clayton Scott. 2015. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS."},{"key":"e_1_3_4_42_2","first-page":"489","volume-title":"COLT","author":"Scott Clayton","year":"2013","unstructured":"Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. 2013. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT. 489\u2013511."},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/2940716.2940790"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1177\/0963721414534257"},{"key":"e_1_3_4_45_2","article-title":"Learning in the presence of corruption","author":"Rooyen Brendan van","year":"2015","unstructured":"Brendan van Rooyen and Robert C. Williamson. 2015. Learning in the presence of corruption. arXiv preprint:1504.00091 (2015).","journal-title":"arXiv preprint:1504.00091"},{"key":"e_1_3_4_46_2","article-title":"Forecast aggregation via peer prediction","author":"Wang Juntao","year":"2019","unstructured":"Juntao Wang, Yang Liu, and Yiling Chen. 2019. Forecast aggregation via peer prediction. arXiv preprint arXiv:1910.03779 (2019).","journal-title":"arXiv preprint arXiv:1910.03779"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1969.10501037"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.10590"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2229012.2229085"},{"key":"e_1_3_4_50_2","volume-title":"Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI\u201912)","author":"Witkowski Jens","year":"2012","unstructured":"Jens Witkowski and David C. Parkes. 2012. A robust Bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI\u201912)."},{"key":"e_1_3_4_51_2","volume-title":"Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce","volume":"14","author":"Witkowski Jens","year":"2013","unstructured":"Jens Witkowski and David C. Parkes. 2013. Learning the prior in minimal peer prediction. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce, Vol. 14."}],"container-title":["ACM Transactions on Economics and Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3565559","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3565559","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:51Z","timestamp":1750182531000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3565559"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,30]]},"references-count":50,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9,30]]}},"alternative-id":["10.1145\/3565559"],"URL":"https:\/\/doi.org\/10.1145\/3565559","relation":{},"ISSN":["2167-8375","2167-8383"],"issn-type":[{"value":"2167-8375","type":"print"},{"value":"2167-8383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,30]]},"assertion":[{"value":"2020-08-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-09-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}