{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T15:59:52Z","timestamp":1790783992901,"version":"4.1.0"},"reference-count":79,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000879","name":"Cognitive Economics at Work","doi-asserted-by":"publisher","award":["False"],"award-info":[{"award-number":["False"]}],"id":[{"id":"10.13039\/100000879","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Behav Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Crowdsourcing offers a fast and cost-efficient approach to obtaining human-labeled datasets. However, crowdsourced datasets and the models trained on them can inherit the cognitive constraints and biases of their annotators. In a process we refer to as cognitive-inspired data engineering, we investigate whether ideas from cognitive science can be applied to mitigate the presence of cognitive constraints and cognitive biases in crowdsourced datasets and, as a result, improve the performance of models trained on these datasets. We evaluate our approach by crowdsourcing labels for medical image diagnostic tasks using two different crowdsourcing platforms across two experiments. In Experiment 1, we collect subjective probability judgments from novice annotators through Amazon Mechanical Turk and, in Experiment 2, we collect subjective probability judgments and binary classifications from skilled annotators through DiagnosUs, a crowdsourcing platform specializing in medical and scientific data annotation. In both experiments, we find that recalibrating subjective probability judgments reduces bias, yielding more accurate crowdsourced datasets and more accurate models trained on them. Our results suggest that cognitive-inspired data engineering offers a promising avenue to improve the quality of crowdsourced datasets, with consistent downstream benefits for machine learning models.<\/jats:p>","DOI":"10.3758\/s13428-026-03160-4","type":"journal-article","created":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T20:40:20Z","timestamp":1788295220000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Improving crowdsourcing for AI through cognitive-inspired data engineering"],"prefix":"10.3758","volume":"58","author":[{"given":"Gunnar P.","family":"Epping","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Caplin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erik","family":"Duhaime","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William R.","family":"Holmes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Martin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer S.","family":"Trueblood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,1]]},"reference":[{"key":"3160_CR1","doi-asserted-by":"crossref","unstructured":"Armstrong, J. S. (2001). Combining forecasts. principles of forecasting: A handbook for researchers and practitioners. Kluwer Academic Publishing (pp. 417\u2013439)","DOI":"10.1007\/978-0-306-47630-3_19"},{"key":"3160_CR2","doi-asserted-by":"crossref","unstructured":"Atanasov, P., Rescober, P., Stone, E., Swift, S. A., Servan-Schreiber, E., Tetlock, P., & Mellers, B. (2017). Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management science,\u00a063(3), 691\u2013706.","DOI":"10.1287\/mnsc.2015.2374"},{"issue":"3","key":"3160_CR3","first-page":"929","volume":"24","author":"JV Baranski","year":"1998","unstructured":"Baranski, J. V., & Petrusic, W. M. (1998). Probing the locus of confidence judgments: experiments on the time to determine confidence. Journal of Experimental Psychology: Human Perception and Performance, 24(3), 929.","journal-title":"Journal of Experimental Psychology: Human Perception and Performance"},{"issue":"2","key":"3160_CR4","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1287\/deca.2014.0293","volume":"11","author":"J Baron","year":"2014","unstructured":"Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E., & Ungar, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis, 11(2), 133\u2013145.","journal-title":"Decision Analysis"},{"key":"3160_CR5","doi-asserted-by":"crossref","unstructured":"Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 acm conference on fairness, accountability, and transparency (pp. 610\u2013623).","DOI":"10.1145\/3442188.3445922"},{"issue":"1","key":"3160_CR6","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1006\/obhd.1996.0067","volume":"67","author":"MH Birnbaum","year":"1996","unstructured":"Birnbaum, M. H., & McIntosh, W. R. (1996). Violations of branch independence in choices between gambles. Organizational Behavior and Human Decision Processes, 67(1), 91\u2013110.","journal-title":"Organizational Behavior and Human Decision Processes"},{"key":"3160_CR7","unstructured":"Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems (p. 29)."},{"issue":"1","key":"3160_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","volume":"78","author":"GW Brier","year":"1950","unstructured":"Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1), 1\u20133.","journal-title":"Monthly weather review"},{"key":"3160_CR9","doi-asserted-by":"crossref","unstructured":"Budescu, D. V., & Chen, E. (2015). Identifying expertise to extract the wisdom of crowds. Management science,61(2), 267\u2013280.","DOI":"10.1287\/mnsc.2014.1909"},{"issue":"3","key":"3160_CR10","first-page":"1","volume":"5","author":"JJ Chen","year":"2011","unstructured":"Chen, J. J., Menezes, N. J., Bradley, A. D., & North, T. (2011). Opportunities for crowdsourcing research on amazon mechanical turk. Interfaces, 5(3), 1.","journal-title":"Interfaces"},{"key":"3160_CR11","doi-asserted-by":"crossref","unstructured":"Collins, K. M., Bhatt, U., & Weller, A. (2022). Eliciting and learning with soft labels from every annotator. In Proceedings of the aaai conference on human computation and crowdsourcing (Vol. 10, pp. 40\u201352)","DOI":"10.1609\/hcomp.v10i1.21986"},{"key":"3160_CR12","doi-asserted-by":"crossref","unstructured":"Collins, R. N., Mandel, D. R., & Budescu, D. V. (2023). Performance-weighted aggregation: Ferreting out wisdom within the crowd. Judgment in predictive analytics (pp. 185\u2013214). Springer.","DOI":"10.1007\/978-3-031-30085-1_7"},{"key":"3160_CR13","doi-asserted-by":"publisher","first-page":"14609","DOI":"10.52202\/075280-0643","volume":"36","author":"N Corvelo Benz","year":"2023","unstructured":"Corvelo Benz, N., & Rodriguez, M. (2023). Human-aligned calibration for ai-assisted decision making. Advances in Neural Information Processing Systems, 36, 14609\u201314636.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"3160_CR14","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1037\/dec0000004","volume":"1","author":"CP Davis-Stober","year":"2014","unstructured":"Davis-Stober, C. P., Budescu, D. V., Dana, J., & Broomell, S. B. (2014). When is a crowd wise? Decision, 1(2), 79.","journal-title":"Decision"},{"issue":"11","key":"3160_CR15","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1007\/s10994-025-06860-8","volume":"114","author":"S de Vries","year":"2025","unstructured":"de Vries, S., & Thierens, D. (2025). Learning with confidence: training better classifiers from soft labels. Machine Learning, 114(11), 238.","journal-title":"Machine Learning"},{"key":"3160_CR16","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248\u2013255).","DOI":"10.1109\/CVPR.2009.5206848"},{"issue":"2","key":"3160_CR17","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1016\/j.jfineco.2012.05.012","volume":"106","author":"NA Doherty","year":"2012","unstructured":"Doherty, N. A., Kartasheva, A. V., & Phillips, R. D. (2012). Information effect of entry into credit ratings market: The case of insurers\u2019 ratings. Journal of Financial Economics, 106(2), 308\u2013330.","journal-title":"Journal of Financial Economics"},{"issue":"2","key":"3160_CR18","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1016\/j.jarmac.2018.03.005","volume":"7","author":"IE Dror","year":"2018","unstructured":"Dror, I. E., Kukucka, J., Kassin, S. M., & Zapf, P. A. (2018). No one is immune to contextual bias\u2013not even forensic pathologists. Journal of Applied Research in Memory and Cognition, 7(2), 316\u2013317.","journal-title":"Journal of Applied Research in Memory and Cognition"},{"key":"3160_CR19","doi-asserted-by":"crossref","unstructured":"Duhaime, E. P., Jin, M., Moulton, T., Weber, J., Kurtansky, N. R., Halpern, A., & Rotemberg, V. (2023). Nonexpert crowds outperform expert individuals in diagnostic accuracy on a skin lesion diagnosis task. In 2023 IEEE 20th international symposium on biomedical imaging (isbi) (pp. 1\u20135)","DOI":"10.1109\/ISBI53787.2023.10230646"},{"key":"3160_CR20","doi-asserted-by":"crossref","unstructured":"Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012). Quality through flow and immersion: gamifying crowdsourced relevance assessments. In Proceedings of the 35th international acm sigir conference on research and development in information retrieval (pp. 871\u2013880).","DOI":"10.1145\/2348283.2348400"},{"key":"3160_CR21","doi-asserted-by":"crossref","unstructured":"Epping, G. P., Caplin, A., Duhaime, E., Holmes, W., Martin, D., & Trueblood, J. S. (2026). Harnessing human uncertainty to train more accurate and aligned ai systems. Decision Analysis.","DOI":"10.1287\/deca.2025.0395"},{"key":"3160_CR22","doi-asserted-by":"crossref","unstructured":"Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015). Certifying and removing disparate impact. In Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining (pp. 259\u2013268)","DOI":"10.1145\/2783258.2783311"},{"issue":"1","key":"3160_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1037\/dec0000059","volume":"5","author":"M Galesic","year":"2018","unstructured":"Galesic, M., Barkoczi, D., & Katsikopoulos, K. (2018). Smaller crowds outperform larger crowds and individuals in realistic task conditions. Decision, 5(1), 1.","journal-title":"Decision"},{"key":"3160_CR24","doi-asserted-by":"publisher","first-page":"450","DOI":"10.1038\/075450a0","volume":"75","author":"F Galton","year":"1907","unstructured":"Galton, F. (1907). Vox populi. Nature, 75, 450\u2013451.","journal-title":"Nature"},{"key":"3160_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41044-016-0014-0","volume":"1","author":"S Garc\u00eda","year":"2016","unstructured":"Garc\u00eda, S., Ram\u00edrez-Gallego, S., Luengo, J., Ben\u00edtez, J. M., & Herrera, F. (2016). Big data preprocessing: methods and prospects. Big data analytics, 1, 1\u201322.","journal-title":"Big data analytics"},{"key":"3160_CR26","unstructured":"Goh, G., Cotter, A., Gupta, M., & Friedlander, M. P. (2016). Satisfying real-world goals with dataset constraints. Advances in neural information processing systems (p. 29)."},{"issue":"1","key":"3160_CR27","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1006\/cogp.1998.0710","volume":"38","author":"R Gonzalez","year":"1999","unstructured":"Gonzalez, R., & Wu, G. (1999). On the shape of the probability weighting function. Cognitive psychology, 38(1), 129\u2013166.","journal-title":"Cognitive psychology"},{"issue":"3","key":"3160_CR28","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1016\/0010-0285(92)90013-R","volume":"24","author":"D Griffin","year":"1992","unstructured":"Griffin, D., & Tversky, A. (1992). The weighing of evidence and the determinants of confidence. Cognitive psychology, 24(3), 411\u2013435.","journal-title":"Cognitive psychology"},{"issue":"1","key":"3160_CR29","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1017\/S1930297500009049","volume":"17","author":"Y Han","year":"2022","unstructured":"Han, Y., & Budescu, D. V. (2022). Recalibrating probabilistic forecasts to improve theiraccuracy. Judgment and Decision Making, 17(1), 91\u2013123. https:\/\/doi.org\/10.1017\/S1930297500009049","journal-title":"Judgment and Decision Making"},{"key":"3160_CR30","unstructured":"Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in neural information processing systems (p. 29)."},{"issue":"2","key":"3160_CR31","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/S1364-6613(97)01014-0","volume":"1","author":"N Harvey","year":"1997","unstructured":"Harvey, N. (1997). Confidence in judgment. Trends in cognitive sciences, 1(2), 78\u201382.","journal-title":"Trends in cognitive sciences"},{"issue":"2","key":"3160_CR32","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1111\/tops.12588","volume":"14","author":"E Hasan","year":"2022","unstructured":"Hasan, E., Eichbaum, Q., Seegmiller, A. C., Stratton, C., & Trueblood, J. S. (2022). Improving medical image decision-making by leveraging metacognitive processes and representational similarity. Topics in Cognitive Science, 14(2), 400\u2013413.","journal-title":"Topics in Cognitive Science"},{"issue":"1","key":"3160_CR33","first-page":"31","volume":"9","author":"E Hasan","year":"2024","unstructured":"Hasan, E., Duhaime, E., & Trueblood, J. S. (2024). Boosting wisdom of the crowd for medical image annotation using training performance and task features. Cognitive Research: Principles and Implications, 9(1), 31.","journal-title":"Cognitive Research: Principles and Implications"},{"issue":"8","key":"3160_CR34","doi-asserted-by":"publisher","first-page":"3953","DOI":"10.3758\/s13428-022-01999-x","volume":"55","author":"DJ Hauser","year":"2023","unstructured":"Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2023). Evaluating cloudresearch\u2019s approved group as a solution for problematic data quality on mturk. Behavior Research Methods, 55(8), 3953\u20133964.","journal-title":"Behavior Research Methods"},{"issue":"5","key":"3160_CR35","doi-asserted-by":"publisher","first-page":"1223","DOI":"10.1037\/xge0001340","volume":"152","author":"M Himmelstein","year":"2023","unstructured":"Himmelstein, M., Budescu, D. V., & Ho, E. H. (2023). The wisdom of many in few: Finding individuals who are as wise as the crowd. Journal of Experimental Psychology: General, 152(5), 1223.","journal-title":"Journal of Experimental Psychology: General"},{"key":"3160_CR36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s42113-019-00042-1","volume":"3","author":"WR Holmes","year":"2020","unstructured":"Holmes, W. R., O\u2019Daniels, P., & Trueblood, J. S. (2020). A joint deep neural network and evidence accumulation modeling approach to human decision-making with naturalistic images. Computational Brain & Behavior, 3, 1\u201312.","journal-title":"Computational Brain & Behavior"},{"key":"3160_CR37","doi-asserted-by":"crossref","unstructured":"Hong, L., & Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences,101(46), 16385\u201316389.","DOI":"10.1073\/pnas.0403723101"},{"key":"3160_CR38","unstructured":"Jiang, H., Nachum, O. (2020). Identifying and correcting label bias in machine learning. In International conference on artificial intelligence and statistics (pp. 702\u2013712)."},{"key":"3160_CR39","doi-asserted-by":"crossref","unstructured":"Kamiran, F., & Calders, T. (2009). Classifying without discriminating. In 2009 2nd international conference on computer, control and communication (pp. 1\u20136).","DOI":"10.1109\/IC4.2009.4909197"},{"key":"3160_CR40","doi-asserted-by":"crossref","unstructured":"Kamishima, T., Akaho, S., Asoh, H.,& Sakuma, J. (2012). Fairness-aware classifier with prejudice remover regularizer. In Machine learning and knowledge discovery in databases: European conference, ecml pkdd 2012, bristol, uk, september 24-28, 2012. proceedings, part ii 23 (pp. 35\u201350).","DOI":"10.1007\/978-3-642-33486-3_3"},{"key":"3160_CR41","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1002\/(SICI)1099-0771(199709)10:3<269::AID-BDM281>3.0.CO;2-L","volume":"10","author":"G Keren","year":"1997","unstructured":"Keren, G. (1997). On the calibration of probability judgements. some critical comments and alternative perspectives. Journal of Behavioral Decision Making, 10, 269\u2013279.","journal-title":"Journal of Behavioral Decision Making"},{"key":"3160_CR42","doi-asserted-by":"crossref","unstructured":"Litman, L., Robinson, J., & Abberbock, T. (2017). Turkprime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior research methods, 49(2), 433\u2013442.","DOI":"10.3758\/s13428-016-0727-z"},{"key":"3160_CR43","doi-asserted-by":"crossref","unstructured":"Maier-Hein, L., Mersmann, S., Kondermann, D., Bodenstedt, S., Sanchez, A., Stock, C.,... Speidel, S. (2014). Can masses of non-experts train highly accurate image classifiers? a crowdsourcing approach to instrument segmentation in laparoscopic images. In Medical image computing and computer-assisted intervention\u2013miccai 2014: 17th international conference, boston, ma, usa, september 14-18, 2014, proceedings, part ii 17 (pp. 438\u2013445).","DOI":"10.1007\/978-3-319-10470-6_55"},{"key":"3160_CR44","doi-asserted-by":"crossref","unstructured":"Murphy, A. H. (1972). Scalar and vector partitions of the probability score: Part i. two-state situation. Journal of Applied Meteorology and Climatology,\u00a011(2), 273\u2013282.","DOI":"10.1175\/1520-0450(1972)011<0273:SAVPOT>2.0.CO;2"},{"key":"3160_CR45","doi-asserted-by":"crossref","unstructured":"\u00d8rting, S., Doyle, A., van Hilten, A., Hirth, M., Inel, O., Madan, C. R.,... Cheplygina, V. (2019). A survey of crowdsourcing in medical image analysis. arXiv preprint arXiv:1902.09159","DOI":"10.15346\/hc.v7i1.111"},{"issue":"3","key":"3160_CR46","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1080\/00221300209602099","volume":"129","author":"G Pallier","year":"2002","unstructured":"Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. The Journal of general psychology, 129(3), 257\u2013299.","journal-title":"The Journal of general psychology"},{"issue":"7","key":"3160_CR47","doi-asserted-by":"publisher","first-page":"1837","DOI":"10.1109\/JBHI.2020.2991043","volume":"24","author":"AS Panayides","year":"2020","unstructured":"Panayides, A. S., Amini, A., Filipovic, N. D., Sharma, A., Tsaftaris, S. A., Young, A., et al. (2020). Ai in medical imaging informatics: current challenges and future directions. IEEE journal of biomedical and health informatics, 24(7), 1837\u20131857.","journal-title":"IEEE journal of biomedical and health informatics"},{"key":"3160_CR48","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G.,... others (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32."},{"key":"3160_CR49","doi-asserted-by":"crossref","unstructured":"Peterson, J. C., Battleday, R. M., Griffiths, T. L., & Russakovsky, O. (2019). Human uncertainty makes classification more robust. In Proceedings of the ieee\/cvf international conference on computer vision (pp. 9617\u20139626).","DOI":"10.1109\/ICCV.2019.00971"},{"issue":"3","key":"3160_CR50","doi-asserted-by":"publisher","first-page":"864","DOI":"10.1037\/a0019737","volume":"117","author":"TJ Pleskac","year":"2010","unstructured":"Pleskac, T. J., & Busemeyer, J. R. (2010). Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. Psychological review, 117(3), 864.","journal-title":"Psychological review"},{"key":"3160_CR51","unstructured":"Press, G. (2021). Centaur labs gets \\$15 million to improve data for health care ai."},{"issue":"2","key":"3160_CR52","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1287\/isre.7.2.233","volume":"7","author":"MC Roy","year":"1996","unstructured":"Roy, M. C., & Lerch, F. J. (1996). Overcoming ineffective mental representations in base-rate problems. Information Systems Research, 7(2), 233\u2013247.","journal-title":"Information Systems Research"},{"key":"3160_CR53","doi-asserted-by":"crossref","unstructured":"Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021). \u201ceveryone wants to do the model work, not the data work\u201d: Data cascades in high-stakes ai. In Proceedings of the 2021 chi conference on human factors in computing systems (pp. 1\u201315).","DOI":"10.1145\/3411764.3445518"},{"key":"3160_CR54","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1023\/A:1009957816843","volume":"1","author":"R Selten","year":"1998","unstructured":"Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 1, 43\u201361.","journal-title":"Experimental Economics"},{"issue":"1","key":"3160_CR55","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.tics.2020.10.007","volume":"25","author":"M Shekhar","year":"2021","unstructured":"Shekhar, M., & Rahnev, D. (2021). Sources of metacognitive inefficiency. Trends in Cognitive Sciences, 25(1), 12\u201323.","journal-title":"Trends in Cognitive Sciences"},{"issue":"10","key":"3160_CR56","doi-asserted-by":"publisher","first-page":"6637","DOI":"10.1287\/mnsc.2021.02009","volume":"70","author":"D Soule","year":"2024","unstructured":"Soule, D., Grushka-Cockayne, Y., & Merrick, J. (2024). A heuristic for combining correlated experts when there are few data. Management Science, 70(10), 6637\u20136668.","journal-title":"Management Science"},{"issue":"2","key":"3160_CR57","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1006\/obhd.1996.0074","volume":"67","author":"L Suantak","year":"1996","unstructured":"Suantak, L., Bolger, F., & Ferrell, W. R. (1996). The hard-easy effect in subjective probability calibration. Organizational Behavior and Human Decision Processes, 67(2), 201\u2013221.","journal-title":"Organizational Behavior and Human Decision Processes"},{"issue":"1","key":"3160_CR58","doi-asserted-by":"publisher","first-page":"1","DOI":"10.37105\/sd.214","volume":"10","author":"J Surma","year":"2024","unstructured":"Surma, J. (2024). Deep learning in military applications. Safety & Defense, 10(1), 1\u20137.","journal-title":"Safety & Defense"},{"key":"3160_CR59","unstructured":"Surowiecki, J. (2005). The wisdom of crowds. Anchor."},{"key":"3160_CR60","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 1\u20139)","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"3160_CR61","doi-asserted-by":"publisher","first-page":"188","DOI":"10.56294\/dm2024188","volume":"3","author":"R Thomas-Acaro","year":"2024","unstructured":"Thomas-Acaro, R., & Meneses-Claudio, B. (2024). Technological assistance in highly competitive sports for referee decision making: A systematic literature review. Data and Metadata, 3, 188\u2013188.","journal-title":"Data and Metadata"},{"issue":"1","key":"3160_CR62","first-page":"1","volume":"3","author":"JS Trueblood","year":"2018","unstructured":"Trueblood, J. S., Holmes, W. R., Seegmiller, A. C., Douds, J., Compton, M., Szentirmai, E., & Eichbaum, Q. (2018). The impact of speed and bias on the cognitive processes of experts and novices in medical image decision-making. Cognitive Research: Principles and Implications, 3(1), 1\u201314.","journal-title":"Cognitive Research: Principles and Implications"},{"key":"3160_CR63","doi-asserted-by":"publisher","first-page":"104713","DOI":"10.1016\/j.cognition.2021.104713","volume":"212","author":"JS Trueblood","year":"2021","unstructured":"Trueblood, J. S., Eichbaum, Q., Seegmiller, A. C., Stratton, C., O\u2019Daniels, P., & Holmes, W. R. (2021). Disentangling prevalence induced biases in medical image decision-making. Cognition, 212, 104713.","journal-title":"Cognition"},{"issue":"8","key":"3160_CR64","doi-asserted-by":"publisher","first-page":"1229","DOI":"10.1038\/s41591-020-0942-0","volume":"26","author":"P Tschandl","year":"2020","unstructured":"Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., et al. (2020). Human-computer collaboration for skin cancer recognition. Nature Medicine, 26(8), 1229\u20131234.","journal-title":"Nature Medicine"},{"key":"3160_CR65","doi-asserted-by":"publisher","first-page":"e6762","DOI":"10.7717\/peerj.6762","volume":"7","author":"JD Tucker","year":"2019","unstructured":"Tucker, J. D., Day, S., Tang, W., & Bayus, B. (2019). Crowdsourcing in medical research: concepts and applications. PeerJ, 7, e6762.","journal-title":"PeerJ"},{"key":"3160_CR66","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1007\/s10994-013-5401-4","volume":"95","author":"BM Turner","year":"2014","unstructured":"Turner, B. M., Steyvers, M., Merkle, E. C., Budescu, D. V., & Wallsten, T. S. (2014). Forecast aggregation via recalibration. Machine learning, 95, 261\u2013289.","journal-title":"Forecast aggregation via recalibration. Machine learning"},{"issue":"2","key":"3160_CR67","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1037\/0033-295X.102.2.269","volume":"102","author":"A Tversky","year":"1995","unstructured":"Tversky, A., & Fox, C. R. (1995). Weighing risk and uncertainty. Psychological review, 102(2), 269.","journal-title":"Psychological review"},{"key":"3160_CR68","doi-asserted-by":"crossref","unstructured":"Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., Van Gool, L. (2020). Scan: Learning to classify images without labels. In European conference on computer vision (pp. 268\u2013285).","DOI":"10.1007\/978-3-030-58607-2_16"},{"key":"3160_CR69","volume-title":"Decision processes in visual perception","author":"D Vickers","year":"2014","unstructured":"Vickers, D. (2014). Decision processes in visual perception. Academic Press."},{"key":"3160_CR70","unstructured":"Vukadinovic, D., & Anderson, D. (2022). X-ray baggage screening and artificial intelligence (ai). Publications Office of the European Union."},{"key":"3160_CR71","unstructured":"Wagstaff, K. (2012). Machine learning that matters. arXiv preprint. arXiv:1206.4656"},{"key":"3160_CR72","unstructured":"Wang, A. (2024). Scale\u2019s series f: Expanding the data foundry for ai."},{"issue":"2","key":"3160_CR73","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1287\/deca.1110.0206","volume":"8","author":"G Wang","year":"2011","unstructured":"Wang, G., Kulkarni, S. R., Poor, H. V., & Osherson, D. N. (2011). Aggregating large sets of probabilistic forecasts by weighted coherent adjustment. Decision Analysis, 8(2), 128\u2013144.","journal-title":"Decision Analysis"},{"key":"3160_CR74","doi-asserted-by":"crossref","unstructured":"Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., & Lungren, M. P. (2020). Preparing medical imaging data for machine learning. Radiology,295(1), 4\u201315.","DOI":"10.1148\/radiol.2020192224"},{"key":"3160_CR75","first-page":"211","volume-title":"Combining judgements from correlated experts","author":"KJ Wilson","year":"2018","unstructured":"Wilson, K. J., & Farrow, M. (2018). Combining judgements from correlated experts (pp. 211\u2013240). Elicitation: The science and art of structuring judgement."},{"issue":"2","key":"3160_CR76","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/j.cub.2009.11.066","volume":"20","author":"JM Wolfe","year":"2010","unstructured":"Wolfe, J. M., & Van Wert, M. J. (2010). Varying target prevalence reveals two dissociable decision criteria in visual search. Current biology, 20(2), 121\u2013124.","journal-title":"Current biology"},{"issue":"4","key":"3160_CR77","doi-asserted-by":"publisher","first-page":"623","DOI":"10.1037\/0096-3445.136.4.623","volume":"136","author":"JM Wolfe","year":"2007","unstructured":"Wolfe, J. M., Horowitz, T. S., Van Wert, M. J., Kenner, N. M., Place, S. S., & Kibbi, N. (2007). Low target prevalence is a stubborn source of errors in visual search tasks. Journal of experimental psychology: General, 136(4), 623.","journal-title":"Journal of experimental psychology: General"},{"key":"3160_CR78","unstructured":"Zafar, M. B., Valera, I., Rogriguez, M. G., Gummadi, K. P. (2017). Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics (pp. 962\u2013970)."},{"key":"3160_CR79","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/fnins.2012.00001","volume":"6","author":"H Zhang","year":"2012","unstructured":"Zhang, H., & Maloney, L. T. (2012). Ubiquitous log odds: a common representation of probability and frequency distortion in perception, action, and cognition. Frontiers in neuroscience, 6, 1.","journal-title":"Frontiers in neuroscience"}],"container-title":["Behavior Research Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-026-03160-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-026-03160-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-026-03160-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T15:02:39Z","timestamp":1790780559000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.3758\/s13428-026-03160-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,1]]},"references-count":79,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["3160"],"URL":"https:\/\/doi.org\/10.3758\/s13428-026-03160-4","relation":{},"ISSN":["1554-3528"],"issn-type":[{"value":"1554-3528","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,1]]},"assertion":[{"value":"13 February 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"There are no conflicts of interest.","order":1,"name":"Ethics","label":"Conflicts of Interest\/Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Both experiments were approved by the Institutional Review Board (IRB) at Indiana University Bloomington (protocol #18017).","order":2,"name":"Ethics","label":"Ethics Approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"In Experiment\n                      \n                      , participants were presented with a study information sheet before beginning the study. Informed consent was not required because the IRB classified this work as exempt research. In Experiment\n                      \n                      , participants agreed to the terms of service agreement on DiagnosUs where they consented to their data being used for commercial and academic purposes.","order":3,"name":"Ethics","label":"Consent to Participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Centaur Labs, the developers of DiagnosUs, has consented to the publication of screenshots of DiagnosUs.","order":4,"name":"Ethics","label":"Consent for Publication","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"283"}}