{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T18:00:00Z","timestamp":1775152800594,"version":"3.50.1"},"reference-count":87,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T00:00:00Z","timestamp":1596499200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T00:00:00Z","timestamp":1596499200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Bus Inf Syst Eng"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Research has shown that aggregation of independent expert judgments significantly improves the quality of forecasts as compared to individual expert forecasts. This \u201cwisdom of crowds\u201d (WOC) has sparked substantial interest. However, previous studies on strengths and weaknesses of aggregation algorithms have been restricted by limited empirical data and analytical complexity. Based on a comprehensive analysis of existing knowledge on WOC and aggregation algorithms, this paper describes the design and implementation of a static stochastic simulation model to emulate WOC scenarios with a wide range of parameters. The model has been thoroughly evaluated: the assumptions are validated against propositions derived from literature, and the model has a computational representation. The applicability of the model is demonstrated by investigating aggregation algorithm behavior on a detailed level, by assessing aggregation algorithm performance, and by exploring previously undiscovered suppositions on WOC. The simulation model helps expand the understanding of WOC, where previous research was restricted. Additionally, it gives directions for developing aggregation algorithms and contributes to a general understanding of the WOC phenomenon.<\/jats:p>","DOI":"10.1007\/s12599-020-00664-x","type":"journal-article","created":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T08:02:37Z","timestamp":1596528157000},"page":"329-348","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment"],"prefix":"10.1007","volume":"63","author":[{"given":"Patrick","family":"Afflerbach","sequence":"first","affiliation":[]},{"given":"Christopher","family":"van Dun","sequence":"additional","affiliation":[]},{"given":"Henner","family":"Gimpel","sequence":"additional","affiliation":[]},{"given":"Dominik","family":"Parak","sequence":"additional","affiliation":[]},{"given":"Johannes","family":"Seyfried","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,8,4]]},"reference":[{"key":"664_CR1","doi-asserted-by":"publisher","first-page":"1499","DOI":"10.1287\/mnsc.31.12.1499","volume":"31","author":"AH Ashton","year":"1985","unstructured":"Ashton AH, Ashton RH (1985) Aggregating subjective forecasts: some empirical results. Manag Sci 31:1499\u20131508","journal-title":"Manag Sci"},{"key":"664_CR2","volume-title":"Discrete-event system simulation","author":"J Banks","year":"2010","unstructured":"Banks J, Carson II, Nelson BL, Nicol DM (2010) Discrete-event system simulation, 5th edn. Prentice Hall, Upper Saddle River","edition":"5"},{"key":"664_CR3","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1057\/jors.1969.103","volume":"20","author":"JM Bates","year":"1969","unstructured":"Bates JM, Granger CWJ (1969) The combination of forecasts. J Oper Res Soc 20:451\u2013468","journal-title":"J Oper Res Soc"},{"key":"664_CR4","doi-asserted-by":"publisher","first-page":"503","DOI":"10.1007\/s12599-018-0529-1","volume":"61","author":"J Beese","year":"2019","unstructured":"Beese J, Haki MK, Aier S, Winter R (2019) Simulation-based research in information systems. Bus Inf Syst Eng 61:503\u2013521. https:\/\/doi.org\/10.1007\/s12599-018-0529-1","journal-title":"Bus Inf Syst Eng"},{"key":"664_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s12599-013-0309-x","volume":"6","author":"M Bichler","year":"2014","unstructured":"Bichler M, Hess T, Krishnan R, Loos P (2014) Emerging research areas in business and information systems engineering. Bus Inf Syst Eng 6:1\u20132. https:\/\/doi.org\/10.1007\/s12599-013-0309-x","journal-title":"Bus Inf Syst Eng"},{"key":"664_CR6","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1287\/deca.1070.0089","volume":"4","author":"JE Bickel","year":"2007","unstructured":"Bickel JE (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decis Anal 4:49\u201365","journal-title":"Decis Anal"},{"key":"664_CR7","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1006\/obhd.1996.0021","volume":"65","author":"LA Brenner","year":"1996","unstructured":"Brenner LA, Koehler DJ, Liberman V, Tversky A (1996) Overconfidence in probability and frequency judgement: a critical examination. Organ Behav Hum Decis Process 65:212\u2013219","journal-title":"Organ Behav Hum Decis Process"},{"key":"664_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1126\/science.27.693.594","volume":"78","author":"GW Brier","year":"1950","unstructured":"Brier GW (1950) Verification of forecasts expersses in terms of probaility. Mon Weather Rev 78:1\u20133. https:\/\/doi.org\/10.1126\/science.27.693.594","journal-title":"Mon Weather Rev"},{"key":"664_CR9","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1175\/waf966.1","volume":"22","author":"J Br\u00f6cker","year":"2007","unstructured":"Br\u00f6cker J, Smith LA (2007) Scoring probabilistic forecasts: the importance of being proper. Weather Forecast 22:382\u2013388. https:\/\/doi.org\/10.1175\/waf966.1","journal-title":"Weather Forecast"},{"key":"664_CR10","doi-asserted-by":"publisher","DOI":"10.1007\/s11336-009-9118-z","author":"SB Broomell","year":"2009","unstructured":"Broomell SB, Budescu DV (2009) Why are experts correlated? Decomposing correlations between judges. Psychometrika. https:\/\/doi.org\/10.1007\/s11336-009-9118-z","journal-title":"Psychometrika"},{"key":"664_CR11","first-page":"327","volume-title":"Information sampling and adaptive cognition","author":"DV Budescu","year":"2006","unstructured":"Budescu DV (2006) Confidence in aggregation of opinions from multiple sources. In: Fiedler K, Juslin P (eds) Information sampling and adaptive cognition. Cambridge University Press, Cambridge, pp 327\u2013352"},{"issue":"2","key":"664_CR12","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1287\/mnsc.2014.1909","volume":"61","author":"DV Budescu","year":"2015","unstructured":"Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Manag Sci 61(2):267\u2013280","journal-title":"Manag Sci"},{"key":"664_CR13","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1002\/for.3980010207","volume":"1","author":"R Carbone","year":"1982","unstructured":"Carbone R, Armstrong JS (1982) Evaluation of extrapolative forecasting methods: results of a survey of academicians and practitioners. J Forecast 1:215\u2013217. https:\/\/doi.org\/10.1002\/for.3980010207","journal-title":"J Forecast"},{"issue":"2","key":"664_CR14","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1287\/deca.2016.0329","volume":"13","author":"E Chen","year":"2016","unstructured":"Chen E, Budescu DV, Lakshmikanth SK et al (2016) Validating the contribution-weighted model: robustness and cost-benefit analyses. Decis Anal 13(2):128\u2013152","journal-title":"Decis Anal"},{"key":"664_CR15","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1016\/0169-2070(89)90012-5","volume":"5","author":"RT Clemen","year":"1989","unstructured":"Clemen RT (1989) Combining forecast: a review and annotated bibliography. Int J Forecast 5:559\u2013583","journal-title":"Int J Forecast"},{"key":"664_CR16","doi-asserted-by":"publisher","first-page":"39","DOI":"10.2307\/1391385","volume":"4","author":"RT Clemen","year":"1986","unstructured":"Clemen RT, Winkler RL (1986) Combining Economic Forecasts. J Bus Econ Stat 4:39\u201346. https:\/\/doi.org\/10.2307\/1391385","journal-title":"J Bus Econ Stat"},{"key":"664_CR17","first-page":"155","volume":"19","author":"RT Clemen","year":"1999","unstructured":"Clemen RT, Winkler RL (1999) Combining probability distributiond from experts in risk analysis. Risk Anal 19:155\u2013156","journal-title":"Risk Anal"},{"key":"664_CR18","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/j.ress.2017.02.003","volume":"163","author":"AR Colson","year":"2017","unstructured":"Colson AR, Cooke RM (2017) Cross validation for the classical model of structured expert judgment. Reliab Eng Syst Saf 163:109\u2013120. https:\/\/doi.org\/10.1016\/j.ress.2017.02.003","journal-title":"Reliab Eng Syst Saf"},{"key":"664_CR19","doi-asserted-by":"publisher","first-page":"657","DOI":"10.1016\/j.ress.2007.03.005","volume":"93","author":"RM Cooke","year":"2008","unstructured":"Cooke RM, Goossens LLHJ (2008) TU Delft expert judgment data base. Reliab Eng Syst Saf 93:657\u2013674. https:\/\/doi.org\/10.1016\/j.ress.2007.03.005","journal-title":"Reliab Eng Syst Saf"},{"key":"664_CR20","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/0007-6813(75)90043-9","volume":"18","author":"DJ Dalrymple","year":"1975","unstructured":"Dalrymple DJ (1975) Sales forecasting methods and accuracy. Bus Horiz 18:69\u201373","journal-title":"Bus Horiz"},{"key":"664_CR21","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1287\/deca.2015.0315","volume":"12","author":"J Dana","year":"2015","unstructured":"Dana J, Broomell SB, Budescu DV, Davis-Stober CP (2015) The composition of optimally wise crowds. Decis Anal 12:130\u2013143. https:\/\/doi.org\/10.1287\/deca.2015.0315","journal-title":"Decis Anal"},{"key":"664_CR22","doi-asserted-by":"publisher","first-page":"480","DOI":"10.5465\/amr.2007.24351453","volume":"32","author":"JP Davis","year":"2007","unstructured":"Davis JP, Eusebgardt KM, Binghaman CB (2007) Developing theory through simulation methods. Acad Manag Rev 32:480\u2013499. https:\/\/doi.org\/10.5465\/amr.2007.24351453","journal-title":"Acad Manag Rev"},{"key":"664_CR23","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1057\/jit.2015.19","volume":"31","author":"RM Davison","year":"2016","unstructured":"Davison RM, Martinsons MG (2016) Context is king! Considering particularism in research design and reporting. J Inf Technol 31:241\u2013249. https:\/\/doi.org\/10.1057\/jit.2015.19","journal-title":"J Inf Technol"},{"key":"664_CR24","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1037\/dec0000004","volume":"1","author":"CP Davis-Stober","year":"2014","unstructured":"Davis-Stober CP, Budescu DV, Dana J, Broomell SB (2014) When is a crowd wise? Decision 1:79\u2013101","journal-title":"Decision"},{"key":"664_CR25","unstructured":"de Condorcet N (1785) Essai sur l\u2019application de l\u2019analyse \u00e0 la probabilit\u00e9 des d\u00e9cisions rendues \u00e0 la pluralit\u00e9 des voix. Reprint by Cambridge University Press, Cambridge"},{"key":"664_CR26","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1016\/s0377-2217(98)00380-4","volume":"120","author":"LM de Menezes","year":"2000","unstructured":"de Menezes LM, Bunn WD, Taylor JW (2000) Review of guidelines for the use of combined forecasts. Eur J Oper Res 120:190\u2013204. https:\/\/doi.org\/10.1016\/s0377-2217(98)00380-4","journal-title":"Eur J Oper Res"},{"key":"664_CR27","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.ress.2013.07.015","volume":"121","author":"JW Eggstaff","year":"2014","unstructured":"Eggstaff JW, Mazzuchi TA, Sarkani S (2014) The effect of the number of seed variables on the performance of Cooke\u2019s classical model. Reliab Eng Syst Saf 121:72\u201382. https:\/\/doi.org\/10.1016\/j.ress.2013.07.015","journal-title":"Reliab Eng Syst Saf"},{"key":"664_CR28","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1037\/0033-2909.84.1.158","volume":"84","author":"HJ Einhorn","year":"1977","unstructured":"Einhorn HJ, Hogarth RM, Klempner E (1977) Quality of group judgment. Psychol Bull 84:158\u2013172","journal-title":"Psychol Bull"},{"key":"664_CR29","doi-asserted-by":"publisher","first-page":"985","DOI":"10.1175\/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2","volume":"8","author":"ES Epstein","year":"1969","unstructured":"Epstein ES (1969) A scoring system for probability forecasts of ranked categories. J Appl Meteorol 8:985\u2013987","journal-title":"J Appl Meteorol"},{"key":"664_CR30","unstructured":"European Central Bank (2017) ECB survey of professional forecasters. https:\/\/www.ecb.europa.eu\/stats\/ecb_surveys\/survey_of_professional_forecasters\/html\/index.en.html"},{"key":"664_CR31","doi-asserted-by":"crossref","unstructured":"Fildes R, Hastings R (1994) The organization and improvement of market forecasting. J Oper Res Soc 1\u201316","DOI":"10.1057\/jors.1994.1"},{"key":"664_CR32","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1016\/0030-5073(81)90017-9","volume":"110","author":"GW Fischer","year":"1981","unstructured":"Fischer GW (1981) When oracles fail\u2014a comparison of four procedures for aggregating subjective probability forecasts. Organ Behav Hum Perform 110:96\u2013110","journal-title":"Organ Behav Hum Perform"},{"key":"664_CR33","doi-asserted-by":"publisher","first-page":"1292","DOI":"10.1016\/j.ress.2011.05.012","volume":"96","author":"F Flandoli","year":"2011","unstructured":"Flandoli F, Giorgi E, Aspinall WP, Neri A (2011) Comparison of a new expert elicitation model with the classical model, equal weights and single experts, using a cross-validation technique. Reliab Eng Syst Saf 96:1292\u20131310. https:\/\/doi.org\/10.1016\/j.ress.2011.05.012","journal-title":"Reliab Eng Syst Saf"},{"key":"664_CR34","doi-asserted-by":"publisher","first-page":"450","DOI":"10.1038\/075450a0","volume":"75","author":"F Galton","year":"1907","unstructured":"Galton F (1907) Vox populi\u2014the wisdom of crowds. Nature 75:450\u2013451. https:\/\/doi.org\/10.1038\/075450a0","journal-title":"Nature"},{"key":"664_CR35","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1214\/ss\/1177013831","volume":"1","author":"C Genest","year":"1986","unstructured":"Genest C, Zidek JV (1986) Combining probability distributions: a critique and an annotated bibliography. Stat Sci 1:147\u2013148. https:\/\/doi.org\/10.1214\/ss\/1177013831","journal-title":"Stat Sci"},{"key":"664_CR36","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1198\/016214506000001437","volume":"102","author":"T Gneiting","year":"2007","unstructured":"Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc 102:359\u2013378. https:\/\/doi.org\/10.1198\/016214506000001437","journal-title":"J Am Stat Assoc"},{"key":"664_CR37","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1111\/j.1539-6924.2012.01833.x","volume":"33","author":"JK Hammitt","year":"2013","unstructured":"Hammitt JK, Zhang Y (2013) Combining experts\u2019 judgments: comparison of algorithmic methods using synthetic data. Risk Anal 33:109\u2013120","journal-title":"Risk Anal"},{"key":"664_CR38","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1126\/science.183.4130.1141-a","volume":"6","author":"J Harling","year":"1958","unstructured":"Harling J (1958) Simulation techniques in operations research\u2014a review. Oper Res 6:307\u2013319. https:\/\/doi.org\/10.1126\/science.183.4130.1141-a","journal-title":"Oper Res"},{"key":"664_CR39","doi-asserted-by":"publisher","first-page":"1229","DOI":"10.5465\/amr.2007.26586485","volume":"32","author":"JR Harrison","year":"2007","unstructured":"Harrison JR, Lin Z, Carroll GR, Carley KM (2007) Simulation modeling in organizational and management research. Acad Manag Rev 32:1229\u20131245. https:\/\/doi.org\/10.5465\/amr.2007.26586485","journal-title":"Acad Manag Rev"},{"key":"664_CR40","doi-asserted-by":"publisher","first-page":"494","DOI":"10.1037\/0033-295x.112.2.494","volume":"112","author":"R Hastie","year":"2005","unstructured":"Hastie R, Kameda T (2005) The robust beauty of majority rules in group decisions. Psychol Rev 112:494\u2013508. https:\/\/doi.org\/10.1037\/0033-295x.112.2.494","journal-title":"Psychol Rev"},{"key":"664_CR41","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9280.2009.02271.x","author":"SM Herzog","year":"2009","unstructured":"Herzog SM, Hertwig R (2009) The wisdom of many in one mind: improving individual judgments with dialectical bootstrapping. Psychol Sci. https:\/\/doi.org\/10.1111\/j.1467-9280.2009.02271.x","journal-title":"Psychol Sci"},{"key":"664_CR42","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1017\/S1930297500002096","volume":"6","author":"SM Herzog","year":"2011","unstructured":"Herzog SM, Hertwig R (2011) The wisdom of ignorant crowds: predicting sport outcomes by mere recognition. Judgm Decis Mak 6:58\u201372","journal-title":"Judgm Decis Mak"},{"key":"664_CR43","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1016\/0030-5073(78)90037-5","volume":"21","author":"RM Hogarth","year":"1978","unstructured":"Hogarth RM (1978) A note on aggregating opinions. Organ Behav Hum Perform 21:40\u201346","journal-title":"Organ Behav Hum Perform"},{"key":"664_CR44","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1287\/mnsc.27.2.115","volume":"27","author":"M Hogarth","year":"1981","unstructured":"Hogarth M, Makridakis S (1981) Forecasting and planning: an evaluation. Manag Sci 27:115\u2013138","journal-title":"Manag Sci"},{"key":"664_CR45","doi-asserted-by":"publisher","DOI":"10.1287\/deca.2013.0282","author":"SC Hora","year":"2013","unstructured":"Hora SC, Fransen BR, Hawkins N, Susel I (2013) Median aggregation of distribution functions. Decis Anal. https:\/\/doi.org\/10.1287\/deca.2013.0282","journal-title":"Decis Anal"},{"key":"664_CR46","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1016\/s0377-2217(01)00226-0","volume":"140","author":"WJ Hurley","year":"2002","unstructured":"Hurley WJ, Lior DU (2002) Combining expert judgment: on the performance of trimmed mean vote aggregation procedures in the presence of strategic voting. Eur J Oper Res 140:142\u2013147. https:\/\/doi.org\/10.1016\/s0377-2217(01)00226-0","journal-title":"Eur J Oper Res"},{"key":"664_CR47","doi-asserted-by":"publisher","first-page":"444","DOI":"10.1287\/opre.44.3.444","volume":"44","author":"MN Jouini","year":"1996","unstructured":"Jouini MN, Clemen RT (1996) Copula models for aggregating expert opinions. Oper Res 44:444\u2013457","journal-title":"Oper Res"},{"key":"664_CR48","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1037\/a0013550","volume":"134","author":"N Karelaia","year":"2008","unstructured":"Karelaia N, Hogarth RM (2008) Determinants of linear judgment: a meta-analysis of lens model studies. Psychol Bull 134:404\u2013426. https:\/\/doi.org\/10.1037\/a0013550","journal-title":"Psychol Bull"},{"key":"664_CR49","doi-asserted-by":"publisher","unstructured":"Keuschnigg M, Ganser C (2017) Crowd wisdom relies on agents\u2019 ability in small groups with a voting aggregation rule. Manag Sci 63:mnsc.2015.2364. https:\/\/doi.org\/10.1287\/mnsc.2015.2364","DOI":"10.1287\/mnsc.2015.2364"},{"key":"664_CR50","doi-asserted-by":"crossref","unstructured":"Kittur A, Kraut RE (2008) Harnessing the wisdom of crowds in Wikipedia: quality through coordination. In: Proceedings of the 2008 ACM conference on Computer Supported Cooperative Work, ACM, pp 37\u201346","DOI":"10.1145\/1460563.1460572"},{"key":"664_CR51","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1109\/wsc.2000.899697","volume":"82","author":"JPC Kleijnen","year":"1995","unstructured":"Kleijnen JPC (1995) Verification and validation of simulation models. Eur J Oper Res 82:145\u2013162. https:\/\/doi.org\/10.1109\/wsc.2000.899697","journal-title":"Eur J Oper Res"},{"key":"664_CR52","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1287\/mnsc.1060.0518","volume":"52","author":"RP Larrick","year":"2006","unstructured":"Larrick RP, Soll JB (2006) Intuitions about combining opinions: misappreciation of the averaging principle. Manag Sci 52:111\u2013127. https:\/\/doi.org\/10.1287\/mnsc.1060.0518","journal-title":"Manag Sci"},{"key":"664_CR53","first-page":"227","volume-title":"Frontiers of social psychology. Social judgment and decision making","author":"RP Larrick","year":"2011","unstructured":"Larrick RP, Mannes AE, Soll JB, Krueger JI (2011) The social psychology of the wisdom of crowds. In: Krueger JI (ed) Frontiers of social psychology. Social judgment and decision making. Psychology Press, Hove, pp 227\u2013242"},{"key":"664_CR54","volume-title":"Simulation modeling & analysis","author":"AM Law","year":"2007","unstructured":"Law AM, Kelton DW (2007) Simulation modeling & analysis. McGraw Hill, Boston"},{"key":"664_CR55","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1016\/j.ijforecast.2006.03.007","volume":"22","author":"M Lawrence","year":"2006","unstructured":"Lawrence M, Goodwin P, O\u2019Connor M, Oenkal D (2006) Judgmental forecasting: a review of progress over the last 25 years. Int J Forecast 22:493\u2013518. https:\/\/doi.org\/10.1016\/j.ijforecast.2006.03.007","journal-title":"Int J Forecast"},{"key":"664_CR56","doi-asserted-by":"publisher","first-page":"914","DOI":"10.3758\/s13421-010-0059-7","volume":"39","author":"MD Lee","year":"2011","unstructured":"Lee MD, Zhang S, Shi J (2011) The wisdom of the crowd playing the price is right. Mem Cognit 39:914\u2013923","journal-title":"Mem Cognit"},{"key":"664_CR57","doi-asserted-by":"publisher","unstructured":"Lee JS, Filatova T, Ligmann-Zielinska A, et al (2015) The complexities of agent-based modeling output analysis. JASSS. https:\/\/doi.org\/10.18564\/jasss.2897","DOI":"10.18564\/jasss.2897"},{"key":"664_CR58","first-page":"245","volume":"2","author":"JM Leimeister","year":"2010","unstructured":"Leimeister JM (2010) Collective intelligence. Bus Inf. Syst Eng 2:245\u2013248","journal-title":"Syst Eng"},{"key":"664_CR59","doi-asserted-by":"publisher","DOI":"10.1007\/s10588-011-9097-3","author":"I Lorscheid","year":"2012","unstructured":"Lorscheid I, Heine BO, Meyer M (2012) Opening the \u201cBlack Box\u201d of simulations: increased transparency and effective communication through the systematic design of experiments. Comput Math Organ Theory. https:\/\/doi.org\/10.1007\/s10588-011-9097-3","journal-title":"Comput Math Organ Theory"},{"key":"664_CR60","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.jsis.2009.01.002","volume":"18","author":"HC Lucas","year":"2009","unstructured":"Lucas HC, Goh JM (2009) Disruptive technology: how Kodak missed the digital photography revolution. J Strateg Inf Syst 18:46\u201355. https:\/\/doi.org\/10.1016\/j.jsis.2009.01.002","journal-title":"J Strateg Inf Syst"},{"key":"664_CR61","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1016\/0169-2070(88)90102-1","volume":"4","author":"V Mahajan","year":"1988","unstructured":"Mahajan V, Wind Y (1988) New product forecasting models. Int J Forecast 4:341\u2013358. https:\/\/doi.org\/10.1016\/0169-2070(88)90102-1","journal-title":"Int J Forecast"},{"key":"664_CR62","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1037\/a0036677","volume":"107","author":"AE Mannes","year":"2014","unstructured":"Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J Pers Soc Psychol 107:276\u2013299. https:\/\/doi.org\/10.1037\/a0036677","journal-title":"J Pers Soc Psychol"},{"key":"664_CR63","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1016\/j.obhdp.2008.02.007","volume":"107","author":"CRM McKenzie","year":"2008","unstructured":"McKenzie CRM, Liersch MJ, Yaniv I (2008) Overconfidence in interval estimates: what does expertise buy you? Organ Behav Hum Decis Process 107:179\u2013191. https:\/\/doi.org\/10.1016\/j.obhdp.2008.02.007","journal-title":"Organ Behav Hum Decis Process"},{"key":"664_CR64","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1108\/eb038541","volume":"1","author":"PA Morris","year":"1986","unstructured":"Morris PA (1986) Combining probability distributions: a critique and an annotated bibliography: comment. Stat Sci 1:141\u2013144. https:\/\/doi.org\/10.1108\/eb038541","journal-title":"Stat Sci"},{"key":"664_CR65","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1175\/1520-0493(1970)098<0917:TRPSAT>2.3.CO;2","volume":"98","author":"AH Murphy","year":"1970","unstructured":"Murphy AH (1970) The ranked probability score and the probability score: a comparison. Mon Weather Rev 98:917\u2013924","journal-title":"Mon Weather Rev"},{"key":"664_CR66","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1109\/mcse.2007.58","volume":"9","author":"TE Oliphant","year":"2007","unstructured":"Oliphant TE (2007) Python for scientific computing. Comput Sci Eng 9:10\u201320. https:\/\/doi.org\/10.1109\/mcse.2007.58","journal-title":"Comput Sci Eng"},{"issue":"5","key":"664_CR67","doi-asserted-by":"publisher","first-page":"1949","DOI":"10.1287\/mnsc.2018.3047","volume":"65","author":"AB Palley","year":"2019","unstructured":"Palley AB, Soll JB (2019) Extracting the wisdom of crowds when information is shared. Manag Sci 65(5):1949\u20132443. https:\/\/doi.org\/10.1287\/mnsc.2018.3047","journal-title":"Manag Sci"},{"issue":"2","key":"664_CR68","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1017\/S1930297500003910","volume":"10","author":"S Park","year":"2015","unstructured":"Park S, Budescu DV (2015) Aggregating multiple probability intervals to improve calibration. Judgm Decis Mak 10(2):130\u2013143","journal-title":"Judgm Decis Mak"},{"key":"664_CR69","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1016\/s0377-2217(98)00058-7","volume":"109","author":"D Petrovic","year":"1998","unstructured":"Petrovic D, Roy R, Petrovic R (1998) Modelling and simulation of a supply chain in an uncertain environment. Eur J Oper Res 109:299\u2013309. https:\/\/doi.org\/10.1016\/s0377-2217(98)00058-7","journal-title":"Eur J Oper Res"},{"key":"664_CR70","first-page":"32","volume":"38","author":"NR Sanders","year":"1997","unstructured":"Sanders NR (1997) The status of forecasting in manufacturing firms. Prod Invent Manag J 38:32\u201335","journal-title":"Prod Invent Manag J"},{"key":"664_CR71","doi-asserted-by":"publisher","unstructured":"Sargent RG (1987) An overview of verification and validation of simulation models. In: Proc 19th Conf Winter Simulation Conference, pp 33\u201339. https:\/\/doi.org\/10.1145\/318371.318379","DOI":"10.1145\/318371.318379"},{"key":"664_CR72","doi-asserted-by":"publisher","unstructured":"Sargent RG (2005) Verification and validation of simulation models. In: Proc 37th Winter Simulation Conference, pp 130\u2013143. https:\/\/doi.org\/10.1109\/wsc.2000.899697","DOI":"10.1109\/wsc.2000.899697"},{"key":"664_CR73","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1057\/s41265-016-0003-9","volume":"31","author":"S Sarker","year":"2016","unstructured":"Sarker S (2016) Building on Davison and Martinsons\u2019 concerns: a call for balance between contextual specificity and generality in IS research. J Inf Technol 31:250\u2013253. https:\/\/doi.org\/10.1057\/s41265-016-0003-9","journal-title":"J Inf Technol"},{"key":"664_CR74","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1086\/592550","volume":"75","author":"G Schurz","year":"2008","unstructured":"Schurz G (2008) The meta-inductivist\u2019s winning strategy in the prediction game: a new approach to Hume\u2019s problem. Philos Sci 75:278\u2013305. https:\/\/doi.org\/10.1086\/592550","journal-title":"Philos Sci"},{"key":"664_CR75","volume-title":"Operations management","author":"N Slack","year":"2007","unstructured":"Slack N, Chambers S, Johnston R (2007) Operations management, 5th edn. Pearson Education, Essex","edition":"5"},{"key":"664_CR76","unstructured":"Surowiecki J (2005) The wisdom of crowds. Anchor"},{"key":"664_CR77","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1002\/1099-131X(200007)19:4<235::AID-FOR772>3.0.CO;2-L","volume":"19","author":"AS Tay","year":"2000","unstructured":"Tay AS, Wallis KF (2000) Density forecasting: a survey. J Forecast 19:235\u2013254","journal-title":"J Forecast"},{"issue":"4157","key":"664_CR78","doi-asserted-by":"publisher","first-page":"1124","DOI":"10.1126\/science.185.4157.1124","volume":"185","author":"A Tversky","year":"1974","unstructured":"Tversky A, Kahneman D (1974) Judgment under uncertainty: heuristics and biases. Science 185(4157):1124\u20131131","journal-title":"Science"},{"issue":"1","key":"664_CR79","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1177\/002224299606000105","volume":"60","author":"GL Urban","year":"1996","unstructured":"Urban GL, Weinberg BD, Hauser JR (1996) Premarket forecasting of really-new products. J Market 60(1):47\u201360","journal-title":"J Market"},{"key":"664_CR80","doi-asserted-by":"publisher","first-page":"2937","DOI":"10.2139\/ssrn.2257995","volume":"62","author":"ED Van Wesep","year":"2016","unstructured":"Van Wesep ED (2016) The quality of expertise. Manag Sci 62:2937\u20132951. https:\/\/doi.org\/10.2139\/ssrn.2257995","journal-title":"Manag Sci"},{"key":"664_CR81","doi-asserted-by":"publisher","unstructured":"Wagner C, Suh A (2014) The wisdom of crowds: impact of collective size and expertise transfer on collective performance. In: Proceedings Annual Hawaii International Conference on System Sciences, pp 594\u2013603. https:\/\/doi.org\/10.1109\/hicss.2014.80","DOI":"10.1109\/hicss.2014.80"},{"key":"664_CR82","unstructured":"Wagner C, Vinaimont T (2010) Evaluating the wisdom of crowds. Proc Issues Inf Syst XI:724\u2013732"},{"key":"664_CR83","doi-asserted-by":"publisher","DOI":"10.1111\/j.1600-0706.2013.01073.x","author":"JW White","year":"2014","unstructured":"White JW, Rassweiler A, Samhouri JF et al (2014) Ecologists should not use statistical significance tests to interpret simulation model results. Oikos. https:\/\/doi.org\/10.1111\/j.1600-0706.2013.01073.x","journal-title":"Oikos"},{"key":"664_CR84","doi-asserted-by":"publisher","first-page":"150","DOI":"10.2307\/2982011","volume":"146","author":"RL Winkler","year":"1983","unstructured":"Winkler RL, Makridakis S (1983) The combination of forecasts. J Royal Stat Soc 146:150\u2013157","journal-title":"J Royal Stat Soc"},{"key":"664_CR85","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1007\/s12599-008-0024-1","volume":"1","author":"R Winter","year":"2009","unstructured":"Winter R (2009) What in fact is fundamental research in business and information systems engineering? Bus Inf Syst Eng 1:192\u2013199. https:\/\/doi.org\/10.1007\/s12599-008-0024-1","journal-title":"Bus Inf Syst Eng"},{"issue":"6004","key":"664_CR86","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1126\/science.1193147","volume":"330","author":"AW Woolley","year":"2010","unstructured":"Woolley AW, Chabris CF, Pentland A et al (2010) Evidence for a collective intelligence factor in the performance of human groups. Sci 330(6004):686\u2013688","journal-title":"Sci"},{"key":"664_CR87","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/0749-5978(91)90042-r","volume":"49","author":"JF Yates","year":"1991","unstructured":"Yates JF, McDaniel LS, Brown ES (1991) Probabilistic forecasts of stock prices and earnings: the hazards of nascent expertise. Organ Behav Hum Decis Process 49:60\u201379. https:\/\/doi.org\/10.1016\/0749-5978(91)90042-r","journal-title":"Organ Behav Hum Decis Process"}],"container-title":["Business &amp; Information Systems Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12599-020-00664-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12599-020-00664-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12599-020-00664-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T07:33:20Z","timestamp":1696491200000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12599-020-00664-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,4]]},"references-count":87,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,8]]}},"alternative-id":["664"],"URL":"https:\/\/doi.org\/10.1007\/s12599-020-00664-x","relation":{},"ISSN":["2363-7005","1867-0202"],"issn-type":[{"value":"2363-7005","type":"print"},{"value":"1867-0202","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,4]]},"assertion":[{"value":"20 December 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 May 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 August 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}