{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:07:36Z","timestamp":1750306056279,"version":"3.41.0"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2017,6,12]],"date-time":"2017-06-12T00:00:00Z","timestamp":1497225600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ANR-Labex IAST"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2017,8,31]]},"abstract":"<jats:p>On the Web, there is always a need to aggregate opinions from the crowd (as in posts, social networks, forums, etc.). Different mechanisms have been implemented to capture these opinions such as Like in Facebook, Favorite in Twitter, thumbs-up\/-down, flagging, and so on. However, in more contested domains (e.g., Wikipedia, political discussion, and climate change discussion), these mechanisms are not sufficient, since they only deal with each issue independently without considering the relationships between different claims. We can view a set of conflicting arguments as a graph in which the nodes represent arguments and the arcs between these nodes represent the defeat relation. A group of people can then collectively evaluate such graphs. To do this, the group must use a rule to aggregate their individual opinions about the entire argument graph. Here we present the first experimental evaluation of different principles commonly employed by aggregation rules presented in the literature. We use randomized controlled experiments to investigate which principles people consider better at aggregating opinions under different conditions. Our analysis reveals a number of factors, not captured by traditional formal models, that play an important role in determining the efficacy of aggregation. These results help bring formal models of argumentation closer to real-world application.<\/jats:p>","DOI":"10.1145\/3053371","type":"journal-article","created":{"date-parts":[[2017,6,13]],"date-time":"2017-06-13T12:18:36Z","timestamp":1497356316000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Experimental Assessment of Aggregation Principles in Argumentation-Enabled Collective Intelligence"],"prefix":"10.1145","volume":"17","author":[{"given":"Edmond","family":"Awad","sequence":"first","affiliation":[{"name":"The Media Lab, Massachusetts Institute of Technology; and Masdar Institute, Cambridge, MA, USA"}]},{"given":"Jean-Fran\u00e7ois","family":"Bonnefon","sequence":"additional","affiliation":[{"name":"Toulouse School of Economics, Center for Research in Management, Institute for Advanced Study in Toulouse, University of Toulouse Capitole, Toulouse Cedex 6, France"}]},{"given":"Martin","family":"Caminada","sequence":"additional","affiliation":[{"name":"School of Computer Science 8 Informatics, Cardiff University, Cardiff, UK"}]},{"given":"Thomas W.","family":"Malone","sequence":"additional","affiliation":[{"name":"Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, USA"}]},{"given":"Iyad","family":"Rahwan","sequence":"additional","affiliation":[{"name":"The Media Lab and Institute for Data, Systems, 8 Society, Massachusetts Institute of Technology; and Masdar Institute, Cambridge, MA, USA"}]}],"member":"320","published-online":{"date-parts":[[2017,6,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/11518655_24"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0020902"},{"volume-title":"Social Choice and Individual Values","author":"Arrow Kenneth J.","key":"e_1_2_1_3_1","unstructured":"Kenneth J. Arrow . 1951. Social Choice and Individual Values . Wiley , New York, NY . Kenneth J. Arrow. 1951. Social Choice and Individual Values. Wiley, New York, NY."},{"key":"e_1_2_1_4_1","unstructured":"Kenneth J. Arrow A. K. Sen and K. Suzumura (Eds.). 2002. Handbook of Social Choice and Welfare. Vol. 1. Elsevier Science Publishers (North-Holland).  Kenneth J. Arrow A. K. Sen and K. Suzumura (Eds.). 2002. Handbook of Social Choice and Welfare. Vol. 1. Elsevier Science Publishers (North-Holland)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1093\/logcom\/exv055"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/0951629813514300"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11229-005-9069-6"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9280.2007.01972.x"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00355-009-0421-5"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11238-007-9050-6"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 14th International Conference on Principles of Knowledge Representation and Reasoning (KR\u201914)","author":"Booth Richard","year":"2014","unstructured":"Richard Booth , Edmond Awad , and Iyad Rahwan . 2014 . Interval methods for judgment aggregation in argumentation . In Proceedings of the 14th International Conference on Principles of Knowledge Representation and Reasoning (KR\u201914) . 594--597. Richard Booth, Edmond Awad, and Iyad Rahwan. 2014. Interval methods for judgment aggregation in argumentation. In Proceedings of the 14th International Conference on Principles of Knowledge Representation and Reasoning (KR\u201914). 594--597."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11229-006-0005-1"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2nd International Conference on Computational Models of Argument (COMMA\u201908)","author":"Shum Simon Buckingham","year":"2008","unstructured":"Simon Buckingham Shum . 2008 . Cohere: Towards Web 2.0 argumentation . In Proceedings of the 2nd International Conference on Computational Models of Argument (COMMA\u201908) , Philippe Besnard, Sylvie Doutre, and Anothony Hunter (Eds.). IOS Press, Amsterdam, The Netherlands, 97--108. Simon Buckingham Shum. 2008. Cohere: Towards Web 2.0 argumentation. In Proceedings of the 2nd International Conference on Computational Models of Argument (COMMA\u201908), Philippe Besnard, Sylvie Doutre, and Anothony Hunter (Eds.). IOS Press, Amsterdam, The Netherlands, 97--108."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/11853886_11"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11225-009-9218-x"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-009-9116-7"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jet.2009.10.015"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.aam.2010.02.005"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622673.1622682"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(94)00041-X"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_2_1_22_1","volume-title":"BNAIC 2013: Proceedings of the 25th Benelux Conference on Artificial Intelligence","author":"Endriss Ulle","year":"2013","unstructured":"Ulle Endriss and Raquel Fern\u00e1ndez . 2013 . Collective annotation of linguistic resources: Basic principles and a formal model . In BNAIC 2013: Proceedings of the 25th Benelux Conference on Artificial Intelligence , Delft, The Netherlands , November 7-8, 2013. Delft University of Technology (TU Delft); under the auspices of the Benelux Association for Artificial Intelligence (BNVKI) and the Dutch Research School for Information and Knowledge Systems (SIKS). Ulle Endriss and Raquel Fern\u00e1ndez. 2013. Collective annotation of linguistic resources: Basic principles and a formal model. In BNAIC 2013: Proceedings of the 25th Benelux Conference on Artificial Intelligence, Delft, The Netherlands, November 7-8, 2013. Delft University of Technology (TU Delft); under the auspices of the Benelux Association for Artificial Intelligence (BNVKI) and the Dutch Research School for Information and Knowledge Systems (SIKS)."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.2307\/1954111"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02425262"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-01568-7","volume-title":"Judgment Aggregation: A Primer. Morgan 8 Claypool.","author":"Grossi Davide","year":"2014","unstructured":"Davide Grossi and Gabriella Pigozzi . 2014 . Judgment Aggregation: A Primer. Morgan 8 Claypool. Davide Grossi and Gabriella Pigozzi. 2014. Judgment Aggregation: A Primer. Morgan 8 Claypool."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.2307\/2586020"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0176-2680(98)00002-0"},{"volume-title":"Collaboration Technologies and Systems (CTS), 2011 International Conference on. IEEE, 40--47","author":"Introne Joshua","key":"e_1_2_1_28_1","unstructured":"Joshua Introne , Robert Laubacher , Gary Olson , and Thomas W. Malone . 2011. The Climate CoLab: Large scale model-based collaborative planning . In Collaboration Technologies and Systems (CTS), 2011 International Conference on. IEEE, 40--47 . Joshua Introne, Robert Laubacher, Gary Olson, and Thomas W. Malone. 2011. The Climate CoLab: Large scale model-based collaborative planning. In Collaboration Technologies and Systems (CTS), 2011 International Conference on. IEEE, 40--47."},{"key":"e_1_2_1_29_1","volume-title":"Supporting collaborative deliberation using a large-scale argumentation system: The MIT collaboratorium. Available at SSRN 1099082 4691-08","author":"Klein Mark","year":"2008","unstructured":"Mark Klein and Luca Iandoli . 2008. Supporting collaborative deliberation using a large-scale argumentation system: The MIT collaboratorium. Available at SSRN 1099082 4691-08 ( 2008 ). Mark Klein and Luca Iandoli. 2008. Supporting collaborative deliberation using a large-scale argumentation system: The MIT collaboratorium. Available at SSRN 1099082 4691-08 (2008)."},{"key":"e_1_2_1_30_1","first-page":"157","article-title":"The \u201cidentified victim\u201d effect: an identified group, or just a single individual","volume":"18","author":"Kogut Tehila","year":"2006","unstructured":"Tehila Kogut and Ilana Ritov . 2006 . The \u201cidentified victim\u201d effect: an identified group, or just a single individual ? J. Behav . Dec. Making 18 (2006), 157 -- 167 . Tehila Kogut and Ilana Ritov. 2006. The \u201cidentified victim\u201d effect: an identified group, or just a single individual? J. Behav. Dec. Making 18 (2006), 157--167.","journal-title":"J. Behav"},{"key":"e_1_2_1_31_1","volume-title":"Mendelzon","author":"Lin Jinxin","year":"1999","unstructured":"Jinxin Lin and Alberto O . Mendelzon . 1999 . Knowledge base merging by majority. In Dynamic Worlds. Springer , 195--218. Jinxin Lin and Alberto O. Mendelzon. 1999. Knowledge base merging by majority. In Dynamic Worlds. Springer, 195--218."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1086\/498466"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11229-011-0025-3"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jet.2010.02.001"},{"volume-title":"The Handbook of Rational and Social Choice","author":"List Christian","key":"e_1_2_1_35_1","unstructured":"Christian List and Clemens Puppe . 2009. Judgment aggregation: A survey . In The Handbook of Rational and Social Choice , Paul Anand, Clemens Puppe, and Prasanta Pattanaik (Eds.). Oxford University Press , Oxford, UK . Christian List and Clemens Puppe. 2009. Judgment aggregation: A survey. In The Handbook of Rational and Social Choice, Paul Anand, Clemens Puppe, and Prasanta Pattanaik (Eds.). Oxford University Press, Oxford, UK."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1013213108"},{"volume-title":"Classics of Social Choice","author":"McLean Iain","key":"e_1_2_1_37_1","unstructured":"Iain McLean , Arnold B. Urken , and Fiona Hewitt . 1995. Classics of Social Choice . University of Michigan Press. Iain McLean, Arnold B. Urken, and Fiona Hewitt. 1995. Classics of Social Choice. University of Michigan Press."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejpoleco.2014.09.005"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0085798"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1111\/0029-4624.35.s1.11"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11229-006-9063-7"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.econlet.2011.02.008"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2010.01123.x"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","volume":"1","author":"Rahwan Iyad","year":"2010","unstructured":"Iyad Rahwan and Fernando Tohm\u00e9 . 2010 . Collective argument evaluation as judgement aggregation . In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems : Volume 1-Volume 1 . International Foundation for Autonomous Agents and Multiagent Systems, 417--424. Iyad Rahwan and Fernando Tohm\u00e9. 2010. Collective argument evaluation as judgement aggregation. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 417--424."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2007.04.015"},{"volume-title":"The Fallacy of Corporate Moral Agency","author":"R\u00f6nnegard David","key":"e_1_2_1_46_1","unstructured":"David R\u00f6nnegard . 2015. The Fallacy of Corporate Moral Agency . Springer . David R\u00f6nnegard. 2015. The Fallacy of Corporate Moral Agency. Springer."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 17th Annual Conference of the Cognitive Science Society, J. D. Moore and J. Fain Lehman (Eds.)","author":"Stenning Keith","year":"1995","unstructured":"Keith Stenning and Richard Cox . 1995 . Attitudes to logical independence: Traits in quantifier interpretation . In Proceedings of the 17th Annual Conference of the Cognitive Science Society, J. D. Moore and J. Fain Lehman (Eds.) . Hillsdale, NJ, 742--747. Lawrence Erlbaum Associates, Inc. Keith Stenning and Richard Cox. 1995. Attitudes to logical independence: Traits in quantifier interpretation. In Proceedings of the 17th Annual Conference of the Cognitive Science Society, J. D. Moore and J. Fain Lehman (Eds.). Hillsdale, NJ, 742--747. Lawrence Erlbaum Associates, Inc."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00355-010-0448-7"}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3053371","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3053371","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:03:35Z","timestamp":1750215815000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3053371"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,12]]},"references-count":48,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2017,8,31]]}},"alternative-id":["10.1145\/3053371"],"URL":"https:\/\/doi.org\/10.1145\/3053371","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"type":"print","value":"1533-5399"},{"type":"electronic","value":"1557-6051"}],"subject":[],"published":{"date-parts":[[2017,6,12]]},"assertion":[{"value":"2016-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-06-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}