{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T02:47:48Z","timestamp":1768877268016,"version":"3.49.0"},"reference-count":71,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,10,24]],"date-time":"2021-10-24T00:00:00Z","timestamp":1635033600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004281","name":"Narodowe Centrum Nauki","doi-asserted-by":"publisher","award":["UMO-2018\/28\/C\/HS5\/00543"],"award-info":[{"award-number":["UMO-2018\/28\/C\/HS5\/00543"]}],"id":[{"id":"10.13039\/501100004281","id-type":"DOI","asserted-by":"publisher"}]},{"name":"POTENTIAL program","award":["26\/EIM\/2021\/POT"],"award-info":[{"award-number":["26\/EIM\/2021\/POT"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The domain of policymaking, which used to be limited to small groups of specialists, is now increasingly opening up to the participation of wide collectives, which are not only influencing government decisions, but also enhancing citizen engagement and transparency, improving service delivery and gathering the distributed wisdom of diverse participants. Although collective intelligence has become a more common approach to policymaking, the studies on this subject have not been conducted in a systematic way. Nevertheless, we hypothesized that methods and strategies specific to different types of studies in this field could be identified and analyzed. Based on a systematic literature review, as well as qualitative and statistical analyses, we identified 15 methods and revealed the dependencies between them. The review indicated the most popular approaches, and the underrepresented ones that can inspire future research.<\/jats:p>","DOI":"10.3390\/e23111391","type":"journal-article","created":{"date-parts":[[2021,10,24]],"date-time":"2021-10-24T22:06:16Z","timestamp":1635113176000},"page":"1391","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Organisational Structure and Created Values. Review of Methods of Studying Collective Intelligence in Policymaking"],"prefix":"10.3390","volume":"23","author":[{"given":"Rafa\u0142","family":"Olszowski","sequence":"first","affiliation":[{"name":"Faculty of Humanities, AGH University of Science and Technology, Gramatyka 8a, 30-071 Krak\u00f3w, Poland"},{"name":"Center for Collective Intelligence, Massachusetts Institute of Technology, 245 First Street, E94, Cambridge, MA 02142, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8490-7117","authenticated-orcid":false,"given":"Piotr","family":"Pi\u0119ta","sequence":"additional","affiliation":[{"name":"Faculty of Humanities, AGH University of Science and Technology, Gramatyka 8a, 30-071 Krak\u00f3w, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastian","family":"Baran","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Cracow University of Economics, Rakowicka 27, 31-510 Krak\u00f3w, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcin","family":"Chmielowski","sequence":"additional","affiliation":[{"name":"Fundacja Wolno\u015bci i Przedsi\u0119biorczo\u015bci, Ul. Asnyka 6, 40-696 Katowice, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,24]]},"reference":[{"key":"ref_1","unstructured":"Bernstein, M.S. (2015). Handbook of Collective Intelligence, The MIT Press."},{"key":"ref_2","unstructured":"Levy, P. (1997). Collective Intelligence: Mankind\u2019s Emerging World in Cyberspace, Plenum."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"16385","DOI":"10.1073\/pnas.0403723101","article-title":"Groups of diverse problem-solvers can outperform groups of high-ability problem-solvers","volume":"101","author":"Hong","year":"2004","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mulgan, G. (2018). Big Mind: How Collective Intelligence Can Change Our World, Princeton University Press.","DOI":"10.1515\/9781400888511"},{"key":"ref_5","first-page":"21","article-title":"The collective intelligence genome","volume":"51","author":"Malone","year":"2010","journal-title":"MIT Sloan Manag. Rev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1126\/science.1193147","article-title":"Evidence for a Collective Intelligence Factor in the Performance of Human Groups","volume":"330","author":"Woolley","year":"2010","journal-title":"Science"},{"key":"ref_7","first-page":"45","article-title":"Decisions 2.0: The Power of Collective Intelligence","volume":"50","author":"Bonabeau","year":"2009","journal-title":"MIT Sloan Manag. Rev."},{"key":"ref_8","unstructured":"Surowiecki, J. (2005). The Wisdom of Crowds, Anchor Books."},{"key":"ref_9","unstructured":"Howe, J. (2008). Crowdsourcing: Why the Power of The Crowd Is Driving the Future of Business, Crown Business."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gonz\u00e1lez, J.R., Pelta, D.A., Cruz, C., Terrazas, G., and Krasnogor, N. (2020). Using Entropy for Evaluating Swarm Intelligence Algorithms. Nature Inspired Cooperative Strategies for Optimization (NICSO 2010), Springer. Studies in Computational Intelligence.","DOI":"10.1007\/978-3-642-12538-6"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"20150037","DOI":"10.1098\/rsif.2015.0037","article-title":"The entropic basis of collective behaviour","volume":"12","author":"Mann","year":"2015","journal-title":"J. R. Soc. Interface"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kang, H., Bei, F., Shen, Y., Sun, X., and Chen, Q. (2021). A Diversity Model Based on Dimension Entropy and Its Application to Swarm Intelligence Algorithm. Entropy, 23.","DOI":"10.3390\/e23040397"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1016\/j.giq.2007.04.007","article-title":"The shape of eParticipation: Characterizing an emerging research area","volume":"25","author":"Rose","year":"2008","journal-title":"Gov. Inf. Q."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sonntagbauer, P., Nazemi, K., Sonntagbauer, S., Prister, G., and Burkhardt, D. (2014). A Living Roadmap for Policymaking 2.0. Handbook of Research on Advanced ICT Integration for Governance and Policy Modeling, IGI Global. [1st ed.].","DOI":"10.4018\/978-1-4666-6236-0"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1016\/j.giq.2018.09.008","article-title":"Mapping the challenges of artificial intelligence in the public sector: Evidence from public healthcare","volume":"36","author":"Sun","year":"2019","journal-title":"Gov. Inf. Q."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1728","DOI":"10.1080\/09640568.2015.1085841","article-title":"Public participation mechanisms and sustainable policy-making: A case study analysis of Mexico City\u2019s Plan Verde","volume":"59","author":"Madero","year":"2016","journal-title":"J. Environ. Plan. Manag."},{"key":"ref_17","unstructured":"Aitamurto, T. (2012). Crowdsourcing for Democracy: New Era in Policy\u2013Making. Publications of the Committee for the Future, Parliament of Finland."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1111\/jopp.12032","article-title":"Inclusive Constitution-Making: The Icelandic Experiment","volume":"23","author":"Landemore","year":"2015","journal-title":"J. Political Philos."},{"key":"ref_19","unstructured":"Greenemeier, L. (2021, February 22). Smart Machines Join Humans in Tracking Africa Ebola Outbreak. Available online: https:\/\/www.scientificamerican.com\/article\/smart-machines-join-humans-in-tracking-africa-ebola-outbreak\/."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"43","DOI":"10.22230\/cjc.2019v44n2a3509","article-title":"Artificial intelligence policy innovations at the Canadian Federal Government","volume":"44","author":"McKelvey","year":"2019","journal-title":"Can. J. Commun."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"101509","DOI":"10.1016\/j.giq.2020.101509","article-title":"Assessing the public policy-cycle framework in the age of artificial intelligence: From agenda-setting to policy evaluation","volume":"37","author":"Criado","year":"2020","journal-title":"Gov. Inf. Q."},{"key":"ref_22","unstructured":"Joyner-Roberson, E. (2020, September 22). What Do Drones, AI and Proactive Policing Have in Common?. Available online: https:\/\/www.sas.com\/en_za\/insights\/articles\/risk-fraud\/drones-ai-proactive-policing.html."},{"key":"ref_23","unstructured":"Grothaus, M. (2021, February 22). China\u2019s Airport Facial Recognition Kiosks Should Make Us Fear for Our Privacy. Available online: https:\/\/www.fastcompany.com\/90324512\/chinas-airport-facial-recognition-kiosks-should-make-us-fear-for-ourprivacy."},{"key":"ref_24","first-page":"22","article-title":"Sustainable Policy Making: A Strategic Challenge for Artificial Intelligence","volume":"35","author":"Milano","year":"2014","journal-title":"AI Mag."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1016\/j.giq.2013.12.006","article-title":"An empirical analysis of e-participation. The role of social networks and e-government over citizens\u2019 online engagement","volume":"31","author":"Vicente","year":"2014","journal-title":"Gov. Inf. Q."},{"key":"ref_26","first-page":"1","article-title":"Varieties of Participatory Democracy and Democratic Theory","volume":"16","author":"Wolfe","year":"1986","journal-title":"Political Sci. Rev."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1017\/S1537592711004877","article-title":"Participatory Democracy Revisited","volume":"10","author":"Pateman","year":"2012","journal-title":"Perspect. Politics"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1111\/j.1468-2427.2008.00777.x","article-title":"Participatory budgeting in Europe: Potentials and challenges","volume":"32","author":"Sintomer","year":"2008","journal-title":"Int. J. Urban Reg. Res."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1093\/jopart\/mum032","article-title":"Collaborative Governance in Theory and Practice","volume":"18","author":"Ansell","year":"2007","journal-title":"J. Public Adm. Res. Theory"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/jopart\/mur011","article-title":"An Integrative Framework for Collaborative Governance","volume":"22","author":"Emerson","year":"2021","journal-title":"J. Public Adm. Res. Theory"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"6","DOI":"10.7326\/0003-4819-151-4-200908180-00135","article-title":"Preferred Reporting Items for Systematic Reviews and Meta Analyses: The PRISMA statement","volume":"151","author":"Moher","year":"2009","journal-title":"Ann. Intern. Med."},{"key":"ref_32","unstructured":"B\u00fccheler, T., F\u00fcchslin, R.M., Pfeifer, R., and Sieg, J.H. (2010, January 19\u201323). Crowdsourcing, Open Innovation and Collective Intelligence in the scientific method: A research agenda and operational framework. Proceedings of the Artificial Life XII\u2014Twelfth International Conference on the Synthesis and Simulation of Living Systems, Odense, Denmark."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/j.jbusres.2019.07.039","article-title":"Literature review as a research methodology: An overview and guidelines","volume":"104","author":"Snyder","year":"2019","journal-title":"J. Bus. Res."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1057\/ejis.2011.51","article-title":"Using grounded theory as a method for rigorously reviewing literature","volume":"22","author":"Wolfswinkel","year":"2013","journal-title":"Eur. J. Inf. Syst."},{"key":"ref_35","unstructured":"Corbin, J., and Strauss, A. (2008). Basics of Qualitative Research, Sage. [3rd ed.]."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1002\/(SICI)1097-4571(199306)44:5<298::AID-ASI5>3.0.CO;2-A","article-title":"KeyWords Plus Algorithmic Derivative Indexing","volume":"44","author":"Garfield","year":"1993","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1002\/poi3.102","article-title":"The Fundamentals of Policy Crowdsourcing","volume":"7","author":"Taeihagh","year":"2015","journal-title":"Policy Internet"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1007\/s11077-017-9303-3","article-title":"Crowdsourcing: A New Tool for Policy-Making?","volume":"50","author":"Taeihagh","year":"2017","journal-title":"Policy Sci. J."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kerzner, H. (2017). Project Management Organisational Structures. Project Management Case Studies, Willey.","DOI":"10.1002\/9781119389040"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/20508840.2017.1282665","article-title":"The value of crowdsourcing in public policymaking: Epistemic, democratic and economic value","volume":"5","author":"Aitamurto","year":"2017","journal-title":"Theory Pract. Legis."},{"key":"ref_41","first-page":"68","article-title":"Value Creation Through Collective Intelligence: Managing Intellectual Capital","volume":"18","author":"Iacuzzil","year":"2020","journal-title":"Electron. J. Knowl. Manag."},{"key":"ref_42","unstructured":"White, J. (2007). Managing Information in the Public Sector, M.E. Sharpe."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1080\/1369118X.2016.1228993","article-title":"Unmasking the crowd: Participants\u2019 motivation factors, expectations, and profile in a crowdsourced law reform","volume":"20","author":"Aitamurto","year":"2017","journal-title":"Inf. Commun. Soc."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1177\/0894439314560851","article-title":"Opening Government: Designing Open Innovation Processes to Collaborate with External Problem Solvers","volume":"33","author":"Mergel","year":"2015","journal-title":"Soc. Sci. Comput. Rev."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.giq.2014.10.002","article-title":"Open government objectives and participation motivations","volume":"32","author":"Wijnhoven","year":"2015","journal-title":"Gov. Inf. Q."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1080\/03637751.2017.1359748","article-title":"Finding the diamond in the rough: Exploring communication and platform in crowdsourcing performance","volume":"84","author":"Guth","year":"2017","journal-title":"Commun. Monogr."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1320","DOI":"10.1177\/1461444817691509","article-title":"Supporting argumentation in online political debate: Evidence from an experiment of collective deliberation","volume":"20","author":"Iandoli","year":"2018","journal-title":"New Media Soc."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1108\/FS-07-2014-0050","article-title":"New forms of innovation: Critical issues for future pathways","volume":"18","author":"Leitner","year":"2016","journal-title":"Foresight"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1016\/j.bushor.2013.12.009","article-title":"Open innovation requires integrated competition-community ecosystems: Lessons learned from civic open innovation","volume":"57","author":"Almirall","year":"2014","journal-title":"Bus. Horiz."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.cogsys.2017.01.002","article-title":"Public policy and the wisdom of crowds","volume":"43","author":"Epp","year":"2017","journal-title":"Cogn. Syst. Res."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.cobeha.2017.03.004","article-title":"Collective decision-making","volume":"16","author":"Bose","year":"2017","journal-title":"Curr. Opin. Behav. Sci."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1111\/puar.12808","article-title":"Crowdsourcing Government: Lessons from Multiple Disciplines","volume":"77","author":"Liu","year":"2017","journal-title":"Public Admin. Rev."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1080\/10967494.2018.1488780","article-title":"Barriers for Crowd\u2019s Impact in Crowdsourced Policymaking: Civic Data Overload and Filter Hierarchy","volume":"22","author":"Chen","year":"2019","journal-title":"Int. Public Manag. J."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"446","DOI":"10.1016\/j.giq.2012.06.003","article-title":"From e-government to we-government: Defining a typology for citizen coproduction in the age of social media","volume":"29","author":"Linders","year":"2012","journal-title":"Gov. Inf. Q."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Ojo, A., and Millard, J. (2017). Governance, Transparency and the Collaborative Design of Open Data Collaboration Platforms: Understanding Barriers, Options, and Needs. Government 3.0\u2014Next Generation Government Technology Infrastructure and Services, Springer.","DOI":"10.1007\/978-3-319-63743-3"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.im.2005.01.002","article-title":"The role played by perceived usability, satisfaction and consumer trust on website loyalty","volume":"43","author":"Gurrea","year":"2006","journal-title":"Inf. Manag."},{"key":"ref_57","first-page":"17","article-title":"Co-Designing participatory tools for a New Age: A proposal for combining collective and artificial intelligences","volume":"5","author":"Aguilar","year":"2018","journal-title":"Int. J. Public Adm. Digit. Age"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Lenart-Gansiniec, R., and Su\u0142kowski, \u0141. (2018). Crowdsourcing\u2014A New Paradigm of Organisational Learning of Public Organisations. Sustainability, 10.","DOI":"10.20944\/preprints201808.0467.v1"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"217","DOI":"10.2307\/2983604","article-title":"Contingency table involving small numbers and the \u03c72 test","volume":"1","author":"Yates","year":"1934","journal-title":"Suppl. J. R. Stat. Soc."},{"key":"ref_60","unstructured":"(2021, May 12). Web of Science Core Collection Help. Available online: https:\/\/images.webofknowledge.com\/images\/help\/WOS\/contents.html."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Landemore, H. (2013). Democratic Reason: Politics, Collective Intelligence, and the Rule of the Many, Princeton University Press.","DOI":"10.23943\/princeton\/9780691155654.001.0001"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Landemore, H. (2020). Open Democracy: Reinventing Popular Rule for the Twenty-First Century, Princeton University Press.","DOI":"10.23943\/princeton\/9780691181998.001.0001"},{"key":"ref_63","unstructured":"Noveck, B.S., Harvey, R., and Dinesh, A. (2019). The Open Policymaking Playbook, New York University. Available online: https:\/\/www.thegovlab.org\/static\/files\/publications\/openpolicymaking-april29.pdf."},{"key":"ref_64","unstructured":"Ryan, M., Gambrell, D., and Noveck, B.S. (2020). Using Collective Intelligence to Solve Public Problems, Nesta."},{"key":"ref_65","unstructured":"Noveck, B.S., Konopacki, M., Dinesh, A., Ryan, M., Munozcano, B.R., Kornberg, M., Gambrell, D., Hervey, R., Joerger, G., and DeJohn, S. (2020). Crowdlaw for Congress. Strategies for 21st Century Lawmaking, New York University. Available online: https:\/\/congress.crowd.law\/files\/crowdlaw_playbook_Oct2020.pdf."},{"key":"ref_66","unstructured":"Aitamurto, T. (2014). Crowdsourced Off.-Road Traffic Law Experiment in Finland."},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Noveck, B.S. (2015). Smart Citizens, Smarter State: The Technologies of Expertise and the Future of Governing, Harvard University Press.","DOI":"10.4159\/9780674915435"},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Mulgan, G. (2019). Social Innovation: How Societies Find the Power to Change. Policy Press.","DOI":"10.56687\/9781447353805"},{"key":"ref_69","unstructured":"Verhulst, S.G., Zahuranec, A.J., and Young, A. (2019). Identifying Citizens\u2019 Needs by Combining AI and CI, New York University. Available online: https:\/\/thegovlab.org\/static\/files\/publications\/CI-AI_oct2019.pdf."},{"key":"ref_70","unstructured":"(2021, July 24). Transforming Our World: The 2030 Agenda for Sustainable Development. Available online: https:\/\/sdgs.un.org\/2030agenda."},{"key":"ref_71","unstructured":"Malone, T. (2018). Superminds: The Surprising Power of People and Computers Thinking Together, Little, Brown and Co."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/11\/1391\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:22:16Z","timestamp":1760167336000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/11\/1391"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,24]]},"references-count":71,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["e23111391"],"URL":"https:\/\/doi.org\/10.3390\/e23111391","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,24]]}}}