{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T21:10:31Z","timestamp":1771103431072,"version":"3.50.1"},"reference-count":100,"publisher":"Maximum Academic Press","license":[{"start":{"date-parts":[[2017,8,22]],"date-time":"2017-08-22T00:00:00Z","timestamp":1503360000000},"content-version":"unspecified","delay-in-days":233,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The Knowledge Engineering Review"],"published-print":{"date-parts":[[2017]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Complex systems often exhibit emergent behaviour, unexpected macro-level behaviour caused by the interaction of micro-level components. In multiagent systems, these micro-level components may be autonomous agents and the emergent behaviour may be expressed as norms\u2014patterns of behaviour that arise among the agents in response to their environment and each other. These emergent norms may be beneficial (e.g. by encouraging cooperative behaviour), or detrimental, but in either case it is useful to recognize these norms as they emerge and either encourage or discourage their establishment. We term this process engineering the emergence of norms and have identified three steps: the identification of a possible norm, evaluation of its benefit and its encouragement (or discouragement). This paper is an attempt to provide a survey of existing research related to these steps. We also provide an analysis of the approaches based upon their suitability for a variety of normative systems: we examine the requirements for agents to have autonomy over their choice of norms, the degree of observability required in the system, and the norm enforcement methods. The paper concludes with an discussion of open issues.<\/jats:p>","DOI":"10.1017\/s0269888917000169","type":"journal-article","created":{"date-parts":[[2017,8,24]],"date-time":"2017-08-24T01:56:49Z","timestamp":1503539809000},"source":"Crossref","is-referenced-by-count":19,"title":["Engineering the emergence of norms: a review"],"prefix":"10.48130","volume":"32","author":[{"given":"Chris","family":"Haynes","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Luck","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"McBurney","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samhar","family":"Mahmoud","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tom\u00e1\u0161","family":"V\u00edtek","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Simon","family":"Miles","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"27968","published-online":{"date-parts":[[2017,8,22]]},"reference":[{"key":"S0269888917000169_ref25","doi-asserted-by":"crossref","first-page":"233","DOI":"10.3233\/AIC-2011-0502","article-title":"Open issues for normative multi-agent systems","volume":"24","author":"Criado","year":"2011","journal-title":"AI Communications"},{"key":"S0269888917000169_ref43","unstructured":"Griffiths N. & Luck M. 2010. Changing neighbours: improving tag-based cooperation. In Proceedings of the Ninth International Conference on Autonomous Agents and Multiagent Systems: Volume 1, 249\u2013256. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S0269888917000169_ref41","doi-asserted-by":"publisher","DOI":"10.1086\/223933"},{"key":"S0269888917000169_ref2","doi-asserted-by":"crossref","first-page":"325","DOI":"10.3233\/AIC-2010-0477","article-title":"Norm internalization in artificial societies","volume":"23","author":"Andrighetto","year":"2010","journal-title":"AI Communications"},{"key":"S0269888917000169_ref38","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-012-9193-x"},{"key":"S0269888917000169_ref67","doi-asserted-by":"publisher","DOI":"10.1145\/2720024"},{"key":"S0269888917000169_ref21","unstructured":"Chen C.-C. , Nagl S. B. & Clack C. D. 2007. Specifying, detecting and analysing emergent behaviours in multi-level agent-based simulations. In Proceedings of the 2007 Summer Computer Simulation Conference, 969\u2013976. Society for Computer Simulation International."},{"key":"S0269888917000169_ref32","unstructured":"dos Santos Neto B. F. , da Silva V. T. & de Lucena C. J. P. 2012. An architectural model for autonomous normative agents. In Advances in Artificial Intelligence-SBIA 2012, Barros L. N., Finger M., Pozo A. T., Gimen\u00e9nez-Lugo G. A, Castilho M. (eds). Springer, 152\u2013161."},{"key":"S0269888917000169_ref50","unstructured":"Itaiwi A. M. K. , Ahmad M. S. , Tang A. Y. C. & Mahmoud M. A. 2014. A proposed norms\u2019 benefits awareness framework for norms adoption. In International Conference on Information Technology and Multimedia, pages 287\u2013292."},{"key":"S0269888917000169_ref12","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74459-7_15"},{"key":"S0269888917000169_ref20","first-page":"40","volume-title":"Agents and Computational Autonomy","author":"Castelfranchi","year":"2003"},{"key":"S0269888917000169_ref70","unstructured":"Parunak H. V. D. & VanderBok R. S. 1997. Managing emergent behavior in distributed control systems. In Proceedings of ISA Tech'97, Instrument Society of America, 1\u20138."},{"key":"S0269888917000169_ref56","unstructured":"Mahmoud S. , Barakat L. , Miles S. , Taweel A. , Delaney B. & Luck M. 2014. Information-based incentivisation when rewards are inadequate. In ECAI, 591\u2013596."},{"key":"S0269888917000169_ref96","unstructured":"Walker A. & Wooldridge M. 1995. Understanding the emergence of conventions in multi-agent systems. In Proceedings of the First International Conference on Multiagent Systems, 384\u2013389."},{"key":"S0269888917000169_ref54","doi-asserted-by":"publisher","DOI":"10.1162\/106454603321489518"},{"key":"S0269888917000169_ref46","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-14220-3_6"},{"key":"S0269888917000169_ref36","doi-asserted-by":"publisher","DOI":"10.1162\/002081898550789"},{"key":"S0269888917000169_ref61","unstructured":"Mahmoud S. , Miles S. , Taweel A. , Delaney B. & Luck M. 2015b. Norm establishment constrained by limited resources. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1819\u20131820. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S0269888917000169_ref80","unstructured":"Sen S. & Airiau S. 2007. Emergence of norms through social learning. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, 1507\u20131512."},{"key":"S0269888917000169_ref73","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-79003-7_15"},{"key":"S0269888917000169_ref40","unstructured":"Garlick M. & Chli M. 2009. The effect of social influence and curfews on civil violence. In Proceedings of the Eighth International Conference on Autonomous Agents and Multiagent Systems \u2013 Volume 2, 1335\u20131336. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S0269888917000169_ref28","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(02)00262-X"},{"key":"S0269888917000169_ref39","first-page":"2","article-title":"Appearances can be deceiving: lessons learned re-implementing Axelrod\u2019s \u2018evolutionary approach to norms\u2019","volume":"8","author":"Gal\u00e1n","year":"2005","journal-title":"Journal of Artificial Societies and Social Simulation"},{"key":"S0269888917000169_ref63","volume-title":"Utilitarianism","author":"Mill","year":"1863"},{"key":"S0269888917000169_ref94","unstructured":"Villatoro D. , Sabater-Mir J. & Sen S. 2011b. Social instruments for robust convention emergence. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Volume 11, 420\u2013425."},{"key":"S0269888917000169_ref87","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2007.03.004"},{"key":"S0269888917000169_ref89","unstructured":"Tinnemeier N. , Dastani M. & Meyer J.-J. 2010. Programming norm change. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1, 957\u2013964. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S0269888917000169_ref83","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(94)00007-N"},{"key":"S0269888917000169_ref65","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-015-9167-9"},{"key":"S0269888917000169_ref3","doi-asserted-by":"crossref","unstructured":"Artikis A. 2009. Formalising dynamic protocols for open agent systems. In Proceedings of the Twelth International Conference on Artificial Intelligence and Law, ICAIL \u201909, 68\u201377. ACM.","DOI":"10.1145\/1568234.1568243"},{"key":"S0269888917000169_ref82","first-page":"545","article-title":"Measuring emergence via nonlinear Granger causality","volume":"11","author":"Seth","year":"2008","journal-title":"Artificial Life"},{"key":"S0269888917000169_ref1","unstructured":"Andrighetto G. , Campenni M. , Cecconi F. & Conte R. 2010a. The complex loop of norm emergence: a simulation model. In Simulating Interacting Agents and Social Phenomena, Agent-Based Social Systems 7, Takadama K., Cioffi-Revilla C., Deffuant G. (eds). Springer, 19\u201335."},{"key":"S0269888917000169_ref6","doi-asserted-by":"publisher","DOI":"10.2307\/1960858"},{"key":"S0269888917000169_ref52","first-page":"1","volume-title":"Lectures in Complex Systems: The Proceedings of the 1993 Complex Systems Summer School, Santa Fe Institute Studies in the Sciences of Complexity Lecture Volume VI, Santa Fe Institute","author":"Kittock","year":"1994"},{"key":"S0269888917000169_ref97","first-page":"77","volume-title":"Engineering Self-Organising Systems","author":"Wolf","year":"2005"},{"key":"S0269888917000169_ref98","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-25867-4_16"},{"key":"S0269888917000169_ref60","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-015-9176-8"},{"key":"S0269888917000169_ref42","doi-asserted-by":"publisher","DOI":"10.1257\/jep.25.4.191"},{"key":"S0269888917000169_ref78","unstructured":"Savarimuthu B. T. R. , Cranefield S. , Purvis M. & Purvis M. 2007. Norm emergence in agent societies formed by dynamically changing networks. In Proceedings of the IEEE\/WIC\/ACM International Conference on Intelligent Agent Technology, 464\u2013470."},{"key":"S0269888917000169_ref91","doi-asserted-by":"crossref","unstructured":"Vasconcelos W. , Kollingbaum M. J. & Norman T. J. 2007. Resolving conflict and inconsistency in norm-regulated virtual organizations. In Proceedings of the Sixth International Joint Conference on Autonomous Agents and Multiagent Systems, 632\u2013639. ACM.","DOI":"10.1145\/1329125.1329236"},{"key":"S0269888917000169_ref34","doi-asserted-by":"publisher","DOI":"10.1016\/j.irle.2007.06.005"},{"key":"S0269888917000169_ref79","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14962-7_14"},{"key":"S0269888917000169_ref71","doi-asserted-by":"crossref","unstructured":"Riveret R. , Artikis A. , Busquets D. & Pitt J. 2014. Self-governance by transfiguration: from learning to prescriptions. In International Conference on Deontic Logic in Computer Science, 177\u2013191. Springer.","DOI":"10.1007\/978-3-319-08615-6_14"},{"key":"S0269888917000169_ref59","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-37756-3_8"},{"key":"S0269888917000169_ref49","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.1750"},{"key":"S0269888917000169_ref88","first-page":"79","article-title":"Multiagent systems","volume":"19","author":"Sycara","year":"1998","journal-title":"AI magazine"},{"key":"S0269888917000169_ref31","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-01991-3_11"},{"key":"S0269888917000169_ref90","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-023X(03)00066-1"},{"key":"S0269888917000169_ref14","doi-asserted-by":"crossref","DOI":"10.1017\/9781139171854","volume-title":"Metric Spaces","author":"Bryant","year":"1985"},{"key":"S0269888917000169_ref68","doi-asserted-by":"publisher","DOI":"10.1007\/s10588-011-9085-7"},{"key":"S0269888917000169_ref22","unstructured":"Chen C.-C. , Nagl S. B. & Clack C. D. 2009. A formalism for multi-level emergent behaviours in designed component-based systems and agent-based simulations. In From System Complexity to Emergent Properties, Aziz-Alaoui M.A., Bertelle C. (eds). Springer, 101\u2013114."},{"key":"S0269888917000169_ref48","first-page":"17","article-title":"Complex adaptive systems","volume":"121","author":"Holland","year":"1992","journal-title":"Daedalus"},{"key":"S0269888917000169_ref51","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-79003-7_21"},{"key":"S0269888917000169_ref16","unstructured":"Campillo-Sanchez P. & Gomez-Sanz J. J. 2015. A framework for developing multi-agent systems in ambient intelligence scenarios. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1949\u20131950. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S0269888917000169_ref44","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-013-0428-6"},{"key":"S0269888917000169_ref24","unstructured":"Cranefield S. , Savarimuthu B. , Meneguzzi F. & Oren N. 2015. A Bayesian approach to norm identification. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1743\u20131744. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S0269888917000169_ref19","doi-asserted-by":"crossref","unstructured":"Castelfranchi C. & Falcone R. 1998. Principles of trust for MAS: cognitive anatomy, social importance, and quantification. In The Proceedings of the International Conference on Multi Agent Systems, 72\u201379. IEEE.","DOI":"10.1109\/ICMAS.1998.699034"},{"key":"S0269888917000169_ref33","doi-asserted-by":"publisher","DOI":"10.1080\/713670491"},{"key":"S0269888917000169_ref45","doi-asserted-by":"publisher","DOI":"10.1093\/he\/9780199644704.001.0001"},{"key":"S0269888917000169_ref29","first-page":"131","article-title":"Emergence of coordination in scale-free networks","volume":"1","author":"Delgado","year":"2003","journal-title":"Web Intelligence and Agent Systems"},{"key":"S0269888917000169_ref17","doi-asserted-by":"publisher","DOI":"10.1016\/j.jebo.2009.03.022"},{"key":"S0269888917000169_ref7","unstructured":"Balke T. , De Vos M. & Padget J. 2012. Normative run-time reasoning for institutionally-situated BDI agents. In Coordination, Organizations, Institutions, and Norms in Agent System VII, Cranefield S., van Riemsdijk M.B., V\u00e1zquez-Salceda J., Noriega P. (eds). Springer, 129\u2013148."},{"key":"S0269888917000169_ref53","first-page":"71","article-title":"Association rules mining: a recent overview","volume":"32","author":"Kotsiantis","year":"2006","journal-title":"GESTS International Transactions on Computer Science and Engineering"},{"key":"S0269888917000169_ref37","unstructured":"Fitzek F. H. P. & Katz M. D. 2007. Cellular controlled peer to peer communications: overview and potentials. In Cognitive Wireless Networks, Fitzek F.H.P., Katz M.D. (eds). Springer, 31\u201359."},{"key":"S0269888917000169_ref72","doi-asserted-by":"publisher","DOI":"10.3233\/MGS-2011-0167"},{"key":"S0269888917000169_ref58","unstructured":"Mahmoud S. , Griffiths N. , Keppens J. & Luck M. 2012b. Efficient norm emergence through experiential dynamic punishment. In Proceedings of the Twentieth European Conference on Artificial Intelligence, Volume 12, 576\u2013581."},{"key":"S0269888917000169_ref9","doi-asserted-by":"publisher","DOI":"10.1016\/j.agsy.2005.06.002"},{"key":"S0269888917000169_ref13","unstructured":"Broughton J. 1990. Restraint Use by Car Occupants. Transport and Road Research Laboratory Research Report, 289."},{"key":"S0269888917000169_ref15","doi-asserted-by":"publisher","DOI":"10.1109\/4235.956713"},{"key":"S0269888917000169_ref30","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2006.05.011"},{"key":"S0269888917000169_ref64","doi-asserted-by":"publisher","DOI":"10.1002\/smj.4250060306"},{"key":"S0269888917000169_ref4","doi-asserted-by":"publisher","DOI":"10.1093\/logcom\/exr018"},{"key":"S0269888917000169_ref57","unstructured":"Mahmoud S. , Griffiths N. , Keppens J. & Luck M. 2010. An analysis of norm emergence in Axelrod\u2019s model. In Proceedings of 8th European Workshop on Multi-Agent Systems."},{"key":"S0269888917000169_ref55","unstructured":"Mahmoud M. A. , Ahmad M. S. , Ahmad A. , Yusoff M. Z. M. & Mustapha A. 2012a. A norms mining approach to norms detection in multi-agent systems. In International Conference on Computer Information Science, Volume 1, 458\u2013463."},{"key":"S0269888917000169_ref5","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2010.05.010"},{"key":"S0269888917000169_ref10","doi-asserted-by":"publisher","DOI":"10.2307\/2392667"},{"key":"S0269888917000169_ref27","doi-asserted-by":"publisher","DOI":"10.1159\/000094185"},{"key":"S0269888917000169_ref35","doi-asserted-by":"publisher","DOI":"10.1016\/j.geb.2012.10.011"},{"key":"S0269888917000169_ref8","doi-asserted-by":"publisher","DOI":"10.1111\/0029-4624.31.s11.17"},{"key":"S0269888917000169_ref47","unstructured":"Hoffmann M. J. 2005. Self-organized criticality and norm avalanches. In Proceedings of the Symposium on Normative Multi-Agent Systems, AISB05: Social Intelligence and Interaction in Animals, Robots and Agents, 117\u2013125."},{"key":"S0269888917000169_ref11","volume-title":"The Grammar of Society: The Nature and Dynamics of Social Norms","author":"Bicchieri","year":"2006"},{"key":"S0269888917000169_ref18","first-page":"3","article-title":"Normative reputation and the costs of compliance","volume":"1","author":"Castelfranchi","year":"1998","journal-title":"Journal of Artificial Societies and Social Simulation"},{"key":"S0269888917000169_ref26","unstructured":"Criado N. , Argente E. , Noriega P. & Botti V. 2010. Towards a normative BDI architecture for norm compliance. In Proceedings of the Multi-Agent Logics, Languages, and Organisations Federated Workshops, 2010, 65\u201381."},{"key":"S0269888917000169_ref77","doi-asserted-by":"crossref","unstructured":"Savarimuthu B. , Padget J. & Purvis M. 2013b. Social norm recommendation for virtual agent societies. In International Conference on Principles and Practice of Multi-Agent Systems, 308\u2013323. Springer.","DOI":"10.1007\/978-3-642-44927-7_21"},{"key":"S0269888917000169_ref62","doi-asserted-by":"publisher","DOI":"10.2202\/1555-5879.1011"},{"key":"S0269888917000169_ref23","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008310107755"},{"key":"S0269888917000169_ref86","first-page":"191","volume-title":"Normative Multi-Agent Systems, Volume 4","author":"Singh","year":"2013"},{"key":"S0269888917000169_ref76","doi-asserted-by":"crossref","unstructured":"Savarimuthu B. & Dam H. K. 2013. Towards mining norms in open source software repositories. In International Workshop on Agents and Data Mining Interaction, 26\u201339. Springer.","DOI":"10.1007\/978-3-642-55192-5_3"},{"key":"S0269888917000169_ref84","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(97)00028-3"},{"key":"S0269888917000169_ref66","doi-asserted-by":"publisher","DOI":"10.1145\/1218063.1217964"},{"key":"S0269888917000169_ref92","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-008-9070-9"},{"key":"S0269888917000169_ref85","first-page":"21","article-title":"Norms as a basis for governing sociotechnical systems","volume":"5","author":"Singh","year":"2013","journal-title":"ACM Transactions on Intelligent Systems and Technology (TIST)"},{"key":"S0269888917000169_ref75","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-012-9126-7"},{"key":"S0269888917000169_ref95","doi-asserted-by":"crossref","unstructured":"Villatoro D. , Sen S. & Sabater-Mir J. 2009. Topology and memory effect on convention emergence. In Proceedings of the 2009 IEEE\/WIC\/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, 233\u2013240. IEEE Computer Society.","DOI":"10.1109\/WI-IAT.2009.155"},{"key":"S0269888917000169_ref93","unstructured":"Villatoro D. , Andrighetto G. , Sabater-Mir J. & Conte R. 2011a. Dynamic sanctioning for robust and cost-efficient norm compliance. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Volume 11, 414\u2013419."},{"key":"S0269888917000169_ref99","doi-asserted-by":"crossref","unstructured":"Zhang Y. & Leezer J. 2009. Emergence of social norms in complex networks. In Proceedings of the International Conference on Computational Science and Engineering, Volume 4, 549\u2013555. IEEE.","DOI":"10.1109\/CSE.2009.392"},{"key":"S0269888917000169_ref81","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2011.11.005"},{"key":"S0269888917000169_ref69","unstructured":"Oren N. & Meneguzzi F. 2013. Norm identification through plan recognition. In Proceedings of the Workshop on Coordination, Organization, Institutions and Norms in Agent Systems (COIN 2013@ AAMAS)."},{"key":"S0269888917000169_ref74","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15420-1_5"},{"key":"S0269888917000169_ref100","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.72.056118"}],"container-title":["The Knowledge Engineering Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0269888917000169","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T14:42:07Z","timestamp":1767624127000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0269888917000169\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017]]},"references-count":100,"alternative-id":["S0269888917000169"],"URL":"https:\/\/doi.org\/10.1017\/s0269888917000169","relation":{},"ISSN":["0269-8889","1469-8005"],"issn-type":[{"value":"0269-8889","type":"print"},{"value":"1469-8005","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017]]},"article-number":"e18"}}