{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T13:34:51Z","timestamp":1769088891588,"version":"3.49.0"},"reference-count":69,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2021,1,26]],"date-time":"2021-01-26T00:00:00Z","timestamp":1611619200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>The field of Artificial Intelligence (AI) is going through a period of great expectations, introducing a certain level of anxiety in research, business and also policy. This anxiety is further energised by an AI race narrative that makes people believe they might be missing out. Whether real or not, a belief in this narrative may be detrimental as some stake-holders will feel obliged to cut corners on safety precautions, or ignore societal consequences just to \u201cwin\u201d. Starting from a baseline model that describes a broad class of technology races where winners draw a significant benefit compared to others (such as AI advances, patent race, pharmaceutical technologies), we investigate here how positive (rewards) and negative (punishments) incentives may beneficially influence the outcomes. We uncover conditions in which punishment is either capable of reducing the development speed of unsafe participants or has the capacity to reduce innovation through over-regulation. Alternatively, we show that, in several scenarios, rewarding those that follow safety measures may increase the development speed while ensuring safe choices. Moreover, in the latter regimes, rewards do not suffer from the issue of over-regulation as is the case for punishment. Overall, our findings provide valuable insights into the nature and kinds of regulatory actions most suitable to improve safety compliance in the contexts of both smooth and sudden technological shifts.<\/jats:p>","DOI":"10.1371\/journal.pone.0244592","type":"journal-article","created":{"date-parts":[[2021,1,26]],"date-time":"2021-01-26T18:23:16Z","timestamp":1611685396000},"page":"e0244592","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":31,"title":["Mediating artificial intelligence developments through negative and positive incentives"],"prefix":"10.1371","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3095-7714","authenticated-orcid":true,"given":"The Anh","family":"Han","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7880-4322","authenticated-orcid":true,"given":"Lu\u00eds Moniz","family":"Pereira","sequence":"additional","affiliation":[]},{"given":"Tom","family":"Lenaerts","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9103-2862","authenticated-orcid":true,"given":"Francisco C.","family":"Santos","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2021,1,26]]},"reference":[{"issue":"2","key":"pone.0244592.ref001","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1007\/s00146-015-0590-y","article-title":"Racing to the precipice: a model of artificial intelligence development","volume":"31","author":"S Armstrong","year":"2016","journal-title":"AI & society"},{"key":"pone.0244592.ref002","doi-asserted-by":"crossref","unstructured":"Cave S, \u00d3h\u00c9igeartaigh S. An AI Race for Strategic Advantage: Rhetoric and Risks. In: AAAI\/ACM Conference on Artificial Intelligence, Ethics and Society; 2018. p. 36\u201340.","DOI":"10.1145\/3278721.3278780"},{"key":"pone.0244592.ref003","unstructured":"AI-Roadmap-Institute. Report from the AI Race Avoidance Workshop, Tokyo. 2017."},{"key":"pone.0244592.ref004","unstructured":"Shulman C, Armstrong S. Arms control and intelligence explosions. In: 7th European Conference on Computing and Philosophy (ECAP), Bellaterra, Spain, July; 2009. p. 2\u20134."},{"issue":"51","key":"pone.0244592.ref005","doi-asserted-by":"crossref","first-page":"14515","DOI":"10.1073\/pnas.1604989113","article-title":"Coordination vs. voluntarism and enforcement in sustaining international environmental cooperation","volume":"113","author":"S Barrett","year":"2016","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"pone.0244592.ref006","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1007\/s10640-012-9581-3","article-title":"Enforcing compliance with environmental agreements in the absence of strong institutions: An experimental analysis","volume":"54","author":"TL Cherry","year":"2013","journal-title":"Environmental and Resource Economics"},{"key":"pone.0244592.ref007","volume-title":"Evolution and the capacity for commitment. Foundation series on trust","author":"RM Nesse","year":"2001"},{"issue":"4","key":"pone.0244592.ref008","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1007\/s00146-016-0677-0","article-title":"On the promotion of safe and socially beneficial artificial intelligence","volume":"32","author":"SD Baum","year":"2017","journal-title":"AI & Society"},{"issue":"7701","key":"pone.0244592.ref009","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1038\/d41586-018-04602-6","article-title":"Regulate artificial intelligence to avert cyber arms race","volume":"556","author":"M Taddeo","year":"2018","journal-title":"Nature"},{"issue":"5","key":"pone.0244592.ref010","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1080\/00963402.2016.1216672","article-title":"It\u2019s already too late to stop the AI arms race: We must manage it instead","volume":"72","author":"EM Geist","year":"2016","journal-title":"Bulletin of the Atomic Scientists"},{"issue":"233","key":"pone.0244592.ref011","article-title":"The role of artificial intelligence in achieving the Sustainable Development Goals","volume":"11","author":"R Vinuesa","year":"2020","journal-title":"Nature Communications"},{"key":"pone.0244592.ref012","unstructured":"Askell A, Brundage M, Hadfield G. The Role of Cooperation in Responsible AI Development. arXiv preprint arXiv:190704534. 2019."},{"key":"pone.0244592.ref013","doi-asserted-by":"crossref","first-page":"881","DOI":"10.1613\/jair.1.12225","article-title":"To Regulate or Not: A Social Dynamics Analysis of an Idealised AI Race","volume":"69","author":"TA Han","year":"2020","journal-title":"Journal of Artificial Intelligence Research"},{"key":"pone.0244592.ref014","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511806292","volume-title":"Evolution and the Theory of Games","author":"J Maynard-Smith","year":"1982"},{"key":"pone.0244592.ref015","doi-asserted-by":"crossref","DOI":"10.2307\/j.ctvjghw98","volume-title":"Evolutionary Dynamics: Exploring the Equations of Life","author":"MA Nowak","year":"2006"},{"key":"pone.0244592.ref016","doi-asserted-by":"crossref","DOI":"10.1515\/9781400832255","volume-title":"The Calculus of Selfishness","author":"K Sigmund","year":"2010"},{"issue":"5","key":"pone.0244592.ref017","doi-asserted-by":"crossref","first-page":"1133","DOI":"10.1162\/jeea_a_00017","article-title":"On the winner-take-all principle in innovation races","volume":"8","author":"V Denicol\u00f2","year":"2010","journal-title":"Journal of the European Economic Association"},{"issue":"3","key":"pone.0244592.ref018","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1080\/10438599.2013.825427","article-title":"Technological races and stock market value: evidence from the pharmaceutical industry","volume":"23","author":"S Campart","year":"2014","journal-title":"Economics of Innovation and New Technology"},{"key":"pone.0244592.ref019","doi-asserted-by":"crossref","unstructured":"Lemley MA. The myth of the sole inventor. Michigan Law Review. 2012; p. 709\u2013760.","DOI":"10.2139\/ssrn.1856610"},{"key":"pone.0244592.ref020","volume-title":"Global challenges: 12 risks that threaten human civilization","author":"D Pamlin","year":"2015"},{"issue":"3","key":"pone.0244592.ref021","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1080\/0952813X.2014.895105","article-title":"The errors, insights and lessons of famous AI predictions\u2013and what they mean for the future","volume":"26","author":"S Armstrong","year":"2014","journal-title":"Journal of Experimental & Theoretical Artificial Intelligence"},{"key":"pone.0244592.ref022","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1613\/jair.1.11222","article-title":"When will AI exceed human performance? Evidence from AI experts","volume":"62","author":"K Grace","year":"2018","journal-title":"Journal of Artificial Intelligence Research"},{"key":"pone.0244592.ref023","doi-asserted-by":"crossref","DOI":"10.4337\/9781849801843","volume-title":"Global pharmaceutical policy: ensuring medicines for tomorrow\u2019s world","author":"FM Abbott","year":"2009"},{"key":"pone.0244592.ref024","doi-asserted-by":"crossref","unstructured":"Burrell R, Kelly C. The COVID-19 pandemic and the challenge for innovation policy. Available at SSRN 3576481. 2020.","DOI":"10.2139\/ssrn.3576481"},{"issue":"15","key":"pone.0244592.ref025","doi-asserted-by":"crossref","first-page":"158104","DOI":"10.1103\/PhysRevLett.108.158104","article-title":"Emergence of fairness in repeated group interactions","volume":"108","author":"S Van Segbroeck","year":"2012","journal-title":"Phys Rev Lett"},{"issue":"4","key":"pone.0244592.ref026","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1162\/ARTL_a_00072","article-title":"Corpus-based intention recognition in cooperation dilemmas","volume":"18","author":"TA Han","year":"2012","journal-title":"Artificial Life"},{"key":"pone.0244592.ref027","doi-asserted-by":"crossref","first-page":"11909","DOI":"10.1103\/PhysRevE.74.011909","article-title":"Stochastic Dynamics of Invasion and Fixation","volume":"74","author":"A Traulsen","year":"2006","journal-title":"Phys Rev E"},{"issue":"1","key":"pone.0244592.ref028","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-43102-z","article-title":"Computation and simulation of evolutionary Game Dynamics in Finite populations","volume":"9","author":"L Hindersin","year":"2019","journal-title":"Scientific reports"},{"key":"pone.0244592.ref029","doi-asserted-by":"crossref","first-page":"10797","DOI":"10.1073\/pnas.0502589102","article-title":"Evolutionary cycles of cooperation and defection","volume":"102","author":"LA Imhof","year":"2005","journal-title":"Proc Natl Acad Sci USA"},{"key":"pone.0244592.ref030","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1038\/nature02414","article-title":"Emergence of cooperation and evolutionary stability in finite populations","volume":"428","author":"MA Nowak","year":"2004","journal-title":"Nature"},{"key":"pone.0244592.ref031","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1038\/415137a","article-title":"Altruistic punishment in humans","volume":"415","author":"E Fehr","year":"2002","journal-title":"Nature"},{"issue":"19","key":"pone.0244592.ref032","doi-asserted-by":"crossref","first-page":"10757","DOI":"10.1073\/pnas.161155698","article-title":"Reward and punishment","volume":"98","author":"K Sigmund","year":"2001","journal-title":"P Natl Acad Sci USA"},{"issue":"5978","key":"pone.0244592.ref033","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1126\/science.1183665","article-title":"Coordinated punishment of defectors sustains cooperation and can proliferate when rare","volume":"328","author":"R Boyd","year":"2010","journal-title":"Science"},{"key":"pone.0244592.ref034","doi-asserted-by":"crossref","first-page":"7308","DOI":"10.1038\/nature09203","article-title":"Social learning promotes institutions for governing the commons","volume":"466","author":"K Sigmund","year":"2010","journal-title":"Nature"},{"key":"pone.0244592.ref035","doi-asserted-by":"crossref","DOI":"10.1038\/srep00458","article-title":"Emergence of responsible sanctions without second order free riders, antisocial punishment or spite","volume":"2","author":"C Hilbe","year":"2012","journal-title":"Scientific reports"},{"issue":"4","key":"pone.0244592.ref036","first-page":"041021","article-title":"Correlation of positive and negative reciprocity fails to confer an evolutionary advantage: Phase transitions to elementary strategies","volume":"3","author":"A Szolnoki","year":"2013","journal-title":"Phys Rev X"},{"issue":"1","key":"pone.0244592.ref037","first-page":"1","article-title":"Reward and punishment in climate change dilemmas","volume":"9","author":"AR G\u00f3is","year":"2019","journal-title":"Sci Rep"},{"key":"pone.0244592.ref038","doi-asserted-by":"crossref","unstructured":"Han TA, Lynch S, Tran-Thanh L, Santos FC. Fostering Cooperation in Structured Populations Through Local and Global Interference Strategies. In: IJCAI-ECAI\u20192018; 2018. p. 289\u2013295.","DOI":"10.24963\/ijcai.2018\/40"},{"issue":"102","key":"pone.0244592.ref039","doi-asserted-by":"crossref","first-page":"20140935","DOI":"10.1098\/rsif.2014.0935","article-title":"First carrot, then stick: how the adaptive hybridization of incentives promotes cooperation","volume":"12","author":"X Chen","year":"2015","journal-title":"Journal of The Royal Society Interface"},{"issue":"156","key":"pone.0244592.ref040","doi-asserted-by":"crossref","first-page":"20190127","DOI":"10.1098\/rsif.2019.0127","article-title":"Evolution of coordinated punishment to enforce cooperation from an unbiased strategy space","volume":"16","author":"J Garc\u00eda","year":"2019","journal-title":"Journal of the Royal Society Interface"},{"key":"pone.0244592.ref041","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.physrep.2017.05.004","article-title":"Statistical physics of human cooperation","volume":"687","author":"M Perc","year":"2017","journal-title":"Phys Rep"},{"key":"pone.0244592.ref042","doi-asserted-by":"crossref","unstructured":"Han TA. Emergence of Social Punishment and Cooperation through Prior Commitments. In: AAAI\u20192016; 2016. p. 2494\u20132500.","DOI":"10.1609\/aaai.v30i1.10120"},{"key":"pone.0244592.ref043","doi-asserted-by":"crossref","unstructured":"Cimpeanu T, Han TA. Making an Example: Signalling Threat in the Evolution of Cooperation. In: 2020 IEEE Congress on Evolutionary Computation (CEC). IEEE; 2020. p. 1\u20138.","DOI":"10.1109\/CEC48606.2020.9185749"},{"key":"pone.0244592.ref044","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.physrep.2016.10.006","article-title":"Statistical physics of vaccination","volume":"664","author":"Z Wang","year":"2016","journal-title":"Physics Reports"},{"issue":"10","key":"pone.0244592.ref045","doi-asserted-by":"crossref","first-page":"e45653","DOI":"10.1371\/journal.pone.0045653","article-title":"The interplay of public intervention and private choices in determining the outcome of vaccination programmes","volume":"7","author":"A d\u2019Onofrio","year":"2012","journal-title":"PLoS One"},{"issue":"9","key":"pone.0244592.ref046","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1038\/nclimate1927","article-title":"A bottom-up institutional approach to cooperative governance of risky commons","volume":"3","author":"VV Vasconcelos","year":"2013","journal-title":"Nature Climate Change"},{"issue":"2","key":"pone.0244592.ref047","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1111\/0034-6527.00287","article-title":"Arms races and negotiations","volume":"71","author":"S Baliga","year":"2004","journal-title":"The Review of Economic Studies"},{"issue":"1","key":"pone.0244592.ref048","doi-asserted-by":"crossref","first-page":"018001","DOI":"10.1088\/0031-8949\/90\/1\/018001","article-title":"Responses to catastrophic AGI risk: a survey","volume":"90","author":"K Sotala","year":"2014","journal-title":"Physica Scripta"},{"issue":"6","key":"pone.0244592.ref049","doi-asserted-by":"crossref","first-page":"858","DOI":"10.1111\/1468-2230.12095","article-title":"Public rewards and innovation policy: lessons from the eighteenth and early nineteenth centuries","volume":"77","author":"R Burrell","year":"2014","journal-title":"The Modern Law Review"},{"key":"pone.0244592.ref050","unstructured":"Brundage M, Avin S, Wang J, Belfield H, Krueger G, Hadfield G, et al. Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv preprint arXiv:200407213. 2020."},{"key":"pone.0244592.ref051","doi-asserted-by":"crossref","unstructured":"Han TA, Pereira LM, Lenaerts T. Modelling and Influencing the AI Bidding War: A Research Agenda. In: Proceedings of the AAAI\/ACM conference AI, Ethics and Society; 2019. p. 5\u201311.","DOI":"10.1145\/3306618.3314265"},{"key":"pone.0244592.ref052","volume-title":"The social control of technology","author":"D Collingridge","year":"1980"},{"issue":"7805","key":"pone.0244592.ref053","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1038\/d41586-020-01221-y","article-title":"The race for coronavirus vaccines: a graphical guide","volume":"580","author":"E Callaway","year":"2020","journal-title":"Nature"},{"key":"pone.0244592.ref054","unstructured":"World Health Organization. Medical device regulations: global overview and guiding principles. World Health Organization; 2003."},{"key":"pone.0244592.ref055","first-page":"93","article-title":"Regulation of Innovation Under Follow-On Biologics Legislation: FDA Exclusivity as an Efficient Incentive Mechanisms","volume":"11","author":"MR Morgan","year":"2010","journal-title":"Colum Sci & Tech L Rev"},{"key":"pone.0244592.ref056","first-page":"353","article-title":"Race-ing patents\/patenting race: an emerging political geography of intellectual property in biotechnology","volume":"92","author":"J Kahn","year":"2006","journal-title":"Iowa L Rev"},{"key":"pone.0244592.ref057","doi-asserted-by":"crossref","unstructured":"Pereira LM, Santos FC. Counterfactual thinking in cooperation dynamics. In: International conference on Model-Based Reasoning. Springer; 2018. p. 69\u201382.","DOI":"10.1007\/978-3-030-32722-4_5"},{"issue":"3","key":"pone.0244592.ref058","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1016\/j.jtbi.2007.03.027","article-title":"Tit-for-tat or win-stay, lose-shift?","volume":"247","author":"LA Imhof","year":"2007","journal-title":"Journal of Theoretical Biology"},{"key":"pone.0244592.ref059","unstructured":"Han TA, Pereira LM, Santos FC, Lenaerts T. Why Is It So Hard to Say Sorry: The Evolution of Apology with Commitments in the Iterated Prisoner\u2019s Dilemma. In: IJCAI\u20192013. AAAI Press; 2013. p. 177\u2013183."},{"issue":"10639","key":"pone.0244592.ref060","article-title":"Apology and forgiveness evolve to resolve failures in cooperative agreements","volume":"5","author":"LA Martinez-Vaquero","year":"2015","journal-title":"Scientific reports"},{"key":"pone.0244592.ref061","volume-title":"Beyond revenge: The evolution of the forgiveness instinct","author":"M McCullough","year":"2008"},{"key":"pone.0244592.ref062","doi-asserted-by":"crossref","first-page":"9","DOI":"10.3389\/frobt.2018.00009","article-title":"When it\u2019s good to feel bad: An evolutionary model of guilt and apology","volume":"5","author":"S Rosenstock","year":"2018","journal-title":"Frontiers in Robotics and AI"},{"issue":"1","key":"pone.0244592.ref063","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-018-04968-1","article-title":"Immediate action is the best strategy when facing uncertain climate change","volume":"9","author":"M Abou Chakra","year":"2018","journal-title":"Nature communications"},{"issue":"7695","key":"pone.0244592.ref064","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1038\/nature25763","article-title":"Social norm complexity and past reputations in the evolution of cooperation","volume":"555","author":"FP Santos","year":"2018","journal-title":"Nature"},{"key":"pone.0244592.ref065","doi-asserted-by":"crossref","unstructured":"Santos FP, Pacheco JM, Santos FC. Indirect Reciprocity and Costly Assessment in Multiagent Systems. In: Thirty-Second AAAI Conference on Artificial Intelligence; 2018. p. 4727\u20134734.","DOI":"10.1609\/aaai.v32i1.12148"},{"key":"pone.0244592.ref066","unstructured":"Pereira LM, Lenaerts T, Martinez-Vaquero LA, Han TA. Social manifestation of guilt leads to stable cooperation in multi-agent systems. In: AAMAS; 2017. p. 1422\u20131430."},{"issue":"1","key":"pone.0244592.ref067","first-page":"1","article-title":"Cost-effective external interference for promoting the evolution of cooperation","volume":"8","author":"TA Han","year":"2018","journal-title":"Scientific reports"},{"issue":"4","key":"pone.0244592.ref068","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1177\/1059712316653451","article-title":"A synergy of costly punishment and commitment in cooperation dilemmas","volume":"24","author":"TA Han","year":"2016","journal-title":"Adaptive Behavior"},{"key":"pone.0244592.ref069","doi-asserted-by":"crossref","first-page":"104914","DOI":"10.1016\/j.cnsns.2019.104914","article-title":"Exploring optimal institutional incentives for public cooperation","volume":"79","author":"S Wang","year":"2019","journal-title":"Communications in Nonlinear Science and Numerical Simulation"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0244592","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,13]],"date-time":"2022-12-13T12:53:16Z","timestamp":1670935996000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0244592"}},"subtitle":[],"editor":[{"given":"Alberto","family":"Antonioni","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2021,1,26]]},"references-count":69,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,1,26]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0244592","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,26]]}}}