{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:08:16Z","timestamp":1760148496012,"version":"build-2065373602"},"reference-count":38,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T00:00:00Z","timestamp":1683763200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation CAREER","award":["CMMI-1943998"],"award-info":[{"award-number":["CMMI-1943998"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Games"],"abstract":"<jats:p>Autonomous driving (AV) technology has elicited discussion on social dilemmas where trade-offs between individual preferences, social norms, and collective interests may impact road safety and efficiency. In this study, we aim to identify whether social dilemmas exist in AVs\u2019 sequential decision making, which we call \u201csequential driving dilemmas\u201d (SDDs). Identifying SDDs in traffic scenarios can help policymakers and AV manufacturers better understand under what circumstances SDDs arise and how to design rewards that incentivize AVs to avoid SDDs, ultimately benefiting society as a whole. To achieve this, we leverage a social learning framework, where AVs learn through interactions with random opponents, to analyze their policy learning when facing SDDs. We conduct numerical experiments on two fundamental traffic scenarios: an unsignalized intersection and a highway. We find that SDDs exist for AVs at intersections, but not on highways.<\/jats:p>","DOI":"10.3390\/g14030041","type":"journal-article","created":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T04:29:01Z","timestamp":1683779341000},"page":"41","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Social Learning for Sequential Driving Dilemmas"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1006-0926","authenticated-orcid":false,"given":"Xu","family":"Chen","sequence":"first","affiliation":[{"name":"Department of Civil Engineering and Engineering Mechanics, Columbia University, New York, NY 10027, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2925-7697","authenticated-orcid":false,"given":"Xuan","family":"Di","sequence":"additional","affiliation":[{"name":"Department of Civil Engineering and Engineering Mechanics, Columbia University, New York, NY 10027, USA"},{"name":"Data Science Institute, Columbia University, New York, NY 10027, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1534-1740","authenticated-orcid":false,"given":"Zechu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY 10027, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,11]]},"reference":[{"key":"ref_1","unstructured":"Sadigh, D., Sastry, S., Seshia, S.A., and Dragan, A.D. (2016, January 12\u201316). Planning for Autonomous Cars that Leverage Effects on Human Actions. Proceedings of the Robotics: Science and Systems, Ann Arbor, MI, USA."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Fisac, J.F., Bronstein, E., Stefansson, E., Sadigh, D., Sastry, S.S., and Dragan, A.D. (2019, January 20\u201324). Hierarchical Game-Theoretic Planning for Autonomous Vehicles. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794007"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"103008","DOI":"10.1016\/j.trc.2021.103008","article-title":"A survey on autonomous vehicle control in the era of mixed-autonomy: From physics-based to AI-guided driving policy learning","volume":"125","author":"Di","year":"2021","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"103189","DOI":"10.1016\/j.trc.2021.103189","article-title":"Dynamic driving and routing games for autonomous vehicles on networks: A mean field game approach","volume":"128","author":"Huang","year":"2021","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"103560","DOI":"10.1016\/j.trc.2022.103560","article-title":"Multi-agent reinforcement learning for Markov routing games: A new modeling paradigm for dynamic traffic assignment","volume":"137","author":"Shou","year":"2022","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_6","unstructured":"Pedersen, P.A. (2001). A Game Theoretical Approach to Road Safety, University of Kent. Technical Report, Department of Economics Discussion Paper."},{"key":"ref_7","first-page":"47","article-title":"Moral hazard in traffic games","volume":"37","author":"Pedersen","year":"2003","journal-title":"J. Transp. Econ. Policy (JTEP)"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"121","DOI":"10.3141\/2386-14","article-title":"Evolutionary game theoretic approach to rear-end events on congested freeway","volume":"2386","author":"Chatterjee","year":"2013","journal-title":"Transp. Res. Rec. J. Transp. Res. Board"},{"key":"ref_9","unstructured":"Chatterjee, I. (2016). Understanding Driver Contributions to Rear-End Crashes on Congested Freeways and Their Implications for Future Safety Measures. [PhD Thesis, University of Minnesota]."},{"key":"ref_10","unstructured":"Yoo, J.H., and Langari, R. (2012, January 17\u201319). Stackelberg game based model of highway driving. Proceedings of the ASME 2012 5th Annual Dynamic Systems and Control Conference joint with the JSME 2012 11th Motion and Vibration Conference, Fort Lauderdale, FL, USA."},{"key":"ref_11","unstructured":"Yoo, J.H. (2014). A Game Theory Based Model of Human Driving with Application to Autonomous and Mixed Driving. [Doctoral Dissertation, Texas A & M University]."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/j.trc.2015.07.007","article-title":"Modeling Lane-Changing Behavior in a Connected Environment: A Game Theory Approach","volume":"59","author":"Talebpour","year":"2015","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.trc.2018.01.016","article-title":"A human-like game theory-based controller for automatic lane changing","volume":"88","author":"Yu","year":"2018","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_14","unstructured":"Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017, January 8\u201312). Multi-agent Reinforcement Learning in Sequential Social Dilemmas. Proceedings of the AAMAS \u201917, 16th International Conference on Autonomous Agents and MultiAgent Systems, Sao Paulo, Brazil."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1038\/nature21723","article-title":"Evolutionary dynamics on any population structure","volume":"544","author":"Allen","year":"2017","journal-title":"Nature"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1038\/s41562-020-0881-2","article-title":"Social goods dilemmas in heterogeneous societies","volume":"4","author":"McAvoy","year":"2020","journal-title":"Nat. Hum. Behav."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"25398","DOI":"10.1073\/pnas.1908936116","article-title":"Evolutionary dynamics with game transitions","volume":"116","author":"Su","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1038\/s41586-018-0277-x","article-title":"Evolution of cooperation in stochastic games","volume":"559","author":"Hilbe","year":"2018","journal-title":"Nature"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1573","DOI":"10.1126\/science.aaf2654","article-title":"The social dilemma of autonomous vehicles","volume":"352","author":"Bonnefon","year":"2016","journal-title":"Science"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"24972","DOI":"10.1073\/pnas.1820676116","article-title":"Social behavior for autonomous vehicles","volume":"116","author":"Schwarting","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_21","unstructured":"Eccles, T., Hughes, E., Kram\u00e1r, J., Wheelwright, S., and Leibo, J.Z. (2019). Learning Reciprocity in Complex Sequential Social Dilemmas. arXiv."},{"key":"ref_22","unstructured":"Badjatiya, P., Sarkar, M., Sinha, A., Singh, S., Puri, N., Subramanian, J., and Krishnamurthy, B. (2020). Inducing Cooperative behaviour in Sequential-Social dilemmas through Multi-Agent Reinforcement Learning using Status-Quo Loss. arXiv."},{"key":"ref_23","unstructured":"Gupta, G. (2020). Obedience-Based Multi-Agent Cooperation for Sequential Social Dilemmas. [Master Thesis, University of Waterloo]."},{"key":"ref_24","unstructured":"Sen, S., and Airiau, S. (2007, January 6\u201312). Emergence of Norms through Social Learning. Proceedings of the IJCAI\u201907, 20th International Joint Conference on Artifical Intelligence, Hyderabad, India."},{"key":"ref_25","unstructured":"Lewis, D. (1970). Convention: A Philosophical Study, Wiley."},{"key":"ref_26","unstructured":"Boella, G., and Lesmo, L. (2001). A Game Theoretic Approach to Norms and Agents, Universita di Torino."},{"key":"ref_27","unstructured":"Boella, G., and van der Torre, L. (2003, January 13\u201317). Norm governed multiagent systems: The delegation of control to autonomous agents. Proceedings of the IAT 2003, IEEE\/WIC International Conference on Intelligent Agent Technology, Halifax, NSA, Canada."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1023\/A:1013810410243","article-title":"Learning to Be Thoughtless: Social Norms and Individual Computation","volume":"18","author":"Epstein","year":"2001","journal-title":"Comput. Econ."},{"key":"ref_29","unstructured":"O\u2019Callaghan, D., and Mannion, P. (2021). Tunable Behaviours in Sequential Social Dilemmas Using Multi-Objective Reinforcement Learning, International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/S0004-3702(02)00262-X","article-title":"Emergence of social conventions in complex networks","volume":"141","author":"Delgado","year":"2002","journal-title":"Artif. Intell."},{"key":"ref_31","unstructured":"Villatoro, D., Sabater-Mir, J., and Sen, S. (2011, January 16\u201322). Social Instruments for Robust Convention Emergence. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Barcelona, Spain."},{"key":"ref_32","unstructured":"Yu, C., Zhang, M., Ren, F., and Luo, X. (2013, January 6\u201310). Emergence of Social Norms through Collective Learning in Networked Agent Societies. Proceedings of the AAMAS \u201913, 2013 International Conference on Autonomous Agents and Multi-Agent Systems, St. Paul, MN, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1007\/s10458-012-9193-x","article-title":"Manipulating convention emergence using influencer agents","volume":"26","author":"Franks","year":"2013","journal-title":"Auton. Agents-Multi-Agent Syst."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, X., Li, Z., and Di, X. (2022, January 4\u20139). Social Learning In Markov Games: Empowering Autonomous Driving. Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany.","DOI":"10.1109\/IV51971.2022.9827289"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"7229","DOI":"10.1073\/pnas.092080099","article-title":"Learning Dynamics in Social Dilemmas","volume":"99","author":"Macy","year":"2002","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1142\/S0129183119500189","article-title":"Evolutionary dilemma game for conflict resolution at unsignalized traffic intersection","volume":"30","author":"Bouderba","year":"2019","journal-title":"Int. J. Mod. Phys."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"417","DOI":"10.3390\/futuretransp3020025","article-title":"Legal Framework for Rear-End Crashes in Mixed-Traffic Platooning: A Matrix Game Approach","volume":"3","author":"Chen","year":"2023","journal-title":"Future Transp."}],"container-title":["Games"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-4336\/14\/3\/41\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:32:48Z","timestamp":1760124768000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-4336\/14\/3\/41"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,11]]},"references-count":38,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["g14030041"],"URL":"https:\/\/doi.org\/10.3390\/g14030041","relation":{},"ISSN":["2073-4336"],"issn-type":[{"type":"electronic","value":"2073-4336"}],"subject":[],"published":{"date-parts":[[2023,5,11]]}}}