{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T15:35:10Z","timestamp":1780500910013,"version":"3.54.1"},"reference-count":50,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T00:00:00Z","timestamp":1763596800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>Humans now routinely work alongside AI in environments where the ethical consequences of decisions are profound, yet there remains limited understanding of how long-term collaboration with a robotic teammate shapes individuals\u2019 moral judgment. Prior studies have demonstrated that people can be influenced by a robot\u2019s moral recommendations, but such investigations have largely focused on single dilemmas or brief encounters conducted in laboratory settings. To address this gap, we conducted a three-month teaming program with 62\u202fU.S. military cadets who interacted extensively with a Socially Intelligent and Ethical Mission Assistant (SIEMA) embodied either as a humanoid robot or as a human advisor in a field setting. After this sustained collaboration, cadets completed a graded moral dilemma that required balancing the lives of soldiers against those of civilians, during which they received a written recommendation from their SIEMA promoting a utilitarian option. Each participant recorded an initial judgment, then a second judgment after receiving SIEMA\u2019s advice, and finally a third judgment following an opposing recommendation that emphasized civilian protection. Approximately half of the cadets shifted toward the utilitarian option after advice, regardless of whether the source was robotic or human. When subsequently presented with the recommendation to prioritize civilian protection, most of these cadets shifted again, often returning to their original stance. Qualitative analyses of open-ended explanations revealed that cadets justified their choices by invoking outcome-based reasoning, duties of protection, trust in their teammate, and personal values. Our findings demonstrate that robotic advisors can influence nuanced moral decisions and that such influence contributes to shaping future judgments. Accordingly, moral-AI design should present trade-offs transparently, surface competing values concurrently, and rely on human reflection rather than assuming isolated AI prompts will durably reset moral priorities.<\/jats:p>","DOI":"10.3389\/frai.2025.1694772","type":"journal-article","created":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T06:15:19Z","timestamp":1763619319000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Robots in the moral loop: a field study of AI advisors in ethical military decision-making"],"prefix":"10.3389","volume":"8","author":[{"given":"Chad C.","family":"Tossell","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Kuennen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Momen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory","family":"Funke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Tolston","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ewart J.","family":"De Visser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,11,20]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"8458","DOI":"10.1038\/s41598-024-58087-7","article-title":"Attributions toward artificial agents in a modified moral turing test","volume":"14","author":"Aharoni","year":"2024","journal-title":"Sci. Rep."},{"key":"ref2","doi-asserted-by":"crossref","DOI":"10.1201\/9781420085952","article-title":"Governing lethal behavior in autonomous robots","author":"Arkin","year":"2009"},{"key":"ref3","doi-asserted-by":"crossref","DOI":"10.1515\/9781400837991","volume-title":"Blind spots: Why we fail to do what's right and what to do about it","author":"Bazerman","year":"2011"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1146\/annurev-psych-030123-113559","article-title":"The moral psychology of artificial intelligence","volume":"75","author":"Bonnefon","year":"2024","journal-title":"Annu. Rev. Psychol."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1111\/j.1520-8583.2009.00160.x","article-title":"Moderate deontology and moral gaps","volume":"23","author":"Brennan","year":"2009","journal-title":"Philos. Perspect."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1007\/s12369-014-0235-1","article-title":"How robots can affect human behavior: investigating the effects of robotic displays of protest and distress","volume":"6","author":"Briggs","year":"2014","journal-title":"Int. J. Soc. Robot."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1080\/17470919.2011.614000","article-title":"Finding faults: how moral dilemmas illuminate cognitive structure","volume":"7","author":"Cushman","year":"2012","journal-title":"Soc. Neurosci."},{"key":"ref10","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9540.001.0001","volume-title":"Unifying the mind: cognitive representations as graphical models","author":"Danks","year":"2014"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"4084","DOI":"10.1038\/s41598-025-86510-0","article-title":"AI language model rivals expert ethicist in perceived moral expertise","volume":"15","author":"Dillon","year":"2025","journal-title":"Sci. Rep."},{"key":"ref13","first-page":"321","article-title":"All robots are not created equal: the design and perception of humanoid robot heads","author":"DiSalvo","year":"2002"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"1107306","DOI":"10.3389\/fpsyg.2023.1107306","article-title":"How automation level influences moral decisions of humans collaborating with industrial robots in different scenarios","volume":"14","author":"Eich","year":"2023","journal-title":"Front. Psychol."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"106028","DOI":"10.1016\/j.cognition.2024.106028","article-title":"People expect artificial moral advisors to be more utilitarian and distrust utilitarian moral advisors","volume":"256","author":"Everett","year":"2025","journal-title":"Cognition"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"724","DOI":"10.1111\/j.2044-8309.2011.02082.x","article-title":"Social categorization of social robots: anthropomorphism as a function of robot group membership","volume":"51","author":"Eyssel","year":"2012","journal-title":"Br. J. Soc. Psychol."},{"key":"ref002","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1037\/pspa0000086","article-title":"Consequences, norms, and generalized inaction in moral dilemmas: The CNI model of moral decision-making","volume":"113","author":"Gawronski","year":"2017","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref17","first-page":"35","article-title":"The secret joke of Kant\u2019s soul","volume-title":"Moral psychology","author":"Greene","year":"2008"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1016\/j.neuron.2004.09.027","article-title":"The neural bases of cognitive conflict and control in moral judgment","volume":"44","author":"Greene","year":"2004","journal-title":"Neuron"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"2105","DOI":"10.1126\/science.1062872","article-title":"An fMRI investigation of emotional engagement in moral judgment","volume":"293","author":"Greene","year":"2001","journal-title":"Science"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"107645","DOI":"10.1016\/j.chb.2022.107645","article-title":"Robots as a source of moral influence: the impact of robot authority on obedience","volume":"141","author":"Grzyb","year":"2023","journal-title":"Comput. Human Behav."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"e2214005119","DOI":"10.1073\/pnas.2214005119","article-title":"A moral trade-off system produces intuitive judgments that are rational and coherent","volume":"119","author":"Guzm\u00e1n","year":"2022","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref23","article-title":"Authority and persuasion in robot advisors: perceptions of competence and trust","author":"Hanson","year":"","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref24","first-page":"125","article-title":"The power of advice: differential blame for human and robot advisors and deciders in a moral advising context","author":"Hanson","year":""},{"key":"ref25","doi-asserted-by":"publisher","first-page":"2037","DOI":"10.1007\/s12369-021-00754-0","article-title":"How do people perceive and trust a lifelike robot compared to a human teammate?","volume":"13","author":"Haring","year":"2021","journal-title":"Int. J. Soc. Robot."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"19751","DOI":"10.1038\/s41598-024-69771-z","article-title":"Over-trust in AI recommendations about whether or not to kill","volume":"14","author":"Holbrook","year":"2024","journal-title":"Sci. Rep."},{"key":"ref27","volume-title":"The design of implicit interactions","author":"Ju","year":"2014"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1080\/17470919.2015.1023400","article-title":"Sidetracked by trolleys: why sacrificial moral dilemmas tell us little (or nothing) about utilitarian judgment","volume":"10","author":"Kahane","year":"2015","journal-title":"Soc. Neurosci."},{"key":"ref29","doi-asserted-by":"crossref","DOI":"10.1145\/2157689.2157696","article-title":"Do people hold a humanoid robot morally accountable for the harm it causes?","author":"Kahn","year":"2012"},{"key":"ref30","doi-asserted-by":"publisher","first-page":"4569","DOI":"10.1038\/s41598-023-31341-0","article-title":"Chatgpt\u2019s inconsistent moral advice influences users\u2019 judgment","volume":"13","author":"Kr\u00fcgel","year":"2023","journal-title":"Sci. Rep."},{"key":"ref31","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1177\/09637214231205866","article-title":"The moral psychology of artificial intelligence","volume":"32","author":"Ladak","year":"2023","journal-title":"Curr. Dir. Psychol. Sci."},{"key":"ref32","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1145\/3719211","article-title":"Examining the impact of robot norm violations on participants' trust, discomfort, behaviour and physiological responses\u2014a mixed method approach","volume":"14","author":"Lawrence","year":"2025","journal-title":"ACM Trans. Hum. Robot Interact."},{"key":"ref34","first-page":"125","article-title":"Which robot am I thinking about? The impact of action and appearance on people\u2019s evaluations of a moral robot","author":"Malle","year":"2016"},{"key":"ref35","author":"Mill","year":"1998"},{"key":"ref001","doi-asserted-by":"crossref","DOI":"10.24251\/HICSS.2023.063","article-title":"Trusting the moral judgments of a robot: Perceived moral competence and humanlikeness of a GPT-3 enabled AI","author":"Momen","year":"2023"},{"key":"ref36","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.concog.2014.04.011","article-title":"Bringing free will down to earth: people's psychological concept of free will and its role in moral judgment","volume":"27","author":"Monroe","year":"2014","journal-title":"Conscious. Cogn."},{"key":"ref37","doi-asserted-by":"crossref","first-page":"106028","DOI":"10.1016\/j.cognition.2024.106028","article-title":"People expect artificial moral advisors to be more utilitarian and distrust utilitarian moral advisors","volume":"256","author":"Myers","year":"2025","journal-title":"Cognition"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1080\/07370024.2022.2080552","article-title":"Can you count on a calculator? The role of agency and affect in judgments of robots as moral agents","volume":"38","author":"Nijssen","year":"2023","journal-title":"Hum. Comp. Inter."},{"key":"ref40","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s43681-022-00214-z","article-title":"Ethics in human\u2013AI teaming: principles and perspectives","volume":"3","author":"Pfanzer","year":"2022","journal-title":"AI Ethics"},{"key":"ref41","first-page":"158","article-title":"Eliciting information from people with a gendered humanoid robot","author":"Powers","year":"2005"},{"key":"ref42","article-title":"Reasons, indeterminacy, and compromise","volume-title":"Articulating the moral community: Toward a constructive ethical pragmatism","author":"Richardson","year":"2018"},{"key":"ref43","first-page":"101","article-title":"Overtrust of robots in emergency evacuation scenarios","author":"Robinette","year":"2016"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"eabd5186","DOI":"10.1126\/scirobotics.abd5186","article-title":"Persuasive robots should avoid authority: the effects of formal and real authority on persuasion in human-robot interaction","volume":"6","author":"Saunderson","year":"2021","journal-title":"Sci. Robot."},{"key":"ref46","first-page":"219","article-title":"No fair!! An interaction with a cheating robot","author":"Short","year":"2016"},{"key":"ref47","article-title":"I need your advice... Human perceptions of robot moral advising behaviors","author":"Starr","year":"2021"},{"key":"ref48","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1016\/j.cognition.2011.01.018","article-title":"Time and moral judgment","volume":"119","author":"Suter","year":"2011","journal-title":"Cognition"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"853","DOI":"10.1037\/0022-3514.78.5.853","article-title":"The psychology of taboo trade-offs","volume":"78","author":"Tetlock","year":"2000","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref50","doi-asserted-by":"publisher","first-page":"204","DOI":"10.5840\/monist197659224","article-title":"Killing, letting die, and the trolley problem","volume":"59","author":"Thomson","year":"1976","journal-title":"Monist"},{"key":"ref51","doi-asserted-by":"publisher","first-page":"1395","DOI":"10.2307\/796133","article-title":"The trolley problem","volume":"94","author":"Thomson","year":"1985","journal-title":"Yale Law J."},{"key":"ref52","doi-asserted-by":"publisher","first-page":"599","DOI":"10.5840\/soctheorpract201137435","article-title":"The relevance of trust for moral justification","volume":"37","author":"Tobin","year":"2011","journal-title":"Soc. Theory Pract."},{"key":"ref53","first-page":"1","article-title":"Moral advice from AI: trust, responsibility, and moral decision-making","volume-title":"Proceedings of CHI 2022","author":"Tolmeijer","year":"2022"},{"key":"ref54","first-page":"127","article-title":"War\u2019s means and the importance of fighting well","volume-title":"Just and unjust wars: a moral argument with historical illustrations","author":"Walzer","year":"2015"},{"key":"ref55","first-page":"1480","article-title":"Perceived agency of a social norm-violating robot","volume":"311","author":"Yasuda","year":"2020","journal-title":"Proceedings of the 42nd Annual Meeting of the Cognitive Science Society (CogSci 2020)"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1694772\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T06:15:32Z","timestamp":1763619332000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1694772\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,20]]},"references-count":50,"alternative-id":["10.3389\/frai.2025.1694772"],"URL":"https:\/\/doi.org\/10.3389\/frai.2025.1694772","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,20]]},"article-number":"1694772"}}