{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T06:08:09Z","timestamp":1788761289650,"version":"build-2803163510"},"reference-count":52,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T00:00:00Z","timestamp":1788739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>The AI industry is advancing toward an Artificial General Intelligence (AGI) that will possess cognitive capabilities that are equal to or stronger than those of humans. While a purely computational AGI may be cognitively powerful, it will probably struggle to understand the human social environment. This gap needs both a theoretical and an applicative solution. This article provides the theoretical solution, specifying what such an AGI must contain and how it must be formed. We propose a theoretical construct of an AGI that will not only be computational but will also be capable to develop internal social understanding, internalize norms in a manner similar to humans, and act upon this social understanding. We call such AGI a \u201cSocial AGI.\u201d We argue that Social AGI cannot be developed only by using technical alignment or external governance. Instead, it must undergo a staged development in which visible social behavior gradually becomes social cognition and norm internalization. This process cannot be separated from the parallel support that human society must provide the effort of developing Social AGI. While AI systems may become more familiar and embedded in institutions and society itself, the public perceptions of risk will probably decline. This normalization will provide the basis for granting the Social AGI progressively higher levels of autonomy. In order to capture this process of mutual change, we develop a socio-technical concept of human-AGI co-formation. We differentiate co-formation from co-existence, which describes humans and AI operating side by side, and from co-evolution, which describes reciprocal and cyclic influence. Co-formation refers to a process in which at the beginning Social AGI is shaped exclusively by human society. However, as the process progresses, they shape together the basic conditions under which the AGI is developed. On the AGI side, this will require the capabilities to internalize norms, and to be able to function according to its roles. On the societal side, this will require to create several mechanisms of role assignment, training and socializing the AGI, and monitoring its value alignment. The most important characteristics society should develop is a public acceptance and justification for increasing the AGI\u2019s autonomy. Society may learn not only how to supervise the AGI, but also it may learn to trust it.<\/jats:p>","DOI":"10.3389\/frai.2026.1904357","type":"journal-article","created":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T05:33:03Z","timestamp":1788759183000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["AGI imagined: social AGI and the co-formation of human-AGI society"],"prefix":"10.3389","volume":"9","author":[{"given":"Jacob","family":"Zilberman","sequence":"first","affiliation":[{"name":"Science, Technology and Society Program, Faculty of interdisciplinary studies, Bar-Ilan University","place":["Ramat Gan, Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,7]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1080\/10508420903035380","article-title":"A Meta-analysis of ethics instruction effectiveness in the sciences","volume":"19","author":"Antes","year":"2009","journal-title":"Ethics Behav."},{"key":"ref2","article-title":"Constitutional AI: harmlessness from AI feedback","author":"Bai","year":"2022","journal-title":"arXiv"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"1250725","DOI":"10.3389\/frai.2023.1250725","article-title":"Defining human-AI teaming the human-centered way: a scoping review and network analysis","volume":"6","author":"Berretta","year":"2023","journal-title":"Front. Artificial Intelligence"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1007\/s13347-017-0263-5","article-title":"Algorithmic accountability and public reason","volume":"31","author":"Binns","year":"2018","journal-title":"Philos. Technol."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1177\/17816858211059249","article-title":"The ethical dimension of human\u2013artificial intelligence collaboration","volume":"20","author":"Boni","year":"2021","journal-title":"Eur. View"},{"key":"ref7","article-title":"Open problems and fundamental limitations of reinforcement learning from human feedback","author":"Casper","year":"2023"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1038\/s42256-019-0020-9","article-title":"Hopes and fears for intelligent machines in fiction and reality","volume":"1","author":"Cave","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref9","article-title":"On the measure of intelligence","author":"Chollet","year":"2019","journal-title":"arXiv preprint arXiv:1911.01547"},{"key":"ref10","author":"Christiano","year":"2017"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1038\/d41586-021-01170-0","article-title":"Cooperative AI: machines must learn to find common ground","volume":"593","author":"Dafoe","year":"2021","journal-title":"Nature"},{"key":"ref12","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1109\/EURBOT.1999.827639","volume-title":"1999 Third European Workshop on Advanced Mobile Robots (Eurobot'99). Proceedings (Cat. No. 99EX355)","author":"Dautenhahn","year":"1999"},{"key":"ref13","first-page":"1","volume-title":"Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization","author":"Delgado","year":"2023"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1007\/s11023-020-09539-2","article-title":"Artificial intelligence, values, and alignment","volume":"30","author":"Gabriel","year":"2020","journal-title":"Minds Mach."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"6","DOI":"10.3390\/soc15010006","article-title":"AI tools in society: impacts on cognitive offloading and the future of critical thinking","volume":"15","author":"Gerlich","year":"2025","journal-title":"Societies"},{"key":"ref16","volume-title":"The Constitution of Society: Outline of the Theory of Structuration","author":"Giddens","year":"1984"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1111\/tops.12679","article-title":"Fostering collective intelligence in human-AI collaboration: laying the groundwork for COHUMAIN","volume":"17","author":"Gupta","year":"2025","journal-title":"Top. Cogn. Sci."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"4885","DOI":"10.1038\/s41467-026-73289-5","article-title":"SuperARC: a test for artificial superintelligence based on compressed modelling, recursive prediction and problem complexity","volume":"17","author":"Hern\u00e1ndez-Espinosa","year":"2026","journal-title":"Nat. Commun."},{"key":"ref19","volume-title":"Ethics Guidelines for Trustworthy AI","year":"2019"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1016\/j.bushor.2018.03.007","article-title":"Artificial intelligence and the future of work: human-AI symbiosis in organizational decision making","volume":"61","author":"Jarrahi","year":"2018","journal-title":"Bus. Horiz."},{"key":"ref21","article-title":"A survey of reinforcement learning from human feedback","volume-title":"arXiv preprint","author":"Kaufmann","year":"2023"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1177\/002200275800200106","article-title":"Compliance, identification, and internalization: three processes of attitude change","volume":"2","author":"Kelman","year":"1958","journal-title":"J. Confl. Resolut."},{"key":"ref23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1057\/s41599-025-04532-5","article-title":"Why human-AI relationships need socioaffective alignment","volume":"12","author":"Kirk","year":"2025","journal-title":"Humanit. Soc. Sci. Commun."},{"key":"ref24","doi-asserted-by":"publisher","first-page":"1205465","DOI":"10.3389\/frai.2023.1205465","article-title":"Legal framework for the coexistence of humans and conscious AI","volume":"6","author":"Kiskis","year":"2023","journal-title":"Frontiers in artificial intelligence"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"1587","DOI":"10.1111\/j.1467-6494.2006.00421.x","article-title":"Children\u2019s conscience and self-regulation","volume":"74","author":"Kochanska","year":"2006","journal-title":"J. Pers."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"1229","DOI":"10.1111\/j.1467-8624.2004.00735.x","article-title":"Maternal parenting and children\u2019s conscience: early security as moderator","volume":"75","author":"Kochanska","year":"2004","journal-title":"Child Dev."},{"key":"ref27","doi-asserted-by":"publisher","first-page":"1320","DOI":"10.1037\/a0020381","article-title":"Children\u2019s conscience during toddler and preschool years, moral self, and a competent, adaptive developmental trajectory","volume":"46","author":"Kochanska","year":"2010","journal-title":"Dev. Psychol."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1518\/hfes.46.1.50.30392","article-title":"Trust in automation: designing for appropriate reliance","volume":"46","author":"Lee","year":"2004","journal-title":"Hum. Factors"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1177\/10597123030112002","article-title":"Social situatedness of natural and artificial intelligence: Vygotsky and beyond","volume":"11","author":"Lindblom","year":"2003","journal-title":"Adapt. Behav."},{"key":"ref30","volume-title":"Mind, Self, and Society: From the Standpoint of a Social Behaviourist","author":"Mead","year":"1934"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1016\/j.im.2015.06.006","article-title":"Exploring information privacy regulation, risks, trust, and behavior","volume":"52","author":"Miltgen","year":"2015","journal-title":"Inf. Manag."},{"key":"ref32","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1007\/s44206-025-00203-9","article-title":"Where is agency moving to? Exploring the interplay between AI technologies in education and human agency","volume":"4","author":"Mouta","year":"2025","journal-title":"Digit. Soc."},{"key":"ref33","first-page":"2","article-title":"Algorithms for inverse reinforcement learning","volume":"1","author":"Ng","year":"2000","journal-title":"In Icml"},{"key":"ref34","volume-title":"The Borrowed Mind: Reclaiming Human Thought in the Age of AI","author":"Nosta","year":"2026"},{"key":"ref35","first-page":"27730","volume-title":"Advances in Neural Information Processing Systems","author":"Ouyang","year":"2022"},{"key":"ref36","doi-asserted-by":"crossref","first-page":"104244","DOI":"10.1016\/j.artint.2024.104244","article-title":"Human-AI coevolution","volume":"339","author":"Pedreschi","year":"2025","journal-title":"Artif. Intell."},{"key":"ref37","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s10676-017-9430-8","article-title":"Society-in-the-loop: programming the algorithmic social contract","volume":"20","author":"Rahwan","year":"2018","journal-title":"Ethics Inf. Technol."},{"key":"ref38","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1038\/s41586-019-1138-y","article-title":"Machine behaviour","volume":"568","author":"Rahwan","year":"2019","journal-title":"Nature"},{"key":"ref39","first-page":"148","article-title":"Embodied artificial general intelligence architectures: an integrative approach to cognitive, sensorimotor, and environmental interaction systems","volume":"1","author":"Reddy","year":"2025","journal-title":"Cognitive Singularity: Journal of Artificial General Intelligence"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1007\/s13280-010-0134-0","article-title":"Coping with complexity, uncertainty and ambiguity in risk governance: a synthesis","volume":"40","author":"Renn","year":"2011","journal-title":"Ambio"},{"key":"ref41","article-title":"Value internalization: learning and generalizing from social reward","volume-title":"Reinforcement Learning Journal (RLJ)","author":"Rong","year":"2024"},{"key":"ref9001","doi-asserted-by":"publisher","first-page":"755","DOI":"10.1097\/00001888-198912000-00014","article-title":"The effect of teaching medical ethics on medical students\u2019 moral reasoning","volume":"64","author":"Self","year":"1989","journal-title":"Academic Medicine"},{"key":"ref43","article-title":"Thinking-fast, slow, and artificial: how AI is reshaping human reasoning and the rise of cognitive surrender","author":"Shaw","year":"2026"},{"key":"ref44","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1080\/10447318.2020.1741118","article-title":"Human-centered artificial intelligence: reliable, safe & trustworthy","volume":"36","author":"Shneiderman","year":"2020","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"ref45","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1126\/science.3563507","article-title":"Perception of risk","volume":"236","author":"Slovic","year":"1987","journal-title":"Science"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1007\/s12525-020-00441-4","article-title":"Trustworthy artificial intelligence","volume":"31","author":"Thiebes","year":"2021","journal-title":"Electron. Mark."},{"key":"ref47","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1207\/s15516709cog2403_1","article-title":"Primate cognition: introduction to the issue","volume":"24","author":"Tomasello","year":"2000","journal-title":"Cogn. Sci."},{"key":"ref48","doi-asserted-by":"publisher","first-page":"1864","DOI":"10.1038\/s41562-024-02001-8","article-title":"A new sociology of humans and machines","volume":"8","author":"Tsvetkova","year":"2024","journal-title":"Nat. Hum. Behav."},{"key":"ref49","first-page":"1","article-title":"Advancing human\u2013AI teams: evolving from instrumental tools to trusted partners","author":"Venkatesha Murthy","year":"2026","journal-title":"AI Soc."},{"key":"ref50","volume-title":"Mind in Society: The Development of Higher Psychological Processes","author":"Vygotsky","year":"1978"},{"key":"ref51","first-page":"308","volume-title":"Global Catastrophic Risks","author":"Yudkowsky","year":"2008"},{"key":"ref52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s00146-025-02717-9","article-title":"AGI imagined: how is AGI configured by the theories of mind","author":"Zilberman","year":"2025","journal-title":"AI Soc."},{"key":"ref53","doi-asserted-by":"publisher","first-page":"102747","DOI":"10.1016\/j.technovation.2023.102747","article-title":"Worker and workplace artificial intelligence (AI) coexistence: emerging themes and research agenda","volume":"124","author":"Zirar","year":"2023","journal-title":"Technovation"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1904357\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T05:33:06Z","timestamp":1788759186000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1904357\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,7]]},"references-count":52,"alternative-id":["10.3389\/frai.2026.1904357"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1904357","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,7]]},"article-number":"1904357"}}