{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T15:46:27Z","timestamp":1783352787404,"version":"3.54.6"},"reference-count":72,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T00:00:00Z","timestamp":1706659200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Deep generative models (DGM) are increasingly employed in emergent communication systems. However, their application in multimodal data contexts is limited. This study proposes a novel model that combines multimodal DGM with the Metropolis-Hastings (MH) naming game, enabling two agents to focus jointly on a shared subject and develop common vocabularies. The model proves that it can handle multimodal data, even in cases of missing modalities. Integrating the MH naming game with multimodal variational autoencoders (VAE) allows agents to form perceptual categories and exchange signs within multimodal contexts. Moreover, fine-tuning the weight ratio to favor a modality that the model could learn and categorize more readily improved communication. Our evaluation of three multimodal approaches - mixture-of-experts (MoE), product-of-experts (PoE), and mixture-of-product-of-experts (MoPoE)\u2013suggests an impact on the creation of latent spaces, the internal representations of agents. Our results from experiments with the MNIST + SVHN and Multimodal165 datasets indicate that combining the Gaussian mixture model (GMM), PoE multimodal VAE, and MH naming game substantially improved information sharing, knowledge formation, and data reconstruction.<\/jats:p>","DOI":"10.3389\/frobt.2023.1290604","type":"journal-article","created":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T04:40:16Z","timestamp":1706676016000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game"],"prefix":"10.3389","volume":"10","author":[{"given":"Nguyen Le","family":"Hoang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tadahiro","family":"Taniguchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoshinobu","family":"Hagiwara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akira","family":"Taniguchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,1,31]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2018.00636","article-title":"Bottom-up and top-down attention for image captioning and visual question answering","author":"Anderson","year":"2018"},{"key":"B2","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: a survey and taxonomy","volume":"41","author":"Baltru\u0161aitis","year":"2019","journal-title":"IEEE Trans. Pattern Analysis Mach. Intell."},{"key":"B3","doi-asserted-by":"publisher","first-page":"P06014","DOI":"10.1088\/1742-5468\/2006\/06\/P06014","article-title":"Sharp transition towards shared vocabularies in multi-agent systems","volume":"2006","author":"Baronchelli","year":"2006","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"B4","unstructured":"Towards more human-like ai communication: a review of emergent communication research\n            BrandizziN.\n          2023"},{"key":"B5","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-1-4471-0663-0_1","article-title":"Computer simulation: a new scientific approach to the study of language evolution","author":"Cangelosi","year":"2002","journal-title":"Simulating Evol. Lang."},{"key":"B6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2307\/1166214","article-title":"Social cognition, joint attention, and communicative competence from 9 to 15 months of age","volume":"63","author":"Carpenter","year":"1998","journal-title":"Monogr. Soc. Res. Child Dev."},{"key":"B7","first-page":"4427","article-title":"Compositionality and generalization in emergent languages","author":"Chaabouni","year":"2020"},{"key":"B8","doi-asserted-by":"publisher","first-page":"e2016569118","DOI":"10.1073\/pnas.2016569118","article-title":"Communicating artificial neural networks develop efficient color-naming systems","volume":"118","author":"Chaabouni","year":"2021","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"B9","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ. Psychol. Meas."},{"key":"B10","unstructured":"Co-evolution of language and agents in referential games\n            DaganG.\n            HupkesD.\n            BruniE.\n          2021"},{"key":"B11","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1109\/tpami.1979.4766909","article-title":"A cluster separation measure","volume":"2","author":"Davies","year":"1979","journal-title":"IEEE Trans. Pattern Analysis Mach. Intell."},{"key":"B12","unstructured":"Interpretable agent communication from scratch(with a generic visual processor emerging on the side)\n            Dess\u00ecR.\n            KharitonovE.\n            BaroniM.\n          2021"},{"key":"B13","unstructured":"Emergent communication in a multi-modal, multi-step referential game\n            EvtimovaK.\n            DrozdovA.\n            KielaD.\n            ChoK.\n          2018"},{"key":"B14","unstructured":"Emergent communication for understanding human language evolution: what\u2019s missing?\n            GalkeL.\n            RamY.\n            RavivL.\n          2022"},{"key":"B15","unstructured":"Dynamic population-based meta-learning for multi-agent communication with natural language\n            GuptaA.\n            LanctotM.\n            LazaridouA.\n          2021"},{"key":"B16","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1080\/01691864.2022.2029721","article-title":"Multiagent multimodal categorization for symbol emergence: emergent communication via interpersonal cross-modal inference","volume":"36","author":"Hagiwara","year":"2022","journal-title":"Adv. Robot."},{"key":"B17","doi-asserted-by":"publisher","first-page":"134","DOI":"10.3389\/frobt.2019.00134","article-title":"Symbol emergence as an interpersonal multimodal categorization","volume":"6","author":"Hagiwara","year":"2019","journal-title":"Front. Robotics AI"},{"key":"B18","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1093\/biomet\/57.1.97","article-title":"Monte Carlo sampling methods using Markov chains and their applications","volume":"57","author":"Hastings","year":"1970","journal-title":"Biometrika"},{"key":"B19","unstructured":"Gans trained by a two time-scale update rule converge to a local nash equilibrium\n            HeuselM.\n            RamsauerH.\n            UnterthinerT.\n            NesslerB.\n            HochreiterS.\n          2017"},{"key":"B20","volume-title":"beta-vae: learning basic visual concepts with a constrained variational framework","author":"Higgins","year":"2017"},{"key":"B21","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/bf01908075","article-title":"Comparing partitions","volume":"2","author":"Hubert","year":"1985","journal-title":"J. Classif."},{"key":"B22","volume-title":"The origins of grammar: language in the light of evolution ii","author":"Hurford","year":"2014"},{"key":"B23","unstructured":"Recursive metropolis-hastings naming game: symbol emergence in a multi-agent system based on probabilistic generative models\n            InukaiJ.\n            TaniguchiT.\n            TaniguchiA.\n            HagiwaraY.\n          2023"},{"key":"B24","first-page":"215","article-title":"Data augmenting contrastive learning of speech representations in the time domain","author":"Kharitonov","year":"2021"},{"key":"B25","unstructured":"Auto-encoding variational bayes\n            KingmaD.\n            WellingM.\n          2013"},{"key":"B26","doi-asserted-by":"publisher","first-page":"127","DOI":"10.48550\/arXiv.1411.2539","article-title":"Unifying visual-semantic embeddings with multimodal neural language models","volume":"2","author":"Kiros","year":"2014","journal-title":"Trans. Assoc. Comput. Linguistics"},{"key":"B27","doi-asserted-by":"crossref","DOI":"10.1109\/ICRA.2019.8794458","article-title":"Multi-modal generative models for learning epistemic active sensing","author":"Korthals","year":"2019"},{"key":"B28","article-title":"Imagenet classification with deep convolutional neural networks","volume-title":"Advances in neural information processing systems","author":"Krizhevsky","year":"2012"},{"key":"B29","doi-asserted-by":"publisher","first-page":"37","DOI":"10.26565\/2218-2926-2022-24-03","article-title":"Emergent meaning-making in multimodal discourse: a case for sadness in the horse whisperer","volume":"24","author":"Krysanova","year":"2022","journal-title":"Cogn. Commun. discourse"},{"key":"B30","unstructured":"Emergent multi-agent communication in the deep learning era\n            LazaridouA.\n            BaroniM.\n          2020"},{"key":"B31","unstructured":"Multi-agent cooperation and the emergence of (natural) language\n            LazaridouA.\n            PeysakhovichA.\n            BaroniM.\n          2017"},{"key":"B32","unstructured":"Multi-agent communication meets natural language: synergies between functional and structural language learning\n            LazaridouA.\n            PotapenkoA.\n            TielemanO.\n          2020"},{"key":"B33","unstructured":"Mnist handwritten digit database\n            LeCunY.\n            CortesC.\n            BurgesC.\n          2010"},{"key":"B34","unstructured":"Private-shared disentangled multimodal vae for learning of hybrid latent representations\n            LeeM.\n            PavlovicV.\n          2020"},{"key":"B35","volume-title":"Convention: a philosophical study","author":"Lewis","year":"2008"},{"key":"B36","unstructured":"The effect of efficient messaging and input variability on neural-agent iterated language learning\n            LianY.\n            BisazzaA.\n            VerhoefT.\n          2021"},{"key":"B37","unstructured":"Foundations and trends in multimodal machine learning: principles, challenges, and open questions\n            LiangP. P.\n            ZadehA.\n            MorencyL.-P.\n          2023"},{"key":"B38","volume-title":"Rethinking language, mind, and world dialogically","author":"Linell","year":"2009"},{"key":"B39","doi-asserted-by":"publisher","first-page":"30046","DOI":"10.1073\/pnas.1907367117","article-title":"Emergent linguistic structure in artificial neural networks trained by self-supervision","volume":"117","author":"Manning","year":"2020","journal-title":"Proc. Natl. Acad. Sci."},{"key":"B40","doi-asserted-by":"publisher","first-page":"1043","DOI":"10.1109\/TCDS.2017.2745502","article-title":"Ensemble-of-concept models for unsupervised formation of multiple categories","volume":"10","author":"Nakamura","year":"2018","journal-title":"IEEE Trans. Cognitive Dev. Syst."},{"key":"B41","doi-asserted-by":"publisher","first-page":"25","DOI":"10.3389\/fnbot.2018.00025","article-title":"Serket: an architecture for connecting stochastic models to realize a large-scale cognitive model","volume":"12","author":"Nakamura","year":"2018","journal-title":"Front. Neurorobotics"},{"key":"B42","volume-title":"Reading digits in natural images with unsupervised feature learning","author":"Netzer","year":"2011"},{"key":"B43","unstructured":"Multimodal deep learning\n            NgiamJ.\n            KhoslaA.\n            KimM.\n            NamJ.\n            LeeH.\n            NgA. Y.\n          2011"},{"key":"B44","doi-asserted-by":"publisher","first-page":"721","DOI":"10.1016\/j.robot.2014.03.003","article-title":"Multimodal integration learning of robot behavior using deep neural networks","volume":"62","author":"Noda","year":"2014","journal-title":"Robotics Aut. Syst."},{"key":"B45","unstructured":"Emergence of hierarchical reference systems in multi-agent communication\n            OhmerX.\n            DudaM.\n            BruniE.\n          2022"},{"key":"B46","unstructured":"Metropolis-hastings algorithm in joint-attention naming game: experimental semiotics study\n            OkumuraR.\n            TaniguchiT.\n            HagiwaraY.\n            TaniguchiA.\n          2023"},{"key":"B47","volume-title":"Peirce on signs: writings on semiotic","author":"Peirce","year":"1991"},{"key":"B48","unstructured":"On the role of population heterogeneity in emergent communication\n            RitaM.\n            StrubF.\n            GrillJ.-B.\n            PietquinO.\n            DupouxE."},{"key":"B49","unstructured":"Emergent communication: generalization and overfitting in lewis games\n            RitaM.\n            TallecC.\n            MichelP.\n            GrillJ.-B.\n            PietquinO.\n            DupouxE."},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1911.03393","article-title":"Variational mixture-of-experts autoencoders for multi-modal deep generative models","author":"Shi","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B51","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1016\/S1364-6613(03)00129-3","article-title":"Evolving grounded communication for robots","volume":"7","author":"Steels","year":"2003","journal-title":"Trends Cognitive Sci."},{"key":"B52","doi-asserted-by":"crossref","DOI":"10.26530\/OAPEN_559870","volume-title":"The talking heads experiment: origins of words and meanings","author":"Steels","year":"2015"},{"key":"B53","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1075\/ais.3.04ste","article-title":"The grounded naming game","author":"Steels","year":"2012","journal-title":"Exp. Cult. Lang. Evol."},{"key":"B54","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1109\/mis.2001.956077","article-title":"Language games for autonomous robots","volume":"16","author":"Steels","year":"2001","journal-title":"IEEE Intell. Syst."},{"key":"B55","volume-title":"The merging of the senses","author":"Stein","year":"1993"},{"key":"B56","unstructured":"Generalized multimodal elbo\n            SutterT.\n            DaunhawerI.\n            VogtJ.\n          2021"},{"key":"B57","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1080\/01691864.2022.2035253","article-title":"A survey of multimodal deep generative models","volume":"36","author":"Suzuki","year":"2022","journal-title":"Adv. Robot."},{"key":"B58","unstructured":"Joint multimodal learning with deep generative models\n            SuzukiM.\n            NakayamaK.\n            MatsuoY.\n          2016"},{"key":"B59","doi-asserted-by":"publisher","first-page":"700","DOI":"10.1080\/01691864.2019.1632223","article-title":"Survey on frontiers of language and robotics","volume":"33","author":"Taniguchi","year":"2019","journal-title":"Adv. Robot."},{"key":"B60","doi-asserted-by":"publisher","first-page":"706","DOI":"10.1080\/01691864.2016.1164622","article-title":"Symbol emergence in robotics: a survey","volume":"30","author":"Taniguchi","year":"2016","journal-title":"Adv. Robot."},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.1007\/s00354-019-00084-w","article-title":"Neuro-serket: development of integrative cognitive system through the composition of deep probabilistic generative models","author":"Taniguchi","year":"2020","journal-title":"New Gener. Comput."},{"key":"B62","doi-asserted-by":"publisher","first-page":"1266","DOI":"10.1080\/01691864.2023.2260856","article-title":"Emergent communication through metropolis-hastings naming game with deep generative models","volume":"37","author":"Taniguchi","year":"2023","journal-title":"Adv. Robot."},{"key":"B63","doi-asserted-by":"publisher","first-page":"1454","DOI":"10.2307\/1130423","article-title":"Joint attention and early language","volume":"57","author":"Tomasello","year":"1986","journal-title":"Child. Dev."},{"key":"B64","first-page":"2579","article-title":"Visualizing data using t-sne","volume":"9","author":"van der Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"B65","unstructured":"Attention is all you need\n            VaswaniA.\n            ShazeerN.\n            ParmarN.\n            UszkoreitJ.\n            JonesL.\n            GomezA. N.\n          2017"},{"key":"B66","unstructured":"Generative models of visually grounded imagination\n            VedantamR.\n            FischerI.\n            HuangJ.\n            MurphyK.\n          2017"},{"key":"B67","unstructured":"Graph attention networks\n            Veli\u010dkovi\u0107P.\n            CucurullG.\n            CasanovaA.\n            RomeroA.\n            Li\u00f2P.\n            BengioY.\n          2018"},{"key":"B68","doi-asserted-by":"publisher","first-page":"1089","DOI":"10.1016\/j.jtbi.2006.05.024","article-title":"How to reach linguistic consensus: a proof of convergence for the naming game","author":"Vylder","year":"2008","journal-title":"J. Theor. Biol."},{"key":"B69","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/10597123030111003","article-title":"Progress in the simulation of emergent communication and language","volume":"11","author":"Wagner","year":"2003","journal-title":"Adapt. Behav."},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1802.05335","article-title":"Multimodal generative models for scalable weakly-supervised learning","author":"Wu","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B71","unstructured":"Compositional generalization in unsupervised compositional representation learning: a study on disentanglement and emergent language\n            XuZ.\n            NiethammerM.\n            RaffelC.\n          2022"},{"key":"B72","unstructured":"Unified vision-language pre-training for image captioning and vqa\n            ZhouL.\n            PalangiH.\n            ZhangL.\n            HuH.\n            CorsoJ. J.\n            GaoJ.\n          2020"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1290604\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T11:46:48Z","timestamp":1715773608000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1290604\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,31]]},"references-count":72,"alternative-id":["10.3389\/frobt.2023.1290604"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1290604","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,31]]},"article-number":"1290604"}}