{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,6,8]],"date-time":"2024-06-08T00:32:33Z","timestamp":1717806753671},"reference-count":60,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,6,7]],"date-time":"2024-06-07T00:00:00Z","timestamp":1717718400000},"content-version":"vor","delay-in-days":158,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>When children learn new words, they employ constraints such as the mutual exclusivity (ME) bias: A novel word is mapped to a novel object rather than a familiar one. This bias has been studied computationally, but only in models that use discrete word representations as input, ignoring the high variability of spoken words. We investigate the ME bias in the context of visually grounded speech models that learn from natural images and continuous speech audio. Concretely, we train a model on familiar words and test its ME bias by asking it to select between a novel and a familiar object when queried with a novel word. To simulate prior acoustic and visual knowledge, we experiment with several initialization strategies using pretrained speech and vision networks. Our findings reveal the ME bias across the different initialization approaches, with a stronger bias in models with more prior (in particular, visual) knowledge. Additional tests confirm the robustness of our results, even when different loss functions are considered. Based on detailed analyses to piece out the model\u2019s representation space, we attribute the ME bias to how familiar and novel classes are distinctly separated in the resulting space.<\/jats:p>","DOI":"10.1162\/tacl_a_00672","type":"journal-article","created":{"date-parts":[[2024,6,7]],"date-time":"2024-06-07T17:26:21Z","timestamp":1717781181000},"page":"755-770","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Visually Grounded Speech Models Have a Mutual Exclusivity Bias"],"prefix":"10.1162","volume":"12","author":[{"given":"Leanne","family":"Nortje","sequence":"first","affiliation":[{"name":"Electrical and Electronic Engineering, Stellenbosch University, South Africa. nortjeleanne@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dan","family":"Onea\u0163\u0103","sequence":"additional","affiliation":[{"name":"SpeeD Lab, University Politehnica of Bucharest, Romania. dan.oneata@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yevgen","family":"Matusevych","sequence":"additional","affiliation":[{"name":"CLCG, University of Groningen, The Netherlands. yevgen.matusevych@rug.nl"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Herman","family":"Kamper","sequence":"additional","affiliation":[{"name":"Electrical and Electronic Engineering, Stellenbosch University, South Africa. kamperh@sun.ac.za"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"2024060717254663800_bib1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v067.i01","article-title":"Fitting linear mixed-effects models using lme4","author":"Bates","year":"2015","journal-title":"Journal of Statistical Software"},{"key":"2024060717254663800_bib2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP49357.2023.10096882","article-title":"M-SpeechCLIP: Leveraging large-scale, pre-trained models for multilingual speech to image retrieval","volume-title":"Proceedings of ICASSP","author":"Berry","year":"2023"},{"issue":"5","key":"2024060717254663800_bib3","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1111\/j.1467-7687.2009.00902.x","article-title":"Monolingual, bilingual, trilingual: Infants\u2019 language experience influences the development of a word-learning heuristic","volume":"12","author":"Byers-Heinlein","year":"2009","journal-title":"Developmental Science"},{"key":"2024060717254663800_bib4","article-title":"A simple framework for contrastive learning of visual representations","volume-title":"Proceedings of ICML","author":"Chen","year":"2020"},{"key":"2024060717254663800_bib5","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12967","article-title":"Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques","author":"Chrupa\u0142a","year":"2022","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2024060717254663800_bib6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1057","article-title":"Representations of language in a model of visually grounded speech signal","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","author":"Chrupa\u0142a","year":"2017"},{"key":"2024060717254663800_bib7","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2016-82","article-title":"Unsupervised learning of audio segment representations using sequence-to-sequence recurrent neural networks","volume-title":"Proceedings of Interspeech","author":"Chung","year":"2016"},{"issue":"10","key":"2024060717254663800_bib8","doi-asserted-by":"publisher","first-page":"472","DOI":"10.1016\/j.tics.2004.08.012","article-title":"How language acquisition builds on cognitive development","volume":"8","author":"Clark","year":"2004","journal-title":"Trends in Cognitive Sciences"},{"issue":"5","key":"2024060717254663800_bib9","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1111\/j.1467-7687.2012.01173.x","article-title":"Phonological similarity and mutual exclusivity: On-line recognition of atypical pronunciations in 3\u20135-year-olds","volume":"15","author":"Creel","year":"2012","journal-title":"Developmental Science"},{"issue":"2","key":"2024060717254663800_bib10","doi-asserted-by":"publisher","first-page":"260","DOI":"10.1080\/17470210902888809","article-title":"Speech segmentation is facilitated by visual cues","volume":"63","author":"Cunillera","year":"2010","journal-title":"Quarterly Journal of Experimental Psychology"},{"key":"2024060717254663800_bib11","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848","article-title":"ImageNet: A large-scale hierarchical image database","volume-title":"Proceedings of CVPR","author":"Deng","year":"2009"},{"issue":"4","key":"2024060717254663800_bib12","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1109\/TPAMI.2006.79","article-title":"One-shot learning of object categories","volume":"28","author":"Li","year":"2006","journal-title":"Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2024060717254663800_bib13","article-title":"Mutual exclusivity as a challenge for deep neural networks","volume-title":"Proceedings of NeurIPS","author":"Gandhi","year":"2020"},{"key":"2024060717254663800_bib14","article-title":"Deep daxes: Mutual exclusivity arises through both learning biases and pragmatic strategies in neural networks","volume-title":"Proceedings of Cognitive Science Society","author":"Gulordava","year":"2020"},{"issue":"1","key":"2024060717254663800_bib15","doi-asserted-by":"publisher","first-page":"B23\u2013B34","DOI":"10.1016\/S0010-0277(02)00186-5","article-title":"The development of a word-learning strategy","volume":"87","author":"Halberda","year":"2003","journal-title":"Cognition"},{"key":"2024060717254663800_bib16","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462396","article-title":"Vision as an interlingua: Learning multilingual semantic embeddings of untranscribed speech","volume-title":"Proceedings of ICASSP","author":"Harwath","year":"2018"},{"key":"2024060717254663800_bib17","doi-asserted-by":"publisher","DOI":"10.1109\/ASRU.2015.7404800","article-title":"Deep multimodal semantic embeddings for speech and images","volume-title":"Proceedings of Automatic Speech Recognition and Understanding","author":"Harwath","year":"2015"},{"key":"2024060717254663800_bib18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1047","article-title":"Learning word-like units from joint audio-visual analysis","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Harwath","year":"2017"},{"key":"2024060717254663800_bib19","article-title":"Learning hierarchical discrete linguistic units from visually-grounded speech","volume-title":"Proceedings of International Conference on Learning Representations","author":"Harwath","year":"2020"},{"key":"2024060717254663800_bib20","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01231-1_40","article-title":"Jointly discovering visual objects and spoken words from raw sensory input","volume-title":"Proceedings of ECCV","author":"Harwath","year":"2018"},{"key":"2024060717254663800_bib21","article-title":"Unsupervised learning of spoken language with visual context","volume-title":"Proceedings of NeurIPS","author":"Harwath","year":"2016"},{"key":"2024060717254663800_bib22","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2018-2364","article-title":"Learning word embeddings: Unsupervised methods for fixed-size representations of variable-length speech segments","volume-title":"Proceedings of Interspeech","author":"Holzenberger","year":"2018"},{"key":"2024060717254663800_bib23","article-title":"Scaling up visual and vision-language representation learning with noisy text supervision","volume-title":"Proceedings of ICML","author":"Jia","year":"2021"},{"issue":"1","key":"2024060717254663800_bib24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1006\/cogp.1995.1010","article-title":"Infants\u2019 detection of the sound patterns of words in fluent speech","volume":"29","author":"Jusczyk","year":"1995","journal-title":"Cognitive Psychology"},{"key":"2024060717254663800_bib25","doi-asserted-by":"publisher","DOI":"10.1017\/S1366728914000364","article-title":"The effects of linguistic experience on the flexible use of mutual exclusivity in word learning","author":"Kalashnikova","year":"2015","journal-title":"Bilingualism: Language and Cognition"},{"key":"2024060717254663800_bib26","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683639","article-title":"Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models","volume-title":"Proceedings of ICASSP","author":"Kamper","year":"2019"},{"key":"2024060717254663800_bib27","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2018.2872106","article-title":"Semantic speech retrieval with a visually grounded model of untranscribed speech","author":"Kamper","year":"2019","journal-title":"IEEE\/ACM Transactions on Audio, Speech, and Language Processing"},{"key":"2024060717254663800_bib28","article-title":"Adam: A method for stochastic optimization","volume-title":"Proceedings of ICLR","author":"Kingma","year":"2015"},{"key":"2024060717254663800_bib29","article-title":"CompRess: Self-supervised learning by compressing representations","author":"Koohpayegani","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024060717254663800_bib30","doi-asserted-by":"publisher","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","author":"Krizhevsky","year":"2017","journal-title":"ACM"},{"key":"2024060717254663800_bib31","doi-asserted-by":"publisher","first-page":"104191","DOI":"10.1016\/j.cognition.2020.104191","article-title":"The role of developmental change and linguistic experience in the mutual exclusivity effect","volume":"198","author":"Lewis","year":"2020","journal-title":"Cognition"},{"key":"2024060717254663800_bib32","article-title":"Align before fuse: Vision and language representation learning with momentum distillation","volume-title":"Proceedings of NeurIPS","author":"Li","year":"2021"},{"key":"2024060717254663800_bib33","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-10602-1_48","article-title":"Microsoft COCO: Common objects in context","volume-title":"Proceedings of ECCV","author":"Lin","year":"2014"},{"key":"2024060717254663800_bib34","volume-title":"Categorization and Naming in Children: Problems of Induction","author":"Markman","year":"1989"},{"issue":"2","key":"2024060717254663800_bib35","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/0010-0285(88)90017-5","article-title":"Children\u2019s use of mutual exclusivity to constrain the meanings of words","volume":"20","author":"Markman","year":"1988","journal-title":"Cognitive Computing"},{"issue":"3","key":"2024060717254663800_bib36","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/S0010-0285(03)00034-3","article-title":"Use of the mutual exclusivity assumption by young word learners","volume":"47","author":"Markman","year":"2003","journal-title":"Cognitive Psychology"},{"issue":"1","key":"2024060717254663800_bib37","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1080\/15250000802569702","article-title":"Learning words over time: The role of stimulus repetition in mutual exclusivity","volume":"14","author":"Mather","year":"2009","journal-title":"Infancy"},{"issue":"3\u20134","key":"2024060717254663800_bib38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2307\/1166130","article-title":"The mutual exclusivity bias in children\u2019s word learning","volume":"54","author":"Merriman","year":"1989","journal-title":"Monographs of the Society for Research in Child Development"},{"issue":"3","key":"2024060717254663800_bib39","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1038\/scientificamerican0987-94","article-title":"How children learn words","volume":"257","author":"Miller","year":"1987","journal-title":"Scientific American"},{"issue":"1","key":"2024060717254663800_bib40","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1006\/jecp.2000.2598","article-title":"Object recognition and object segregation in 4.5-month-old infants","volume":"78","author":"Needham","year":"2001","journal-title":"Journal of Experimental Child Psychology"},{"key":"2024060717254663800_bib41","doi-asserted-by":"publisher","DOI":"10.1109\/SLT54892.2023.10023079","article-title":"Towards visually prompted keyword localisation for zero-resource spoken languages","volume-title":"Proceedings of SLT","author":"Nortje","year":"2023"},{"key":"2024060717254663800_bib42","article-title":"Visually grounded few-shot word learning in low-resource settings","author":"Nortje","year":"2023","journal-title":"arXiv preprint arXiv:2306.11371"},{"issue":"1","key":"2024060717254663800_bib43","doi-asserted-by":"publisher","first-page":"e13069","DOI":"10.1111\/cogs.13069","article-title":"Mutual exclusivity in pragmatic agents","volume":"46","author":"Ohmer","year":"2022","journal-title":"Cognitive Science"},{"issue":"6","key":"2024060717254663800_bib44","doi-asserted-by":"publisher","first-page":"1454","DOI":"10.1109\/JSTSP.2022.3180220","article-title":"Keyword localisation in untranscribed speech using visually grounded speech models","volume":"16","author":"Olaleye","year":"2022","journal-title":"Journal of Selected Topics in Signal Processing"},{"key":"2024060717254663800_bib45","article-title":"Representation learning with contrastive predictive coding","author":"van den Oord","year":"2019","journal-title":"arXiv preprint arXiv:1807.03748"},{"key":"2024060717254663800_bib46","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178964","article-title":"Librispeech: An ASR corpus based on public domain audio books","volume-title":"Proceedings of ICASSP","author":"Panayotov","year":"2015"},{"key":"2024060717254663800_bib47","doi-asserted-by":"publisher","first-page":"372","DOI":"10.1162\/tacl_a_00656","article-title":"What do self-supervised speech models know about words?","volume":"12","author":"Pasad","year":"2023","journal-title":"Transactions of the Association for Computational Linguistics (2024)"},{"key":"2024060717254663800_bib48","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP43922.2022.9747103","article-title":"Fast-slow transformer for visually grounding speech","volume-title":"Proceedings of ICASSP","author":"Peng","year":"2022"},{"key":"2024060717254663800_bib49","article-title":"Self-supervised representation learning for speech using visual grounding and masked language modeling","volume-title":"AAAI Conference on Artificial Intelligence SAS Workshop","author":"Peng","year":"2022"},{"key":"2024060717254663800_bib50","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2022-10652","article-title":"Word discovery in visually grounded, self-supervised speech models","volume-title":"Proceedings of Interspeech","author":"Peng","year":"2022"},{"key":"2024060717254663800_bib51","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2023-2044","article-title":"Syllable discovery and cross-lingual generalization in a visually grounded, self-supervised speech mode","volume-title":"Proceedings of Interspeech","author":"Peng","year":"2023"},{"key":"2024060717254663800_bib52","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461326","article-title":"End-to-end audiovisual speech recognition","volume-title":"Proceedings of ICASSP","author":"Petridis","year":"2018"},{"issue":"1","key":"2024060717254663800_bib53","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.specom.2004.09.001","article-title":"The Buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability","volume":"45","author":"Pitt","year":"2005","journal-title":"Speech Communication"},{"key":"2024060717254663800_bib54","article-title":"Learning transferable visual models from natural language supervision","volume-title":"Proceedings of ICML","author":"Radford","year":"2021"},{"key":"2024060717254663800_bib55","doi-asserted-by":"publisher","DOI":"10.1109\/SLT54892.2023.10022954","article-title":"SpeechCLIP: Integrating speech with pre-trained vision and language model","volume-title":"Proceedings of SLT","author":"Shih","year":"2023"},{"issue":"6","key":"2024060717254663800_bib56","doi-asserted-by":"publisher","first-page":"1093","DOI":"10.1111\/j.1551-6709.2010.01118.x","article-title":"Effects of visual information on adults\u2019 and infants\u2019 auditory statistical learning","volume":"34","author":"Thiessen","year":"2010","journal-title":"Cognitive Science"},{"key":"2024060717254663800_bib57","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-1693","article-title":"Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge","volume-title":"Proceedings of Interspeech","author":"van Niekerk","year":"2020"},{"issue":"4","key":"2024060717254663800_bib58","doi-asserted-by":"publisher","first-page":"e13122","DOI":"10.31234\/osf.io\/udbh2","article-title":"Cross-situational word learning with multimodal neural networks","volume":"46","author":"Vong","year":"2022","journal-title":"Cognitive Science"},{"key":"2024060717254663800_bib59","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462002","article-title":"Segmental audio Word2Vec: Representing utterances as sequences of vectors with applications in spoken term detection","volume-title":"Proceedings of ICASSP","author":"Wang","year":"2018"},{"issue":"5","key":"2024060717254663800_bib60","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1111\/j.1467-9280.2007.01915.x","article-title":"Rapid word learning under uncertainty via cross-situational statistics","volume":"18","author":"Chen","year":"2007","journal-title":"Psychological Science"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00672\/2377793\/tacl_a_00672.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00672\/2377793\/tacl_a_00672.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,7]],"date-time":"2024-06-07T17:26:31Z","timestamp":1717781191000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00672\/121542\/Visually-Grounded-Speech-Models-Have-a-Mutual"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":60,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00672","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}