{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T11:06:50Z","timestamp":1786100810204,"version":"3.56.0"},"reference-count":50,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2023,3,15]],"date-time":"2023-03-15T00:00:00Z","timestamp":1678838400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Until recently, most research in the Digital Humanities (DH) was monomodal, meaning that the object of analysis was either textual or visual. Seeking to integrate multimodality theory into the DH, this article demonstrates that recently developed multimodal deep learning models, such as Contrastive Language Image Pre-training (CLIP), offer new possibilities to explore and analyze image\u2013text combinations at scale. These models, which are trained on image and text pairs, can be applied to a wide range of text-to-image, image-to-image, and image-to-text prediction tasks. Moreover, multimodal models show high accuracy in zero-shot classification, i.e. predicting unseen categories across heterogeneous datasets. Based on three exploratory case studies, we argue that this zero-shot capability opens up the way for a multimodal turn in DH research. Moreover, multimodal models allow scholars to move past the artificial separation of text and images that was dominant in the field and analyze multimodal meaning at scale. However, we also need to be aware of the specific (historical) bias of multimodal deep learning that stems from biases in the training data used to train these models.<\/jats:p>","DOI":"10.1093\/llc\/fqad008","type":"journal-article","created":{"date-parts":[[2023,3,15]],"date-time":"2023-03-15T20:56:39Z","timestamp":1678913799000},"page":"1267-1280","source":"Crossref","is-referenced-by-count":37,"title":["A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections"],"prefix":"10.1093","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8579-824X","authenticated-orcid":false,"given":"Thomas","family":"Smits","sequence":"first","affiliation":[{"name":"Faculty of Arts, University of Antwerp , Antwerp, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8177-4582","authenticated-orcid":false,"given":"Melvin","family":"Wevers","sequence":"additional","affiliation":[{"name":"Department of History, Faculty of Humanities, University of Amsterdam , Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,3,15]]},"reference":[{"key":"2023083111393845700_fqad008-B1","doi-asserted-by":"crossref","DOI":"10.4324\/9780203928752","volume-title":"The Family in English Children\u2019s Literature","author":"Alston","year":"2008"},{"key":"2023083111393845700_fqad008-B2","doi-asserted-by":"crossref","first-page":"i3","DOI":"10.1093\/llc\/fqz013","article-title":"Distant viewing: analyzing large visual corpora","volume":"34(Supplement_1)","author":"Arnold","year":"2019","journal-title":"Digital Scholarship in the Humanities"},{"issue":"1","key":"2023083111393845700_fqad008-B3","doi-asserted-by":"crossref","first-page":"127","DOI":"10.3406\/comm.1961.921","article-title":"Le message photographique","volume":"1","author":"Barthes","year":"1961","journal-title":"Communications"},{"key":"2023083111393845700_fqad008-B4","doi-asserted-by":"crossref","DOI":"10.4324\/9781315773971","volume-title":"Text and Image: A Critical Introduction to the Visual\/Verbal Divide","author":"Bateman","year":"2014"},{"issue":"2","key":"2023083111393845700_fqad008-B5","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1093\/tcbh\/hwq007","article-title":"The digitization of newspaper archives: opportunities and challenges for historians","volume":"21","author":"Bingham","year":"2010","journal-title":"Twentieth Century British History"},{"key":"2023083111393845700_fqad008-B6","author":"Birhane","year":"2021"},{"issue":"4","key":"2023083111393845700_fqad008-B151","doi-asserted-by":"crossref","first-page":"171","DOI":"10.18352\/bmgn-lchr.9351","article-title":"Who\u2019s afraid of Patterns?: The Particular versus the Universal and the Meaning of Humanities 3.0","volume":"128","author":"Bod","year":"2013","journal-title":"BMGN - Low Countries Historical Review"},{"key":"2023083111393845700_fqad008-B7","first-page":"i25","article-title":"Digital humanities is text heavy, visualization light, and simulation poor","volume":"32(Suppl_1)","author":"Champion","year":"2017","journal-title":"Digital Scholarship in the Humanities"},{"key":"2023083111393845700_fqad008-B8","author":"CLIP Model Card","year":"2020"},{"key":"2023083111393845700_fqad008-B9","author":"Crawford","year":"2019"},{"key":"2023083111393845700_fqad008-B10","first-page":"11162","author":"Desai","year":"2021"},{"key":"2023083111393845700_fqad008-B11","author":"Duhaime","year":"2017"},{"key":"2023083111393845700_fqad008-B12","volume-title":"L\u2019arch\u00e9ologie du Savoir","author":"Foucault","year":"1969"},{"issue":"2","key":"2023083111393845700_fqad008-B13","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1177\/26349795211007094","article-title":"Distant viewing and multimodality theory: prospects and challenges","volume":"1","author":"Hiippala","year":"2021","journal-title":"Multimodality & Society"},{"key":"2023083111393845700_fqad008-B14","author":"Jain","year":"2019"},{"key":"2023083111393845700_fqad008-B15","first-page":"4904","author":"Jia","year":"2021"},{"key":"2023083111393845700_fqad008-B16","first-page":"306","author":"Jo","year":"2020"},{"issue":"1","key":"2023083111393845700_fqad008-B17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/17460654.2019.1640605","article-title":"The magic lantern: open medium","volume":"17","author":"Kember","year":"2019","journal-title":"Early Popular Visual Culture"},{"issue":"7","key":"2023083111393845700_fqad008-B18","doi-asserted-by":"crossref","first-page":"1956","DOI":"10.1007\/s11263-020-01316-z","article-title":"The open images dataset v4","volume":"128","author":"Kuznetsova","year":"2020","journal-title":"International Journal of Computer Vision"},{"key":"2023083111393845700_fqad008-B19","author":"Leetaru","year":"2008"},{"key":"2023083111393845700_fqad008-B20","first-page":"740","author":"Lin","year":"2014"},{"key":"2023083111393845700_fqad008-B21","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/11214.001.0001","volume-title":"Cultural Analytics","author":"Manovich","year":"2020"},{"issue":"2","key":"2023083111393845700_fqad008-B22","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1177\/1470412905054673","article-title":"There are no visual media","volume":"4","author":"Mitchell","year":"2005","journal-title":"Journal of Visual Culture"},{"key":"2023083111393845700_fqad008-B23","first-page":"54","article-title":"Conjectures on world literature","volume":"1","author":"Moretti","year":"2000","journal-title":"New Left Review"},{"key":"2023083111393845700_fqad008-B24","volume-title":"Distant Reading","author":"Moretti","year":"2015"},{"issue":"1","key":"2023083111393845700_fqad008-B25","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1080\/13688804.2012.752963","article-title":"The digital turn","volume":"19","author":"Nicholson","year":"2013","journal-title":"Media History"},{"key":"2023083111393845700_fqad008-B26","volume-title":"Das visuelle Zeitalter: Punkt und Pixel","author":"Paul","year":"2016"},{"key":"2023083111393845700_fqad008-B27","first-page":"8748","author":"Radford","year":"2021"},{"issue":"9","key":"2023083111393845700_fqad008-B28","doi-asserted-by":"crossref","first-page":"2352","DOI":"10.1162\/neco_a_00990","article-title":"Deep convolutional neural networks for image classification: a comprehensive review","volume":"29","author":"Rawat","year":"2017","journal-title":"Neural Computation"},{"key":"2023083111393845700_fqad008-B29","volume-title":"Inside Amazon\u2019s Clickworker Platform: How Half a Million People Are Being Paid Pennies to Train AI","author":"Reese","year":"2016"},{"issue":"3","key":"2023083111393845700_fqad008-B30","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"International Journal of Computer Vision"},{"key":"2023083111393845700_fqad008-B31","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-476-05758-7","volume-title":"BildWissen\u2014KinderBuch","author":"Schmideler","year":"2021"},{"key":"2023083111393845700_fqad008-B32","author":"Schuhmann","year":"2022"},{"key":"2023083111393845700_fqad008-B33","author":"Schuhmann","year":"2021"},{"key":"2023083111393845700_fqad008-B34","author":"Seguin","year":"2017"},{"issue":"12","key":"2023083111393845700_fqad008-B35","doi-asserted-by":"crossref","first-page":"1349","DOI":"10.1109\/34.895972","article-title":"Content-based image retrieval at the end of the early years","volume":"22","author":"Smeulders","year":"2000","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2023083111393845700_fqad008-B36","author":"Smits","year":"2022"},{"key":"2023083111393845700_fqad008-B37","first-page":"149","volume-title":"Proceedings of the Conference on Computational Humanities Research 2021","author":"Smits","year":"2021"},{"key":"2023083111393845700_fqad008-B38","article-title":"Distant reading 940,000 online circulations of 26 iconic photographs","volume":"14614448211049460","author":"Smits","year":"2021","journal-title":"New Media & Society"},{"issue":"2","key":"2023083111393845700_fqad008-B39","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1177\/1470357221992097","article-title":"The agency of computer vision models as optical instruments","volume":"21","author":"Smits","year":"2021","journal-title":"Visual Communication"},{"key":"2023083111393845700_fqad008-B40","volume-title":"On Photography","author":"Sontag","year":"1977"},{"key":"2023083111393845700_fqad008-B41","volume-title":"Language and Ideology in Children\u2019s Fiction","author":"Stephens","year":"1992"},{"key":"2023083111393845700_fqad008-B42","first-page":"200","article-title":"Multimodal few-shot learning with frozen language models","volume":"34","author":"Tsimpoukelli","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023083111393845700_fqad008-B43","doi-asserted-by":"crossref","DOI":"10.1093\/llc\/fqac003","article-title":"A survey of computational methods for iconic image analysis","author":"van Noord","year":"2022","journal-title":"Digital Scholarship in the Humanities, 37(4): 1316\u20131338"},{"key":"2023083111393845700_fqad008-B44","first-page":"74","volume-title":"Keywords for Children\u2019s Literature","author":"Wesseling","year":"2021"},{"key":"2023083111393845700_fqad008-B45","author":"Wevers","year":"2021"},{"key":"2023083111393845700_fqad008-B46","author":"Wevers","year":"2017"},{"issue":"1","key":"2023083111393845700_fqad008-B47","first-page":"194","article-title":"The visual digital turn. Using neural networks to study historical images","volume":"35","author":"Wevers","year":"2020","journal-title":"Digital Scholarship in the Humanities"},{"issue":"1","key":"2023083111393845700_fqad008-B48","doi-asserted-by":"crossref","DOI":"10.18146\/tmg.815","article-title":"What to do with 2.000.000 historical press photos? The challenges and opportunities of applying a scene detection algorithm to a digitised press photo collection","volume":"25","author":"Wevers","year":"2022","journal-title":"TMG Journal for Media History"},{"issue":"1","key":"2023083111393845700_fqad008-B49","doi-asserted-by":"crossref","DOI":"10.18146\/2213-7653.2017.277","article-title":"Digital humanities and media history: a challenge for historical newspaper research","volume":"20","author":"Wijfjes","year":"2017","journal-title":"TMG Journal for Media History"}],"container-title":["Digital Scholarship in the Humanities"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/dsh\/article-pdf\/38\/3\/1267\/51309490\/fqad008.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/dsh\/article-pdf\/38\/3\/1267\/51309490\/fqad008.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T11:43:20Z","timestamp":1693482200000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/dsh\/article\/38\/3\/1267\/7078540"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,15]]},"references-count":50,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,3,15]]},"published-print":{"date-parts":[[2023,8,31]]}},"URL":"https:\/\/doi.org\/10.1093\/llc\/fqad008","relation":{},"ISSN":["2055-7671","2055-768X"],"issn-type":[{"value":"2055-7671","type":"print"},{"value":"2055-768X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,9,1]]},"published":{"date-parts":[[2023,3,15]]}}}