{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T09:10:11Z","timestamp":1766049011434,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2018,9,12]],"date-time":"2018-09-12T00:00:00Z","timestamp":1536710400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Educational content of many kinds and from many disciplines are increasingly presented in the form of short videos made broadly accessible via platforms such as YouTube. We argue that understanding how such communicative forms function effectively (or not) demands a more thorough theoretical foundation in the principles of multimodal communication that is also capable of engaging with, and driving, empirical studies. We introduce the basic concepts adopted and discuss an empirical study showing how functional measures derived from the theory of multimodality we employ and results from a recipient-based study that we conducted align. We situate these results with respect to the state of the art in cognitive research in multimodal learning and argue that the more complex multimodal interactions and artifacts become, the more a fine-grained view of multimodal communication of the kind we propose will be essential for engaging with such media, both theoretically and empirically.<\/jats:p>","DOI":"10.3390\/mti2030059","type":"journal-article","created":{"date-parts":[[2018,9,12]],"date-time":"2018-09-12T10:26:36Z","timestamp":1536747996000},"page":"59","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["The Communicative Effectiveness of Education Videos: Towards an Empirically-Motivated Multimodal Account"],"prefix":"10.3390","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7209-9295","authenticated-orcid":false,"given":"John A.","family":"Bateman","sequence":"first","affiliation":[{"name":"Faculty of Linguistics and Literary Sciences, Bremen University, 28359 Bremen, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9618-8946","authenticated-orcid":false,"given":"Florian","family":"Schmidt-Borcherding","sequence":"additional","affiliation":[{"name":"Faculty of Pedagogy and Educational Sciences, Bremen University, 28359 Bremen, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2018,9,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Mayer, R.E. (2001). Multimedia Learning, Cambridge University Press.","DOI":"10.1017\/CBO9781139164603"},{"key":"ref_2","unstructured":"Mayer, R.E. (2009). An Integrated Model of Text and Picture Comprehension. The Cambridge Handbook of Multimedia Learning, Cambridge University Press. [2nd ed.]."},{"key":"ref_3","unstructured":"Paivio, A. (1986). Mental Representations: A Dual Coding Approach, Oxford University Press."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Landy, D., Allen, C., and Zednik, C. (2014). A Perceptual Account of Symbolic Reasoning. Front. Psychol., 5.","DOI":"10.3389\/fpsyg.2014.00275"},{"key":"ref_5","unstructured":"Mayer, R.E. (2009). Cognitive Theory of Multimedia Learning. The Cambridge Handbook of Multimedia Learning, Cambridge University Press. [2nd ed.]."},{"key":"ref_6","unstructured":"Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis, Routledge. [2nd ed.]."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bateman, J.A., Wildfeuer, J., and Hiippala, T. (2017). Multimodality\u2014Foundations, Research and Analysis. A Problem-Oriented Introduction, Mouton de Gruyter.","DOI":"10.1515\/9783110479898"},{"key":"ref_8","unstructured":"Kress, G., and van Leeuwen, T. (1996). Reading Images: The Grammar of Visual Design, Routledge."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1080\/10350339909360442","article-title":"Interdependence, Interaction and Metaphor in Multisemiotic Texts","volume":"9","year":"1999","journal-title":"Soc. Semiot."},{"key":"ref_10","unstructured":"Painter, C., Martin, J.R., and Unsworth, L. (2013). Reading Visual Narratives: Image Analysis of Children\u2019s Picture Books, Equinox."},{"key":"ref_11","unstructured":"Bucher, H.J., Gloning, T., and Lehnen, K. (2010). Multimodalit\u00e4t\u2014Eine Universalie des Medienwandels. Problemstellung und Theorien der Multimodalit\u00e4tsforschung. Neue Medien\u2014Neue Formate. Ausdifferenzierung und Konvergenz in der Medienkommunikation, Campus Verlag. Number 10 in Interaktiva. Schriftenreihe des Zentrums f\u00fcr Medien und Interaktivit\u00e4t (ZMI), Gie\u00dfen."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Diekmannshenke, H.J., Klemm, M., and St\u00f6ckl, H. (2011). Multimodales Verstehen oder Rezeption als Interaktion. Theoretische und empirische Grundlagen einer systematischen Analyse der Multimodalit\u00e4t. Bildlinguistik. Theorien\u2014Methoden\u2014Fallbeispiele, Erich Schmidt.","DOI":"10.37307\/b.978-3-503-12264-6"},{"key":"ref_13","unstructured":"Kress, G., and van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication, Arnold."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bezemer, J., and Kress, G. (2016). Multimodality, Learning and Communication: A Social Semiotic Frame, Routledge.","DOI":"10.4324\/9781315687537"},{"key":"ref_15","unstructured":"Royce, T.D., and Bowcher, W.L. (2007). Multiliteracies and multimodal text analysis in classroom work with children\u2019s literature. New Directions in the Analysis of Multimodal Discourse, Lawrence Erlbaum Associates."},{"key":"ref_16","unstructured":"Dreyfus, S., Hood, S., and Stenglin, M. (2011). Integrating visual and verbal meaning in multimodal text comprehension: towards a model of intermodal relations. Semiotic Margins: Reclaiming Meaning, Continuum."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1007\/s13384-011-0023-y","article-title":"Image-language interaction in online reading environments: Challenges for students\u2019 reading comprehension","volume":"38","author":"Chan","year":"2011","journal-title":"Aust. Educ. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1037\/0022-0663.94.2.428","article-title":"Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features, and redundancy","volume":"94","author":"Craig","year":"2002","journal-title":"J. Educ. Psychol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.chb.2017.01.049","article-title":"Instructor presence in instructional video: Effects on visual attention, recall, and perceived learning","volume":"71","author":"Wang","year":"2017","journal-title":"Comput. Hum. Behav."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1007\/s10648-011-9173-8","article-title":"Facilitating Understanding of Movements in Dynamic Visualizations: An Embodied Perspective","volume":"23","author":"Tabbers","year":"2011","journal-title":"Educ. Psychol. Rev."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1037\/0033-2909.123.2.162","article-title":"Situation Models in Language Comprehension and Memory","volume":"123","author":"Zwaan","year":"1998","journal-title":"Psychol. Bull."},{"key":"ref_22","first-page":"80","article-title":"Proving the value of visual design in scientific communication","volume":"23","author":"Cheng","year":"2017","journal-title":"Inf. Des. J."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1287\/mnsc.32.5.554","article-title":"Organizational information requirements, media richness and structural design","volume":"32","author":"Daft","year":"1986","journal-title":"Manag. Sci."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1016\/j.pragma.2014.10.001","article-title":"A multimodal discourse theory of visual narrative","volume":"74","author":"Bateman","year":"2014","journal-title":"J. Pragmat."},{"key":"ref_25","unstructured":"Mayer, R.E. (2009). Navigational Principles in Multimedia Learning. The Cambridge Handbook of Multimedia Learning, Cambridge University Press. [2nd ed.]."},{"key":"ref_26","unstructured":"O\u2019Halloran, K.L., and Smith, B.A. (2011). The Decomposability of Semiotic Modes. Multimodal Studies: Multiple Approaches and Domains, Routledge Studies in Multimodality, Routledge."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Klug, N.M., and St\u00f6ckl, H. (2016). Methodological and theoretical issues for the empirical investigation of multimodality. Handbuch Sprache im Multimodalen Kontext, De Gruyter Mouton. Number 7 in Handbooks of Linguistics and Communication Science (HSK).","DOI":"10.1515\/9783110296099"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chorianopoulos, K., and Giannakos, M.N. (2013, January 24\u201326). Usability Design for Video Lectures. Proceedings of the 11th European Conference on Interactive TV and Video, Como, Italy.","DOI":"10.1145\/2465958.2465982"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1016\/j.dcm.2017.06.009","article-title":"Triangulating transmediality: A multimodal semiotic framework relating media, modes and genres","volume":"20","author":"Bateman","year":"2017","journal-title":"Discourse Context Med."},{"key":"ref_30","unstructured":"Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., and Sloetjes, H. (2006, January 22\u201328). ELAN: A professional framework for multimodality research. Proceedings of the LREC 2006\u2014Fifth International Conference on Language Resources and Evaluation, Genoa, Italy."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Chapelle, C.A. (2013). Multimodal Corpus-Based Approaches. The Encyclopedia of Applied Linguistics, Blackwell Publishing Ltd.","DOI":"10.1002\/9781405198431"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Baker, P., and McEnery, T. (2015). Corpora and Discourse Studies: Integrating Discourse and Corpora, Palgrave Macmillan.","DOI":"10.1057\/9781137431738"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1177\/1470357211424691","article-title":"Multimodal Narrative Construction in Christopher Nolan\u2019s Memento: A Description of Method","volume":"11","author":"Tseng","year":"2012","journal-title":"J. Vis. Commun."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1080\/1472586X.2016.1173892","article-title":"Towards next-generation visual archives: Image, film and discourse","volume":"31","author":"Bateman","year":"2016","journal-title":"Vis. Stud."},{"key":"ref_35","unstructured":"Asher, N., and Lascarides, A. (2003). Logics of Conversation, Cambridge University Press."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Martin, J.R. (1992). English Text: Systems and Structure, Benjamins.","DOI":"10.1075\/z.59"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Balacheff, N., Ludvigsen, S., de Jong, T., Lazonder, A., and Barnes, S. (2009). External representations for learning: headed towards a digital culture. Technology-Enhanced Learning: Principles and Products, Springer.","DOI":"10.1007\/978-1-4020-9827-7"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/s10032-006-0016-y","article-title":"Towards a theory of tables","volume":"8","author":"Hurst","year":"2006","journal-title":"Int. J. Doc. Anal."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Bateman, J.A. (2008). Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents, Palgrave Macmillan.","DOI":"10.1057\/9780230582323_5"},{"key":"ref_40","unstructured":"O\u2019Halloran, K.L. (2005). Mathematical Discourse: Language, Symbolism and Visual Images, Continuum."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3765\/sp.6.2","article-title":"Strategic conversation","volume":"6","author":"Asher","year":"2013","journal-title":"Semant. Pragmat."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Gibbon, D., Mertins, I., and Moore, R.K. (2000). Handbook of Multimodal and Spoken Dialogue Systems. Resources, Terminology and Product Evaluation, Springer Science+Business Media.","DOI":"10.1007\/978-1-4615-4501-9"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1177\/1470357212446409","article-title":"Visualizing science: The reception of PowerPoint presentations","volume":"11","author":"Bucher","year":"2012","journal-title":"Vis. Commun."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1037\/0022-0663.93.2.377","article-title":"Signaling as a cognitive guide in multimedia learning","volume":"93","author":"Mautone","year":"2001","journal-title":"J. Educ. Psychol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.edurev.2015.12.003","article-title":"Signaling text-picture relations in multimedia learning: A comprehensive meta-analysis","volume":"17","author":"Richter","year":"2016","journal-title":"Educ. Res. Rev."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"M\u00fcller, C., Cienki, A., Fricke, E., Ladewig, S.H., McNeill, D., and Bressem, J. (2014). Gesture families. Body\u2013Language\u2013Communication. An International Handbook on Multmodality in Human Interaction, Mouton de Gruyter. Number 38\/2 in Handb\u00fccher zur Sprach- und Kommunikationswissenschaft\/Handbooks of Linguistics and Communication Science (HSK).","DOI":"10.1515\/9783110302028"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"4","DOI":"10.3991\/ijet.v9i1.3354","article-title":"Video-Based learning and open online courses","volume":"9","author":"Giannakos","year":"2014","journal-title":"Int. J. Emerg. Technol. Learn."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"35","DOI":"10.3991\/ijet.v9i1.3349","article-title":"Looking at MOOCs rapid growth through the lens of video-based learning research","volume":"9","author":"Giannakos","year":"2014","journal-title":"Int. J. Emerg. Technol. Learn."},{"key":"ref_49","unstructured":"Bishop, J.L., and Verleger, M.A. (2013, January 23\u201326). The flipped classroom: A survey of the research. Proceedings of the 120th ASEE Annual Conference & Exposition, Atlanta, GA, USA."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.compedu.2014.08.015","article-title":"Effects of different video lecture types on sustained attention, emotion, cognitive load, and learning performance","volume":"80","author":"Chen","year":"2015","journal-title":"Comput. Educ."},{"key":"ref_51","first-page":"1","article-title":"Lern- und Lehrvideos: Gestaltung, Produktion, Einsatz","volume":"Volume 4","author":"Ebner","year":"2017","journal-title":"Handbuch E-Learning"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/s10648-014-9264-4","article-title":"Picture or Text First? Explaining Sequence Effects when Learning with Pictures and Text","volume":"27","author":"Eitel","year":"2015","journal-title":"Educ. Psychol. Rev."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/j.learninstruc.2017.03.001","article-title":"Investigating gaze behavior during processing of inconsistent text-picture information: Evidence for text-picture integration","volume":"49","year":"2017","journal-title":"Learn. Instr."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/2\/3\/59\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:20:05Z","timestamp":1760196005000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/2\/3\/59"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9,12]]},"references-count":53,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2018,9]]}},"alternative-id":["mti2030059"],"URL":"https:\/\/doi.org\/10.3390\/mti2030059","relation":{},"ISSN":["2414-4088"],"issn-type":[{"type":"electronic","value":"2414-4088"}],"subject":[],"published":{"date-parts":[[2018,9,12]]}}}