{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,19]],"date-time":"2026-04-19T19:57:25Z","timestamp":1776628645767,"version":"3.51.2"},"reference-count":33,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T00:00:00Z","timestamp":1730764800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Virtual Real."],"abstract":"<jats:p>Head-mounted mixed reality interfaces, in which a user perceives a seamless blend of real, virtual or remote content, have great potential in a wide range of educational contexts. In this paper, we explore the use of mixed reality (MR) in piano education. We begin with a review of previous examples of virtual and mixed reality for piano teaching and learning, identifying four major categories of functionality: remote teaching, learning to read musical notation, providing alternative notation systems to Western musical notation, and enhancing users\u2019 understanding and experience of music. Following this review, we present an application designed to demonstrate the use of MR for facilitating remote student-teacher piano lessons. Additionally, hand-tracking enables real-time, bi-directional visualization of both the student\u2019s and teacher\u2019s hand movements while playing, which is a crucial communication channel in piano instruction. We also present the Piano Theory Hub, an interactive area for independent learning and practice designed to compliment remote instructions. The Piano Theory Hub uses MR to provide spatial <jats:italic>in situ<\/jats:italic> explanations and exercises on notes, intervals, scales, and chords. User studies were conducted to evaluate both remote and solo aspects of the application. The first study with 10 participants revealed a strong sense of immersion and co-presence with the teacher during remote lessons. Trials also revealed that, in addition to virtual hands, visual highlighting of key-presses was found to improve visualisation of the remote users\u2019 play. The second study, with 15 participants found the Piano Theory Hub to be helpful for beginners and some intermediate players, with feedback suggesting improvements in user experience and highlighting the limitations of replacing traditional piano teachers. Our experiments demonstrate that MR can be successfully employed for the following strategies for piano education: hand interaction coaching, flexible virtual hand and piano visibility, augmented feedback including key highlighting, gamified learning elements, and flexible teacher positioning options. Overall, the findings suggest that mixed reality holds promise as an effective tool for remote piano learning and music education, offering immersive and engaging learning experiences.<\/jats:p>","DOI":"10.3389\/frvir.2024.1397154","type":"journal-article","created":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T04:53:57Z","timestamp":1730782437000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Mixed reality strategies for piano education"],"prefix":"10.3389","volume":"5","author":[{"given":"Vivienne","family":"Amm","sequence":"first","affiliation":[]},{"given":"Krishnan","family":"Chandran","sequence":"additional","affiliation":[]},{"given":"Lars","family":"Engeln","sequence":"additional","affiliation":[]},{"given":"Matthew","family":"McGinity","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,11,5]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1109\/mmul.2022.3232892","article-title":"Passthrough mixed reality with oculus quest 2: a case study on learning piano","volume":"30","author":"Banquiero","year":"2023","journal-title":"IEEE Multimed."},{"key":"B2","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1145\/3130859.3131336","article-title":"Keynvision: exploring piano pedagogy in mixed reality","volume-title":"Extended abstracts publication of the annual symposium on computer-human interaction in play","author":"Birhanu","year":"2017"},{"key":"B3","doi-asserted-by":"publisher","first-page":"73","DOI":"10.5555\/2525493.2525501","article-title":"Music education using augmented reality with a head mounted display","volume":"139","author":"Chow","year":"2013","journal-title":"Proc. Fourteenth Australas. User Interface Conference"},{"key":"B4","volume-title":"Teaching improvisation to piano students of elementary to intermediate levels","author":"Chyu","year":"2004"},{"key":"B5","article-title":"An expert system for teaching piano to novices","volume-title":"Icmc","author":"Dannenberg","year":"1990"},{"key":"B6","first-page":"511","article-title":"Music everywhere\u2013augmented reality piano improvisation learning system","volume-title":"Proceedings of the international conference on new interfaces for musical expression (NIME)","author":"Das","year":"2017"},{"key":"B7","doi-asserted-by":"publisher","first-page":"11","DOI":"10.34293\/education.v10is1-aug.5181","article-title":"A research on the design and use of colored notes for children in music education","volume":"10","author":"Demirel","year":"2022","journal-title":"Shanlax Int. J. Educ."},{"key":"B8","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1386\/jmte.10.1.5_1","article-title":"A new medium for remote music tuition","volume":"10","author":"Duffy","year":"2017","journal-title":"J. Music, Technol. and Educ."},{"key":"B10","first-page":"436","article-title":"Hand-by-hand mentor: an ar based training system for piano performance","author":"Guo","year":"2021"},{"key":"B11","first-page":"140","article-title":"Holokeys-an augmented reality application for learning the piano","volume-title":"Forum media technology","author":"Hackl","year":"2017"},{"key":"B12","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1093\/ml\/xiii.4.371","article-title":"Rhythmics and pianoforte improvisation","author":"Jaques-Dalcroze","year":"1932","journal-title":"Music and Lett."},{"key":"B13","doi-asserted-by":"publisher","first-page":"394","DOI":"10.1177\/0255761413489082","article-title":"A proposal of a color music notation system on a single melody for music beginners","volume":"31","author":"Kuo","year":"2013","journal-title":"Int. J. Music Educ."},{"key":"B14","volume-title":"Following the master\u2019s hands: Capturing piano performances for mixed reality piano learning applications","author":"Labrou","year":"2023"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.12738\/estp.2018.5.070","article-title":"Application of augmented reality technology in piano teaching system design","volume":"18","author":"Li","year":"2018","journal-title":"Educ. Sci. Theory and Pract."},{"key":"B16","doi-asserted-by":"publisher","first-page":"609028","DOI":"10.3389\/fpsyg.2021.609028","article-title":"Chinese pre-service music teachers\u2019 perceptions of augmented reality-assisted musical instrument learning","volume":"12","author":"Mei","year":"2021","journal-title":"Front. Psychol."},{"key":"B17","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1007\/s11042-020-09678-9","article-title":"A novel approach to learning music and piano based on mixed reality and gamification","volume":"80","author":"Molero","year":"2021","journal-title":"Multimedia Tools Appl."},{"key":"B18","first-page":"1","article-title":"Mixed reality piano tutor: a gamified piano practice environment","author":"Molloy","year":"2019"},{"key":"B9","first-page":"1791","article-title":"Conservatory voice department students\u2019 opinions on piano and accompaniment courses in The Covid-19","volume-title":"Gazi \u00dcniversitesi Gazi E\u011fitim Fak\u00fcltesi Dergisi","author":"Okan","year":"2022"},{"key":"B19","article-title":"Learning to create piano performances","volume-title":"NIPS 2017 workshop on machine learning for creativity and design","author":"Oore","year":"2017"},{"key":"B20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3544549.3582738","article-title":"Towards developing a virtual guitar instructor through biometrics informed human-computer interaction","author":"Rhodes","year":"2023","journal-title":"Ext. Abstr. 2023 CHI Conf. Hum. Factors Comput. Syst."},{"key":"B21","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1145\/3441000.3441039","article-title":"piarno-an augmented reality piano tutor","volume-title":"Proceedings of the 32nd Australian conference on human-computer interaction","author":"Rigby","year":"2020"},{"key":"B22","first-page":"149","article-title":"Piano: faster piano learning with interactive projection","author":"Rogers","year":"2014"},{"key":"B23","first-page":"194","article-title":"Enhanced learning of jazz chords with a projector based piano keyboard augmentation","author":"Sandnes","year":"2019"},{"key":"B24","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1162\/comj_a_00372","article-title":"Virtual reality musical instruments: state of the art, design principles, and future directions","volume":"40","author":"Serafin","year":"2016","journal-title":"Comput. Music J."},{"key":"B25","article-title":"Analysis of problems and reform methods in modern piano teaching","author":"Sha","year":"2019"},{"key":"B26","volume-title":"Mapping music: for faster learning and secure memory AR Editions, Inc","author":"Shockley","year":"2001"},{"key":"B27","first-page":"1","article-title":"Holokeys: interactive piano education using augmented reality and iot","author":"Stanbury","year":"2021"},{"key":"B28","doi-asserted-by":"publisher","first-page":"020140","DOI":"10.1063\/1.5055542","article-title":"Development of traditional musical instruments using augmented reality (AR) through mobile learning","volume":"2016","author":"Tan","year":"2018","journal-title":"AIP Conf. Proc."},{"key":"B29","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1177\/0305735608090845","article-title":"Musically untrained college students\u2019 interpretations of musical notation: sound, silence, loudness, duration, and temporal order","volume":"37","author":"Tan","year":"2009","journal-title":"Psychol. Music"},{"key":"B30","article-title":"Ppvr: teaching and learning to play the piano in virtual reality","author":"Wang","year":"2020"},{"key":"B31","doi-asserted-by":"crossref","DOI":"10.1145\/1935701.1935705","volume-title":"Mirrorfugue: communicating hand gesture in remote piano collaboration","author":"Xiao","year":"2010"},{"key":"B32","first-page":"37","article-title":"Andantino: teaching children piano with projected animated characters","author":"Xiao","year":"2016"},{"key":"B33","first-page":"629","article-title":"Andante: walking figures on the piano keyboard to visualize musical motion","volume-title":"Nime","author":"Xiao","year":"2014"}],"container-title":["Frontiers in Virtual Reality"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frvir.2024.1397154\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T04:54:04Z","timestamp":1730782444000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frvir.2024.1397154\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,5]]},"references-count":33,"alternative-id":["10.3389\/frvir.2024.1397154"],"URL":"https:\/\/doi.org\/10.3389\/frvir.2024.1397154","relation":{},"ISSN":["2673-4192"],"issn-type":[{"value":"2673-4192","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,5]]},"article-number":"1397154"}}