{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:22:40Z","timestamp":1760242960382,"version":"build-2065373602"},"reference-count":48,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2015,1,28]],"date-time":"2015-01-28T00:00:00Z","timestamp":1422403200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Advancements in input device and sensor technologies led to the evolution of the traditional human-machine interaction paradigm based on the mouse and keyboard. Touch-, gesture- and voice-based interfaces are integrated today in a variety of applications running on consumer devices (e.g., gaming consoles and smartphones). However, to allow existing applications running on desktop computers to utilize natural interaction, significant re-design and re-coding efforts may be required. In this paper, a framework designed to transparently add multi-modal interaction capabilities to applications to which users are accustomed is presented. Experimental observations confirmed the effectiveness of the proposed framework and led to a classification of those applications that could benefit more from the availability of natural interaction modalities.<\/jats:p>","DOI":"10.3390\/s150202832","type":"journal-article","created":{"date-parts":[[2015,1,28]],"date-time":"2015-01-28T09:54:57Z","timestamp":1422438897000},"page":"2832-2859","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Adding Pluggable and Personalized Natural Control Capabilities to Existing Applications"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7703-1372","authenticated-orcid":false,"given":"Fabrizio","family":"Lamberti","sequence":"first","affiliation":[{"name":"Dipartimento di Automatica e Informatica, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7916-1699","authenticated-orcid":false,"given":"Andrea","family":"Sanna","sequence":"additional","affiliation":[{"name":"Dipartimento di Automatica e Informatica, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gilles","family":"Carlevaris","sequence":"additional","affiliation":[{"name":"Dipartimento di Automatica e Informatica, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Claudio","family":"Demartini","sequence":"additional","affiliation":[{"name":"Dipartimento di Automatica e Informatica, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2015,1,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/MC.2013.352","article-title":"Gestural technology: Moving interfaces in a new direction","volume":"46","author":"Garber","year":"2013","journal-title":"IEEE Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1109\/MIC.2014.36","article-title":"The road to natural conversational speech interfaces","volume":"18","author":"Ortiz","year":"2014","journal-title":"IEEE Internet Comput."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1145\/965105.807503","article-title":"Put-That-There: Voice and gesture at the graphics interface","volume":"14","author":"Bolt","year":"1980","journal-title":"ACM Comput. Graph."},{"key":"ref_4","first-page":"1","article-title":"Application of speech technology in the automotive industry","volume":"10","author":"Cassford","year":"1998","journal-title":"IEEE Colloq. Speech Proc."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1145\/1409360.1409366","article-title":"Touching the future","volume":"51","author":"Selker","year":"2008","journal-title":"Commun. ACM"},{"key":"ref_6","first-page":"14","article-title":"Making sense of sensors","volume":"52","author":"Wright","year":"2009","journal-title":"Commun. ACM"},{"key":"ref_7","unstructured":"Raheja, J.L., Shyam, R., Kumar, U., and Prasad, P.B. (, January 9\u201311February). Real-time robotic hand control using hand gestures. , Bangalore, India."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1016\/j.ijhcs.2011.04.005","article-title":"I'm home: Defining and evaluating a gesture set for smart-home control","volume":"69","author":"Kuhnel","year":"2011","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.imavis.2010.08.006","article-title":"Real-time 3D pointing gesture recognition for mobile robots with cascade HMM and particle","volume":"29","author":"Park","year":"2011","journal-title":"Image Vis. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.knosys.2013.11.007","article-title":"A smart TV system with body-gesture control, tag-based rating and context-aware recommendation","volume":"56","author":"Lee","year":"2014","journal-title":"Knowl.-Based Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1129","DOI":"10.1109\/TCE.2012.6414977","article-title":"Single-camera dedicated television control system using gesture drawing","volume":"58","author":"Soonmook","year":"2012","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1016\/j.jvcir.2011.12.006","article-title":"Playing into the wild: A gesture-based interface for gaming in public spaces","volume":"23","author":"Roccetti","year":"2012","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1016\/j.apergo.2013.10.013","article-title":"Evaluation of the safety and usability of touch gestures in operating in-vehicle information systems with visual occlusion","volume":"45","author":"Kim","year":"2014","journal-title":"Appl. Ergon."},{"key":"ref_14","unstructured":"Paravati, G., Donna Bianco, M., Sanna, A., and Lamberti, F. (, January 1\u20133September). A multitouch solution to build personalized interfaces for the control of remote applications. Palma de Mallorca, Spain."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1016\/j.cag.2012.11.004","article-title":"Adapting user interfaces for gestural interaction with the Flexible Action and Articulated Skeleton Toolkit","volume":"37","author":"Suma","year":"2013","journal-title":"Comput. Graph."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, R.Y., Paris, S., and Popovic, J. (2011, January 16\u201319). 6D Hands: Markerless hand tracking for computer aided design. Santa Barbara, CA, USA.","DOI":"10.1145\/2047196.2047269"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lamberti, F., Sanna, A., Paravati, G., and Demartini, C. (2013, January 11\u201314). Endowing existing desktop applications with customizable body gesture-based interfaces. Las Vegas, NV, USA.","DOI":"10.1109\/ICCE.2013.6487017"},{"key":"ref_18","unstructured":"Microsoft Kinect. Available online: http:\/\/www.microsoft.com\/en-us\/kinectforwindows\/."},{"key":"ref_19","unstructured":"Leap Motion 3D Controller. Available online: https:\/\/www.leapmotion.com."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Estrany, B., Fuster, P., Garcia, A., and Luo, Y. (2008, January 21\u201322). Accurate interaction with computer by eye movement tracking. Seattle, WA, USA.","DOI":"10.1049\/cp:20081109"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1109\/TCE.2012.6227433","article-title":"Real-time eye gaze tracking for gaming design and consumer electronics systems","volume":"58","author":"Corcoran","year":"2012","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_22","unstructured":"Rohs, M., and Zweifel, P. (, January 8\u201313May). A conceptual framework for camera phone-based interaction techniques. Munich, Germany."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Pfleging, B., Schneegass, S., and Schmidt, A. (2012, January 17\u201319). Multimodal interaction in the car: combining speech and gestures on the sting wheel. Portsmouth, NH, USA.","DOI":"10.1145\/2390256.2390282"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.entcom.2012.11.001","article-title":"Evaluating a multi-player brain-computer interface game: Challenge versus co-experience","volume":"4","author":"Gurkok","year":"2013","journal-title":"Entertain. Comput."},{"key":"ref_25","unstructured":"Woods, E., Mason, P., and Billinghurst, M. (, January 11\u201314February). MagicMouse: An inexpensive 6-degree-of-freedom mouse. Melbourne, Victoria, Australia."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/TVCG.2010.91","article-title":"JanusVF: Accurate navigation using SCAAT and virtual fiducials","volume":"17","author":"Hutson","year":"2011","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1439","DOI":"10.1109\/TCE.2009.5278011","article-title":"Actual remote control: A universal remote control using hand motions on a virtual menu","volume":"55","author":"Lee","year":"2009","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zettlemoyer, L.S., and Amant, R.S. (1999, January 15\u201320). A visual medium for programmatic control of interactive applications. Pittsburgh, PA, USA.","DOI":"10.1145\/302979.303039"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/MCG.2008.70","article-title":"Extensible GUIs for remote application control on mobile devices","volume":"28","author":"Lamberti","year":"2008","journal-title":"IEEE Comput. Graph. Appl."},{"key":"ref_30","unstructured":"Bullard, V., Smith, K.T., and Daconta, M.C. (2001). Essential XUL Programming, John Wiley & Sons."},{"key":"ref_31","unstructured":"Generation of the description of a GUI in XUL format with the reported framework. Available online: http:\/\/youtu.be\/rB4orAPanVA."},{"key":"ref_32","unstructured":"Oikonomidis, I., Kyriazis, N., and Argyros, A.A. (September, January 29). Efficient model-based 3D tracking of hand articulations using Kinect. Dundee, United Kingdom."},{"key":"ref_33","unstructured":"Muller, M. (2007). Information Retrievel for Music and Motion, Springer-Verlag New York, Inc."},{"key":"ref_34","unstructured":"Kinect SDK Dynamic Time Warping (DTW) Gesture Recognition. Available online: http:\/\/kinectdtw.codeplex.com."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1145\/2524713.2524719","article-title":"More than a Mouse","volume":"56","author":"Savage","year":"2013","journal-title":"Commun. ACM"},{"key":"ref_36","unstructured":"Cortona3D Viewer. Available online: http:\/\/www.cortona3d.com\/."},{"key":"ref_37","unstructured":"Google Picasa. Available online: http:\/\/picasa.google.com."},{"key":"ref_38","unstructured":"Sample mapping of natural interactions onto the GUI of the Cortona3D Viewer application. Available online: http:\/\/youtu.be\/Bm7O75I7tDU."},{"key":"ref_39","unstructured":"3DUI 2010 Contest 3D model. Available online: http:\/\/conferences.computer.org\/3dui\/3dui2010\/."},{"key":"ref_40","unstructured":"First experiment: Mouse-keyboard interaction with Cortona3D Viewer. Available online: http:\/\/youtu.be\/4g0gFegPsXY."},{"key":"ref_41","unstructured":"First experiment: Hand pose\/gesture-based interaction with Cortona3D Viewer. Available online: http:\/\/youtu.be\/wo0-Kf_g_vA."},{"key":"ref_42","unstructured":"First experiment: Mouse-keyboard interaction with Google Picasa. Available online: http:\/\/youtu.be\/njSieU66CFs."},{"key":"ref_43","unstructured":"First experiment: Hand pose\/gesture-based interaction with Google Picasa. Available online: http:\/\/youtu.be\/syzjWuW-EXQ."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1145\/182987.584629","article-title":"Intuitive equals familiar","volume":"37","author":"Raskin","year":"1994","journal-title":"Commun. ACM"},{"key":"ref_45","unstructured":"(2012). ISO\/TC 159\/SC 4, ISO 9241-9:2000, Ergonomic requirements for office work with visual display terminals (VDTs)-Part 9: Requirements for non-keyboard input devices, International Organization for Standardization."},{"key":"ref_46","unstructured":"Second experiment: Mouse-keyboard interaction with Google Picasa. Available online: http:\/\/youtu.be\/ZjxvR31o_FQ."},{"key":"ref_47","unstructured":"Second experiment: Multi-modal interaction with Google Picasa. Available online: http:\/\/youtu.be\/KXXWsu2yMCY."},{"key":"ref_48","unstructured":"Myo armband by Thalmic Labs. Available online: https:\/\/www.thalmic.com\/en\/myo\/."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/2\/2832\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:42:03Z","timestamp":1760215323000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/2\/2832"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1,28]]},"references-count":48,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2015,2]]}},"alternative-id":["s150202832"],"URL":"https:\/\/doi.org\/10.3390\/s150202832","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2015,1,28]]}}}