{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T10:37:32Z","timestamp":1760524652554,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human\u2013computer interaction (HCI) with screens through gestures is a pivotal method amidst the digitalization trend. In this work, a gesture recognition method is proposed that combines multi-band spectral features with spatial characteristics of screen-reflected light. Based on the method, a red-green-blue (RGB) three-channel spectral gesture recognition system has been developed, composed of a display screen integrated with narrowband spectral receivers as the hardware setup. During system operation, emitted light from the screen is reflected by gestures and received by the narrowband spectral receivers. These receivers at various locations are tasked with capturing multiple narrowband spectra and converting them into light-intensity series. The availability of multi-narrowband spectral data integrates multidimensional features from frequency and spatial domains, enhancing classification capabilities. Based on the RGB three-channel spectral features, this work formulates an RGB multi-channel convolutional neural network long short-term memory (CNN-LSTM) gesture recognition model. It achieves accuracies of 99.93% in darkness and 99.89% in illuminated conditions. This indicates the system\u2019s capability for stable operation across different lighting conditions and accurate interaction. The intelligent gesture recognition method can be widely applied for interactive purposes on various screens such as computers and mobile phones, facilitating more convenient and precise HCI.<\/jats:p>","DOI":"10.3390\/s24175519","type":"journal-article","created":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T07:53:37Z","timestamp":1724658817000},"page":"5519","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Intelligent Gesture Recognition Based on Screen Reflectance Multi-Band Spectral Features"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8087-4856","authenticated-orcid":false,"given":"Peiying","family":"Lin","sequence":"first","affiliation":[{"name":"School of Electrical and Information Engineering, Jiangsu University of Science and Technology, Zhangjiagang 215600, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenrui","family":"Li","sequence":"additional","affiliation":[{"name":"Laboratory of Applied Research on Electromagnetics, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sijie","family":"Chen","sequence":"additional","affiliation":[{"name":"Laboratory of Applied Research on Electromagnetics, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiangtao","family":"Huangfu","sequence":"additional","affiliation":[{"name":"Laboratory of Applied Research on Electromagnetics, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6660-593X","authenticated-orcid":false,"given":"Wei","family":"Yuan","sequence":"additional","affiliation":[{"name":"School of Electrical and Information Engineering, Jiangsu University of Science and Technology, Zhangjiagang 215600, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,26]]},"reference":[{"key":"ref_1","unstructured":"Vrana, J., and Singh, R. (2022). Handbook of Nondestructive Evaluation 4.0, Springer International Publishing."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Hewett, T., Baecker, R., Card, S., Carey, T., Gasen, J., Mantei, M., Perlman, G., Strong, G., and Verplank, W. (1992). ACM SIGCHI Curricula for Human-Computer Interaction, ACM Press.","DOI":"10.1145\/2594128"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mourtzis, D., Angelopoulos, J., and Panopoulos, N. (2023). The future of the human\u2013machine interface (HMI) in society 5.0. Future Internet, 15.","DOI":"10.3390\/fi15050162"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1182","DOI":"10.1109\/TVCG.2020.3030460","article-title":"Personal augmented reality for information visualization on large interactive displays","volume":"27","author":"Reipschlager","year":"2021","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_5","first-page":"39","article-title":"Hand movements using keyboard and mouse","volume":"996","author":"Biele","year":"2022","journal-title":"Hum. Mov. Hum.-Comput. Interact."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1080\/10696679.2022.2158872","article-title":"Touch or click? The effect of direct and indirect human-computer interaction on consumer responses","volume":"32","author":"Wu","year":"2023","journal-title":"J. Mark. Theory Pract."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2576099","article-title":"Up close and personal: Collaborative work on a high-resolution multitouch wall display","volume":"21","author":"Jakobsen","year":"2014","journal-title":"ACM Trans. Comput.-Hum. Interact."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Nunes, J.S., Castro, N., Gon\u00e7alves, S., Pereira, N., Correia, V., and Lanceros-Mendez, S. (2017). Marked object recognition multitouch screen printed touchpad for interactive applications. Sensors, 17.","DOI":"10.3390\/s17122786"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1936","DOI":"10.1109\/TVCG.2016.2592906","article-title":"Evaluating multi-user selection for exploring graph topology on wall-displays","volume":"23","author":"Prouzeau","year":"2016","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Huang, Z., and Huang, X. (2018, January 20\u201321). A study on the application of voice interaction in automotive human machine interface experience design. Proceedings of the AIP Conference, Xi\u2019an, China.","DOI":"10.1063\/1.5033738"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1967","DOI":"10.3906\/elk-1710-128","article-title":"User interaction in hands-free gaming: A comparative study of gaze-voice and touchscreen interface control","volume":"26","year":"2018","journal-title":"Turk. J. Electr. Eng. Comput. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Gao, L., Liu, Y., Le, J., and Liu, R. (2023, January 11\u201313). Research on the application of multi-channel interaction in information system. Proceedings of the 2nd International Conference on Robotics, Artificial Intelligence and Intelligent Control (RAIIC), Mianyang, China.","DOI":"10.1109\/RAIIC59453.2023.10280893"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1939","DOI":"10.1177\/09544054211014492","article-title":"Environmental effects on reliability and accuracy of MFCC based voice recognition for industrial human-robot-interaction","volume":"235","author":"Birch","year":"2021","journal-title":"Proc. Inst. Mech. Eng. B J. Eng. Manuf."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"35089","DOI":"10.1109\/ACCESS.2023.3264260","article-title":"Modified earthworm optimization with deep learning assisted emotion recognition for human computer interface","volume":"11","author":"Alrowais","year":"2023","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Pereira, R., Mendes, C., Ribeiro, J., Ribeiro, R., Miragaia, R., Rodrigues, N., Costa, N., and Pereira, A. (2024). Systematic review of emotion detection with computer vision and deep learning. Sensors, 24.","DOI":"10.3390\/s24113484"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Aghajanzadeh, S., Naidu, R., Chen, S.H., Tung, C., Goel, A., Lu, Y.H., and Thiruvathukal, G.K. (2020, January 25\u201328). Camera placement meeting restrictions of computer vision. Proceedings of the IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/ICIP40778.2020.9190851"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Harshitaa, A., Hansini, P., and Asha, P. (2021, January 4\u20136). Gesture based home appliance control system for disabled people. Proceedings of the Second International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India.","DOI":"10.1109\/ICESC51422.2021.9532973"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"209","DOI":"10.5194\/isprs-archives-XLVIII-2-W3-2023-209-2023","article-title":"Cross-language transfer learning using visual information for automatic sign gesture recognition","volume":"48","author":"Ryumin","year":"2023","journal-title":"Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci."},{"key":"ref_19","first-page":"200262","article-title":"Camera-based interactive wall display using hand gesture recognition","volume":"19","author":"Zahra","year":"2023","journal-title":"Intell. Syst. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Benitez-Garcia, G., Prudente-Tixteco, L., Castro-Madrid, L.C., Toscano-Medina, R., Olivares-Mercado, J., Sanchez-Perez, G., and Villalba, L.J.G. (2021). Improving real-time hand gesture recognition with semantic segmentation. Sensors, 21.","DOI":"10.3390\/s21020356"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"31481","DOI":"10.1109\/ACCESS.2020.2973305","article-title":"HCI on the table: Robust gesture recognition using acoustic sensing in your hand","volume":"8","author":"Luo","year":"2020","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/LSENS.2018.2882642","article-title":"Robust gesture recognition using millimetric-wave radar system","volume":"2","author":"Hazra","year":"2018","journal-title":"IEEE Sens. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Cheng, Y.L., Yeh, W., and Liao, Y.P. (2024). The implementation of a gesture recognition system with a millimeter wave and thermal imager. Sensors, 24.","DOI":"10.3390\/s24020581"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand gesture recognition based on computer vision: A review of techniques. J. Imaging, 6.","DOI":"10.3390\/jimaging6080073"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Galv\u00e1n-Ruiz, J., Travieso-Gonz\u00e1lez, C.M., Tejera-Fettmilch, A., Pinan-Roescher, A., Esteban-Hern\u00e1ndez, L., and Dom\u00ednguez-Quintana, L. (2020). Perspective and evolution of gesture recognition for sign language: A review. Sensors, 20.","DOI":"10.3390\/s20123571"},{"key":"ref_26","first-page":"137","article-title":"A combined method of skin-and depth-based hand gesture recognition","volume":"17","author":"Sokhib","year":"2020","journal-title":"Int. Arab J. Inf. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xu, J., Li, J., Zhang, S., Xie, C., and Dong, J. (2020, January 7\u20139). Skeleton guided conflict-free hand gesture recognition for robot control. Proceedings of the 11th International Conference on Awareness Science and Technology (iCAST), Qingdao, China.","DOI":"10.1109\/iCAST51195.2020.9319483"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3554922","article-title":"Ghosm: Graph-based hybrid outline and skeleton modelling for shape recognition","volume":"19","author":"Alwaely","year":"2023","journal-title":"ACM Trans. Multim. Comput. Commun. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2446","DOI":"10.1109\/TETCI.2024.3363071","article-title":"Spatio-temporal fusion spiking neural network for frame-based and event-based camera sensor fusion","volume":"8","author":"Qiao","year":"2024","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ryumin, D., Ivanko, D., and Ryumina, E. (2023). Audio-visual speech and gesture recognition by sensors of mobile devices. Sensors, 23.","DOI":"10.3390\/s23042284"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hakim, N.L., Shih, T.K., Kasthuri Arachchi, S.P., Aditya, W., Chen, Y.C., and Lin, C.Y. (2019). Dynamic hand gesture recognition using 3DCNN and LSTM with FSM context-aware model. Sensors, 19.","DOI":"10.3390\/s19245429"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1049\/iet-ipr.2019.0230","article-title":"Depth data and fusion of feature descriptors for static gesture recognition","volume":"14","author":"Sharma","year":"2020","journal-title":"IET Image Process."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zengeler, N., Kopinski, T., and Handmann, U. (2019). Hand gesture recognition in automotive human\u2013machine interaction using depth cameras. Sensors, 19.","DOI":"10.3390\/s19010059"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yu, J., Qin, M., and Zhou, S. (2022). Dynamic gesture recognition based on 2D convolutional neural network and feature fusion. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-08133-z"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7\u201312). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"121811","DOI":"10.1109\/ACCESS.2019.2946404","article-title":"A new precise contactless medical image multimodal interaction system for surgical practice","volume":"8","author":"Hui","year":"2020","journal-title":"IEEE Access"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Safavi, S.M., Sundaram, S.M., Heydarigorji, A., Udaiwal, N.S., and Chou, P.H. (2017, January 11\u201315). Application of infrared scanning of the neck muscles to control a cursor in human-computer interface. Proceedings of the 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Republic of Korea.","DOI":"10.1109\/EMBC.2017.8036942"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Singh, J., and Raza, U. (2020, January 21). Passive visible light positioning systems: An overview. Proceedings of the Workshop on Light Up the IoT, London, UK.","DOI":"10.1145\/3412449.3412553"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"46444","DOI":"10.1109\/ACCESS.2024.3382757","article-title":"LEDPOS: Indoor visible light positioning based on LED as sensor and machine learning","volume":"12","author":"Fragner","year":"2024","journal-title":"IEEE Access"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2047","DOI":"10.1109\/COMST.2015.2476474","article-title":"Visible light communication, networking, and sensing: A survey, potential and challenges","volume":"17","author":"Pathak","year":"2015","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1109\/TWC.2020.3023930","article-title":"Telling secrets in the light: An efficient key extraction mechanism via ambient light","volume":"20","author":"Lu","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_42","unstructured":"Liao, Z., Luo, Z., Huang, Q., Zhang, L., Wu, F., Zhang, Q., and Wang, Y. (February, January 31). SMART: Screen-based gesture recognition on commodity mobile devices. Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, New Orleans, LA, USA."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"24439","DOI":"10.1109\/JSEN.2022.3219645","article-title":"LED screen-based intelligent hand gesture recognition system","volume":"22","author":"Lin","year":"2022","journal-title":"IEEE Sens. J."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Jogin, M., Madhulika, M.S., Divya, G.D., Meghana, R.K., and Apoorva, S. (2018, January 18\u201319). Feature extraction using convolution neural networks (CNN) and deep learning. Proceedings of the 3rd IEEE International Conference on Recent Trends in Electronics, Information and Communication Technology (RTEICT), Bangalore, India.","DOI":"10.1109\/RTEICT42901.2018.9012507"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"132306","DOI":"10.1016\/j.physd.2019.132306","article-title":"Fundamentals of recurrent neural network and long short-term memory network","volume":"404","author":"Sherstinsky","year":"2020","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1162\/089976600300015015","article-title":"Learning to forget: Continual prediction with LSTM","volume":"12","author":"Gers","year":"2000","journal-title":"Neural Comput."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"4961","DOI":"10.1007\/s10489-021-02635-5","article-title":"Confidence interval for micro-averaged F1 and macro-averaged F1 scores","volume":"52","author":"Takahashi","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Sokolova, M., Japkowicz, N., and Szpakowicz, S. (2006, January 4\u20138). Beyond accuracy, F-score and ROC: A family of discriminant measures for performance evaluation. Proceedings of the 19th Australasian Joint Conference on Artificial Intelligence, Berlin, Germany.","DOI":"10.1007\/11941439_114"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/17\/5519\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:42:55Z","timestamp":1760110975000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/17\/5519"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,26]]},"references-count":49,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["s24175519"],"URL":"https:\/\/doi.org\/10.3390\/s24175519","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2024,8,26]]}}}