{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T05:00:05Z","timestamp":1785301205554,"version":"3.55.0"},"reference-count":57,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T00:00:00Z","timestamp":1784851200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Brain Sci.","Brain Sciences"],"abstract":"<jats:p>Background: Modern household appliances often present complex interfaces that can be difficult to use, especially for older adults, people with visual impairments, and users with mild cognitive difficulties. In such cases, interacting with appliances may require sustained attention, visuospatial search, working memory, and sequential action planning, while traditional user manuals often provide limited contextual support. Methods: To address this issue, this study presents a proof-of-concept augmented reality (AR) and artificial intelligence (AI)-based system for contextual appliance guidance. The proposed architecture integrates visual sensing, deep learning, and large language models to detect appliance controls, interpret user queries, retrieve relevant information from user manuals, and provide step-by-step guidance directly on the real interface. A YOLOv8 model trained on a custom dataset was used for button detection, YOLO-Seg was employed to enhance visual highlighting through segmentation, and BoT-SORT was used to maintain detection consistency across frames. A Unity-based mobile application displayed real-time AR overlays with customizable visual settings for accessibility needs, such as low vision and color blindness that may be relevant for future accessibility-oriented applications. In addition to its technical pipeline, the system is conceptually relevant as a potential form of external cognitive support because it transforms static manual instructions into situated, sequential, and visually grounded guidance.Results: Experimental results showed promising technical performance for button detection and segmentation, while a preliminary user evaluation in a non-clinical sample suggested good usability, clarity, and acceptability of the interface. Conclusions: These findings support the technical feasibility and preliminary usability of the approach, while cognitive workload, confidence, functional autonomy, and clinical benefit were not directly measured.\u00a0Targeted validation in older adults, people with visual impairments, and clinical populations is therefore still needed. Future developments will include multimodal feedback, read-aloud guidance, and more specific evaluation of workload, confidence, and functional autonomy.<\/jats:p>","DOI":"10.3390\/brainsci16080783","type":"journal-article","created":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T14:08:14Z","timestamp":1785161294000},"page":"783","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["An Augmented Reality and AI-Based System for Contextual Appliance Guidance: Implications for Cognitive Accessibility and Assistive Interaction"],"prefix":"10.3390","volume":"16","author":[{"given":"Kimia","family":"Hafezi","sequence":"first","affiliation":[{"name":"Department of Computer, Control, and Management Engineering \u201cAntonio Ruberti\u201d, Sapienza University of Rome, 00185 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Atra Hossein","family":"Tafreshi","sequence":"additional","affiliation":[{"name":"Department of Computer, Control, and Management Engineering \u201cAntonio Ruberti\u201d, Sapienza University of Rome, 00185 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3336-5853","authenticated-orcid":false,"given":"Christian","family":"Napoli","sequence":"additional","affiliation":[{"name":"Department of Computer, Control, and Management Engineering \u201cAntonio Ruberti\u201d, Sapienza University of Rome, 00185 Rome, Italy"},{"name":"Department of Artificial Intelligence, Czestochowa University of Technology, ul. D\u0105browskiego 69, 42-201 Czestochowa, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5300-3561","authenticated-orcid":false,"given":"Cristian","family":"Randieri","sequence":"additional","affiliation":[{"name":"Department of Computer, Control, and Management Engineering \u201cAntonio Ruberti\u201d, Sapienza University of Rome, 00185 Rome, Italy"},{"name":"Department of Theoretical and Applied Sciences, eCampus University, Via Isimbardi 10, 22060 Novedrate, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1846-9996","authenticated-orcid":false,"given":"Samuele","family":"Russo","sequence":"additional","affiliation":[{"name":"Department of Theoretical and Applied Sciences, eCampus University, Via Isimbardi 10, 22060 Novedrate, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,24]]},"reference":[{"key":"ref_1","unstructured":"(2026, July 03). Bad Energy Habits Are Sweeping the World. PR Newswire, 2022. Industry Survey Conducted with OnePoll. Available online: https:\/\/www.prnewswire.com\/news-releases\/bad-energy-habits-are-sweeping-the-world-301701761.html."},{"key":"ref_2","unstructured":"Alzheimer\u2019s Association (2024). 2024 Alzheimer\u2019s Disease Facts and Figures. Alzheimer\u2019s Dement., 20, 700\u2013789."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1016\/j.cger.2013.07.002","article-title":"Normal Cognitive Aging","volume":"29","author":"Harada","year":"2013","journal-title":"Clin. Geriatr. Med."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Novick, D.G., and Ward, K. (2006). Why don\u2019t people read the manual?. Proceedings of the 24th Annual ACM International Conference on Design of Communication, New York, NY, USA, ACM.","DOI":"10.1145\/1166324.1166329"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liu, S., Wang, S., and Sun, K. (2023). Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional Videos. arXiv.","DOI":"10.1145\/3660245"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Keshav Kollaa, S.S.V., Sanchez, A., and Plapper, P. (2021). Comparing effectiveness of paper-based and Augmented Reality instructions for manual assembly and training tasks. Proceedings of the CIRP Life Cycle Engineering (LCE) Conference 2021, Elsevier.","DOI":"10.2139\/ssrn.3859970"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research","volume":"Volume 52","author":"Hancock","year":"1988","journal-title":"Human Mental Workload"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Jordan, P., Thomas, B., Weerdmeester, B., and McClelland, I. (1996). SUS: A \u201cquick and dirty\u201d usability scale. Usability Evaluation in Industry, Taylor & Francis.","DOI":"10.1201\/9781498710411"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Murthy, S.V., Simoski, A., Rawashdeh, S., and Kim, H. (2024). Augmented Reality User Manual for Assembly Tasks. Proceedings of the 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), IEEE.","DOI":"10.1109\/VRW62533.2024.00056"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"16779","DOI":"10.1038\/s41598-023-42656-3","article-title":"Development of a Mobile-Based Intelligent Module for Identification and Tracking of Household Appliances","volume":"13","author":"Kaplun","year":"2023","journal-title":"Sci. Rep."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"DeTone, D., Malisiewicz, T., and Rabinovich, A. (2018). SuperPoint: Self-Supervised Interest Point Detection and Description. arXiv.","DOI":"10.1109\/CVPRW.2018.00060"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Sarlin, P.E., DeTone, D., Malisiewicz, T., and Rabinovich, A. (2020). SuperGlue: Learning Feature Matching with Graph Neural Networks. arXiv.","DOI":"10.1109\/CVPR42600.2020.00499"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tschakert, H., Lang, F., Wieland, M., Schmidt, A., and Machulla, T.K. (2023). A Dataset and Machine Learning Approach to Classify and Augment Interface Elements of Household Appliances to Support People with Visual Impairment. Proceedings of the 28th International Conference on Intelligent User Interfaces, Sydney, Australia, ACM.","DOI":"10.1145\/3581641.3584038"},{"key":"ref_14","unstructured":"An, H., Kim, H., Hong, S., and Shin, Y. (2024). A Study of Digital Appliances Accessibility for People with Visual Disabilities. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Lee, J.H., Kim, Y.M., Rhiu, I., and Yun, M.H. (2021). A Persona-Based Approach for Identifying Accessibility Issues in Elderly and Disabled Users\u2019 Interaction with Home Appliances. Appl. Sci., 11.","DOI":"10.3390\/app11010368"},{"key":"ref_16","first-page":"243","article-title":"Blind Assistance System for Appliance Control and Public Transport Safety Using CNN, MobileNet V2 and YOLOv8","volume":"20","author":"Srividhya","year":"2024","journal-title":"Int. J. Hybrid Intell. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Estrada, J., Paheding, S., Yang, X., and Niyaz, Q. (2022). Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training. Appl. Sci., 12.","DOI":"10.3390\/app12105159"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Liang, H., Liu, J., Du, W., Jin, D., Lei, W., Wen, Z., and Lv, J. (2023). Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2023.findings-acl.671"},{"key":"ref_19","unstructured":"Corporation, I. (2025, January 02). Computer Vision Annotation Tool (CVAT), Version 2.0. Available online: https:\/\/cvat.org."},{"key":"ref_20","unstructured":"(2025, January 15). Ultralytics YOLOv8. Available online: https:\/\/github.com\/ultralytics\/ultralytics."},{"key":"ref_21","unstructured":"(2025, January 15). YOLOv5 by Ultralytics. Available online: https:\/\/github.com\/ultralytics\/yolov5."},{"key":"ref_22","unstructured":"Yaseen, M. (2024). What is YOLOv9: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mirzaei, B., Nezamabadi-Pour, H., Raoof, A., and Derakhshani, R. (2023). Small Object Detection and Tracking: A Comprehensive Review. Sensors, 23.","DOI":"10.3390\/s23156887"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kisantal, M., Wojna, Z., Murawski, J., Naruniec, J., and Cho, K. (2019). Augmentation for Small Object Detection. arXiv.","DOI":"10.5121\/csit.2019.91713"},{"key":"ref_25","unstructured":"Zhang, H., Zhang, S., and Zou, R. (2024). Select-Mosaic: Data Augmentation Method for Dense Small Object Scenes. arXiv."},{"key":"ref_26","unstructured":"Team, R. (2025, February 05). Roboflow: The Universal Computer Vision Platform. Available online: https:\/\/roboflow.com."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2018). Mask R-CNN. arXiv.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_28","unstructured":"(2025, January 18). Detectron2. Available online: https:\/\/github.com\/facebookresearch\/detectron2."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv.","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"ref_30","unstructured":"Aharon, N., Orfaig, R., and Bobrovsky, B.-Z. (2022). BoT-SORT: Robust Associations Multi-Object Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE."},{"key":"ref_31","unstructured":"AI, T. (2025, February 05). Norfair: Lightweight Tracking Library. GitHub Repository. Available online: https:\/\/github.com\/tryolabs\/norfair."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., and Wang, X. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. arXiv.","DOI":"10.1007\/978-3-031-20047-2_1"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wojke, N., Bewley, A., and Paulus, D. (2017). Simple Online and Realtime Tracking with a Deep Association Metric. Proceedings of the IEEE International Conference on Image Processing (ICIP), IEEE.","DOI":"10.1109\/ICIP.2017.8296962"},{"key":"ref_34","unstructured":"Zhang, Y., Wang, C., Wang, X., Zeng, W., and Liu, W. (2020). FairMOT: On the Fairness of Detection and Re-Identification in Multi-Object Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE."},{"key":"ref_35","unstructured":"(2025, February 10). Gemini: Google\u2019s Next-Generation AI Model. Available online: https:\/\/deepmind.google\/technologies\/gemini\/."},{"key":"ref_36","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models from Natural Language Supervision. arXiv."},{"key":"ref_37","unstructured":"Li, J., Selvaraju, R.R., Gotmare, A.D., Joty, S., Xiong, C., and Hoi, S.C. (2022). BLIP: Bootstrapped Language-Image Pretraining for Unified Vision-Language Understanding and Generation. arXiv."},{"key":"ref_38","first-page":"120","article-title":"The OpenCV Library","volume":"25","author":"Bradski","year":"2000","journal-title":"Dr. Dobb\u2019s J. Softw. Tools"},{"key":"ref_39","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_40","first-page":"5485","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","unstructured":"(2025, January 13). LaMDA: Language Models for Dialog Applications. Available online: https:\/\/blog.google\/technology\/ai\/lamda\/."},{"key":"ref_42","unstructured":"Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv."},{"key":"ref_43","unstructured":"OpenAI (2025, January 13). GPT-4 Technical Report. Available online: https:\/\/openai.com\/research\/gpt-4."},{"key":"ref_44","unstructured":"(2025, January 13). Anthropic. Claude: Constitutional AI and Safer Alignment. Available online: https:\/\/www.anthropic.com\/index\/claude."},{"key":"ref_45","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv."},{"key":"ref_46","unstructured":"Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., and Wang, S. (2024). Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context. arXiv."},{"key":"ref_47","unstructured":"Niantic, I. (2025, February 20). Lightship SDK. Online Documentation. Available online: https:\/\/lightship.dev\/docs\/ardk\/."},{"key":"ref_48","unstructured":"Technologies, U. (2025, February 20). Unity Sentis: AI Inference in Unity. Unity Documentation, 2024. Available online: https:\/\/docs.unity.cn\/Packages\/com.unity.sentis@2.1\/manual\/index.html."},{"key":"ref_49","unstructured":"Technologies, U. (2025, February 20). Unity Barracuda: Neural Network Inference for Unity. GitHub Repository. Available online: https:\/\/github.com\/Unity-Technologies\/barracuda-release."},{"key":"ref_50","unstructured":"Contributors, Z. (2025, February 10). ZXing: Multi-Format 1D\/2D Barcode Image Processing Library. Available online: https:\/\/github.com\/zxing\/zxing."},{"key":"ref_51","unstructured":"World Wide Web Consortium (W3C) (2025, February 20). Web Content Accessibility Guidelines (WCAG) 2.1. Available online: https:\/\/www.w3.org\/TR\/WCAG21\/."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1038\/nmeth.1618","article-title":"Color Blindness","volume":"8","author":"Wong","year":"2011","journal-title":"Nat. Methods"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Iqbal, M.W., Shahzad, S.K., Ahmad, N., Amelio, A., and Brodic, D. (2018). Adaptive Interface for Color-Blind People in Mobile Phones. Proceedings of the 2018 International Conference on Advancements in Computational Sciences (ICACS), IEEE.","DOI":"10.1109\/ICACS.2018.8333488"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Russo, S., Tibermacine, I.E., Randieri, C., Rabehi, A., Alharbi, A.H., El-kenawy, E.S.M., and Napoli, C. (2025). Exploiting facial emotion recognition system for ambient assisted living technologies triggered by interpreting the user\u2019s emotional state. Front. Neurosci., 19.","DOI":"10.3389\/fnins.2025.1622194"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Brata, K.C., Funabiki, N., Riyantoko, P.A., Panduman, Y.Y.F., and Mentari, M. (2024). Performance Investigations of VSLAM and Google Street View Integration in Outdoor Location-Based Augmented Reality under Various Lighting Conditions. Electronics, 13.","DOI":"10.3390\/electronics13152930"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"112908","DOI":"10.1016\/j.engappai.2025.112908","article-title":"Current trends and future directions in eye tracking technology: A literature review","volume":"163","author":"Fiani","year":"2026","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Randieri, C., Pollina, A., Puglisi, A., and Napoli, C. (2025). Smart Glove: A Cost-Effective and Intuitive Interface for Advanced Drone Control. Drones, 9.","DOI":"10.3390\/drones9020109"}],"container-title":["Brain Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3425\/16\/8\/783\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T04:15:43Z","timestamp":1785298543000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3425\/16\/8\/783"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,24]]},"references-count":57,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["brainsci16080783"],"URL":"https:\/\/doi.org\/10.3390\/brainsci16080783","relation":{},"ISSN":["2076-3425"],"issn-type":[{"value":"2076-3425","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,24]]}}}