{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T15:53:06Z","timestamp":1774453986282,"version":"3.50.1"},"reference-count":28,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,10,28]],"date-time":"2022-10-28T00:00:00Z","timestamp":1666915200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science and Technology Council (NSTC), Taiwan","award":["110-2221-E-259-016"],"award-info":[{"award-number":["110-2221-E-259-016"]}]},{"name":"National Science and Technology Council (NSTC), Taiwan","award":["111-2221-E-259-012"],"award-info":[{"award-number":["111-2221-E-259-012"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Cooking at home is a critical survival skill. We propose a new cooking assistance system in which a user only needs to wear an all-in-one augmented reality (AR) headset without having to install any external sensors or devices in the kitchen. Utilizing the built-in camera and cutting-edge computer vision (CV) technology, the user can direct the AR headset to recognize available food ingredients by simply looking at them. Based on the types of the recognized food ingredients, suitable recipes are suggested accordingly. A step-by-step video tutorial providing details of the selected recipe is then displayed with the AR glasses. The user can conveniently interact with the proposed system using eight kinds of natural hand gestures without needing to touch any devices throughout the entire cooking process. Compared with the deep learning models ResNet and ResNeXt, experimental results show that the YOLOv5 achieves lower accuracy for ingredient recognition, but it can locate and classify multiple ingredients in one shot and make the scanning process easier for users. Twenty participants test the prototype system and provide feedback via two questionnaires. Based on the analysis results, 19 of the 20 participants would recommend others to use the proposed system, and all participants are overall satisfied with the prototype system.<\/jats:p>","DOI":"10.3390\/s22218290","type":"journal-article","created":{"date-parts":[[2022,10,30]],"date-time":"2022-10-30T10:47:57Z","timestamp":1667126877000},"page":"8290","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Augmented Reality Based Interactive Cooking Guide"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8747-0253","authenticated-orcid":false,"given":"Isaias","family":"Majil","sequence":"first","affiliation":[{"name":"Department of Computer Science & Information Engineering, National Dong Hwa University, Hualien 974301, Taiwan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1859-6730","authenticated-orcid":false,"given":"Mau-Tsuen","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Computer Science & Information Engineering, National Dong Hwa University, Hualien 974301, Taiwan"}]},{"given":"Sophia","family":"Yang","sequence":"additional","affiliation":[{"name":"Interdisciplinary Program of Electrical Engineering & Computer Science, National Tsing-Hua University, Hsinchu 300044, Taiwan"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,28]]},"reference":[{"key":"ref_1","unstructured":"Magic Leap (2022, September 01). Get Started with Unity. Available online: https:\/\/ml1-developer.magicleap.com\/en-us\/learn\/guides\/unity-overview."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"012007","DOI":"10.1088\/1757-899X\/384\/1\/012007","article-title":"Automation and Monitoring Smart Kitchen Based on Internet of Things (IoT)","volume":"384","author":"Nugroho","year":"2018","journal-title":"IOP Conf. Ser. Mater. Sci. Eng."},{"key":"ref_3","unstructured":"Hashimoto, A., Mori, N., Funatomi, T., Yamakata, Y., Kakusho, K., and Minoh, M. (2008, January 15\u201319). Smart kitchen: A user centric cooking support system. Proceedings of the International Conference on Information Processing and Management of Uncertainty (IPMU), Montpellier, France."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1469","DOI":"10.1016\/j.procs.2020.09.157","article-title":"Eye and Voice Control for an Augmented Reality Cooking Experience","volume":"176","author":"Iftene","year":"2020","journal-title":"Procedia Comput. Sci."},{"key":"ref_5","first-page":"45","article-title":"CounterIntelligence: Augmented Reality Kitchen","volume":"2239","author":"Lee","year":"2005","journal-title":"Comput. Hum. Interact. (CHI)"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Stander, M., Hadjakos, A., Lochschmidt, N., Klos, C., Renner, B., and Muhlhauser, M. (2012, January 10\u201312). A Smart Kitchen Infrastructure. Proceedings of the 2012 IEEE International Symposium on Multimedia, Irvine, CA, USA.","DOI":"10.1109\/ISM.2012.27"},{"key":"ref_7","first-page":"378","article-title":"Smart Kitchen System using IOT","volume":"04","author":"Palandurkar","year":"2020","journal-title":"Int. J. Eng. Appl. Sci. Technol. (IJEAST)"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hassan, C.A.U., Iqbal, J., Khan, M.S., Hussain, S., Akhunzada, A., Ali, M., Gani, A., Uddin, M., and Ullah, S.S. (2022). Design and Implementation of Real-Time Kitchen Monitoring and Automation System Based on Internet of Things. Energies, 15.","DOI":"10.3390\/en15186778"},{"key":"ref_9","first-page":"13","article-title":"IOT based Smart Kitchen","volume":"7","author":"Sundarapandiyan","year":"2019","journal-title":"Int. J. Comput. Sci. Trends Technol. (IJCST)"},{"key":"ref_10","unstructured":"Logeshwaran, M., and Sheela, J. (2022, January 29\u201331). Designing an IoT based Kitchen Monitoring and Automation System for Gas and Fire Detection. Proceedings of the International Conference on Computing Methodologies and Communication, Erode, India."},{"key":"ref_11","unstructured":"Watts, D. (2022, September 01). How Smart Kitchens Are Improving Our Lives. The AI Journal, Available online: https:\/\/aijourn.com\/how-smart-kitchens-are-improving-our-lives\/."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Hasada, H., Zhang, J., Yamamoto, K., Ryskeldiev, B., and Ochiai, Y. (2019). AR Cooking: Comparing Display Methods for the Instructions of Cookwares on AR Goggles. International Conference on Human-Computer Interaction, Springer.","DOI":"10.1007\/978-3-030-22649-7_11"},{"key":"ref_13","unstructured":"(2022, September 01). Microsoft HoloLens. Available online: https:\/\/www.microsoft.com\/zh-tw\/hololens."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhai, K., Cao, Y., Hou, W., and Li, X. (2020). Interactive Mixed Reality Cooking Assistant for Unskilled Operating Scenario. HCI International Conference, Lecture Notes in Computer Science, Springer.","DOI":"10.1007\/978-3-030-49698-2_13"},{"key":"ref_15","unstructured":"Reisinho, P., Silva, C., Vairinhos, M., Oliveira, A., and Zagalo, N. (August, January 4\u20136). Tangible Interfaces and Augmented Reality in a Nutrition Serious Game for Kids. Proceedings of the IEEE International Conference on Serious Games and Applications for Health, Dubai, United Arab Emirates."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ricci, M., Scarcelli, A., Introno, A.D., Strippoli, V., Cariati, S., and Fiorentino, M. (2022). A Human-Centred Design Approach for Designing Augmented Reality Enabled Interactive Systems: A Kitchen Machine Case Study. Advances on Mechanics, Design Engineering and Manufacturing IV, Springer.","DOI":"10.1007\/978-3-031-15928-2_123"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"216","DOI":"10.3390\/digital1040016","article-title":"Augmented Reality in Food Promotion and Analysis: Review and Potentials","volume":"1","author":"Styliaras","year":"2021","journal-title":"Digital"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1016\/j.tifs.2022.04.021","article-title":"Augmented\/Mixed Reality Technologies for Food: A Review","volume":"124","author":"Chai","year":"2022","journal-title":"Trends Food Sci. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Balaji, A., Sathyasri, B., Vanaja, S., Manasa, M.N., Malavega, M., and Maheswari, S. (2020, January 10\u201312). Smart Kitchen Wardrobe System Based on IoT. Proceedings of the International Conference on Smart Electronics and Communication, Trichy, Tamilnadu, India.","DOI":"10.1109\/ICOSEC49089.2020.9215459"},{"key":"ref_20","unstructured":"Dormehl, L. (2022, September 01). Samsung\u2019s New Food A.I. Can Suggest Recipes Based on What\u2019s in Your Fridge. Digital Trends, Available online: https:\/\/www.digitaltrends.com\/home\/samsung-fridge-ai-suggest-recipes-ces-2020\/."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You Only Look Once, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_22","unstructured":"Jocher, G., Stoken, A., Chaurasia, A., Borovec, J., Kwon, Y., Michael, K., Liu, C., Fang, J., Abhiram, V., and Skalski, S.P. (2021). YOLOv5n \u2018Nano\u2019 models. Zenodo."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wang, C., Liao, H., Wu, Y., Chen, P., Hsieh, J., and Yeh, I. (2020, January 13\u201319). CSPNet: A New Backbone That Can Enhance Learning Capability of CNN. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00203"},{"key":"ref_24","unstructured":"Wang, K., Liew, J., Zou, Y., Zhou, D., and Feng, J. (November, January 27). PANet: Few-shot Image Semantic Segmentation with Prototype Alignment. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_25","unstructured":"(2022, September 01). Q-100, Ingredients-Classification. GitHub Repository. Available online: https:\/\/github.com\/Q-100\/ingredients-classification?fbclid=IwAR2_Qu5XRjKFV_FerUzu7Ubqm_GWLX3KoHLTSXOQHGkvgGCFNeYsXhDnIDc."},{"key":"ref_26","unstructured":"(2022, September 01). OpenCV for Unity, Unity Asset Store. Available online: https:\/\/assetstore.unity.com\/packages\/tools\/integration\/opencv-for-unity-21088."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Xie, S., Girshick, R., Dollar, P., Tu, Z., and He, K. (2017, January 21\u201326). Aggregated Residual Transformations for Deep Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.634"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8290\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:05:25Z","timestamp":1760144725000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8290"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,28]]},"references-count":28,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22218290"],"URL":"https:\/\/doi.org\/10.3390\/s22218290","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,28]]}}}