{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T06:12:01Z","timestamp":1776319921837,"version":"3.50.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2022,9,29]],"date-time":"2022-09-29T00:00:00Z","timestamp":1664409600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,29]],"date-time":"2022-09-29T00:00:00Z","timestamp":1664409600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Universidad Polit\u00e8cnica de Val\u00e8ncia"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"published-print":{"date-parts":[[2023,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With its various available frameworks and possible devices, augmented reality is a proven useful tool in various industrial processes such as maintenance, repairing, training, reconfiguration, and even monitoring tasks of production lines in large factories. Despite its advantages, augmented reality still does not usually give meaning to the elements it complements, staying in a physical or geometric layer of its environment and without providing information that may be of great interest to industrial operators in carrying out their work. An expert\u2019s remote human assistance is becoming an exciting complement in these environments, but this is expensive or even impossible in many cases. This paper shows how a machine learning semantic layer can complement augmented reality solutions in the industry by providing an intelligent layer, sometimes even beyond some expert\u2019s skills. This layer, using state-of-the-art models, can provide visual validation and new inputs, natural language interaction, and automatic anomaly detection. All this new level of semantic context can be integrated into almost any current augmented reality system, improving the operator\u2019s job with additional contextual information, new multimodal interaction and validation, increasing their work comfort, operational times, and security.<\/jats:p>","DOI":"10.1007\/s11042-022-13803-1","type":"journal-article","created":{"date-parts":[[2022,9,29]],"date-time":"2022-09-29T01:02:40Z","timestamp":1664413360000},"page":"15875-15901","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Towards achieving a high degree of situational awareness and multimodal interaction with AR and semantic AI in industrial applications"],"prefix":"10.1007","volume":"82","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0076-7001","authenticated-orcid":false,"given":"Juan","family":"Izquierdo-Domenech","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jordi","family":"Linares-Pellicer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jorge","family":"Orta-Lopez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,9,29]]},"reference":[{"issue":"7","key":"13803_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3390\/math8071104","volume":"8","author":"A Alexeev","year":"2020","unstructured":"Alexeev A, Kukharev G, Matveev Y, Matveev A (2020) A highly efficient neural network solution for automated detection of pointer meters with different analog scales operating in different conditions. Mathematics 8(7):1\u201312. https:\/\/doi.org\/10.3390\/math8071104","journal-title":"Mathematics"},{"key":"13803_CR2","doi-asserted-by":"crossref","unstructured":"Baldauf M, B\u00f6sch R, Frei C, Hautle F, Jenny M (2018) Exploring requirements and opportunities of conversational user interfaces for the cognitively impaired. In: MobileHCI 2018 - beyond mobile: the next 20 years - 20th international conference on human-computer interaction with mobile devices and services, conference proceedings adjunct. Assoc Comput Mach, Inc, pp 119\u2013126","DOI":"10.1145\/3236112.3236128"},{"key":"13803_CR3","unstructured":"Ball G H, Hall D J (1965) ISODATA, a novel method of data analysis and pattern classification. Technical report NTIS AD 699616. Stanford Research Institute, Stanford, CA"},{"key":"13803_CR4","doi-asserted-by":"crossref","unstructured":"Barakonyi I, Psik T, Schmalstieg D (2004) Agents that talk and hit back: animated agents in augmented reality. In: ISMAR 2004: proceedings of the third IEEE and ACM international symposium on mixed and augmented reality, pp 141\u2013150","DOI":"10.1109\/ISMAR.2004.11"},{"key":"13803_CR5","doi-asserted-by":"publisher","first-page":"428","DOI":"10.1016\/j.renene.2012.12.043","volume":"55","author":"S Benbelkacem","year":"2013","unstructured":"Benbelkacem S, Belhocine M, Bellarbi A, Zenati-Henda N, Tadjine M (2013) Augmented reality for photovoltaic pumping systems maintenance tasks. Renew Energy 55:428\u2013437. https:\/\/doi.org\/10.1016\/j.renene.2012.12.043https:\/\/doi.org\/10.1016\/j.renene.2012.12.043","journal-title":"Renew Energy"},{"issue":"3","key":"13803_CR6","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1080\/24725854.2018.1493244","volume":"51","author":"E Bottani","year":"2019","unstructured":"Bottani E, Vignali G (2019) Augmented reality technology in the manufacturing industry: a review of the last decade. IISE Trans 51(3):284\u2013310. https:\/\/doi.org\/10.1080\/24725854.2018.1493244","journal-title":"IISE Trans"},{"key":"13803_CR7","first-page":"3","volume":"3","author":"J Brooke","year":"1996","unstructured":"Brooke J (1996) SUS - a quick and dirty usability scale. Usability Eval Ind 3:3","journal-title":"Usability Eval Ind"},{"key":"13803_CR8","doi-asserted-by":"crossref","unstructured":"Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the Twenty-Fifth Hawaii international conference on system sciences. http:\/\/ieeexplore.ieee.org\/document\/183317\/, pp 659\u2013669","DOI":"10.1109\/HICSS.1992.183317"},{"key":"13803_CR9","doi-asserted-by":"publisher","unstructured":"Chalapathy R, Chawla S (2019) Deep learning for anomaly detection: a survey, 1\u201350. https:\/\/doi.org\/10.48550\/arXiv.1901.03407https:\/\/doi.org\/10.48550\/arXiv.1901.03407","DOI":"10.48550\/arXiv.1901.03407 10.48550\/arXiv.1901.03407"},{"key":"13803_CR10","doi-asserted-by":"publisher","unstructured":"Coli E, Melluso N, Fantoni G, Mazzei D (2020) Towards automatic building of human-machine conversational system to support maintenance processes. https:\/\/doi.org\/10.48550\/arXiv.2005.06517","DOI":"10.48550\/arXiv.2005.06517"},{"key":"13803_CR11","doi-asserted-by":"publisher","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. https:\/\/doi.org\/10.48550\/arXiv.1810.04805","DOI":"10.48550\/arXiv.1810.04805"},{"issue":"2","key":"13803_CR12","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1109\/MIE.2014.2312079","volume":"8","author":"R Drath","year":"2014","unstructured":"Drath R, Horch A (2014) Industrie 4.0: hit or hype? IEEE Ind Electron Mag 8(2):56\u201358. https:\/\/doi.org\/10.1109\/MIE.2014.2312079","journal-title":"IEEE Ind Electron Mag"},{"key":"13803_CR13","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1016\/j.eswa.2016.07.006","volume":"63","author":"V Elia","year":"2016","unstructured":"Elia V, Gnoni M G, Lanzilotto A (2016) Evaluating the application of augmented reality devices in manufacturing from a process point of view: an AHP based model. Expert Syst Appl 63:187\u2013197. https:\/\/doi.org\/10.1016\/j.eswa.2016.07.006","journal-title":"Expert Syst Appl"},{"issue":"1","key":"13803_CR14","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1518\/001872095779049543","volume":"37","author":"MR Endsley","year":"1995","unstructured":"Endsley M R (1995) Toward a theory of situation awareness in dynamic systems. Hum Factors 37(1):32\u201364. https:\/\/doi.org\/10.1518\/001872095779049543","journal-title":"Hum Factors"},{"key":"13803_CR15","doi-asserted-by":"publisher","first-page":"13358","DOI":"10.1109\/ACCESS.2018.2808326","volume":"6","author":"P Fraga-Lamas","year":"2018","unstructured":"Fraga-Lamas P, Fern\u00e1ndez-Caram\u00e9s T M, Blanco-Novoa O, Vilar-Montesinos M A (2018) A review on industrial augmented reality systems for the industry 4.0 shipyard. IEEE Access 6:13358\u201313375. https:\/\/doi.org\/10.1109\/ACCESS.2018.2808326","journal-title":"IEEE Access"},{"key":"13803_CR16","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1016\/j.procs.2013.11.019","volume":"25","author":"LE Garza","year":"2013","unstructured":"Garza L E, Pantoja G, Ram\u00edrez P, Ram\u00edrez H, Rodr\u00edguez N, Gonz\u00e1lez E, Quintal R, P\u00e9rez J A (2013) Augmented reality application for the maintenance of a flapper valve of a fuller-kynion type m pump. Procedia Comput Sci 25:154\u2013160. https:\/\/doi.org\/10.1016\/j.procs.2013.11.019","journal-title":"Procedia Comput Sci"},{"key":"13803_CR17","doi-asserted-by":"crossref","unstructured":"Gonzalez A V, Koh S, Kapalo K, Sottilare R, Garrity P, Billinghurst M, Laviola J (2019) A comparison of desktop and augmented reality scenario based training authoring tools. In: Proceedings - 2019 IEEE international symposium on mixed and augmented reality, ISMAR 2019, pp 339\u2013350","DOI":"10.1109\/VR.2019.8797973"},{"key":"13803_CR18","doi-asserted-by":"crossref","unstructured":"Gorecky D, Schmitt M, Loskyll M, Z\u00fchlke D (2014) Human-machine-interaction in the industry 4.0 era. In: 2014 12th IEEE international conference on industrial informatics (INDIN), pp 289\u2013294","DOI":"10.1109\/INDIN.2014.6945523"},{"key":"13803_CR19","doi-asserted-by":"crossref","unstructured":"Guerreiro B V, Lins R G, Sun J, Schmitt R (2018) Definition of smart retrofitting: first steps for a company to deploy aspects of industry 4.0. In: Advances in manufacturing. Springer, pp 161\u2013170","DOI":"10.1007\/978-3-319-68619-6_16"},{"issue":"4","key":"13803_CR20","doi-asserted-by":"publisher","first-page":"386","DOI":"10.1016\/j.vrih.2019.07.002","volume":"1","author":"L Jinyu","year":"2019","unstructured":"Jinyu L, Bangbang Y, Danpeng C, Nan W, Guofeng Z, Hujun B (2019) Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality. Virt Real Intell Hardware 1(4):386\u2013410. https:\/\/doi.org\/10.1016\/j.vrih.2019.07.002","journal-title":"Virt Real Intell Hardware"},{"key":"13803_CR21","doi-asserted-by":"crossref","unstructured":"Kagermann H, Helbig J, Hellinger A, Wahlster W (2013) Recommendations for implementing the strategic initiative INDUSTRIE 4.0: securing the future of German manufacturing industry; final report of the Industrie 4.0 Working Group. Forschungsunion","DOI":"10.3390\/sci4030026"},{"key":"13803_CR22","doi-asserted-by":"crossref","unstructured":"Kamat P, Sugandhi R (2020) Anomaly detection for predictive maintenance in industry 4.0-A survey. In: E3S Web of Conferences, vol 170, pp 1\u20138","DOI":"10.1051\/e3sconf\/202017002007"},{"key":"13803_CR23","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.jmsy.2020.02.010","volume":"55","author":"ZH Lai","year":"2020","unstructured":"Lai Z H, Tao W, Leu M C, Yin Z (2020) Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing. J Manuf Syst 55:69\u201381. https:\/\/doi.org\/10.1016\/j.jmsy.2020.02.010","journal-title":"J Manuf Syst"},{"key":"13803_CR24","doi-asserted-by":"publisher","unstructured":"Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) RoBERTa: a robustly optimized BERT pretraining approach. https:\/\/doi.org\/10.48550\/arXiv.1907.11692","DOI":"10.48550\/arXiv.1907.11692"},{"issue":"1","key":"13803_CR25","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.cirp.2016.04.038","volume":"65","author":"S Makris","year":"2016","unstructured":"Makris S, Karagiannis P, Koukas S, Matthaiakis A S (2016) Augmented reality system for operator support in human-robot collaborative assembly. CIRP Ann Manuf Technol 65(1):61\u201364. https:\/\/doi.org\/10.1016\/j.cirp.2016.04.038","journal-title":"CIRP Ann Manuf Technol"},{"key":"13803_CR26","doi-asserted-by":"publisher","first-page":"5","DOI":"10.3390\/app10051855","volume":"10","author":"D Mourtzis","year":"2020","unstructured":"Mourtzis D, Siatras V, Angelopoulos J (2020) Real-time remote maintenance support based on augmented reality (AR). Appl Sci (Switzerland) 10:5. https:\/\/doi.org\/10.3390\/app10051855","journal-title":"Appl Sci (Switzerland)"},{"issue":"1","key":"13803_CR27","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/j.cirp.2009.03.020","volume":"58","author":"SK Ong","year":"2009","unstructured":"Ong S K, Shen Y (2009) A mixed reality environment for collaborative product design and development. CIRP Ann Manuf Technol 58(1):139\u2013142. https:\/\/doi.org\/10.1016\/j.cirp.2009.03.020","journal-title":"CIRP Ann Manuf Technol"},{"key":"13803_CR28","unstructured":"Pianta E, Bentivogli L, Girardi C (2002) MultiWordNet: developing an aligned multilingual database. In: First international conference on global WordNet, pp 293\u2013302"},{"issue":"5","key":"13803_CR29","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1080\/10447318.2014.994194","volume":"31","author":"R Radkowski","year":"2015","unstructured":"Radkowski R, Herrema J, Oliver J (2015) Augmented reality-based manual assembly support with visual features for different degrees of difficulty. Int J Human-Comput Interact 31(5):337\u2013349. https:\/\/doi.org\/10.1080\/10447318.2014.994194","journal-title":"Int J Human-Comput Interact"},{"key":"13803_CR30","doi-asserted-by":"publisher","unstructured":"Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu P J (2019) Exploring the limits of transfer learning with a unified text-to-text transformer. https:\/\/doi.org\/10.48550\/arXiv.1910.10683https:\/\/doi.org\/10.48550\/arXiv.1910.10683","DOI":"10.48550\/arXiv.1910.10683 10.48550\/arXiv.1910.10683"},{"key":"13803_CR31","doi-asserted-by":"publisher","unstructured":"Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) SQuAD: 100,000+ questions for machine comprehension of text. https:\/\/doi.org\/10.48550\/arXiv.1606.05250","DOI":"10.48550\/arXiv.1606.05250"},{"key":"13803_CR32","doi-asserted-by":"publisher","unstructured":"Sanh V, Debut L, Chaumond J, Wolf T (2019) DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. https:\/\/doi.org\/10.48550\/arXiv.1910.01108","DOI":"10.48550\/arXiv.1910.01108"},{"key":"13803_CR33","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1016\/j.compind.2018.02.001","volume":"98","author":"GW Scurati","year":"2018","unstructured":"Scurati G W, Gattullo M, Fiorentino M, Ferrise F, Bordegoni M, Uva A E (2018) Converting maintenance actions into standard symbols for augmented reality applications in industry 4.0. Comput Ind 98:68\u201379. https:\/\/doi.org\/10.1016\/j.compind.2018.02.001","journal-title":"Comput Ind"},{"key":"13803_CR34","doi-asserted-by":"crossref","unstructured":"Simon G, Fitzgibbon A W, Zisserman A (2000) Markerless tracking using planar structures in the scene. In: Proceedings - IEEE and ACM international symposium on augmented reality, ISAR 2000. Institute of Electrical and Electronics Engineers Inc, pp 120\u2013128","DOI":"10.1109\/ISAR.2000.880935"},{"key":"13803_CR35","doi-asserted-by":"crossref","unstructured":"Tony Liu F, Ming Ting K, Zhou Z-H (2008) Isolation forest. In: Eighth IEEE international conference on data mining. IEEE, pp 413\u2013422","DOI":"10.1109\/ICDM.2008.17"},{"issue":"4","key":"13803_CR36","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1109\/JRA.1987.1087109","volume":"3","author":"RY Tsai","year":"1987","unstructured":"Tsai R Y (1987) A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J Robot Autom 3(4):323\u2013344. https:\/\/doi.org\/10.1109\/JRA.1987.1087109","journal-title":"IEEE J Robot Autom"},{"key":"13803_CR37","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, vol 30, pp 6000\u20136010"},{"issue":"1","key":"13803_CR38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s40436-015-0131-4","volume":"4","author":"X Wang","year":"2016","unstructured":"Wang X, Ong S K, Nee A YC (2016) A comprehensive survey of augmented reality assembly research. Adv Manuf 4(1):1\u201322. https:\/\/doi.org\/10.1007\/s40436-015-0131-4","journal-title":"Adv Manuf"},{"key":"13803_CR39","doi-asserted-by":"crossref","unstructured":"Yu W, Wu L, Deng Y, Mahindru R, Zeng Q, Guven S, Jiang M (2020) A technical question answering system with transfer learning. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. https:\/\/github.com\/wyu97\/TTQA, pp 92\u201399","DOI":"10.18653\/v1\/2020.emnlp-demos.13"},{"issue":"7","key":"13803_CR40","doi-asserted-by":"publisher","first-page":"1745","DOI":"10.1080\/00207540600972935","volume":"46","author":"ML Yuan","year":"2008","unstructured":"Yuan M L, Ong S K, Nee A YC (2008) Augmented reality for assembly guidance using a virtual interactive tool. Int J Prod Res 46(7):1745\u20131767. https:\/\/doi.org\/10.1080\/00207540600972935","journal-title":"Int J Prod Res"},{"key":"13803_CR41","doi-asserted-by":"publisher","unstructured":"Zafrir O, Larey A, Boudoukh G, Shen H, Wasserblat M (2021) Prune once for all: sparse pre-trained language models. https:\/\/doi.org\/10.48550\/arXiv.2111.05754","DOI":"10.48550\/arXiv.2111.05754"},{"key":"13803_CR42","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1016\/j.cie.2020.106889","volume":"12","author":"T Zonta","year":"2020","unstructured":"Zonta T, da Costa C A, da Rosa Righi R, de Lima M J, da Trindade E S, Li G P (2020) Predictive maintenance in the Industry 4.0: a systematic literature review. Comput Ind Eng 12:150. https:\/\/doi.org\/10.1016\/j.cie.2020.106889","journal-title":"Comput Ind Eng"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-022-13803-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-022-13803-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-022-13803-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T11:25:54Z","timestamp":1679657154000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-022-13803-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,29]]},"references-count":42,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2023,4]]}},"alternative-id":["13803"],"URL":"https:\/\/doi.org\/10.1007\/s11042-022-13803-1","relation":{},"ISSN":["1380-7501","1573-7721"],"issn-type":[{"value":"1380-7501","type":"print"},{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,29]]},"assertion":[{"value":"12 May 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 August 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 September 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 September 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflicts of interest to report regarding the present study.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}]}}