{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T16:47:24Z","timestamp":1782492444395,"version":"3.54.5"},"reference-count":33,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2020,9,29]],"date-time":"2020-09-29T00:00:00Z","timestamp":1601337600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Intelligent agents that can interact with users using natural language are becoming increasingly common. Sometimes an intelligent agent may not correctly understand a user command or may not perform it properly. In such cases, the user might try a second time by giving the agent another, slightly different command. Giving an agent the ability to detect such user corrections might help it fix its own mistakes and avoid making them in the future. In this work, we consider the problem of automatically detecting user corrections using deep learning. We develop a multimodal architecture called SAIF, which detects such user corrections, taking as inputs the user\u2019s voice commands as well as their transcripts. Voice inputs allow SAIF to take advantage of sound cues, such as tone, speed, and word emphasis. In addition to sound cues, our model uses transcripts to determine whether a command is a correction to the previous command. Our model also obtains internal input from the agent, indicating whether the previous command was executed successfully or not. Finally, we release a unique dataset in which users interacted with an intelligent agent assistant, by giving it commands. This dataset includes labels on pairs of consecutive commands, which indicate whether the latter command is in fact a correction of the former command. We show that SAIF outperforms current state-of-the-art methods on this dataset.<\/jats:p>","DOI":"10.3390\/s20195577","type":"journal-article","created":{"date-parts":[[2020,9,29]],"date-time":"2020-09-29T08:43:27Z","timestamp":1601369007000},"page":"5577","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["SAIF: A Correction-Detection Deep-Learning Architecture for Personal Assistants"],"prefix":"10.3390","volume":"20","author":[{"given":"Amos","family":"Azaria","sequence":"first","affiliation":[{"name":"Data Science Center, Ariel University, Ariel 40700, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7451-5945","authenticated-orcid":false,"given":"Keren","family":"Nivasch","sequence":"additional","affiliation":[{"name":"Data Science Center, Ariel University, Ariel 40700, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,29]]},"reference":[{"key":"ref_1","unstructured":"Oard, D.W., and Kim, J. (2001, January 3\u20138). Modeling information content using observable behavior. Proceedings of the 64th Annual Meeting of the American Society for Information Science and Technology, Washington, DC, USA."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1145\/959258.959260","article-title":"Implicit feedback for inferring user preference: A bibliography","volume":"Volume 37","author":"Kelly","year":"2003","journal-title":"ACM SIGIR Forum"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Jannach, D., Lerche, L., and Zanker, M. (2018). Recommending Based on Implicit Feedback. Social Information Access-Systems and Technologies, Springer.","DOI":"10.1007\/978-3-319-90092-6_14"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Levitan, R., and Elson, D. (2014, January 22\u201327). Detecting retries of voice search queries. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, MD, USA.","DOI":"10.3115\/v1\/P14-2038"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zweig, G. (2009, January 6\u201310). New methods for the analysis of repeated utterances. Proceedings of the 10th Annual Conference of the International Speech Communication Association (Interspeech 2009), Brighton, UK.","DOI":"10.21437\/Interspeech.2009-713"},{"key":"ref_6","first-page":"527","article-title":"Speech repairs, intonational phrases, and discourse markers: Modeling speakers\u2019 utterances in spoken dialogue","volume":"25","author":"Heeman","year":"1999","journal-title":"Comput. Linguist."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bechet, F., and Favre, B. (2013, January 26\u201331). ASR error segment localization for spoken recovery strategy. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6638986"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ogawa, A., and Hori, T. (2015, January 19\u201324). ASR error detection and recognition rate estimation using deep bidirectional recurrent neural networks. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia.","DOI":"10.1109\/ICASSP.2015.7178796"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Dolan, B., Quirk, C., and Brockett, C. (2004, January 23\u201327). Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. Proceedings of the 20th international conference on Computational Linguistics, Association for Computational Linguistics, Geneva, Switzerland.","DOI":"10.3115\/1220355.1220406"},{"key":"ref_10","unstructured":"Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. (2015, January 7\u201312). Skip-thought vectors. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"922","DOI":"10.1016\/j.ipm.2018.06.005","article-title":"A deep network model for paraphrase detection in short text messages","volume":"54","author":"Agarwal","year":"2018","journal-title":"Inf. Process. Manag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. (2016, January 11\u201312). Generating Sentences from a Continuous Space. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany.","DOI":"10.18653\/v1\/K16-1002"},{"key":"ref_13","unstructured":"Madnani, N., Tetreault, J., and Chodorow, M. (2012, January 3\u20138). Re-examining machine translation metrics for paraphrase identification. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Montreal, QC, Canada."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Issa, F., Damonte, M., Cohen, S.B., Yan, X., and Chang, Y. (2018, January 2\u20137). Abstract meaning representation for paraphrase detection. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. Volume 1 (Long Papers).","DOI":"10.18653\/v1\/N18-1041"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Duong, P.H., Nguyen, H.T., Duong, H.N., Ngo, K., and Ngo, D. (2018, January 23\u201324). A hybrid approach to paraphrase detection. Proceedings of the 2018 5th NAFOSTED Conference on Information and Computer Science (NICS), Ho Chi Minh City, Vietnam.","DOI":"10.1109\/NICS.2018.8606845"},{"key":"ref_16","first-page":"8887","article-title":"A Hybrid Model for Paraphrase Detection Combines pros of Text Similarity with Deep Learning","volume":"975","author":"Gomaa","year":"2019","journal-title":"Int. J. Comput. Appl."},{"key":"ref_17","unstructured":"Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., and Salakhutdinov, R.R. (2016, January 5\u201310). On multiplicative integration with recurrent neural networks. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_18","unstructured":"(2020, September 27). Quora Question Pairs|Kaggle. Available online: https:\/\/kaggle.com\/quora\/question-pairs-dataset."},{"key":"ref_19","unstructured":"Choudhary, D. (2020, September 27). BERT Fine-Tuning on Quora Question Pairs. Available online: https:\/\/github.com\/drc10723\/bert_quora_question_pairs."},{"key":"ref_20","unstructured":"Sharma, L., Graesser, L., Nangia, N., and Evci, U. (2019). Natural language understanding with the quora question pairs dataset. arXiv."},{"key":"ref_21","unstructured":"Chandra, A., and Stefanus, R. (2020). Experiments on Paraphrase Identification Using Quora Question Pairs Dataset. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal Machine Learning: A Survey and Taxonomy","volume":"41","author":"Baltrusaitis","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","unstructured":"(2020, September 27). Speech Emotion Recognition with Librosa. Available online: https:\/\/data-flair.training\/blogs\/python-mini-project-speech-emotion-recognition\/."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, T.J.J., Azaria, A., and Myers, B.A. (2017, January 6\u201311). SUGILITE: Creating multimodal smartphone automation by demonstration. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA.","DOI":"10.1145\/3025453.3025483"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0196391"},{"key":"ref_26","unstructured":"Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (2, January 28). Multimodal deep learning. Proceedings of the 28th international conference on machine learning (ICML-11), Bellevue, WA, USA."},{"key":"ref_27","unstructured":"Srivastava, N., and Salakhutdinov, R.R. (2012, January 3\u20136). Multimodal learning with deep boltzmann machines. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Azaria, A., Krishnamurthy, J., and Mitchell, T.M. (2016, January 12\u201317). Instructable Intelligent Personal Agent. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10357"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1596","DOI":"10.1080\/10447318.2018.1557972","article-title":"LIA: A Virtual Assistant that Can Be Taught New Commands by Speech","volume":"35","author":"Chkroun","year":"2019","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_30","unstructured":"Azaria, A., and Nivasch, K. (2020, September 27). Correction-Detection GitHub Repository. Available online: https:\/\/github.com\/kerenivasch\/correction-detection."},{"key":"ref_31","unstructured":"(2020, September 27). WebRTC. Available online: https:\/\/webrtc.org\/."},{"key":"ref_32","unstructured":"Burstein, J., Doran, C., and Solorio, T. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, Minneapolis, MN, USA. Volume 1 (Long and Short, Papers)."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"M\u00fcller, M. (2007). Information Retrieval for Music and Motion, Springer.","DOI":"10.1007\/978-3-540-74048-3"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/19\/5577\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:14:45Z","timestamp":1760177685000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/19\/5577"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,29]]},"references-count":33,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2020,10]]}},"alternative-id":["s20195577"],"URL":"https:\/\/doi.org\/10.3390\/s20195577","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,29]]}}}