{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T11:36:25Z","timestamp":1774697785172,"version":"3.50.1"},"reference-count":37,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2019,9,7]],"date-time":"2019-09-07T00:00:00Z","timestamp":1567814400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Biomedical Research Institute grant, Kyungpook National University Hospital (2018)","award":["NA"],"award-info":[{"award-number":["NA"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Videofluoroscopic swallowing study (VFSS) is a standard diagnostic tool for dysphagia. To detect the presence of aspiration during a swallow, a manual search is commonly used to mark the time intervals of the pharyngeal phase on the corresponding VFSS image. In this study, we present a novel approach that uses 3D convolutional networks to detect the pharyngeal phase in raw VFSS videos without manual annotations. For efficient collection of training data, we propose a cascade framework which no longer requires time intervals of the swallowing process nor the manual marking of anatomical positions for detection. For video classification, we applied the inflated 3D convolutional network (I3D), one of the state-of-the-art network for action classification, as a baseline architecture. We also present a modified 3D convolutional network architecture that is derived from the baseline I3D architecture. The classification and detection performance of these two architectures were evaluated for comparison. The experimental results show that the proposed model outperformed the baseline I3D model in the condition where both models are trained with random weights. We conclude that the proposed method greatly reduces the examination time of the VFSS images with a low miss rate.<\/jats:p>","DOI":"10.3390\/s19183873","type":"journal-article","created":{"date-parts":[[2019,9,9]],"date-time":"2019-09-09T04:12:40Z","timestamp":1568002360000},"page":"3873","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Automatic Detection of the Pharyngeal Phase in Raw Videos for the Videofluoroscopic Swallowing Study Using Efficient Data Collection and 3D Convolutional Networks"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6962-3148","authenticated-orcid":false,"given":"Jong Taek","family":"Lee","sequence":"first","affiliation":[{"name":"Artificial Intelligence Application Research Section, Electronics and Telecommunications Research Institute (ETRI), Daegu 42994, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6553-2321","authenticated-orcid":false,"given":"Eunhee","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation Medicine, Kyungpook National University Chilgok Hospital, Daegu 41404, Korea"},{"name":"Department of Rehabilitation Medicine, School of Medicine, Kyungpook National University, Daegu 41944, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tae-Du","family":"Jung","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation Medicine, Kyungpook National University Chilgok Hospital, Daegu 41404, Korea"},{"name":"Department of Rehabilitation Medicine, School of Medicine, Kyungpook National University, Daegu 41944, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,9,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1038\/nrgastro.2015.49","article-title":"Dysphagia: Current reality and scope of the problem","volume":"12","author":"Shaker","year":"2015","journal-title":"Nat. Rev. Gastroenterol. Hepatol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1044\/1092-4388(2002\/034)","article-title":"Oropharyngeal swallow in younger and older women: Videofluoroscopic analysis","volume":"45","author":"Logemann","year":"2002","journal-title":"J. Speech Lang. Hear. Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1056\/NEJM200103013440908","article-title":"Aspiration pneumonitis and aspiration pneumonia","volume":"344","author":"Marik","year":"2001","journal-title":"N. Engl. J. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1118","DOI":"10.1046\/j.1532-5415.2002.50270.x","article-title":"Prevalence of impaired swallowing in institutionalized older people in Taiwan","volume":"50","author":"Lin","year":"2002","journal-title":"J. Am. Geriatr. Soc."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2226","DOI":"10.1016\/S1388-2457(03)00237-2","article-title":"Neurophysiology of swallowing","volume":"114","author":"Ertekin","year":"2003","journal-title":"Clin. Neurophysiol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"507","DOI":"10.2214\/ajr.161.3.8352094","article-title":"Clinical and videofluoroscopic evaluation of swallowing disorders","volume":"161","author":"Ott","year":"1993","journal-title":"AJR Am. J. Roentgenol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1007\/BF01354540","article-title":"A protocol for the videofluorographic swallowing study","volume":"8","author":"Palmer","year":"1993","journal-title":"Dysphagia"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1007\/PL00021291","article-title":"Inter-and intrajudge reliability for videofluoroscopic swallowing evaluation measures","volume":"16","author":"McCullough","year":"2001","journal-title":"Dysphagia"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/BF00262752","article-title":"Videofluoroscopic evidence of aspiration predicts pneumonia and death but not dehydration following stroke","volume":"9","author":"Schmidt","year":"1994","journal-title":"Dysphagia"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"12310","DOI":"10.1038\/s41598-018-30182-6","article-title":"Automatic hyoid bone detection in fluoroscopic images using deep learning","volume":"8","author":"Zhang","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_11","unstructured":"Noorwali, S. (2013). Semi-Automatic Tracking of the Hyoid Bone and the Epiglottis Movements in Digital Videofluoroscopic Images. [Ph.D. Thesis, The University of Western Ontario]."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1007\/s00455-009-9261-9","article-title":"Computer-assisted assessment of hyoid bone motion from videofluoroscopic swallow studies","volume":"25","author":"Kellen","year":"2010","journal-title":"Dysphagia"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1170","DOI":"10.1016\/j.medengphy.2010.08.008","article-title":"Automated anatomical demarcation using an active shape model for videofluoroscopic analysis in swallowing","volume":"32","author":"Aung","year":"2010","journal-title":"Med. Eng. Phys."},{"key":"ref_14","first-page":"1168","article-title":"Two methods for quantifying pharyngeal residue on fluoroscopic swallow studies: Reliability assessment","volume":"4","author":"Leonard","year":"2017","journal-title":"Ann. Otolaryngol. Rhinol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"277","DOI":"10.2214\/ajr.153.2.277","article-title":"Interactive computer program for biomechanical analysis of videoradiographic studies of swallowing","volume":"153","author":"Logemann","year":"1989","journal-title":"Am. J. Roentgenol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1007\/BF02503461","article-title":"Image processing in swallowing and speech research","volume":"6","author":"Dengel","year":"1991","journal-title":"Dysphagia"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1007\/s00455-013-9506-5","article-title":"Kinematic and temporal factors associated with penetration\u2013aspiration in swallowing liquids","volume":"29","author":"Molfenter","year":"2014","journal-title":"Dysphagia"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jardine, M., Miles, A., and Allen, J. (2018). Dysphagia onset in older adults during unrelated hospital admission: Quantitative videofluoroscopic measures. Geriatrics, 3.","DOI":"10.3390\/geriatrics3040066"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1007\/s00455-014-9547-4","article-title":"Automated acoustic analysis in detection of spontaneous swallows in Parkinson\u2019s disease","volume":"29","author":"Golabbakhsh","year":"2014","journal-title":"Dysphagia"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neucom.2017.12.059","article-title":"Deep learning for classification of normal swallows in adults","volume":"285","author":"Dudik","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Imtiaz, U., Yamamura, K., Kong, W., Sessa, S., Lin, Z., Bartolomeo, L., Ishii, H., Zecca, M., Yamada, Y., and Takanishi, A. (2014, January 26\u201330). Application of wireless inertial measurement units and EMG sensors for studying deglutition\u2014Preliminary results. Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Chicago, IL, USA.","DOI":"10.1109\/EMBC.2014.6944842"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Shieh, W.Y., Wang, C.M., Cheng, H.Y.K., and Wang, C.H. (2019). Using Wearable and Non-Invasive Sensors to Measure Swallowing Function: Detection, Verification, and Clinical Application. Sensors, 19.","DOI":"10.3390\/s19112624"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.compbiomed.2015.01.005","article-title":"Monitoring eating habits using a piezoelectric sensor-based necklace","volume":"58","author":"Kalantarian","year":"2015","journal-title":"Comput. Biol. Med."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7\u201313). Learning spatiotemporal features with 3D convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2017, January 21\u201326). Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Li, Y., Zhang, H., and Shen, Q. (2017). Spectral\u2013spatial classification of hyperspectral imagery with 3D convolutional neural network. Remote Sens., 9.","DOI":"10.3390\/rs9010067"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Song, A., Choi, J., Han, Y., and Kim, Y. (2018). Change detection in hyperspectral images using recurrent 3D fully convolutional networks. Remote Sens., 10.","DOI":"10.3390\/rs10111827"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kim, D., Cho, H., Shin, H., Lim, S.C., and Hwang, W. (2019). An Efficient Three-Dimensional Convolutional Neural Network for Inferring Physical Interaction Force from Video. Sensors, 19.","DOI":"10.3390\/s19163579"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lee, J.T., and Park, E. (2018). Detection of the Pharyngeal Phase in the Videofluoroscopic Swallowing Study Using Inflated 3D Convolutional Networks. Machine Learning in Medical Imaging, Springer.","DOI":"10.1007\/978-3-030-00919-9_38"},{"key":"ref_30","unstructured":"Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., and Natsev, P. (2017). The kinetics human action video dataset. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zach, C., Pock, T., and Bischof, H. (2007). A duality based approach for realtime TV-L 1 optical flow. Pattern Recognition, Springer.","DOI":"10.1007\/978-3-540-74936-3_22"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"137","DOI":"10.5201\/ipol.2013.26","article-title":"TV-L1 optical flow estimation","volume":"3","author":"Facciolo","year":"2013","journal-title":"Image Process. Line"},{"key":"ref_33","unstructured":"Loshchilov, I., and Hutter, F. (2017, January 24\u201326). Sgdr: Stochastic gradient descent with warm restarts. Proceedings of the 2017 International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_34","unstructured":"Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. (2011, January 6\u201313). HMDB: A large video database for human motion recognition. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126543"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1044\/jshr.3702.314","article-title":"Oropharyngeal swallow efficiency as a representative measure of swallowing function","volume":"37","author":"Rademaker","year":"1994","journal-title":"J. Speech Lang. Hear. Res."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/18\/3873\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:17:38Z","timestamp":1760188658000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/18\/3873"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,7]]},"references-count":37,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2019,9]]}},"alternative-id":["s19183873"],"URL":"https:\/\/doi.org\/10.3390\/s19183873","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9,7]]}}}