{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T22:40:29Z","timestamp":1789684829084,"version":"4.0.1"},"reference-count":34,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2023,10,13]],"date-time":"2023-10-13T00:00:00Z","timestamp":1697155200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research Council of Science and Technology (NST) grant by the Ministry of Science and ICT (MSIT)","award":["CRC-20-02-KIST"],"award-info":[{"award-number":["CRC-20-02-KIST"]}]},{"name":"National Research Council of Science and Technology (NST) grant by the Ministry of Science and ICT (MSIT)","award":["2021R1A6A3A0108672112"],"award-info":[{"award-number":["2021R1A6A3A0108672112"]}]},{"DOI":"10.13039\/501100003725","name":"Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education","doi-asserted-by":"publisher","award":["CRC-20-02-KIST"],"award-info":[{"award-number":["CRC-20-02-KIST"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003725","name":"Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education","doi-asserted-by":"publisher","award":["2021R1A6A3A0108672112"],"award-info":[{"award-number":["2021R1A6A3A0108672112"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>During the 2019 coronavirus disease pandemic, robotic-based systems for swab sampling were developed to reduce burdens on healthcare workers and their risk of infection. Teleoperated sampling systems are especially appreciated as they fundamentally prevent contact with suspected COVID-19 patients. However, the limited field of view of the installed cameras prevents the operator from recognizing the position and deformation of the swab inserted into the nasal cavity, which highly decreases the operating performance. To overcome this limitation, this study proposes a visual feedback system that monitors and reconstructs the shape of an NP swab using augmented reality (AR). The sampling device contained three load cells and measured the interaction force applied to the swab, while the shape information was captured using a motion-tracking program. These datasets were used to train a one-dimensional convolution neural network (1DCNN) model, which estimated the coordinates of three feature points of the swab in 2D X\u2013Y plane. Based on these points, the virtual shape of the swab, reflecting the curvature of the actual one, was reconstructed and overlaid on the visual display. The accuracy of the 1DCNN model was evaluated on a 2D plane under ten different bending conditions. The results demonstrate that the x-values of the predicted points show errors of under 0.590 mm from P0, while those of P1 and P2 show a biased error of about \u22121.5 mm with constant standard deviations. For the y-values, the error of all feature points under positive bending is uniformly estimated with under 1 mm of difference, when the error under negative bending increases depending on the amount of deformation. Finally, experiments using a collaborative robot validate its ability to visualize the actual swab\u2019s position and deformation on the camera image of 2D and 3D phantoms.<\/jats:p>","DOI":"10.3390\/s23208443","type":"journal-article","created":{"date-parts":[[2023,10,13]],"date-time":"2023-10-13T10:16:12Z","timestamp":1697192172000},"page":"8443","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Deep Neural Network-Based Visual Feedback System for Nasopharyngeal Swab Sampling"],"prefix":"10.3390","volume":"23","author":[{"given":"Suhun","family":"Jung","sequence":"first","affiliation":[{"name":"Artificial Intelligence and Robot Institute, Korea Institute of Science and Technology, 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul 02792, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7751-3179","authenticated-orcid":false,"given":"Yonghwan","family":"Moon","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul 02841, Republic of Korea"},{"name":"Augmented Safety System with Intelligence Sensing and Tracking, Korea Institute of Science and Technology, 5 Hwarang-ro 14-gil, Seongbuk-gu, Seoul 02792, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeongryul","family":"Kim","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Robot Institute, Korea Institute of Science and Technology, 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul 02792, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keri","family":"Kim","sequence":"additional","affiliation":[{"name":"Augmented Safety System with Intelligence Sensing and Tracking, Korea Institute of Science and Technology, 5 Hwarang-ro 14-gil, Seongbuk-gu, Seoul 02792, Republic of Korea"},{"name":"Division of Bio-Medical Science and Technology, University of Science and Technology, 217 Gajeong-ro, Yuseong-gu, Daejeon 34113, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,13]]},"reference":[{"key":"ref_1","unstructured":"World Health Organization (2020). Critical Preparedness, Readiness and Response Actions for COVID-19: Interim Guidance, 22 March 2020 (No. WHO\/2019-nCoV\/Community_Actions\/2020.3), World Health Organization."},{"key":"ref_2","unstructured":"World Health Organization (2020). Laboratory Testing for Coronavirus Disease 2019 (COVID-19) in Suspected Human Cases: Interim Guidance, 2 March 2020 (No. WHO\/COVID-19\/laboratory\/2020.4), World Health Organization."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1186\/s40463-020-00429-2","article-title":"Guidance for otolaryngology health care workers performing aerosol generating medical procedures during the COVID-19 pandemic","volume":"49","author":"Lammers","year":"2020","journal-title":"J. Otolaryngol. Head. Neck Surg."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1111\/irv.12745","article-title":"Medical masks vs. N95 respirators for preventing COVID-19 in healthcare workers: A systematic review and meta-analysis of randomized trials","volume":"14","author":"Bartoszko","year":"2020","journal-title":"Influenza Other Respir. Viruses"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1109\/TMRB.2020.3036461","article-title":"Design of a low-cost miniature robot to assist the COVID-19 na-sopharyngeal swab sampling","volume":"3","author":"Wang","year":"2020","journal-title":"IEEE Trans. Med. Robot. Bionics."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"612167","DOI":"10.3389\/frobt.2021.612167","article-title":"A flexible transoral robot towards COVID-19 swab sampling","volume":"8","author":"Li","year":"2021","journal-title":"Front. Robot. AI"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"11150","DOI":"10.1109\/LRA.2022.3187499","article-title":"Development of an inherently safe nasopharyngeal swab sampling robot using a force restriction mechanism","volume":"7","author":"Maeng","year":"2022","journal-title":"IEEE Robot. Automat. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Seo, J., Shim, S., Park, H., Baek, J., Cho, J.H., and Kim, N.H. (2020). Development of robot-assisted untact swab sampling system for upper respiratory disease. Appl. Sci., 10.","DOI":"10.3390\/app10217707"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Chen, W., Zhou, J., Cheng, S.S., Lu, Y., Zhong, F., Gao, Y., Wang, Y., Xue, L., Tong, M.C., and Liu, Y.H. (2021). Tele-Operated Oropharyngeal Swab (TOOS) Robot Enabled by TSS Soft Hand for Safe and EffectiveCOVID-19 OP Sampling. arXiv.","DOI":"10.1109\/TMRB.2021.3123530"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"103917","DOI":"10.1016\/j.robot.2021.103917","article-title":"A collaborative robot for COVID-19 oropharyngeal swabbing","volume":"148","author":"Chen","year":"2022","journal-title":"Robot. Auton. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2001912","DOI":"10.1183\/13993003.01912-2020","article-title":"Clinical application of an intelligent oropharyngeal swab robot: Implication for the COVID-19 pandemic","volume":"56","author":"Li","year":"2020","journal-title":"Eur. Respir. J."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/s40463-020-00490-x","article-title":"Variability in instructions for performance of nasopharyngeal swabs across Canada in the era of COVID-19\u2013what type of swab is actually being performed?","volume":"50","author":"Hiebert","year":"2021","journal-title":"J. Otolaryngol. Head. Neck Surg."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"eabl6700","DOI":"10.1126\/sciadv.abl6700","article-title":"Electronic skin as wireless human-machine interfaces for robotic VR","volume":"8","author":"Liu","year":"2022","journal-title":"Sci. Adv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ku, J., Chen, C.Y., Ku, J., Chang, H.K., Wu, J.C., and Yen, Y.S. (2021). Iatrogenic cerebrospinal fluid leak after repeated nasal swab tests for COVID-19: Illustrative case. J. Neurosurg. Case Lessons, 2.","DOI":"10.3171\/CASE21421"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"457","DOI":"10.5694\/mja2.51082","article-title":"COVID-19 swab-related skull base injury","volume":"214","author":"Mistry","year":"2021","journal-title":"Med. J. Aust."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e1400","DOI":"10.7717\/peerj-cs.1400","article-title":"The multi-modal fusion in visual question answering: A review of attention mechanisms","volume":"9","author":"Lu","year":"2023","journal-title":"PeerJ Comput. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4621","DOI":"10.1109\/TIP.2023.3302162","article-title":"Reducing Vision-Answer biases for Multiple-choice VQA","volume":"32","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Image Process."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"714318","DOI":"10.3389\/fnins.2021.714318","article-title":"Fuzzy system based medical image processing for brain disease prediction","volume":"15","author":"Hu","year":"2021","journal-title":"Front. Neurosci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1016\/j.neuroimage.2016.01.024","article-title":"Deep MRI brain extraction: A 3D convolutional neural network for skull stripping","volume":"129","author":"Kleesiek","year":"2016","journal-title":"NeuroImage"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., and Chen, T. (2021, January 10\u201317). Big self-supervised models advance medical image classification. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00346"},{"key":"ref_21","unstructured":"Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., and Wang, M. (2021). SwinUnet: Unet-like pure transformer for medical image segmentation. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Hou, Y., Ma, L., Zhu, R., Chen, X., and Zhang, J. (2016). A low-cost iPhone-assisted augmented reality solution for the localization of intracranial lesions. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0159185"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.ejca.2022.04.023","article-title":"Augmented reality-guided localization of a chest wall tumor in a pediatric patient","volume":"170","author":"Spijkerboer","year":"2022","journal-title":"Eur. J. Can."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"18620","DOI":"10.1038\/s41598-020-75676-4","article-title":"Augmented reality improves procedural efficiency and reduces radiation dose for CT-guided lesion targeting: A phantom study using HoloLens 2","volume":"10","author":"Park","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1148\/radiol.2019182210","article-title":"Implementing virtual and augmented reality tools for radiology education and training, communication, and clinical care","volume":"291","author":"Uppot","year":"2019","journal-title":"Radiology"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1016\/S0020-7462(01)00019-1","article-title":"Large deflections of cantilever beams of non-linear elastic material under a combined loading","volume":"37","author":"Lee","year":"2002","journal-title":"Int. J. Non-Linear Mech."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1016\/j.ijnonlinmec.2009.12.004","article-title":"An integral approach for large deflection cantilever beams","volume":"45","author":"Chen","year":"2010","journal-title":"Int. J. Non-Linear Mech."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"100412","DOI":"10.1016\/j.imu.2020.100412","article-title":"A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images","volume":"20","author":"Islam","year":"2020","journal-title":"Inform. Med. Unlocked"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"113692","DOI":"10.1016\/j.sna.2022.113692","article-title":"A deep neural network approach for accurate 3D shape estimation of soft manipulator with vision correction","volume":"344","author":"Zou","year":"2022","journal-title":"Sensor Actuat. Phys."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"105129","DOI":"10.1016\/j.ijmecsci.2019.105129","article-title":"Deep learning for haptic feedback of flexible endoscopic robot without prior knowledge on sheath configuration","volume":"163","author":"Li","year":"2019","journal-title":"Int. J. Mech. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1109\/TASE.2020.3045655","article-title":"Toward teaching by demonstration for robot-assisted minimally invasive surgery","volume":"18","author":"Su","year":"2021","journal-title":"IEEE Trans. Automat. Sci. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"107398","DOI":"10.1016\/j.ymssp.2020.107398","article-title":"1D convolutional neural networks and applications: A survey","volume":"151","author":"Kiranyaz","year":"2021","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Farhani, G., Zhou, Y., Danielson, P., and Trejos, A.L. (2022). Implementing machine learning algorithms to classify postures and forecast motions when using a dynamic chair. Sensors, 22.","DOI":"10.3390\/s22010400"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Roh, J., Park, H.J., Lee, K.J., Hyeong, J., Kim, S., and Lee, B. (2018). Sitting posture monitoring system based on a low-cost load cell using machine learning. Sensors, 18.","DOI":"10.3390\/s18010208"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/20\/8443\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:06:24Z","timestamp":1760130384000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/20\/8443"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,13]]},"references-count":34,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["s23208443"],"URL":"https:\/\/doi.org\/10.3390\/s23208443","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,13]]}}}