{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T14:57:25Z","timestamp":1781017045511,"version":"3.54.1"},"reference-count":32,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2022,7,23]],"date-time":"2022-07-23T00:00:00Z","timestamp":1658534400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62171134"],"award-info":[{"award-number":["62171134"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["JAT200024"],"award-info":[{"award-number":["JAT200024"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Foundation for Middle-aged and Young Educational Committee of Fujian Province","award":["62171134"],"award-info":[{"award-number":["62171134"]}]},{"name":"Foundation for Middle-aged and Young Educational Committee of Fujian Province","award":["JAT200024"],"award-info":[{"award-number":["JAT200024"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The booming haptic data significantly improve the users\u2019 immersion during multimedia interaction. As a result, the study of a Haptic-based Interaction System has attracted the attention of the multimedia community. To construct such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to the user experience. Despite audio-visual synchronization efforts, there is still a lack of a haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic\u2013visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic\u2013visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics and a key frame extraction of visual signals based on deep-object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic\u2013visual signals. Without timestamps in the signal flow, the proposed method is still effective and more robust in complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.<\/jats:p>","DOI":"10.3390\/s22155502","type":"journal-article","created":{"date-parts":[[2022,7,25]],"date-time":"2022-07-25T04:52:47Z","timestamp":1658724767000},"page":"5502","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Timestamp-Independent Haptic\u2013Visual Synchronization Method for Haptic-Based Interaction System"],"prefix":"10.3390","volume":"22","author":[{"given":"Yiwen","family":"Xu","sequence":"first","affiliation":[{"name":"Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China"},{"name":"College of Zhicheng, Fuzhou University, Fuzhou 350108, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liangtao","family":"Huang","sequence":"additional","affiliation":[{"name":"Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tiesong","family":"Zhao","sequence":"additional","affiliation":[{"name":"Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Fang","sequence":"additional","affiliation":[{"name":"Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5900-4175","authenticated-orcid":false,"given":"Liqun","family":"Lin","sequence":"additional","affiliation":[{"name":"Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MWC.2016.1500157RP","article-title":"Realizing the Tactile Internet: Haptic Communications over Next Generation 5G Cellular Networks","volume":"24","author":"Aijaz","year":"2017","journal-title":"IEEE Wirel. Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3034","DOI":"10.1109\/COMST.2018.2851452","article-title":"Toward Haptic Communications Over the 5G Tactile Internet","volume":"20","author":"Antonakoglou","year":"2018","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Qiao, Y., Zheng, Q., Lin, Y., Fang, Y., Xu, Y., and Zhao, T. (2020, January 13\u201316). Haptic Communication: Toward 5G Tactile Internet. Proceedings of the 2020 Cross Strait Radio Science & Wireless Technology Conference (CSRSWTC), Fuzhou, China.","DOI":"10.1109\/CSRSWTC50769.2020.9372659"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1109\/JPROC.2018.2867835","article-title":"Haptic Codecs for the Tactile Internet","volume":"107","author":"Steinbach","year":"2019","journal-title":"Proc. IEEE"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Xu, Y., Huang, Y., Chen, W., Xue, H., and Zhao, T. (2019, January 23\u201325). Error Resilience of Haptic Data in Interactive Systems. Proceedings of the 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP), Xi\u2019an, China.","DOI":"10.1109\/WCSP.2019.8927919"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1469","DOI":"10.1109\/TNSRE.2018.2846814","article-title":"Wearable Haptics and Immersive Virtual Reality Rehabilitation Training in Children With Neuromotor Impairments","volume":"26","author":"Bortone","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhou, H., Wei, L., Cao, R., Hanoun, S., Bhatti, A., Tai, Y., and Nahavandi, S. (2018, January 7\u201310). The Study of Using Eye Movements to Control the Laparoscope Under a Haptically-Enabled Laparoscopic Surgery Simulation Environment. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00513"},{"key":"ref_8","unstructured":"Yu, H., Liu, J., Liu, L., Ju, Z., Liu, Y., and Zhou, D. (2019). Application of Haptic Virtual Fixtures on Hot-Line Work Robot-Assisted Manipulation. Intelligent Robotics and Applications, Publishing House."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Durai, V.S.I., Arjunan, R., and Manivannan, M. (2019, January 23\u201327). The Effect of Audio and Visual Modality Based CPR Skill Training with Haptics Feedback in VR. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.","DOI":"10.1109\/VR.2019.8798006"},{"key":"ref_10","first-page":"117","article-title":"A touch of gloss: Haptic perception of packaging and consumers\u2019 reactions","volume":"11743","author":"Cloonan","year":"2019","journal-title":"J. Prod. Brand Manag."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1007\/978-981-13-3194-7_58","article-title":"Deformation and Friction: 3D Haptic Asset Enhancement in e-Commerce for the Visually Impaired","volume":"535","author":"Wong","year":"2018","journal-title":"Haptic Interact. AsiaHaptics"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1109\/LRA.2017.2738328","article-title":"Haptic Guidance in Dynamic Environments Using Optimal Reciprocal Collision Avoidance","volume":"3","author":"Scheggi","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Da Fonseca, V.P., Monteiro Rocha Lima, B., Alves de Oliveira, T.E., Zhu, Q., Groza, V.Z., and Petriu, E.M. (2019, January 26\u201328). In-Hand Telemanipulation Using a Robotic Hand and Biology-Inspired Haptic Sensing. Proceedings of the 2019 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Istanbul, Turkey.","DOI":"10.1109\/MeMeA.2019.8802139"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4424","DOI":"10.1109\/LRA.2019.2937483","article-title":"Intuitive Control of a Robotic Arm and Hand System With Pneumatic Haptic Feedback","volume":"4","author":"Li","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zeng, Q., Ishibashi, Y., Fukushima, N., Sugawara, S., and Psannis, K.E. (2013, January 1\u20134). Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensemble. Proceedings of the 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE), Tokyo, Japan.","DOI":"10.1109\/GCCE.2013.6664891"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Aung, S.T., Ishibashi, Y., Mya, K.T., Watanabe, H., and Huang, P. (2020, January 16\u201319). Influences of Network Delay on Cooperative Work in Networked Virtual Environment with Haptics. Proceedings of the 2020 IEEE Region 10 Conference (TENCON), Osaka, Japan.","DOI":"10.1109\/TENCON50793.2020.9293934"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"El-Helaly, M., and Amer, A. (October, January 16). Synchronization of Processed Audio-Video Signals using Time-Stamps. Proceedings of the 2007 IEEE International Conference on Image Processing, San Antonio, TX, USA.","DOI":"10.1109\/ICIP.2007.4379554"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1007\/s00530-012-0262-4","article-title":"Assessing the importance of audio\/video synchronization for simultaneous translation of video sequences","volume":"18","author":"Staelens","year":"2012","journal-title":"Multimed. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kikuchi, T., and Ozasa, Y. (2018, January 15\u201320). Watch, Listen Once, and Sync: Audio-Visual Synchronization With Multi-Modal Regression Cnn. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8461853"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"22081","DOI":"10.1109\/ACCESS.2017.2761539","article-title":"3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition","volume":"5","author":"Torfi","year":"2017","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yang, M., Bourbakis, N., Chen, Z., and Trifas, M. (2007, January 2\u20135). An Efficient Audio-Video Synchronization Methodology. Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, Beijing, China.","DOI":"10.1109\/ICME.2007.4284763"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/TIP.2020.3029901","article-title":"Hier R-CNN: Instance-Level Human Parts Detection and A New Benchmark","volume":"30","author":"Yang","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1904","DOI":"10.1109\/TPAMI.2015.2389824","article-title":"Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition","volume":"37","author":"He","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Ullah, A., Xie, H., Farooq, M.O., and Sun, Z. (2018, January 7\u201310). Pedestrian Detection in Infrared Images Using Fast RCNN. Proceedings of the 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA), Xi\u2019an, China.","DOI":"10.1109\/IPTA.2018.8608121"},{"key":"ref_25","unstructured":"Gomzales, R., Machacuay, J., Rotta, P., and Chinguel, C. (2021, January 28\u201330). Faster R-CNN with a cross-validation approach to object detection in radar images. Proceedings of the 2021 IEEE International Conference on Aerospace and Signal Processing(INCAS), Lima, Peru."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ahmad, T., Chen, X., Saqlain, A., and Ma, Y. (2021, January 24\u201326). EDF-SSD: An Improved Feature Fused SSD for Object Detection. Proceedings of the 2021 IEEE 6th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), Chengdu, China.","DOI":"10.1109\/ICCCBDA51879.2021.9442501"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"56416","DOI":"10.1109\/ACCESS.2021.3072211","article-title":"Small-Object Detection Based on YOLO and Dense Block via Image Super-Resolution","volume":"9","author":"Wang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"157818","DOI":"10.1109\/ACCESS.2019.2950053","article-title":"Research on Recognition Method of Electrical Components Based on YOLO V3","volume":"7","author":"Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_29","unstructured":"International Telecommunication Union (2002). Methodology for the Subjective Assessment of the Quality of Television Pictures, International Telecommunication Union."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Montagud, M., Cesar, P., Boronat, F., and Jansen, J. (2018). Media Synchronization in Networked Multisensory Applications with Haptics. MediaSync, Publishing House.","DOI":"10.1007\/978-3-319-65840-7"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Thirumalai, C., Chandhini, S.A., and Vaishnavi, M. (2017, January 20\u201322). Analysing the concrete compressive strength using Pearson and Spearman. Proceedings of the 2017 International Conference of Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India.","DOI":"10.1109\/ICECA.2017.8212799"},{"key":"ref_32","unstructured":"Yang, H., Bao, B., Guo, H., Jiang, Y., and Zhang, J. (2020, January 15\u201319). Spearman Correlation Coefficient Abnormal Behavior Monitoring Technology Based on RNN in 5G Network for Smart City. Proceedings of the 2020 International Wireless Communications and Mobile Computing (IWCMC), Limassol, Cyprus."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5502\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:55:27Z","timestamp":1760140527000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5502"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,23]]},"references-count":32,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22155502"],"URL":"https:\/\/doi.org\/10.3390\/s22155502","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,23]]}}}