{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:09:33Z","timestamp":1787029773625,"version":"build-2736575974"},"reference-count":40,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T00:00:00Z","timestamp":1771891200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U2141217"],"award-info":[{"award-number":["U2141217"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Subtle vibrations induced in everyday objects by ambient sound, especially speech, carry rich acoustic cues that can potentially be transformed into meaningful text, with potential implications for monitoring and security-related scenarios. This paper presents a Vision-based Subtitle Generator (VSG). This is the first attempt to recover text directly from high-speed videos of sound-induced object vibrations using a generative approach. To this end, VSG introduces a phase-based motion estimation (PME) technique that treats each pixel as an \u201cindependent microphone\u201d, and extracts thousands of pseudo-acoustic signals from a single video. Meanwhile, the pretrained Hidden-unit Bidirectional Encoder Representations from Transformers (HuBERT) serves as the encoder of the proposed VSG-Transformer architecture, effectively transferring large-scale acoustic representation knowledge to the vibration-to-text task. These strategies significantly reduce reliance on large volumes of video data. Experimentally, text was generated from vibrations induced in a bag of chips by AISHELL-1 audio samples. Two VSG-Transformer variants with different parameter scales (Base and Large) achieved character error rates of 13.7 and 12.5%, respectively, demonstrating the remarkable effectiveness of the proposed generative approach. Furthermore, experiments using signal upsampling techniques show that the VSG-Transformer maintains effective performance when operating on videos with limited temporal sampling, indicating robustness to lower sampling rates.<\/jats:p>","DOI":"10.3390\/s26051407","type":"journal-article","created":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T12:19:23Z","timestamp":1771935563000},"page":"1407","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Vision-Based Subtitle Generator: Text Reconstruction via Subtle Vibrations from Videos"],"prefix":"10.3390","volume":"26","author":[{"given":"Yan","family":"Wang","sequence":"first","affiliation":[{"name":"School of Mechanical Engineering, Beijing Institute of Technology, Haidian District, Beijing 100081, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingchong","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Beijing Institute of Technology, Haidian District, Beijing 100081, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiuqi","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Beijing Institute of Technology, Haidian District, Beijing 100081, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoyu","family":"Ding","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Beijing Institute of Technology, Haidian District, Beijing 100081, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,2,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2601097.2601119","article-title":"The visual microphone: Passive recovery of sound from video","volume":"33","author":"Davis","year":"2014","journal-title":"ACM Trans. Graph."},{"key":"ref_2","first-page":"4401","article-title":"Lamphone: Real-time passive sound recovery from light bulb vibrations","volume":"2020","author":"Nassi","year":"2020","journal-title":"Cryptol. ePrint Arch."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"516","DOI":"10.1016\/0022-460X(89)90705-0","article-title":"Laser vibrometry: Pseudo-vibrations","volume":"135","author":"Rothberg","year":"1989","journal-title":"J. Sound Vib."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Nassi, B., Swissa, R., Shams, J., Zadov, B., and Elovici, Y. (2023). The little seal bug: Optical sound recovery from lightweight reflective objects. Proceedings of the IEEE Security and Privacy Workshops, San Francisco, CA, USA, 25 May 2023, IEEE.","DOI":"10.1109\/SPW59333.2023.00032"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Nassi, B., Pirutin, Y., Galor, T., Elovici, Y., and Zadov, B. (2021). Glowworm attack: Optical tempest sound recovery via a device\u2019s power indicator led. Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, 15\u201319 November 2021, Association for Computing Machinery.","DOI":"10.1145\/3460120.3484775"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kwong, A., Xu, W., and Fu, K. (2019). Hard drive of hearing: Disks that eavesdrop with a synthesized microphone. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA, 19\u201323 May 2019, IEEE.","DOI":"10.1109\/SP.2019.00008"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"094105","DOI":"10.1117\/1.OE.56.9.094105","article-title":"Efficient subtle motion detection from high-speed video for sound recovery and vibration analysis using singular value decomposition-based approach","volume":"56","author":"Zhang","year":"2017","journal-title":"Opt. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"086111","DOI":"10.1063\/1.4961979","article-title":"Note: Sound recovery from video using svd-based information extraction","volume":"87","author":"Zhang","year":"2016","journal-title":"Rev. Sci. Instrum."},{"key":"ref_9","unstructured":"Guri, M., Solewicz, Y., Daidakulov, A., and Elovici, Y. (2017, January 14\u201315). SPEAKE(a) R: Turn speakers to microphones for fun and profit. Proceedings of the 11th USENIX Workshop on Offensive Technologies, Vancouver, BC, Canada."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"B4016013","DOI":"10.1061\/(ASCE)IS.1943-555X.0000348","article-title":"Video camera\u2013based vibration measurement for civil infrastructure applications","volume":"23","author":"Chen","year":"2017","journal-title":"J. Infrastruct. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Davis, A., Bouman, K., Chen, J., Rubinstein, M., Durand, F., and Freeman, W. (2015). Visual vibrometry: Estimating material properties from small motion in video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7\u201312 June 2015, IEEE.","DOI":"10.1109\/CVPR.2015.7299171"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zona, A. (2021). Vision-based vibration monitoring of structures and infrastructures: An overview of recent applications. Infrastructures, 6.","DOI":"10.3390\/infrastructures6010004"},{"key":"ref_13","unstructured":"Michalevsky, Y., Boneh, D., and Nakibly, G. (2014). Gyrophone: Recognizing speech from gyroscope signals. Proceedings of the 23rd USENIX Security Symposium, San Diego, CA, USA, 20\u201322 August 2014, USENIX Association."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Long, Y., Naghavi, P., Kojusner, B., Butler, K., Rampazzi, S., and Fu, K. (2023). Side eye: Characterizing the limits of pov acoustic eavesdropping from smartphone cameras with rolling shutters and movable lenses. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA, 21\u201325 May 2023, IEEE.","DOI":"10.1109\/SP46215.2023.10179313"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhang, L., Pathak, P., Wu, M., Zhao, Y., and Mohapatra, P. (2015). AccelWord: Energy efficient hotword detection through accelero-meter. Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, Florence, Italy, 18\u201322 May 2015, Association for Computing Machinery.","DOI":"10.1145\/2742647.2742658"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Han, J., Chung, A.J., and Tague, P. (2017). Pitchln: Eavesdropping via intelligible speech reconstruction using non-acoustic sensor fusion. Proceedings of the 16th ACM\/IEEE International Conference on Information Processing in Sensor Networks, Pittsburgh, PA, USA, 18\u201320 April 2017, Association for Computing Machinery.","DOI":"10.1145\/3055031.3055088"},{"key":"ref_17","first-page":"2907","article-title":"We can hear you with Wi-Fi!","volume":"Volume 15","author":"Wang","year":"2016","journal-title":"Proceedings of the IEEE Transactions on Mobile Computing, Piscataway, NJ, USA, 1 November 2016"},{"key":"ref_18","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2\u20137 June 2019, Association for Computational Linguistics."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3451","DOI":"10.1109\/TASLP.2021.3122291","article-title":"Hubert: Self-supervised speech representation learning by masked prediction of hidden units","volume":"29","author":"Hsu","year":"2021","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_20","unstructured":"Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. (2019). XLNet: Generalized autoregressive pretraining for language understanding. Proceedings of the Neural Information Processing Systems, Vancouver, BC, Canada, 8\u201314 December 2019, Curran Associates, Inc."},{"key":"ref_21","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Bu, H., Du, J., Na, X., Wu, B., and Zheng, H. (2017). AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline. Proceedings of the 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I\/O Systems and Assessment, Seoul, Republic of Korea, 1\u20133 November 2017, IEEE.","DOI":"10.1109\/ICSDA.2017.8384449"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1007\/BF00056772","article-title":"Computation of component image velocity from local phase information","volume":"5","author":"Fleet","year":"1990","journal-title":"Int. J. Comput. Vis."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1109\/TNN.2002.1031944","article-title":"A phase-based approach to the estimation of the optical flow field using spatial filtering","volume":"13","author":"Gautama","year":"2002","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1109\/34.93808","article-title":"The design and use of steerable filters","volume":"13","author":"Freeman","year":"1991","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"108786","DOI":"10.1016\/j.ymssp.2021.108786","article-title":"Out-of-plane modal property extraction based on multi-level image pyramid reconstruction using stereophotogrammetry","volume":"169","author":"Chou","year":"2022","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1145\/2461912.2461966","article-title":"Phase based video motion processing","volume":"32","author":"Wadhwa","year":"2013","journal-title":"ACM Trans. Graph."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1109\/LSP.2017.2782270","article-title":"Deep shrinkage convolutional neural network for adaptive noise reduction","volume":"25","author":"Isogawa","year":"2018","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"4681","DOI":"10.1109\/TII.2019.2943898","article-title":"Deep residual shrinkage networks for fault diagnosis","volume":"16","author":"Zhao","year":"2020","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, B., Lv, H., Guo, P., Shao, Q., Yang, C., and Xie, L. (2022). Wenetspeech: A 10000 hours multi-domain mandarin corpus for speech recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore, 23\u201327 May 2022, IEEE.","DOI":"10.1109\/ICASSP43922.2022.9746682"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1162\/tacl_a_00300","article-title":"SpanBERT: Improving pre-training by representing and predicting spans","volume":"8","author":"Joshi","year":"2020","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_32","unstructured":"Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). Wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv."},{"key":"ref_33","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Proceedings of the Neural Information Processing Systems, Vancouver, BC, Canada, 8\u201314 December 2019, Curran Associates, Inc."},{"key":"ref_34","unstructured":"Kingma, D., and Ba, J. (2015). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Chorowski, J., and Jaitly, N. (2017). Towards better decoding and language model integration in sequence to sequence models. arXiv.","DOI":"10.21437\/Interspeech.2017-343"},{"key":"ref_36","unstructured":"Wu, Y., Schuster, M., Chen, Z., Le, Q., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yang, Y., Hira, M., Ni, Z., Astafurov, A., Chen, C., and Puhrsch, C. (2022). Torchaudio: Building blocks for audio and speech processing. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore, 23\u201327 May 2022, IEEE.","DOI":"10.1109\/ICASSP43922.2022.9747236"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Li, K., Huang, Z., Xu, Y., and Lee, C. (2015). DNN-based speech bandwidth expansion and its application to adding high-frequency missing features for automatic speech recognition of narrowband speech. Proceedings of the Interspeech, Dresden, Germany, 6\u201310 September 2015, International Speech Communication Association.","DOI":"10.21437\/Interspeech.2015-555"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Rakotonirina, N. (2021). Self-attention for audio super-resolution. Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing, Gold Coast, Australia, 25\u201328 October 2021, IEEE.","DOI":"10.1109\/MLSP52302.2021.9596082"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Meyer, S., Djelouah, A., McWilliams, B., Sorkine-Hornung, A., Gross, M., and Schroers, C. (2018). Phasenet for video frame interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18\u201322 June 2018, IEEE.","DOI":"10.1109\/CVPR.2018.00059"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/5\/1407\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T12:25:31Z","timestamp":1771935931000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/5\/1407"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,24]]},"references-count":40,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["s26051407"],"URL":"https:\/\/doi.org\/10.3390\/s26051407","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,24]]}}}