{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T01:52:58Z","timestamp":1780451578715,"version":"3.54.1"},"reference-count":50,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2021,9,21]],"date-time":"2021-09-21T00:00:00Z","timestamp":1632182400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100007049","name":"Korea Institute of Ocean Science and Technology","doi-asserted-by":"publisher","award":["pe99942"],"award-info":[{"award-number":["pe99942"]}],"id":[{"id":"10.13039\/501100007049","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>One of the most promising techniques for the analysis of Spatio-Temporal ocean wave fields is stereo vision. Indeed, the reconstruction accuracy and resolution typically outperform other approaches like radars, satellites, etc. However, it is computationally expensive so its application is typically restricted to the analysis of short pre-recorded sequences. What prevents such methodology from being truly real-time is the final 3D surface estimation from a scattered, non-equispaced point cloud. Recently, we studied a novel approach exploiting the temporal dependence of subsequent frames to iteratively update the wave spectrum over time. Albeit substantially faster, the unpredictable convergence time of the optimization involved still prevents its usage as a continuously running remote sensing infrastructure. In this work, we build upon the same idea, but investigating the feasibility of a fully data-driven Machine Learning (ML) approach. We designed a novel Convolutional Neural Network that learns how to produce an accurate surface from the scattered elevation data of three subsequent frames. The key idea is to embed the linear dispersion relation into the model itself to physically relate the sparse points observed at different times. Assuming that the scattered data are uniformly distributed in the spatial domain, this has the same effect of increasing the sample density of each single frame. Experiments demonstrate how the proposed technique, even if trained with purely synthetic data, can produce accurate and physically consistent surfaces at five frames per second on a modern PC.<\/jats:p>","DOI":"10.3390\/rs13183780","type":"journal-article","created":{"date-parts":[[2021,9,21]],"date-time":"2021-09-21T22:35:20Z","timestamp":1632263720000},"page":"3780","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["A Physics-Driven CNN Model for Real-Time Sea Waves 3D Reconstruction"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6273-290X","authenticated-orcid":false,"given":"Mara","family":"Pistellato","sequence":"first","affiliation":[{"name":"Department of Environmental Sciences, Informatics and Statistics-Ca\u2019 Foscari University of Venice, Dorsoduro 3246, 30123 Venice, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6668-1556","authenticated-orcid":false,"given":"Filippo","family":"Bergamasco","sequence":"additional","affiliation":[{"name":"Department of Environmental Sciences, Informatics and Statistics-Ca\u2019 Foscari University of Venice, Dorsoduro 3246, 30123 Venice, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9189-4924","authenticated-orcid":false,"given":"Andrea","family":"Torsello","sequence":"additional","affiliation":[{"name":"Department of Environmental Sciences, Informatics and Statistics-Ca\u2019 Foscari University of Venice, Dorsoduro 3246, 30123 Venice, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francesco","family":"Barbariol","sequence":"additional","affiliation":[{"name":"Istituto di Scienze Marine (ISMAR), Consiglio Nazionale delle Ricerche (CNR), 30122 Venice, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeseon","family":"Yoo","sequence":"additional","affiliation":[{"name":"Korea Institute of Ocean Science and Technology (KIOST), Pusan 49111, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jin-Yong","family":"Jeong","sequence":"additional","affiliation":[{"name":"Korea Institute of Ocean Science and Technology (KIOST), Pusan 49111, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9535-4922","authenticated-orcid":false,"given":"Alvise","family":"Benetazzo","sequence":"additional","affiliation":[{"name":"Istituto di Scienze Marine (ISMAR), Consiglio Nazionale delle Ricerche (CNR), 30122 Venice, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2261","DOI":"10.1175\/JPO-D-15-0017.1","article-title":"Observation of extreme sea waves in a space-time ensemble","volume":"45","author":"Benetazzo","year":"2015","journal-title":"J. Phys. Oceanogr."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/j.ocemod.2017.03.010","article-title":"Space-time extreme wind waves: Analysis and prediction of shape and height","volume":"113","author":"Alvise","year":"2017","journal-title":"Ocean Model."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Benetazzo, A., Barbariol, F., Bergamasco, F., Bertotti, L., Yoo, J., Shim, J.S., and Cavaleri, L. (2021). On the extreme value statistics of spatio-temporal maximum sea waves under cyclone winds. Prog. Oceanogr., 197.","DOI":"10.1016\/j.pocean.2021.102642"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"20190008","DOI":"10.1098\/rsta.2019.0008","article-title":"La Jument Lighthouse: A real scale laboratory for the study of giant waves and their loading on marine structures","volume":"377","author":"Filipot","year":"2019","journal-title":"Philos. Trans. R. Soc."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e2020JC016943","DOI":"10.1029\/2020JC016943","article-title":"A New Probabilistic Wave Breaking Model for Dominant Wind-Sea Waves Based on the Gaussian Field Theory","volume":"126","author":"Stringari","year":"2021","journal-title":"J. Geophys. Res. Ocean."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Douglas, S., Cornett, A., and Nistor, I. (2020). Image-Based Measurement of Wave Interactions with Rubble Mound Breakwaters. J. Mar. Sci. Eng., 8.","DOI":"10.3390\/jmse8060472"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"055503","DOI":"10.1088\/0957-0233\/19\/5\/055503","article-title":"Retrieval of short ocean wave slope using polarimetric imaging","volume":"19","author":"Zappa","year":"2008","journal-title":"Meas. Sci. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1049","DOI":"10.1029\/JC090iC01p01049","article-title":"A Three-Dimensional Analysis of Marine Radar Images for the Determination of Ocean Wave Directionality and Surface Currents","volume":"90","author":"Young","year":"1985","journal-title":"J. Geophys. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.ocemod.2012.10.004","article-title":"Detection of spatio-temporal wave grouping properties by using temporal sequences of X-band radar images of the sea surface","volume":"61","author":"Reichert","year":"2013","journal-title":"Ocean Model."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2197","DOI":"10.1364\/JOSAA.11.002197","article-title":"Imaging of short ocean wind waves: A critical theoretical review","volume":"11","author":"Klinke","year":"1994","journal-title":"J. Opt. Soc. Amer. A Opt. Image Sci. Vis."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1016\/j.coastaleng.2012.01.007","article-title":"Offshore stereo measurements of gravity waves","volume":"64","author":"Benetazzo","year":"2012","journal-title":"Coast. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.cageo.2017.07.001","article-title":"WASS: An open-source pipeline for 3D stereo reconstruction of ocean waves","volume":"107","author":"Bergamasco","year":"2017","journal-title":"Comput. Geosci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4211","DOI":"10.1109\/TIP.2013.2271118","article-title":"Variational stereo imaging of oceanic waves with statistical constraints","volume":"22","author":"Gallego","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1038\/s41597-020-0492-9","article-title":"A data set of sea surface stereo images to resolve space-time wave fields","volume":"7","author":"Ardhuin","year":"2020","journal-title":"Sci. Data"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bergamasco, F., Benetazzo, A., Yoo, J., Torsello, A., Barbariol, F., Jeong, J.Y., Shim, J.S., and Cavaleri, L. (2021). Toward real-time optical estimation of ocean waves\u2019 space-time fields. Comput. Geosci., 147.","DOI":"10.1016\/j.cageo.2020.104666"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Bertero, M., and Boccacci, P. (1998). Introduction to Inverse Problems in Imaging, CRC Press.","DOI":"10.1887\/0750304359"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1080\/00029890.1976.11994053","article-title":"Inverse Problems","volume":"83","author":"Keller","year":"1976","journal-title":"Am. Math. Mon."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"13891","DOI":"10.1029\/JC093iC11p13891","article-title":"Directional Measurement of Short Ocean Waves With Stereophotography","volume":"93","author":"Shemdin","year":"1988","journal-title":"J. Geophys. Res."},{"key":"ref_19","first-page":"311","article-title":"Measuring short surface waves with stereophotography","volume":"93","author":"Shemdin","year":"1992","journal-title":"Photogram. Eng. Remote Sens."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1017\/S0022112089000157","article-title":"Wavenumber spectra of short gravity waves","volume":"198","author":"Banner","year":"1989","journal-title":"J. Fluid Mech."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1023\/A:1014573219977","article-title":"A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms","volume":"47","author":"Scharstein","year":"2002","journal-title":"Int. J. Comput. Vis."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1013","DOI":"10.1016\/j.coastaleng.2006.06.012","article-title":"Measurements of short water waves using stereo matched image sequences","volume":"53","author":"Benetazzo","year":"2006","journal-title":"Coast. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1016\/j.oceaneng.2005.05.006","article-title":"Automated trinocular stereo imaging system for three-dimensional surface wave measurements","volume":"33","author":"Wanek","year":"2006","journal-title":"Ocean Eng."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gallego, G., Benetazzo, A., Yezzi, A., and Fedele, F. (2008, January 15\u201320). Wave Statistics and Spectra via a Variational Wave Acquisition Stereo System. Proceedings of the ASME 2008 27th International Conference on Offshore Mechanics and Arctic Engineering, Estoril, Portugal.","DOI":"10.1115\/OMAE2008-57160"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"4445","DOI":"10.1109\/TGRS.2011.2150230","article-title":"A Variational Stereo Method for the Three-Dimensional Reconstruction of Ocean Waves","volume":"49","author":"Gallego","year":"2011","journal-title":"Geosci. Remote Sens. IEEE Trans."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Vieira, M., Guimar\u00e3es, P., Violante-Carvalho, N., Benetazzo, A., Bergamasco, F., and Pereira, H. (2020). A low-cost stereo video system for measuring directional wind waves. J. Mar. Sci. Eng., 8.","DOI":"10.3390\/jmse8110831"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.ocemod.2018.06.007","article-title":"Characterizing the signature of a spatio-temporal wind wave field","volume":"129","author":"Benetazzo","year":"2018","journal-title":"Ocean Model."},{"key":"ref_28","first-page":"2492","article-title":"Extreme sea state measurements by stereo video system","volume":"3","author":"Leckler","year":"2019","journal-title":"Int. J. Offshore Polar Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pereira, H., Violante-Carvalho, N., Fabbri, R., Babanin, A., Pinho, U., and Skvortsov, A. (2021). An algorithm for tracking drifters dispersion induced by wave turbulence using optical cameras. Comput. Geosci., 148.","DOI":"10.1016\/j.cageo.2020.104654"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"725","DOI":"10.5194\/os-15-725-2019","article-title":"Analysis of the effect of fish oil on wind waves and implications for air-water interaction studies","volume":"15","author":"Benetazzo","year":"2019","journal-title":"Ocean Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.cageo.2016.07.012","article-title":"Multi-view horizon-driven sea plane estimation for stereo wave imaging on moving vessels","volume":"95","author":"Bergamasco","year":"2016","journal-title":"Comput. Geosci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1175\/JPO-D-16-0187.1","article-title":"Sharp-crested breaking surface waves observed from a ship-based stereo video system","volume":"47","author":"Schwendeman","year":"2017","journal-title":"J. Phys. Oceanogr."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zhou, K., Meng, X., and Cheng, B. (2020). Review of Stereo Matching Algorithms Based on Deep Learning. Comput. Intell. Neurosci., 2020.","DOI":"10.1155\/2020\/8562323"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Hagen, H., and Roller, D. (1991). Scattered Data Interpolation and Applications: A Tutorial and Survey. Geometric Modeling, Springer.","DOI":"10.1007\/978-3-642-76404-2"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Li, B., Zhang, T., and Xia, T. (2016). Vehicle Detection from 3D Lidar Using Fully Convolutional Network. arXiv.","DOI":"10.15607\/RSS.2016.XII.042"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ma, F., and Karaman, S. (2018, January 21\u201325). Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.","DOI":"10.1109\/ICRA.2018.8460184"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zweig, S., and Wolf, L. (2017, January 21\u201326). InterpoNet, A brain inspired neural network for optical flow dense interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.674"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., and Geiger, A. (2017, January 10\u201312). Sparsity Invariant CNNs. Proceedings of the 2017 International Conference on 3D Vision (3DV), Qingdao, China.","DOI":"10.1109\/3DV.2017.00012"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3429","DOI":"10.1109\/TIP.2019.2960589","article-title":"HMS-Net: Hierarchical Multi-Scale Sparsity-Invariant Network for Sparse Depth Completion","volume":"29","author":"Huang","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Jaritz, M., de Charette, R., Wirbel, E., Perrotton, X., and Nashashibi, F. (2018, January 5\u20138). Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation. Proceedings of the 2018 International Conference on 3D Vision (3DV), Verona, Italy.","DOI":"10.1109\/3DV.2018.00017"},{"key":"ref_41","unstructured":"Yan, Z., Wang, K., Li, X., Zhang, Z., Xu, B., Li, J., and Yang, J. (2021, September 20). RigNet: Repetitive Image Guided Network for Depth Completion. Available online: https:\/\/arxiv.org\/pdf\/2107.13802.pdf."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Hu, M., Wang, S., Li, B., Ning, S., Fan, L., and Gong, X. (2021, September 20). PENet: Towards Precise and Efficient Image Guided Depth Completion. Available online: https:\/\/arxiv.org\/pdf\/2103.00783.pdf.","DOI":"10.1109\/ICRA48506.2021.9561035"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Shepard, D. (1968, January 27\u201329). A Two-dimensional Interpolation Function for Irregularly-spaced Data. Proceedings of the 1968 23rd ACM National Conference, Las Vegas, NV, USA.","DOI":"10.1145\/800186.810616"},{"key":"ref_44","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. Available online: http:\/\/www.deeplearningbook.org."},{"key":"ref_45","unstructured":"Brodtkorb, P., Johannesson, P., Lindgren, G., Rychlik, I., Ryd\u00e9n, J., and Sj\u00f6, E. (June, January 27). WAFO\u2014A Matlab Toolbox for the Analysis of Random Waves and Loads. Proceedings of the Tenth International Offshore and Polar Engineering Conference, Seattle, WA, USA."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1109\/TCI.2016.2644865","article-title":"Loss Functions for Image Restoration With Neural Networks","volume":"3","author":"Zhao","year":"2017","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image Quality Assessment: From Error Visibility to Structural Similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009, January 14\u201318). Curriculum learning. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.","DOI":"10.1145\/1553374.1553380"},{"key":"ref_49","unstructured":"Graves, A., Bellemare, M.G., Menick, J., Munos, R., and Kavukcuoglu, K. (2017, January 7\u20139). Automated curriculum learning for neural networks. Proceedings of the International Conference On Machine Learning, Sydney, Australia."},{"key":"ref_50","unstructured":"Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M.E., and Stone, P. (2020). Curriculum learning for reinforcement learning domains: A framework and survey. arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/18\/3780\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:02:44Z","timestamp":1760166164000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/18\/3780"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,21]]},"references-count":50,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["rs13183780"],"URL":"https:\/\/doi.org\/10.3390\/rs13183780","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,21]]}}}