{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T16:52:34Z","timestamp":1783702354024,"version":"3.55.0"},"reference-count":52,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2023,8,30]],"date-time":"2023-08-30T00:00:00Z","timestamp":1693353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JSPS KAKENHI","award":["17H06102"],"award-info":[{"award-number":["17H06102"]}]},{"name":"JSPS KAKENHI","award":["18K19818"],"award-info":[{"award-number":["18K19818"]}]},{"name":"JSPS KAKENHI","award":["20K20628"],"award-info":[{"award-number":["20K20628"]}]},{"name":"JSPS KAKENHI","award":["23KJ1050"],"award-info":[{"award-number":["23KJ1050"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>A camera captures multidimensional information of the real world by convolving it into two dimensions using a sensing matrix. The original multidimensional information is then reconstructed from captured images. Traditionally, multidimensional information has been captured by uniform sampling, but by optimizing the sensing matrix, we can capture images more efficiently and reconstruct multidimensional information with high quality. Although compressive video sensing requires random sampling as a theoretical optimum, when designing the sensing matrix in practice, there are many hardware limitations (such as exposure and color filter patterns). Existing studies have found random sampling is not always the best solution for compressive sensing because the optimal sampling pattern is related to the scene context, and it is hard to manually design a sampling pattern and reconstruction algorithm. In this paper, we propose an end-to-end learning approach that jointly optimizes the sampling pattern as well as the reconstruction decoder. We applied this deep sensing approach to the video compressive sensing problem. We modeled the spatio\u2013temporal sampling and color filter pattern using a convolutional neural network constrained by hardware limitations during network training. We demonstrated that the proposed method performs better than the manually designed method in gray-scale video and color video acquisitions.<\/jats:p>","DOI":"10.3390\/s23177535","type":"journal-article","created":{"date-parts":[[2023,8,30]],"date-time":"2023-08-30T10:30:52Z","timestamp":1693391452000},"page":"7535","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Deep Sensing for Compressive Video Acquisition"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2227-6345","authenticated-orcid":false,"given":"Michitaka","family":"Yoshida","sequence":"first","affiliation":[{"name":"Japan Society for the Promotion of Science, Shizuoka University, Hamamatsu 102-0083, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akihiko","family":"Torii","sequence":"additional","affiliation":[{"name":"Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology, Tokyo 152-8550, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masatoshi","family":"Okutomi","sequence":"additional","affiliation":[{"name":"Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology, Tokyo 152-8550, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2588-6894","authenticated-orcid":false,"given":"Rin-ichiro","family":"Taniguchi","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hajime","family":"Nagahara","sequence":"additional","affiliation":[{"name":"Institute of Datability Science, Osaka University, Suita 565-0871, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasushi","family":"Yagi","sequence":"additional","affiliation":[{"name":"Institute of Datability Science, Osaka University, Suita 565-0871, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,30]]},"reference":[{"key":"ref_1","unstructured":"Bayer, B.E. (1976). Color Imaging Array. (U.S. Patent 3,971,065)."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Condat, L. (2009, January 7\u201310). A new random color filter array with good spectral properties. Proceedings of the International Conference on Image Processing (ICIP), IEEE, Cairo, Egypt.","DOI":"10.1109\/ICIP.2009.5413678"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., and Nayar, S.K. (2011, January 6\u201313). Video from a single coded exposure photograph using a learned over-complete dictionary. Proceedings of the International Conference on Computer Vision (ICCV), Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126254"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Sonoda, T., Nagahara, H., Endo, K., Sugiyama, Y., and Taniguchi, R. (2016, January 13\u201315). High-speed imaging using CMOS image sensor with quasi pixel-wise exposure. Proceedings of the International Conference on Computational Photography (ICCP), IEEE, Evanston, IL, USA.","DOI":"10.1109\/ICCPHOT.2016.7492875"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1109\/TCSVT.2012.2207269","article-title":"Motion-Aware Decoding of Compressed-Sensed Video","volume":"23","author":"Liu","year":"2013","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1109\/TCSVT.2015.2418586","article-title":"Multihypothesis Compressed Video Sensing Technique","volume":"26","author":"Azghani","year":"2016","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1182","DOI":"10.1109\/TCSVT.2016.2527181","article-title":"Video Compressive Sensing Reconstruction via Reweighted Residual Sparsity","volume":"27","author":"Zhao","year":"2017","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"4863","DOI":"10.1109\/TIP.2014.2344294","article-title":"Video compressive sensing using Gaussian mixture models","volume":"23","author":"Yang","year":"2014","journal-title":"IEEE Trans. Image Process."},{"key":"ref_9","unstructured":"Chakrabarti, A. (2016, January 5\u201310). Learning sensor multiplexing design through back-propagation. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Nie, S., Gu, L., Zheng, Y., Lam, A., Ono, N., and Sato, I. (2018, January 18). Deeply learned filter response functions for hyperspectral reconstruction. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00501"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.dsp.2017.09.010","article-title":"Deep fully-connected networks for video compressive sensing","volume":"72","author":"Iliadis","year":"2018","journal-title":"Digit. Signal Process."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ma, J., Liu, X., Shou, Z., and Yuan, X. (2019, January 15\u201320). Deep tensor admm-net for snapshot compressive imaging. Proceedings of the International Conference on Computer Vision (ICCV), Long Beach, CA, USA.","DOI":"10.1109\/ICCV.2019.01032"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yuan, X., Liu, Y., Suo, J., and Dai, Q. (2020, January 14\u201319). Plug-and-play algorithms for large-scale snapshot compressive imaging. Proceedings of the Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00152"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Han, X., Wu, B., Shou, Z., Liu, X.Y., Zhang, Y., and Kong, L. (2020, January 13\u201319). Tensor FISTA-Net for real-time snapshot compressive imaging. Proceedings of the AAAI Conference on Artificial Intelligence, Seattle, WA, USA.","DOI":"10.1609\/aaai.v34i07.6726"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Li, Y., Qi, M., Gulve, R., Wei, M., Genov, R., Kutulakos, K.N., and Heidrich, W. (2020, January 24\u201326). End-to-End Video Compressive Sensing Using Anderson-Accelerated Unrolled Networks. Proceedings of the International Conference on Computational Photography (ICCP), IEEE, Saint Louis, MO, USA.","DOI":"10.1109\/ICCP48838.2020.9105237"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Yoshida, M., Torii, A., Okutomi, M., Endo, K., Sugiyama, Y., Taniguchi, R., and Nagahara, H. (2018, January 8\u201314). Joint optimization for compressive video sensing and reconstruction under hardware constraints. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01249-6_39"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Inagaki, Y., Kobayashi, Y., Takahashi, K., Fujii, T., and Nagahara, H. (2018, January 8\u201314). Learning to capture light fields through a coded aperture camera. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_26"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A., and Veeraraghavan, A. (2019, January 15\u201317). PhaseCam3D\u2014Learning Phase Masks for Passive Single View Depth Estimation. Proceedings of the International Conference on Computational Photography (ICCP), IEEE, Tokyo, Japan.","DOI":"10.1109\/ICCPHOT.2019.8747330"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sun, H., Dalca, A.V., and Bouman, K.L. (2020, January 24\u201326). Learning a Probabilistic Strategy for Computational Imaging Sensor Selection. Proceedings of the International Conference on Computational Photography (ICCP), IEEE, Saint Louis, MO, USA.","DOI":"10.1109\/ICCP48838.2020.9105133"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"102591","DOI":"10.1016\/j.dsp.2019.102591","article-title":"Deepbinarymask: Learning a binary mask for video compressive sensing","volume":"96","author":"Iliadis","year":"2020","journal-title":"Digit. Signal Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1647","DOI":"10.3390\/s18051647","article-title":"Sensitivity and resolution improvement in RGBW color filter array sensor","volume":"18","author":"Jee","year":"2018","journal-title":"Sensors"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"15678","DOI":"10.1364\/OE.391253","article-title":"Color reproduction pipeline for an RGBW color filter array sensor","volume":"28","author":"Choi","year":"2020","journal-title":"Opt. Express"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, X., Gunturk, B., and Zhang, L. (2008, January 27\u201331). Image demosaicing: A systematic survey. Proceedings of the Visual Communications and Image Processing 2008, International Society for Optics and Photonics, San Jose, CA, USA.","DOI":"10.1117\/12.766768"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sato, S., Wakai, N., Nobori, K., Azuma, T., Miyata, T., and Nakashizuka, M. (2017, January 8\u201312). Compressive color sensing using random complementary color filter array. Proceedings of the International Conference on Machine Vision Applications (MVA), IEEE, Nagoya, Japan.","DOI":"10.23919\/MVA.2017.7986768"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1876","DOI":"10.1109\/TIP.2008.2002164","article-title":"Spatio-Spectral Color Filter Array Design for Optimal Image Recovery","volume":"17","author":"Hirakawa","year":"2008","journal-title":"IEEE Trans. Image Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Saideni, W., Helbert, D., Courreges, F., and Cances, J.P. (2022). An overview on deep learning techniques for video compressive sensing. Appl. Sci., 12.","DOI":"10.3390\/app12052734"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Xia, K., Pan, Z., and Mao, P. (2022). Video Compressive sensing reconstruction using unfolded LSTM. Sensors, 22.","DOI":"10.3390\/s22197172"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/MSP.2007.914730","article-title":"Single-pixel imaging via compressive sampling","volume":"25","author":"Duarte","year":"2008","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1038\/nmeth.1429","article-title":"Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging","volume":"7","author":"Bub","year":"2010","journal-title":"Nat. Methods"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Gupta, M., Agrawal, A., Veeraraghavan, A., and Narasimhan, S.G. (2010, January 5\u201311). Flexible voxels for motion-aware videography. Proceedings of the European Conference on Computer Vision (ECCV), Crete, Greece.","DOI":"10.1007\/978-3-642-15549-9_8"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1406","DOI":"10.1109\/TCSVT.2016.2540073","article-title":"Compressive Sensing Reconstruction for Video: An Adaptive Approach Based on Motion Estimation","volume":"27","author":"Ding","year":"2017","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wen, J., Huang, J., Chen, X., Huang, K., and Sun, Y. (2023). Transformer-Based Cascading Reconstruction Network for Video Snapshot Compressive Imaging. Appl. Sci., 13.","DOI":"10.3390\/app13105922"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"4961","DOI":"10.3390\/s130404961","article-title":"Compressive sensing image sensors-hardware implementation","volume":"13","author":"Dadkhah","year":"2013","journal-title":"Sensors"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wei, M., Sarhangnejad, N., Xia, Z., Gusev, N., Katic, N., Genov, R., and Kutulakos, K.N. (2018, January 8\u201314). Coded Two-Bucket Cameras for Computer Vision. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01219-9_4"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_37","first-page":"613","article-title":"Beitrag zum Verst\u00e4ndnis der magnetischen Erscheinungen in festen K\u00f6rpern","volume":"21","author":"Lenz","year":"1920","journal-title":"Phys. Z."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"883","DOI":"10.1103\/RevModPhys.39.883","article-title":"History of the Lenz-Ising model","volume":"39","author":"Brush","year":"1967","journal-title":"Rev. Mod. Phys."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Yuan, X. (2016, January 25\u201328). Generalized alternating projection based total variation minimization for compressive sensing. Proceedings of the International Conference on Image Processing (ICIP), IEEE, Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7532817"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Cheng, Z., Chen, B., Liu, G., Zhang, H., Lu, R., Wang, Z., and Yuan, X. (2021, January 20\u201325). Memory-efficient network for large-scale video compressive sensing. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01598"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"5931","DOI":"10.1109\/TCSVT.2022.3164241","article-title":"Video Snapshot Compressive Imaging Using Residual Ensemble Network","volume":"32","author":"Sun","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1642","DOI":"10.1109\/TPAMI.2020.2986944","article-title":"Neural sensors: Learning pixel exposures for hdr imaging and video compressive sensing with programmable sensors","volume":"42","author":"Martel","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Saragadam, V., and Sankaranarayanan, A.C. (2020, January 24\u201326). Programmable Spectrometry: Per-pixel Material Classification using Learned Spectral Filters. Proceedings of the International Conference on Computational Photography (ICCP), IEEE, Saint Louis, MS, USA.","DOI":"10.1109\/ICCP48838.2020.9105281"},{"key":"ref_44","unstructured":"Tan, R., Zhang, K., Zuo, W., and Zhang, L. (2017, January 10\u201314). Color image demosaicking via deep residual learning. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2408","DOI":"10.1109\/TIP.2018.2803341","article-title":"DeepDemosaicking: Adaptive image demosaicking via multiple deep fully convolutional networks","volume":"27","author":"Tan","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2980179.2982399","article-title":"Deep joint demosaicking and denoising","volume":"35","author":"Gharbi","year":"2016","journal-title":"ACM Trans. Graph."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Kokkinos, F., and Lefkimmiatis, S. (2018, January 8\u201314). Deep image demosaicking using a cascade of convolutional residual denoising networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01264-9_19"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"128076","DOI":"10.1109\/ACCESS.2019.2939578","article-title":"Color Filter Array Demosaicking Using Densely Connected Residual Network","volume":"7","author":"Park","year":"2019","journal-title":"IEEE Access"},{"key":"ref_49","unstructured":"Gomez, A.N., Ren, M., Urtasun, R., and Grosse, R.B. (2017). The reversible residual network: Backpropagation without storing activations. arXiv."},{"key":"ref_50","unstructured":"Pont-Tuset, J., Perazzi, F., Caelles, S., Arbel\u00e1ez, P., Sorkine-Hornung, A., and Van Gool, L. (2017). The 2017 DAVIS Challenge on Video Object Segmentation. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16\u201321). Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6248074"},{"key":"ref_52","unstructured":"Hamamatsu Photonics, K.K. (2015). Imaging Device. (Japan Patent JP2015-216594A)."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/17\/7535\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:43:06Z","timestamp":1760128986000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/17\/7535"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,30]]},"references-count":52,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23177535"],"URL":"https:\/\/doi.org\/10.3390\/s23177535","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,30]]}}}