{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T17:35:33Z","timestamp":1785605733946,"version":"3.56.0"},"reference-count":48,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,10,26]],"date-time":"2023-10-26T00:00:00Z","timestamp":1698278400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Common RGBD, CMOS, and CCD-based cameras produce motion blur and incorrect exposure under high-speed and improper lighting conditions. According to the bionic principle, the event camera developed has the advantages of low delay, high dynamic range, and no motion blur. However, due to its unique data representation, it encounters significant obstacles in practical applications. The image reconstruction algorithm based on an event camera solves the problem by converting a series of \u201cevents\u201d into common frames to apply existing vision algorithms. Due to the rapid development of neural networks, this field has made significant breakthroughs in past few years. Based on the most popular Events-to-Video (E2VID) method, this study designs a new network called E2VIDX. The proposed network includes group convolution and sub-pixel convolution, which not only achieves better feature fusion but also the network model size is reduced by 25%. Futhermore, we propose a new loss function. The loss function is divided into two parts, first part calculates the high level features and the second part calculates the low level features of the reconstructed image. The experimental results clearly outperform against the state-of-the-art method. Compared with the original method, Structural Similarity (SSIM) increases by 1.3%, Learned Perceptual Image Patch Similarity (LPIPS) decreases by 1.7%, Mean Squared Error (MSE) decreases by 2.5%, and it runs faster on GPU and CPU. Additionally, we evaluate the results of E2VIDX with application to image classification, object detection, and instance segmentation. The experiments show that conversions using our method can help event cameras directly apply existing vision algorithms in most scenarios.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1277160","type":"journal-article","created":{"date-parts":[[2023,10,26]],"date-time":"2023-10-26T14:45:42Z","timestamp":1698331542000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["E2VIDX: improved bridge between conventional vision and bionic vision"],"prefix":"10.3389","volume":"17","author":[{"given":"Xujia","family":"Hou","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Feihu","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dhiraj","family":"Gulati","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tingfeng","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,10,26]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"33","DOI":"10.3390\/electronics8010033","article-title":"Embedded microcontroller with a ccd camera as a digital lighting control system","volume":"8","author":"Adam","year":"2019","journal-title":"Electronics"},{"key":"B2","first-page":"1","article-title":"\u201cMeta-reinforcement learning in nonstationary and nonparametric environments,\u201d","author":"Bing","year":"","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"B3","doi-asserted-by":"publisher","first-page":"3476","DOI":"10.1109\/TPAMI.2022.3185549","article-title":"Meta-reinforcement learning in non-stationary and dynamic environments","volume":"45","author":"Bing","year":"","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B4","doi-asserted-by":"publisher","first-page":"2147","DOI":"10.1109\/TNNLS.2021.3128380","article-title":"Toward cognitive navigation: Design and implementation of a biologically inspired head direction cell network","volume":"33","author":"Bing","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst"},{"key":"B5","first-page":"9157","article-title":"\u201cYolact: Real-time instance segmentation,\u201d","author":"Bolya","year":"2019","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B6","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1109\/ISCAS.2014.6865228","article-title":"\u201cReal-time, high-speed video decompression using a frame-and event-based davis sensor,\u201d","author":"Brandli","year":"2014","journal-title":"2014 IEEE International Symposium on Circuits and Systems (ISCAS)"},{"key":"B7","doi-asserted-by":"publisher","first-page":"2488","DOI":"10.1109\/TIP.2021.3052070","article-title":"Spade-e2vid: Spatially-adaptive denormalization for event-based video reconstruction","volume":"30","author":"Cadena","year":"2021","journal-title":"IEEE Trans. Image Process"},{"key":"B8","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1109\/TPAMI.2020.3008413","article-title":"Event-based vision: a survey","volume":"44","author":"Gallego","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B9","first-page":"1440","article-title":"\u201cFast r-cnn,\u201d","author":"Girshick","year":"2015","journal-title":"Proceedings of the IEEE International Conference on Computer Vision"},{"key":"B10","first-page":"1312","article-title":"\u201cv2e: From video frames to realistic DVS events,\u201d","author":"Hu","year":"2021","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B11","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1016\/j.autcon.2022.104242","article-title":"Developments, challenges, and perspectives of railway inspection robots","volume":"104242","author":"Jing","year":"2022","journal-title":"Automat. Construct"},{"key":"B12","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv preprint"},{"key":"B13","author":"Krizhevsky","year":"2009","journal-title":"Learning Multiple Layers of Features From Tiny Images"},{"key":"B14","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"B15","article-title":"FSSD: feature fusion single shot multibox detector","author":"Li","year":"2017","journal-title":"arXiv preprint"},{"key":"B16","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","article-title":"\u201cMicrosoft coco: Common objects in context,\u201d","volume-title":"Computer Vision-ECCV 2014: 13th European Conference","author":"Lin","year":"2014"},{"key":"B17","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1016\/j.engappai.2022.105399","article-title":"A review of driver fatigue detection and its advances on the use of rgb-d camera and deep learning","volume":"105399","author":"Liu","year":"2022","journal-title":"Eng. Appl. Artif. Intell"},{"key":"B18","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-319-46448-0_2","article-title":"\u201cSSD: Single shot multibox detector,\u201d","volume-title":"Computer Vision-ECCV 2016: 14th European Conference","author":"Liu","year":"2016"},{"key":"B19","doi-asserted-by":"publisher","first-page":"1381","DOI":"10.1007\/s11263-018-1106-2","article-title":"Real-time intensity-image reconstruction for event cameras using manifold regularisation","volume":"126","author":"Munda","year":"2018","journal-title":"Int. J. Comput. Vis"},{"key":"B20","doi-asserted-by":"publisher","first-page":"9","DOI":"10.3389\/fnins.2015.00437","article-title":"Converting static image datasets to spiking neuromorphic datasets using saccades","volume":"437","author":"Orchard","year":"2015","journal-title":"Front. Neurosci"},{"key":"B21","first-page":"2337","article-title":"\u201cSemantic image synthesis with spatially-adaptive normalization,\u201d","author":"Park","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B22","first-page":"16639","article-title":"Learning to detect objects with a 1 megapixel event camera","volume":"33","author":"Perot","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B23","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1109\/JSSC.2010.2085952","article-title":"A QVGA 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds","volume":"46","author":"Posch","year":"2010","journal-title":"IEEE J. Solid-State Circ"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1470","DOI":"10.1109\/JPROC.2014.2346153","article-title":"Retinomorphic event-based vision sensors: bioinspired cameras with spiking output","volume":"102","author":"Posch","year":"2014","journal-title":"Proc. IEEE"},{"key":"B25","first-page":"969","article-title":"\u201cEsim: an open event camera simulator,\u201d","author":"Rebecq","year":"2018","journal-title":"Conference on Robot Learning"},{"key":"B26","first-page":"3857","article-title":"\u201cEvents-to-video: Bringing modern computer vision to event cameras,\u201d","author":"Rebecq","year":"","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B27","doi-asserted-by":"publisher","first-page":"1964","DOI":"10.1109\/TPAMI.2019.2963386","article-title":"High speed and high dynamic range video with an event camera","volume":"43","author":"Rebecq","year":"","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B28","first-page":"779","article-title":"\u201cYou only look once: Unified, real-time object detection,\u201d","author":"Redmon","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B29","first-page":"7263","article-title":"\u201cYolo9000: better, faster, stronger,\u201d","author":"Redmon","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B30","article-title":"Yolov3: An incremental improvement","author":"Redmon","year":"2018","journal-title":"arXiv preprint"},{"key":"B31","first-page":"28","article-title":"Faster r-cnn: towards real-time object detection with region proposal networks","volume":"2015","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-24574-4_28","article-title":"\u201cU-net: Convolutional networks for biomedical image segmentation,\u201d","volume-title":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","author":"Ronneberger","year":"2015"},{"key":"B33","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis"},{"key":"B34","first-page":"308","article-title":"\u201cContinuous-time intensity estimation using event cameras,\u201d","volume-title":"Asian Conference on Computer Vision","author":"Scheerlinck","year":"2018"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1109\/WACV45572.2020.9093366","article-title":"\u201cFast image reconstruction with an event camera,\u201d","author":"Scheerlinck","year":"2020","journal-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision"},{"key":"B36","first-page":"1874","article-title":"\u201cReal-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,\u201d","author":"Shi","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B37","article-title":"Convolutional lstm network: a machine learning approach for precipitation nowcasting","author":"Shi","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B38","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2014","journal-title":"arXiv preprint"},{"key":"B39","doi-asserted-by":"publisher","first-page":"488","DOI":"10.3390\/s21020488","article-title":"Cmos image sensors in surveillance system applications","volume":"21","author":"Sukhavasi","year":"2021","journal-title":"Sensors"},{"key":"B40","doi-asserted-by":"publisher","first-page":"994","DOI":"10.1109\/LRA.2018.2793357","article-title":"Ultimate slam? combining events, images, and imu for robust visual slam in hdr and high-speed scenarios","volume":"3","author":"Vidal","year":"2018","journal-title":"IEEE Robot. Automat. Lett"},{"key":"B41","first-page":"10081","article-title":"\u201cEvent-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang","year":"2019"},{"key":"B42","doi-asserted-by":"publisher","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: from error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE transactions on image processing"},{"key":"B43","first-page":"1492","article-title":"\u201cAggregated residual transformations for deep neural networks,\u201d","author":"Xie","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B44","first-page":"4968","article-title":"\u201cEventcap: Monocular 3d capture of high-speed human motions using an event camera,\u201d","author":"Xu","year":"2020","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B45","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1016\/j.cviu.2019.102827","article-title":"Assd: attentive single shot multibox detector","volume":"102827","author":"Yi","year":"2019","journal-title":"Comp. Vision Image Underst"},{"key":"B46","first-page":"586","article-title":"\u201cThe unreasonable effectiveness of deep features as a perceptual metric,\u201d","author":"Zhang","year":"2018","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B47","doi-asserted-by":"publisher","first-page":"12274","DOI":"10.3390\/su141912274","article-title":"Real-time vehicle detection based on improved yolo v5","volume":"14","author":"Zhang","year":"2022","journal-title":"Sustainability"},{"key":"B48","first-page":"235","article-title":"\u201cSemi-dense 3D reconstruction with a stereo event camera,\u201d","author":"Zhou","year":"2018","journal-title":"Proceedings of the European Conference on Computer Vision (ECCV)"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1277160\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,26]],"date-time":"2023-10-26T14:46:06Z","timestamp":1698331566000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1277160\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,26]]},"references-count":48,"alternative-id":["10.3389\/fnbot.2023.1277160"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1277160","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,26]]},"article-number":"1277160"}}