{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T10:34:19Z","timestamp":1760524459012,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T00:00:00Z","timestamp":1675814400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"French Public Authorities"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Three-dimensional-integrated focal-plane array image processor chips offer new opportunities to implement highly parallelised computer vision algorithms directly inside sensors. Neural networks in particular can perform highly complex machine vision tasks, and therefore their efficient implementation in such imagers are of significant interest. However, studies with existing pixel-processor array chips have focused on the implementation of a subset of neural network components\u2014notably convolutional kernels\u2014on pixel processor arrays. In this work, we implement a continuous end-to-end pipeline for a convolutional neural network from the digitisation of incoming photons to the output prediction vector on a macropixel processor array chip (where a single processor acts on group of pixels). Our implementation performs inference at a rate between 265 and 309 frames per second, directly inside of the sensor, by exploiting the different levels of parallelism available.<\/jats:p>","DOI":"10.3390\/s23041909","type":"journal-article","created":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T05:37:31Z","timestamp":1675834651000},"page":"1909","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["End-to-End Implementation of a Convolutional Neural Network on a 3D-Integrated Image Sensor with Macropixel Array"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8426-7789","authenticated-orcid":false,"given":"Maria","family":"Lepecq","sequence":"first","affiliation":[{"name":"Universit\u00e9 Paris-Saclay, CEA, List, F-91120 Palaiseau, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0326-2121","authenticated-orcid":false,"given":"Thomas","family":"Dalgaty","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Paris-Saclay, CEA, List, F-91120 Palaiseau, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5365-699X","authenticated-orcid":false,"given":"William","family":"Fabre","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Paris-Saclay, CEA, List, F-91120 Palaiseau, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6907-097X","authenticated-orcid":false,"given":"St\u00e9phane","family":"Chevobbe","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Paris-Saclay, CEA, List, F-91120 Palaiseau, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,8]]},"reference":[{"key":"ref_1","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_3","first-page":"91","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume":"8","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8\u201314). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Horowitz, M. (2014, January 9\u201313). 1.1 computing\u2019s energy problem (and what we can do about it). Proceedings of the 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), San Francisco, CA, USA.","DOI":"10.1109\/ISSCC.2014.6757323"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1145\/2654822.2541967","article-title":"Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning","volume":"42","author":"Chen","year":"2014","journal-title":"ACM Sigarch Comput. Archit. News"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1145\/3007787.3001177","article-title":"Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks","volume":"44","author":"Chen","year":"2016","journal-title":"ACM Sigarch Comput. Archit. News"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1109\/JETCAS.2019.2910232","article-title":"Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices","volume":"9","author":"Chen","year":"2019","journal-title":"IEEE J. Emerg. Sel. Top. Circuits Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1109\/JETCAS.2019.2952457","article-title":"CNNP-v2: A Memory-Centric Architecture for Low-Power CNN Processor on Domain-Specific Mobile Devices","volume":"9","author":"Choi","year":"2019","journal-title":"IEEE J. Emerg. Sel. Top. Circuits Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Das, S., Roy, A., Chandrasekharan, K.K., Deshwal, A., and Lee, S. (2020, January 10\u201321). A Systolic Dataflow Based Accelerator for CNNs. Proceedings of the 2020 IEEE International Symposium on Circuits and Systems (ISCAS), Sevilla, Spain.","DOI":"10.1109\/ISCAS45731.2020.9180403"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Liu, P., and Song, Y. (2021). A Hybrid Vision Processing Unit with a Pipelined Workflow for Convolutional Neural Network Accelerating and Image Signal Processing. Electronics, 10.","DOI":"10.3390\/electronics10232989"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3460776","article-title":"Configurable Multi-Directional Systolic Array Architecture for Convolutional Neural Networks","volume":"18","author":"Xu","year":"2021","journal-title":"ACM Trans. Archit. Code Optim."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Jouppi, N.P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., and Borchers, A. (2017, January 24\u201328). In-datacenter performance analysis of a tensor processing unit. Proceedings of the 44th Annual International Symposium on Computer Architecture, Toronto, ON, Canada.","DOI":"10.1145\/3079856.3080246"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1109\/TCSI.2004.840093","article-title":"A general-purpose processor-per-pixel analog SIMD vision chip","volume":"52","author":"Dudek","year":"2005","journal-title":"IEEE Trans. Circuits Syst. I Regul. Pap."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1109\/JETCAS.2021.3121259","article-title":"Near-Sensor Distributed DNN Processing for Augmented and Virtual Reality","volume":"11","author":"Pinkham","year":"2021","journal-title":"IEEE J. Emerg. Sel. Top. Circuits Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1109\/JSSC.2018.2886325","article-title":"A 5500-frames\/s 85-GOPS\/W 3-D stacked bsi vision chip based on parallel in-focal-plane acquisition and processing","volume":"54","author":"Millet","year":"2019","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_19","unstructured":"Chevobbe, S., Lepecq, M., Benchehida, K., Darouich, M., Dombek, T., Guellec, F., and Millet, L. (2019, January 23\u201327). A versatile 3D stacked vision chip with massively parallel processing enabling low latency image analysis. Proceedings of the 2019 International Image Sensor Workshop, Snowbird, UT, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nose, A., Yamazaki, T., Katayama, H., Uehara, S., Kobayashi, M., Shida, S., Odahara, M., Takamiya, K., Matsumoto, S., and Miyashita, L. (2018). Design and performance of a 1 ms high-speed vision chip with 3D-stacked 140 GOPS column-parallel PEs. Sensors, 18.","DOI":"10.3390\/s18051313"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Eki, R., Yamada, S., Ozawa, H., Kai, H., Okuike, K., Gowtham, H., Nakanishi, H., Almog, E., Livne, Y., and Yuval, G. (2021, January 13\u201322). 9.6 A 1\/2.3 inch 12.3 Mpixel with on-chip 4.97 TOPS\/W CNN processor back-illuminated stacked CMOS image sensor. Proceedings of the 2021 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA.","DOI":"10.1109\/ISSCC42613.2021.9365965"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Gontard, L.C., Carmona-Gal\u00e1n, R., and Rodr\u00edguez-V\u00e1zquez, A. (2020, January 10\u201321). Cellular-Neural-Network Focal-Plane Processor as Pre-Processor for ConvNet Inference. Proceedings of the 2020 IEEE International Symposium on Circuits and Systems (ISCAS), Sevilla, Spain.","DOI":"10.1109\/ISCAS45731.2020.9181102"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bose, L., Dudek, P., Chen, J., Carey, S.J., and Mayol-Cuevas, W.W. (2020, January 23\u201328). Fully embedding fast convolutional networks on pixel processor arrays. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58526-6_29"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_25","unstructured":"(2022, February 01). N2D2: Post-Training Quantization. Available online: https:\/\/cea-list.github.io\/N2D2-docs\/quant\/post.html."},{"key":"ref_26","unstructured":"Bose, L., Chen, J., Carey, S.J., Dudek, P., and Mayol-Cuevas, W. (November, January 27). A camera that CNNs: Towards embedded neural networks on pixel processor arrays. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Lu, Y. (2022, January 19\u201330). On-Sensor Binarized Fully Convolutional Neural Network for Localisation and Coarse Segmentation. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA.","DOI":"10.1109\/CVPRW56347.2022.00407"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/1909\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:28:11Z","timestamp":1760120891000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/1909"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,8]]},"references-count":27,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23041909"],"URL":"https:\/\/doi.org\/10.3390\/s23041909","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,2,8]]}}}