{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:18:30Z","timestamp":1760242710627,"version":"build-2065373602"},"reference-count":23,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2016,4,6]],"date-time":"2016-04-06T00:00:00Z","timestamp":1459900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000006","name":"ONR","doi-asserted-by":"publisher","award":["00014-10-1-0933"],"award-info":[{"award-number":["00014-10-1-0933"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Gordon and Betty Moore"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Novel image sensors transduce the stream of photons directly into asynchronous electrical pulses, rather than forming an image. Classical approaches to vision start from a good quality image and therefore it is tempting to consider image reconstruction as a first step to image analysis. We propose that, instead, one should focus on the task at hand (e.g., detection, tracking or control) and design algorithms that compute the relevant variables (class, position, velocity) directly from the stream of photons. We discuss three examples of such computer vision algorithms and test them on simulated data from photon-counting sensors. Such algorithms work just-in-time, i.e., they complete classification, search and tracking with high accuracy as soon as the information is sufficient, which is typically before there are enough photons to form a high-quality image. We argue that this is particularly useful when the photons are few or expensive, e.g., in astronomy, biological imaging, surveillance and night vision.<\/jats:p>","DOI":"10.3390\/s16040484","type":"journal-article","created":{"date-parts":[[2016,4,6]],"date-time":"2016-04-06T12:54:18Z","timestamp":1459947258000},"page":"484","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Vision without the Image"],"prefix":"10.3390","volume":"16","author":[{"given":"Bo","family":"Chen","sequence":"first","affiliation":[{"name":"Computation and Neural Systems, California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125, USA"}]},{"given":"Pietro","family":"Perona","sequence":"additional","affiliation":[{"name":"Computation and Neural Systems, California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125, USA"}]}],"member":"1968","published-online":{"date-parts":[[2016,4,6]]},"reference":[{"doi-asserted-by":"crossref","unstructured":"Hall, E., and Brenner, D. (2014). Cancer risks from diagnostic radiology. Cancer, 81.","key":"ref_1","DOI":"10.1259\/bjr\/01948454"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1126\/science.1082160","article-title":"Light microscopy techniques for live cell imaging","volume":"300","author":"Stephens","year":"2003","journal-title":"Science"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1038\/nphoton.2010.29","article-title":"Experimental realization of sub-shot-noise quantum imaging","volume":"4","author":"Brida","year":"2010","journal-title":"Nat. Photonics"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.sna.2007.06.021","article-title":"Principles and features of single-photon avalanche diode arrays","volume":"140","author":"Zappa","year":"2007","journal-title":"Sens. Actuators A Phys."},{"doi-asserted-by":"crossref","unstructured":"Fossum, E. (2011, January 10\u201314). The quanta image sensor (QIS): Concepts and challenges. Proceedings of the Computational Optical Sensing and Imaging 2011, Toronto, ON, Canada.","key":"ref_5","DOI":"10.1364\/COSI.2011.JTuE1"},{"unstructured":"Fossum, E.R. (2015, January 8\u201312). Multi-Bit Quanta Image Sensors. Proceedings of the International Image Sensor Workshop, Vaals, The Netherlands.","key":"ref_6"},{"doi-asserted-by":"crossref","unstructured":"Sbaiz, L., Yang, F., Charbon, E., S\u00fcsstrunk, S., and Vetterli, M. (2009, January 19\u201324). The gigavision camera. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.","key":"ref_7","DOI":"10.1109\/ICASSP.2009.4959778"},{"doi-asserted-by":"crossref","unstructured":"Morris, P.A., Aspden, R.S., Bell, J.E., Boyd, R.W., and Padgett, M.J. (2015). Imaging with a small number of photons. Nat. Commun., 6.","key":"ref_8","DOI":"10.1038\/ncomms6913"},{"doi-asserted-by":"crossref","unstructured":"Chen, B., and Perona, P. (2015, January 7\u201313). Scotopic Visual Recognition. Proceedings of the IEEE International Conference on Computer Vision Workshops, Santiago, Chile.","key":"ref_9","DOI":"10.1109\/ICCVW.2015.88"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1364\/JOSAA.23.000272","article-title":"Methods to detect objects in photon-limited images","volume":"23","author":"Galatsanos","year":"2006","journal-title":"JOSA A"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"IEEE Proc."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1214\/aoms\/1177731118","article-title":"Sequential tests of statistical hypotheses","volume":"16","author":"Wald","year":"1945","journal-title":"Ann. Math. Stat."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/BF00114844","article-title":"Distributed representations, simple recurrent networks, and grammatical structure","volume":"7","author":"Elman","year":"1991","journal-title":"Mach. Learn."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1167\/15.16.9","article-title":"Speed versus accuracy in visual search: Optimal performance and neural architecture","volume":"15","author":"Chen","year":"2015","journal-title":"J. Vis."},{"key":"ref_15","first-page":"35","article-title":"A new approach to linear filtering and prediction problems","volume":"82","author":"Kalman","year":"1960","journal-title":"J. Fluids Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1080\/01621459.1999.10474153","article-title":"Filtering via simulation: Auxiliary particle filters","volume":"94","author":"Pitt","year":"1999","journal-title":"J. Amer. Stat. Assoc."},{"unstructured":"Liberzon, D. (2012). Switching in Systems and Control, Springer Science & Business Media.","key":"ref_17"},{"unstructured":"Vedaldi, A., and Lenc, K. MatConvNet\u2014Convolutional Neural Networks for MATLAB. Available online: http:\/\/arxiv.org\/abs\/1412.4564.","key":"ref_18"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1088\/0004-637X\/786\/2\/106","article-title":"Intergalactic medium emission observations with the Cosmic Web Imager. I. The circum-QSO medium of QSO 1549+19, and evidence for a filamentary gas inflow","volume":"786","author":"Martin","year":"2014","journal-title":"Astrophys. J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1038\/nbt1278","article-title":"Controlled light-exposure microscopy reduces photobleaching and phototoxicity in fluorescence live-cell imaging","volume":"25","author":"Hoebe","year":"2007","journal-title":"Nat. Biotechnol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1038\/nmeth.1175","article-title":"High-speed, low-photodamage nonlinear imaging using passive pulse splitters","volume":"5","author":"Ji","year":"2008","journal-title":"Nat. Methods"},{"doi-asserted-by":"crossref","unstructured":"Cheng, D., Price, B., Cohen, S., and Brown, M.S. (2015, January 7\u201312). Effective Learning-Based Illuminant Estimation Using Simple Features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","key":"ref_22","DOI":"10.1109\/CVPR.2015.7298702"},{"unstructured":"Ovtcharov, K., Ruwase, O., Kim, J.Y., Fowers, J., Strauss, K., and Chung, E.S. Accelerating Deep Convolutional Neural Networks Using Specialized Hardware, Microsoft Research Whitepaper. Available online: http:\/\/research.microsoft.com\/apps\/pubs\/?id=240715.","key":"ref_23"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/4\/484\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:21:49Z","timestamp":1760210509000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/4\/484"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,4,6]]},"references-count":23,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2016,4]]}},"alternative-id":["s16040484"],"URL":"https:\/\/doi.org\/10.3390\/s16040484","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2016,4,6]]}}}