{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T16:43:53Z","timestamp":1777567433951,"version":"3.51.4"},"reference-count":35,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2018,3,5]],"date-time":"2018-03-05T00:00:00Z","timestamp":1520208000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The photometric stereo method enables estimation of surface normals from images that have been captured using different but known lighting directions. The classical photometric stereo method requires at least three images to determine the normals in a given scene. However, this method cannot be applied to dynamic scenes because it is assumed that the scene remains static while the required images are captured. In this work, we present a dynamic photometric stereo method for estimation of the surface normals in a dynamic scene. We use a multi-tap complementary metal-oxide-semiconductor (CMOS) image sensor to capture the input images required for the proposed photometric stereo method. This image sensor can divide the electrons from the photodiode from a single pixel into the different taps of the exposures and can thus capture multiple images under different lighting conditions with almost identical timing. We implemented a camera lighting system and created a software application to enable estimation of the normal map in real time. We also evaluated the accuracy of the estimated surface normals and demonstrated that our proposed method can estimate the surface normals of dynamic scenes.<\/jats:p>","DOI":"10.3390\/s18030786","type":"journal-article","created":{"date-parts":[[2018,3,6]],"date-time":"2018-03-06T07:37:25Z","timestamp":1520321845000},"page":"786","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["The Dynamic Photometric Stereo Method Using a Multi-Tap CMOS Image Sensor"],"prefix":"10.3390","volume":"18","author":[{"given":"Takuya","family":"Yoda","sequence":"first","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hajime","family":"Nagahara","sequence":"additional","affiliation":[{"name":"Institute for Datability Science, Osaka University, Osaka 565-0871, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rin-ichiro","family":"Taniguchi","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keiichiro","family":"Kagawa","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 432-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keita","family":"Yasutomi","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 432-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shoji","family":"Kawahito","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 432-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,3,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2666","DOI":"10.1016\/j.patcog.2010.03.004","article-title":"A State of the Art in Structured Light Patterns for Surface Profilometry","volume":"43","author":"Salvi","year":"2010","journal-title":"Pattern Recogn."},{"key":"ref_2","unstructured":"Gokturk, S., Yalcin, H., and Bamji, C. (July, January 27). A time-of-flight depth sensor-system description, issues and solutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1117\/12.7972479","article-title":"Photometric method for determining surface orientation from multiple images","volume":"19","author":"Woodham","year":"1980","journal-title":"Opt. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1016\/j.imavis.2006.02.009","article-title":"3D shape-based face representation and feature extraction for face recognition","volume":"24","author":"Akarun","year":"2006","journal-title":"Image Vis. Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"076017","DOI":"10.1117\/1.JBO.18.7.076017","article-title":"Photometric stereo endoscopy","volume":"18","author":"Parot","year":"2013","journal-title":"J. Biomed. Opt."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/BF01427152","article-title":"Three-dimensional shape from color photometric stereo","volume":"13","author":"Christensen","year":"1994","journal-title":"Int. J. Comput. Vis."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez, C., Vogiatzis, G., Brostow, G.J., Stenger, B., and Cipolla, R. (2007, January 14\u201321). Non-rigid Photometric Stereo with Colored Lights. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.","DOI":"10.1109\/ICCV.2007.4408939"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Smith, M., and Smith, L. (2005, January 6\u20138). Dynamic Photometric Stereo. Proceedings of the Image Analysis and Processing\u2014 ICIAP 2005, Cagliari, Italy.","DOI":"10.1007\/11553595_101"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Jank\u00f3, Z., Delaunoy, A., and Prados, E. (2011). Colour Dynamic Photometric Stereo for Textured Surfaces. Computer Vision\u2014ACCV 2010, Springer.","DOI":"10.1007\/978-3-642-19309-5_5"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kim, H., Wilburn, B., and Ben-Ezra, M. (2010, January 5\u201311). Photometric Stereo for Dynamic Surface Orientations. Proceedings of the 11th European Conference on Computer Vision, Heraklion, Crete, Greece.","DOI":"10.1007\/978-3-642-15549-9_5"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1145\/1618452.1618520","article-title":"Dynamic shape capture using multi-view photometric stereo","volume":"28","author":"Vlasic","year":"2009","journal-title":"ACM Trans. Graph."},{"key":"ref_12","unstructured":"Malzbender, T., Wilburn, B., Gelb, D., and Ambrisco, B. (2006, January 26\u201328). Surface Enhancement Using Real-time Photometric Stereo and Reflectance Transformation. Proceedings of the 17th Eurographics Conference on Rendering Techniques, Eurographics Association, Aire-la-Ville, Switzerland."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yoda, T., Nagahara, H., Taniguchi, R.i., Kagawa, K., Yasutomi, K., and Kawahito, S. (2016, January 4\u20138). Dynamic Photometric Stereo Method using Multi-tap CMOS Image Sensor. Proceedings of the 23rd International Conference on Pattern Recognition, Canc\u00fan, Mexico.","DOI":"10.1109\/ICPR.2016.7899988"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1060","DOI":"10.1109\/TPAMI.2009.102","article-title":"Shape and Spatially-Varying BRDFs from Photometric Stereo","volume":"32","author":"Goldman","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., and Ma, Y. (2010, January 8\u20139). Robust Photometric Stereo via Low-Rank Matrix Completion and Recovery. Proceedings of the Computer Vision\u2014ACCV 2010, Queenstown, New Zealand.","DOI":"10.1007\/978-3-642-19318-7_55"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ikehata, S., Wipf, D., Matsushita, Y., and Aizawa, K. (2012, January 16\u201321). Robust photometric stereo using sparse regression. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6247691"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Chandraker, M., Agarwal, S., and Kriegman, D. (2007, January 17\u201322). ShadowCuts: Photometric Stereo with Shadows. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383288"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1016\/0146-664X(82)90001-6","article-title":"Obtaining 3-dimensional shape of textured and specular surfaces using four-source photometry","volume":"18","author":"Coleman","year":"1982","journal-title":"Comput. Graph. Image Process."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1109\/34.491627","article-title":"Extracting the shape and roughness of specular lobe objects using four light photometric stereo","volume":"18","author":"Solomon","year":"1996","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1109\/TPAMI.2003.1233898","article-title":"The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows","volume":"25","author":"Barsky","year":"2003","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sunkavalli, K., Zickler, T., and Pfister, H. (2010, January 5\u201311). Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows. Proceedings of the Computer Vision\u2014ECCV 2010, Crete, Greece.","DOI":"10.1007\/978-3-642-15552-9_19"},{"key":"ref_22","unstructured":"Iwahori, Y., Sugie, H., and Ishii, N. (1990, January 16\u201321). Reconstructing shape from shading images under point light source illumination. Proceedings of the 10th International Conference on Pattern Recognition, Atlantic, NJ, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Xie, W., Dai, C., and Wang, C.C.L. (2015, January 7\u201312). Photometric stereo with near point lighting: A solution by mesh deformation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299089"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Qu\u00e9au, Y., Wu, T., and Cremers, D. (2017). Semi-calibrated Near-Light Photometric Stereo. Scale Space and Variational Methods in Computer Vision, Springer International Publishing.","DOI":"10.1007\/978-3-319-58771-4_52"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cho, D., Matsushita, Y., Tai, Y.W., and Kweon, I. (2016, January 11\u201314). Photometric Stereo Under Non-uniform Light Intensities and Exposures. Proceedings of the Computer Vision\u2014ECCV 2016, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_11"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Qu\u00e9au, Y., Wu, T., Lauze, F., Durou, J.D., and Cremers, D. (2017, January 21\u201326). A Non-convex Variational Approach to Photometric Stereo under Inaccurate Lighting. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.45"},{"key":"ref_27","first-page":"1","article-title":"Deviation correction method for close-range photometric stereo with nonuniform illumination","volume":"56","author":"Fan","year":"2017","journal-title":"Opt. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/s11263-006-8815-7","article-title":"Photometric Stereo with General, Unknown Lighting","volume":"72","author":"Basri","year":"2007","journal-title":"Int. J. Comput. Vis."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shi, B., Matsushita, Y., Wei, Y., Xu, C., and Tan, P. (2010, January 13\u201318). Self-calibrating photometric stereo. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5540091"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Lu, F., Matsushita, Y., Sato, I., Okabe, T., and Sato, Y. (2013, January 23\u201328). Uncalibrated Photometric Stereo for Unknown Isotropic Reflectances. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.196"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1031","DOI":"10.1109\/JSSC.2012.2185189","article-title":"CMOS Image Sensors With Multi-Bucket Pixels for Computational Photography","volume":"47","author":"Wan","year":"2012","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1109\/JEDS.2014.2382689","article-title":"A Time-of-Flight Range Image Sensor With Background Canceling Lock-in Pixels Based on Lateral Electric Field Charge Modulation","volume":"3","author":"Han","year":"2015","journal-title":"IEEE J. Electron Devices Soc."},{"key":"ref_33","unstructured":"(2017, November 27). Kinect for Xbox One. Available online: http:\/\/www.xbox.com\/en-US\/xbox-one\/accessories\/kinect."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"4275","DOI":"10.1109\/JSEN.2015.2416651","article-title":"Evaluating and improving the depth accuracy of Kinect for Windows v2","volume":"15","author":"Yang","year":"2015","journal-title":"IEEE Sens. J."},{"key":"ref_35","unstructured":"(2017, November 27). ARDUINO UNO REV3. Available online: https:\/\/store.arduino.cc\/arduino-uno-rev3."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/3\/786\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:57:38Z","timestamp":1760194658000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/3\/786"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,3,5]]},"references-count":35,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2018,3]]}},"alternative-id":["s18030786"],"URL":"https:\/\/doi.org\/10.3390\/s18030786","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,3,5]]}}}