{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T23:44:25Z","timestamp":1774655065601,"version":"3.50.1"},"reference-count":63,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"67","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2022,6,29]]},"abstract":"<jats:p>Flesh encodes a variety of haptic information including deformation, temperature, vibration, and damage stimuli using a multisensory array of mechanoreceptors distributed on the surface of the human body. Currently, soft sensors are capable of detecting some haptic stimuli, but whole-body multimodal perception at scales similar to a human adult (surface area ~17,000 square centimeters) is still a challenge in artificially intelligent agents due to the lack of encoding. This encoding is needed to reduce the wiring required to send the vast amount of information transmitted to the processor. We created a robotic flesh that could be further developed for use in these agents. This engineered flesh is an optical, elastomeric matrix \u201cinnervated\u201d with stretchable lightguides that encodes haptic stimuli into light: temperature into wavelength due to thermochromic dyes and forces into intensity due to mechanical deformation. By exploiting the optical properties of the constitutive materials and using machine learning, we infer spatiotemporal, haptic information from light that is read by an image sensor. We demonstrate the capabilities of our system in various assemblies to estimate temperature, contact location, normal and shear force, gestures, and damage from temporal snapshots of light coming from the entire haptic sensor with errors &lt;5%.<\/jats:p>","DOI":"10.1126\/scirobotics.abi6745","type":"journal-article","created":{"date-parts":[[2022,6,8]],"date-time":"2022-06-08T17:57:47Z","timestamp":1654711067000},"source":"Crossref","is-referenced-by-count":59,"title":["Haptic perception using optoelectronic robotic flesh for embodied artificially intelligent agents"],"prefix":"10.1126","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6558-8656","authenticated-orcid":true,"given":"Jose A.","family":"Barreiros","sequence":"first","affiliation":[{"name":"Department of Systems Science and Engineering, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4576-742X","authenticated-orcid":true,"given":"Artemis","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9151-137X","authenticated-orcid":true,"given":"Sofya","family":"Pugach","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6510-4228","authenticated-orcid":true,"given":"Narahari","family":"Iyengar","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7455-8918","authenticated-orcid":true,"given":"Graeme","family":"Troxell","sequence":"additional","affiliation":[{"name":"Department of Systems Science and Engineering, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4115-508X","authenticated-orcid":true,"given":"Alexander","family":"Cornwell","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA."}]},{"given":"Samantha","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0666-3123","authenticated-orcid":true,"given":"Bart","family":"Selman","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Cornell University, Ithaca, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0631-9587","authenticated-orcid":true,"given":"Robert F.","family":"Shepherd","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA."}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.2307\/3331482"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00424-014-1651-7"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abi4667"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2016.05.006"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2006.06.011"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1037\/0096-3445.116.4.356"},{"key":"e_1_3_2_8_2","unstructured":"Hsiao Steven Gomez-Ramirez Manuel Neurobiology of Sensation and Reward (CRC Press 2011)."},{"key":"e_1_3_2_9_2","unstructured":"G. Colombetti E. Thompson in Developmental Perspectives on Embodiment and Consciousness W. F. Overton U. M\u00fcller J. L. Newman Eds. (Erlbaum 2008) pp. 8\u201310 12."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"R. Koiva T. Schwank G. Walck M. Meier R. Haschke H. Ritter Barometer-based tactile skin for anthropomorphic robot hand in 2020 IEEE International Conference on Intelligent Robots and Systems (IEEE 2020) pp. 9821\u20139826.","DOI":"10.1109\/IROS45743.2020.9341691"},{"key":"e_1_3_2_11_2","first-page":"311","article-title":"Texture characterization including warm\/cool sensation using force-, light-, and temperature-sensitive micro-electromechanical systems sensor","volume":"29","author":"Sato F.","year":"2017","unstructured":"F. Sato, K. Takahashi, T. Abe, M. Okuyama, H. Noma, M. Sohgawa, Texture characterization including warm\/cool sensation using force-, light-, and temperature-sensitive micro-electromechanical systems sensor. Sensors and Materials. 29, 311\u2013321 (2017).","journal-title":"Sensors and Materials."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.3390\/s18040948"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.7567\/1347-4065\/ab3903"},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","unstructured":"P. Weiner C. Neef T. Asfour A multimodal embedded sensor system for scalable robotic and prosthetic fingers in IEEE-RAS International Conference on Humanoid Robots . 2018 November (IEEE 2019) pp. 286\u2013292.","DOI":"10.1109\/HUMANOIDS.2018.8624955"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2012.2200790"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1002\/adma.201400334"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaf8810"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"J. Wissman T. Lu C. Majidi Soft-matter electronics with stencil lithography in Proceedings of IEEE Sensors (IEEE 2013) pp. 1\u20134.","DOI":"10.1109\/ICSENS.2013.6688217"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.1602200"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aai9251"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aba5132"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aai7529"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaw6304"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aba5504"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"C. To T. L. Hellebrekers Y. L. Park Highly stretchable optical sensors for pressure strain and curvature measurement in 2015 IEEE International Conference on Intelligent Robots and Systems (IEEE 2015) pp. 5898\u20135903.","DOI":"10.1109\/IROS.2015.7354215"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aau2489"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.3390\/s17122762"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2977257"},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"D. F. Gomes Z. Lin S. Luo GelTip: A finger-shaped optical tactile sensor for robotic manipulation in 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE 2020) pp. 9903\u20139909.","DOI":"10.1109\/IROS45743.2020.9340881"},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","unstructured":"C. Trueeb C. Sferrazza R. D\u2019Andrea Towards vision-based robotic skins: A data-driven multi-camera tactile sensor in 2020 Third IEEE International Conference on Soft Robotics (RoboSoft) (IEEE 2020) pp. 333\u2013338.","DOI":"10.1109\/RoboSoft48309.2020.9116060"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2956882"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"A. Mkhitaryan D. Burschka Vision based haptic multisensor for manipulation of soft fragile objects in 2012 IEEE SENSORS (IEEE 2012) pp. 1\u20134.","DOI":"10.1109\/ICSENS.2012.6411035"},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","unstructured":"H. Sun K. J. Kuchenbecker G. Martius A soft thumb-sized vision-based sensor with accurate all-round force perception. arXiv:2111.05934 [cs.RO] (10 November 2021).","DOI":"10.1038\/s42256-021-00439-3"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","unstructured":"A. Agarwal T. Man W. Yuan Simulation of vision-based tactile sensors using physics based rendering. arXiv:2012.13184 [cs.RO] (24 December 2020).","DOI":"10.1109\/ICRA48506.2021.9561122"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"A. Alspach K. Hashimoto N. Kuppuswamy R. Tedrake Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation. arXiv:1904.02252 [cs.RO] (3 April 2019).","DOI":"10.1109\/ROBOSOFT.2019.8722713"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/0166-2236(83)90011-5"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-12161-1"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1039\/b905832j"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1111\/cote.12020"},{"key":"e_1_3_2_40_2","unstructured":"J. D. Willard X. Jia S. Xu M. Steinbach V. Kumar Integrating physics-based modeling with machine learning: A survey. arXiv:2003.04919 [physics.comp-ph] (10 March 2020)."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1038\/srep28665"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1590\/1678-7757-2016-0462"},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"C. Zhang A. Bedri G. Reyes B. Bercik O. T. Inan T. E. Starner G. D. Abowd TapSkin: Recognizing on-skin input for smartwatches in Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces: Nature Meets Interactive Surfaces ISS 2016 (ACM 2016) pp. 13\u201322.","DOI":"10.1145\/2992154.2992187"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.3758\/BF03212187"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.22649.1"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1600-0668.1991.00009.x"},{"key":"e_1_3_2_47_2","first-page":"354","article-title":"The effect of rate of temperature change and adapting temperature on thermal sensitivity","volume":"1","author":"Molinari H.","year":"1977","unstructured":"H. Molinari, J. D. Greenspan, D. Krenshalo, The effect of rate of temperature change and adapting temperature on thermal sensitivity. Sens. Processes 1, 354\u2013362 (1977).","journal-title":"Sens. Processes"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1002\/app.49748"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1616377114"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1063\/1.5020325"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.3934\/matersci.2016.3.851"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature09963"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1002\/adma.201702616"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1039\/C9TC05612B"},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","unstructured":"Y. Bengio J. Louradour R. Collobert J. Weston Curriculum learning in Proceedings of the 26th Annual International Conference on Machine Learning (2009) pp. 41\u201348.","DOI":"10.1145\/1553374.1553380"},{"key":"e_1_3_2_56_2","doi-asserted-by":"crossref","unstructured":"J. Tobin R. Fong A. Ray J. Schneider W. Zaremba P. Abbeel Domain randomization for transferring deep neural networks from simulation to the real world in 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE 2017) pp. 23\u201330.","DOI":"10.1109\/IROS.2017.8202133"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03356-9_3"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40092-017-0234-6"},{"key":"e_1_3_2_59_2","unstructured":"Google Inc. Welcome to Colaboratory (2020); https:\/\/colab.research.google.com."},{"key":"e_1_3_2_60_2","first-page":"2825","article-title":"Scikit-learn: Machine learning in python","volume":"12","author":"Pedregosa F.","year":"2011","unstructured":"F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, \u00c9. Duchesnay, Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijimpeng.2004.11.010"},{"key":"e_1_3_2_62_2","unstructured":"K. Larson Can you estimate modulus from durometer hardness for silicones? White paper Dow Corning Corporation (2016)."},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1002\/adfm.201303288"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2018.03.067"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.abi6745","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,16]],"date-time":"2024-01-16T12:50:53Z","timestamp":1705409453000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.abi6745"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,29]]},"references-count":63,"journal-issue":{"issue":"67","published-print":{"date-parts":[[2022,6,29]]}},"alternative-id":["10.1126\/scirobotics.abi6745"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.abi6745","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,29]]},"article-number":"eabi6745"}}