{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:22:41Z","timestamp":1776115361557,"version":"3.50.1"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2017,9,11]],"date-time":"2017-09-11T00:00:00Z","timestamp":1505088000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1029679"],"award-info":[{"award-number":["IIS-1029679"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000893","name":"Simons Foundation","doi-asserted-by":"publisher","award":["336363"],"award-info":[{"award-number":["336363"]}],"id":[{"id":"10.13039\/100000893","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2017,9,11]]},"abstract":"<jats:p>Eye contact is a crucial element of non-verbal communication that signifies interest, attention, and participation in social interactions. As a result, measures of eye contact arise in a variety of applications such as the assessment of the social communication skills of children at risk for developmental disorders such as autism, or the analysis of turn-taking and social roles during group meetings. However, the automated measurement of visual attention during naturalistic social interactions is challenging due to the difficulty of estimating a subject\u2019s looking direction from video. This paper proposes a novel approach to eye contact detection during adult-child social interactions in which the adult wears a point-of-view camera which captures an egocentric view of the child\u2019s behavior. By analyzing the child\u2019s face regions and inferring their head pose we can accurately identify the onset and duration of the child\u2019s looks to their social partner\u2019s eyes. We introduce the Pose-Implicit CNN, a novel deep learning architecture that predicts eye contact while implicitly estimating the head pose. We present a fully automated system for eye contact detection that solves the sub-problems of end-to-end feature learning and pose estimation using deep neural networks. To train our models, we use a dataset comprising 22 hours of 156 play session videos from over 100 children, half of whom are diagnosed with Autism Spectrum Disorder. We report an overall precision of 0.76, recall of 0.80, and an area under the precision-recall curve of 0.79, all of which are significant improvements over existing methods.<\/jats:p>","DOI":"10.1145\/3131902","type":"journal-article","created":{"date-parts":[[2017,9,11]],"date-time":"2017-09-11T12:12:26Z","timestamp":1505131946000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":62,"title":["Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment"],"prefix":"10.1145","volume":"1","author":[{"given":"Eunji","family":"Chong","sequence":"first","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katha","family":"Chanda","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhefan","family":"Ye","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Audrey","family":"Southerland","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nataniel","family":"Ruiz","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rebecca M.","family":"Jones","sequence":"additional","affiliation":[{"name":"Weill Cornell Medicine"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Agata","family":"Rozga","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James M.","family":"Rehg","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,9,11]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"distance and affiliation. Sociometry","author":"Argyle Michael","year":"1965","unstructured":"Michael Argyle and Janet Dean . 1965. Eye-contact , distance and affiliation. Sociometry ( 1965 ), 289--304. Michael Argyle and Janet Dean. 1965. Eye-contact, distance and affiliation. Sociometry (1965), 289--304."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10593-2_39"},{"key":"e_1_2_1_3_1","first-page":"137","article-title":"Early mother-infant reciprocity","volume":"3","author":"Brazelton T Berry","year":"1975","unstructured":"T Berry Brazelton , Edward Tronick , Lauren Adamson , Heidelise Als , and Susan Wise . 1975 . Early mother-infant reciprocity . Parent-Infant Interaction 3 (1975), 137 . T Berry Brazelton, Edward Tronick, Lauren Adamson, Heidelise Als, and Susan Wise. 1975. Early mother-infant reciprocity. Parent-Infant Interaction 3 (1975), 137.","journal-title":"Parent-Infant Interaction"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10803-009-0803-7"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ridd.2015.10.011"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/1362361313480277"},{"key":"e_1_2_1_9_1","volume-title":"Intraface. In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG)","volume":"1","author":"la Torre Fernando De","year":"2015","unstructured":"Fernando De la Torre , Wen-Sheng Chu , Xuehan Xiong , Francisco Vicente , Xiaoyu Ding , and Jeffrey Cohn . 2015 . Intraface. In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG) , Vol. 1 . IEEE, 1--8. Fernando De la Torre, Wen-Sheng Chu, Xuehan Xiong, Francisco Vicente, Xiaoyu Ding, and Jeffrey Cohn. 2015. Intraface. In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1--8."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2010.5540094"},{"key":"e_1_2_1_11_1","volume-title":"Pattern classification","author":"Duda Richard O","unstructured":"Richard O Duda , Peter E Hart , and David G Stork . 2012. Pattern classification . John Wiley 8 Sons. Richard O Duda, Peter E Hart, and David G Stork. 2012. Pattern classification. John Wiley 8 Sons."},{"key":"e_1_2_1_12_1","unstructured":"Sarah R Edmunds Agata Rozga Yin Li Elizabeth A Karp Lisa V Ibanez James M Rehg and Wendy L Stone. 2017. Brief Report: Using a Point-of-View Camera to Measure Eye Gaze in Young Children with Autism Spectrum Disorder During Naturalistic Social Interactions: A Pilot Study. Journal of Autism and Developmental Disorders (2017) 1--7.  Sarah R Edmunds Agata Rozga Yin Li Elizabeth A Karp Lisa V Ibanez James M Rehg and Wendy L Stone. 2017. Brief Report: Using a Point-of-View Camera to Measure Eye Gaze in Young Children with Autism Spectrum Disorder During Naturalistic Social Interactions: A Pilot Study. Journal of Autism and Developmental Disorders (2017) 1--7."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-009-0275-4"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2009.167"},{"key":"e_1_2_1_15_1","unstructured":"Centers for Disease Control and Prevention. 2016. Summary of Autism Spectrum Disorder Prevalence Studies. https:\/\/www.cdc.gov\/ncbddd\/autism\/documents\/ASDPrevalenceDataTable2016.pdf. (2016). Accessed: 2017-05-03.  Centers for Disease Control and Prevention. 2016. Summary of Autism Spectrum Disorder Prevalence Studies. https:\/\/www.cdc.gov\/ncbddd\/autism\/documents\/ASDPrevalenceDataTable2016.pdf. (2016). Accessed: 2017-05-03."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.visres.2011.07.002"},{"key":"e_1_2_1_17_1","volume-title":"Deep Learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep Learning . MIT Press . http:\/\/www.deeplearningbook.org. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http:\/\/www.deeplearningbook.org."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10803-016-2782-9"},{"key":"e_1_2_1_19_1","volume-title":"Development of the Brief Observation of Social Communication Change (BOSCC) for Verbally Able Children with ASD. Biennial Meeting of the Society for Research on Child Development (SRCD)","author":"Grzadzinski R","year":"2017","unstructured":"R Grzadzinski , K Martinez , G Gunin , E Ajodan , S Kim , and C Lord . 2017 . Development of the Brief Observation of Social Communication Change (BOSCC) for Verbally Able Children with ASD. Biennial Meeting of the Society for Research on Child Development (SRCD) (2017). R Grzadzinski, K Martinez, G Gunin, E Ajodan, S Kim, and C Lord. 2017. Development of the Brief Observation of Social Communication Change (BOSCC) for Verbally Able Children with ASD. Biennial Meeting of the Society for Research on Child Development (SRCD) (2017)."},{"key":"e_1_2_1_20_1","volume-title":"Visual social attention in autism spectrum disorder: Insights from eye tracking studies. Neuroscience 8 Biobehavioral Reviews 42","author":"Guillon Quentin","year":"2014","unstructured":"Quentin Guillon , Nouchine Hadjikhani , Sophie Baduel , and Bernadette Rog\u00e9 . 2014. Visual social attention in autism spectrum disorder: Insights from eye tracking studies. Neuroscience 8 Biobehavioral Reviews 42 ( 2014 ), 279--297. Quentin Guillon, Nouchine Hadjikhani, Sophie Baduel, and Bernadette Rog\u00e9. 2014. Visual social attention in autism spectrum disorder: Insights from eye tracking studies. Neuroscience 8 Biobehavioral Reviews 42 (2014), 279--297."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2009.30"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1542\/peds.2011-2278"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10803-011-1262-5"},{"key":"e_1_2_1_24_1","volume-title":"Fddb: A benchmark for face detection in unconstrained settings. UMass Amherst Technical Report","author":"Jain Vidit","year":"2010","unstructured":"Vidit Jain and Erik G Learned-Miller . 2010 . Fddb: A benchmark for face detection in unconstrained settings. UMass Amherst Technical Report (2010). Vidit Jain and Erik G Learned-Miller. 2010. Fddb: A benchmark for face detection in unconstrained settings. UMass Amherst Technical Report (2010)."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_2_1_26_1","volume-title":"Face detection with the faster R-CNN. arXiv preprint arXiv:1606.03473","author":"Jiang Huaizu","year":"2016","unstructured":"Huaizu Jiang and Erik Learned-Miller . 2016. Face detection with the faster R-CNN. arXiv preprint arXiv:1606.03473 ( 2016 ). Huaizu Jiang and Erik Learned-Miller. 2016. Face detection with the faster R-CNN. arXiv preprint arXiv:1606.03473 (2016)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1001\/archpsyc.65.8.946"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.100.1.78"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1001\/archpsyc.59.9.809"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.239"},{"key":"e_1_2_1_31_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS). 1097--1105.   Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS). 1097--1105."},{"key":"e_1_2_1_32_1","volume-title":"Looking and acting: vision and eye movements in natural behaviour","author":"Land Michael","unstructured":"Michael Land and Benjamin Tatler . 2009. Looking and acting: vision and eye movements in natural behaviour . Oxford University Press . Michael Land and Benjamin Tatler. 2009. Looking and acting: vision and eye movements in natural behaviour. Oxford University Press."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_2_1_34_1","unstructured":"Catherine Lord Pamela C DiLavore and Katherine Gotham. 2012. Autism diagnostic observation schedule. Western Psychological Services Torrance CA.  Catherine Lord Pamela C DiLavore and Katherine Gotham. 2012. Autism diagnostic observation schedule. Western Psychological Services Torrance CA."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2013.00840"},{"key":"e_1_2_1_36_1","volume-title":"Joint attention, social engagement, and the development of social competence. The Development of Social Engagement: Neurobiological Perspectives","author":"Mundy P","year":"2006","unstructured":"P Mundy and C Fran\u00e7oise Acra . 2006. Joint attention, social engagement, and the development of social competence. The Development of Social Engagement: Neurobiological Perspectives ( 2006 ), 81--117. P Mundy and C Fran\u00e7oise Acra. 2006. Joint attention, social engagement, and the development of social competence. The Development of Social Engagement: Neurobiological Perspectives (2006), 81--117."},{"key":"e_1_2_1_37_1","volume-title":"Meg Venezia, Anne Hogan, and Jeffrey Seibert.","author":"Mundy Peter","year":"2003","unstructured":"Peter Mundy , Christine Delgado , Jessica Block , Meg Venezia, Anne Hogan, and Jeffrey Seibert. 2003 . Early social communication scales (ESCS) . Coral Gables, FL : University of Miami (2003) . Peter Mundy, Christine Delgado, Jessica Block, Meg Venezia, Anne Hogan, and Jeffrey Seibert. 2003. Early social communication scales (ESCS). Coral Gables, FL: University of Miami (2003)."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0044144"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5244\/C.29.41"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1001\/archgenpsychiatry.2010.113"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.438"},{"key":"e_1_2_1_43_1","unstructured":"Shaoqing Ren Kaiming He Ross Girshick and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS). 91--99.   Shaoqing Ren Kaiming He Ross Girshick and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS). 91--99."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10803-010-1051-6"},{"key":"e_1_2_1_45_1","first-page":"30","article-title":"Autism diagnostic interview-revised","volume":"29","author":"Rutter Michael","year":"2003","unstructured":"Michael Rutter , A Le Couteur , and C Lord . 2003 . Autism diagnostic interview-revised . Los Angeles, CA: Western Psychological Services 29 (2003), 30 . Michael Rutter, A Le Couteur, and C Lord. 2003. Autism diagnostic interview-revised. Los Angeles, CA: Western Psychological Services 29 (2003), 30.","journal-title":"Los Angeles, CA: Western Psychological Services"},{"key":"e_1_2_1_46_1","first-page":"e3675","article-title":"Eye tracking young children with autism","volume":"61","author":"Sasson Noah J","year":"2012","unstructured":"Noah J Sasson and Jed T Elison . 2012 . Eye tracking young children with autism . Journal of Visualized Experiments 61 (2012), e3675 -- e3675 . Noah J Sasson and Jed T Elison. 2012. Eye tracking young children with autism. Journal of Visualized Experiments 61 (2012), e3675--e3675.","journal-title":"Journal of Visualized Experiments"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/968363.968384"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0021963098002935"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-7610.1986.tb00189.x"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2501994"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.235"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.1991.139758"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:VISI.0000013087.49260.fb"},{"key":"e_1_2_1_54_1","unstructured":"OMRON OKAO vision. 2017. https:\/\/www.omron.com\/ecb\/products\/mobile\/okao01.html. (2017). Accessed: 2017-05-03.  OMRON OKAO vision. 2017. https:\/\/www.omron.com\/ecb\/products\/mobile\/okao01.html. (2017). Accessed: 2017-05-03."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.419"},{"key":"e_1_2_1_56_1","volume-title":"WIDER FACE: A Face Detection Benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Yang Shuo","year":"2016","unstructured":"Shuo Yang , Ping Luo , Chen Change Loy , and Xiaoou Tang . 2016 . WIDER FACE: A Face Detection Benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. 2016. WIDER FACE: A Face Detection Benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2370216.2370368"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG)","volume":"1","author":"Ye Zhefan","year":"2015","unstructured":"Zhefan Ye , Yin Li , Yun Liu , Chanel Bridges , Agata Rozga , and James M Rehg . 2015 . Detecting bids for eye contact using a wearable camera . In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG) , Vol. 1 . IEEE, 1--8. Zhefan Ye, Yin Li, Yun Liu, Chanel Bridges, Agata Rozga, and James M Rehg. 2015. Detecting bids for eye contact using a wearable camera. In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1--8."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2016.2603342"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299081"},{"key":"e_1_2_1_61_1","volume-title":"It\u2019s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. arXiv preprint arXiv:1611.08860","author":"Zhang Xucong","year":"2016","unstructured":"Xucong Zhang , Yusuke Sugano , Mario Fritz , and Andreas Bulling . 2016. It\u2019s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. arXiv preprint arXiv:1611.08860 ( 2016 ). Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. 2016. It\u2019s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. arXiv preprint arXiv:1611.08860 (2016)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbr.2013.04.004"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3131902","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3131902","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3131902","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:33Z","timestamp":1750217433000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3131902"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,9,11]]},"references-count":61,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2017,9,11]]}},"alternative-id":["10.1145\/3131902"],"URL":"https:\/\/doi.org\/10.1145\/3131902","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,9,11]]},"assertion":[{"value":"2017-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-09-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}