{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T14:00:08Z","timestamp":1780322408722,"version":"3.54.1"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2018,6,1]],"date-time":"2018-06-01T00:00:00Z","timestamp":1527811200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2018,6,1]],"date-time":"2018-06-01T00:00:00Z","timestamp":1527811200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digital Med"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Current tools for objectively measuring young children\u2019s observed behaviors are expensive, time-consuming, and require extensive training and professional administration. The lack of scalable, reliable, and validated tools impacts access to evidence-based knowledge and limits our capacity to collect population-level data in non-clinical settings. To address this gap, we developed mobile technology to collect videos of young children while they watched movies designed to elicit autism-related behaviors and then used automatic behavioral coding of these videos to quantify children\u2019s emotions and behaviors. We present results from our iPhone study Autism &amp; Beyond, built on ResearchKit\u2019s open-source platform. The entire study\u2014from an e-Consent process to stimuli presentation and data collection\u2014was conducted within an iPhone-based app available in the Apple Store. Over 1 year, 1756 families with children aged 12\u201372 months old participated in the study, completing 5618 caregiver-reported surveys and uploading 4441 videos recorded in the child\u2019s natural settings. Usable data were collected on 87.6% of the uploaded videos. Automatic coding identified significant differences in emotion and attention by age, sex, and autism risk status. This study demonstrates the acceptability of an app-based tool to caregivers, their willingness to upload videos of their children, the feasibility of caregiver-collected data in the home, and the application of automatic behavioral encoding to quantify emotions and attention variables that are clinically meaningful and may be refined to screen children for autism and developmental disorders outside of clinical settings. This technology has the potential to transform how we screen and monitor children\u2019s development.<\/jats:p>","DOI":"10.1038\/s41746-018-0024-6","type":"journal-article","created":{"date-parts":[[2018,5,10]],"date-time":"2018-05-10T09:22:06Z","timestamp":1525944126000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":90,"title":["Automatic emotion and attention analysis of young children at home: a ResearchKit autism feasibility study"],"prefix":"10.1038","volume":"1","author":[{"given":"Helen L.","family":"Egger","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geraldine","family":"Dawson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jordan","family":"Hashemi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kimberly L. H.","family":"Carpenter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven","family":"Espinosa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathleen","family":"Campbell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Brotkin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jana","family":"Schaich-Borg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Qiu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mariano","family":"Tepper","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey P.","family":"Baker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6902-2396","authenticated-orcid":false,"suffix":"Jr.","given":"Richard A.","family":"Bloomfield","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillermo","family":"Sapiro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2018,6,1]]},"reference":[{"key":"24_CR1","first-page":"1","volume":"65","author":"DL Christensen","year":"2016","unstructured":"Christensen, D. L. et al. Prevalence and characteristics of autism spectrum disorder among children aged 8 years\u2014Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2012. Morb. Mortal. Wkly. Rep. Surveill. Summ. 65, 1\u201323 (2016).","journal-title":"Morb. Mortal. Wkly. Rep. Surveill. Summ."},{"key":"24_CR2","doi-asserted-by":"publisher","first-page":"694","DOI":"10.1001\/archpsyc.63.6.694","volume":"63","author":"C Lord","year":"2006","unstructured":"Lord, C. et al. Autism from 2 to 9 years of age. Arch. Gen. Psychiatry 63, 694\u2013701 (2006).","journal-title":"Arch. Gen. Psychiatry"},{"key":"24_CR3","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1542\/peds.2009-0958","volume":"125","author":"G Dawson","year":"2010","unstructured":"Dawson, G. et al. Randomized, controlled trial of an intervention for toddlers with autism: the Early Start Denver Model. Pediatrics 125, 17\u201323 (2010).","journal-title":"Pediatrics"},{"key":"24_CR4","doi-asserted-by":"publisher","first-page":"1150","DOI":"10.1016\/j.jaac.2012.08.018","volume":"51","author":"G Dawson","year":"2012","unstructured":"Dawson, G. et al. Early behavioral intervention is associated with normalized brain activity in young children with autism. J. Am. Acad. Child Adolesc. Psychiatry 51, 1150\u20131159 (2012).","journal-title":"J. Am. Acad. Child Adolesc. Psychiatry"},{"key":"24_CR5","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/BF02211841","volume":"19","author":"C Lord","year":"1989","unstructured":"Lord, C. et al. Autism diagnostic observation schedule: a standardized observation of communicative and social behavior. J. Autism Dev. Disord. 19, 185\u2013212 (1989).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR6","doi-asserted-by":"publisher","first-page":"731","DOI":"10.1007\/s10803-007-0440-y","volume":"38","author":"SE Bryson","year":"2008","unstructured":"Bryson, S. E. et al. The Autism Observation Scale for infants: scale development and reliability data. J. Autism Dev. Disord. 38, 731\u2013738 (2008).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2014\/935686","volume":"2014","author":"J Hashemi","year":"2014","unstructured":"Hashemi, J. et al. Computer vision tools for low-cost and non-invasive measurement of autism-related behaviors in infants. Autism Res. Treat. 2014, 1\u201312 (2014).","journal-title":"Autism Res. Treat."},{"key":"24_CR8","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1016\/j.jpeds.2017.01.021","volume":"183","author":"J Campbell","year":"2017","unstructured":"Campbell, J. et al. Use of a digital M-CHAT-R\/F to improve quality of screening for autism. J. Pediatr. 183, 133\u2013139 (2017).","journal-title":"J. Pediatr."},{"key":"24_CR9","doi-asserted-by":"publisher","unstructured":"Campbell, J. et al. Computer vision analysis detects inconsistent and delayed social orienting in toddlers with autism. Autism. pp 1\u201310, https:\/\/doi.org\/10.1177\/1362361318766247 (2018).","DOI":"10.1177\/1362361318766247"},{"key":"24_CR10","first-page":"23","volume":"2015","author":"J Hashemi","year":"2015","unstructured":"Hashemi, J. et al. A scalable app for measuring autism risk behaviors in young children: a technical validity and feasibility study. MobiHealth 2015, 23\u201327 (2015).","journal-title":"MobiHealth"},{"key":"24_CR11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1542\/peds.2013-1813","volume":"133","author":"DL Robins","year":"2013","unstructured":"Robins, D. L. et al. Validation of the modified checklist for autism in toddlers, revised with follow-up (M-CHAT-R\/F). Pediatrics 133, 37\u201345 (2013).","journal-title":"Pediatrics"},{"key":"24_CR12","unstructured":"Campbell, K., et al. Computer vision detects delayed social orienting in toddlers with autism. 2016 Annual Meeting of the International Society for Autism Research (INSAR, Baltimore, MD, 2016)."},{"key":"24_CR13","unstructured":"Egger, H., et al. Feasibility of a mobile phone-delivered study of social and emotional behaviors in young children at risk for autism. Annual Meeting of the International Society for Autism Research (INSAR, San Francisco, CA, 2017)."},{"key":"24_CR14","unstructured":"Hashemi, J., et al. A ResearchKit app with automatic detection of facial affect and social behaviors from videos of children with autism. Annual Meeting of the International Society for Autism Research (INSAR, San Francisco, CA, 2017)."},{"key":"24_CR15","doi-asserted-by":"crossref","unstructured":"Chu, W.-S., De la Torre, F. & Cohn, J. F. Selective transfer machine for personalized facial action unit detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, Oregon, 2013).","DOI":"10.1109\/CVPR.2013.451"},{"key":"24_CR16","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1007\/BF00916569","volume":"18","author":"G Dawson","year":"1990","unstructured":"Dawson, G., Hill, D., Galpert, L., Spencer, A. & Watson, L. Affective exchanges between young autistic children and their mothers. J. Abnorm. Child Psychol. 18, 335\u2013345 (1990).","journal-title":"J. Abnorm. Child Psychol."},{"key":"24_CR17","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1037\/0012-1649.40.2.271","volume":"40","author":"G Dawson","year":"2004","unstructured":"Dawson, G. et al. Social attention impairments in autism: social orienting, joint attention, and attention to distress. Dev. Psychol. 40, 271\u2013283 (2004).","journal-title":"Dev. Psychol."},{"key":"24_CR18","first-page":"2","volume":"18:8","author":"EJH Jones","year":"2016","unstructured":"Jones, E. J. H. et al. Reduced engagement with social stimuli in 6-month-old infants with later autism spectrum disorder: a longitudinal prospective study of infants at high familial risk. J. Neurodev. Disord. 18:8, 2\u201320 (2016).","journal-title":"J. Neurodev. Disord."},{"key":"24_CR19","doi-asserted-by":"publisher","first-page":"961","DOI":"10.1002\/aur.1754","volume":"10","author":"EJH Jones","year":"2017","unstructured":"Jones, E. J. H., Dawson, G., Kelly, J., Estes, A. & Webb, S. J. Parent-delivered early intervention in infants at risk for ASD: effects on electrophysiological and habituation measures of social attention. Autism Res. 10, 961\u2013972 (2017).","journal-title":"Autism Res."},{"key":"24_CR20","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1007\/BF02172225","volume":"24","author":"J Osterling","year":"1994","unstructured":"Osterling, J. & Dawson, G. Early recognition of children with autism: a study of first birthday home videotapes. J. Autism Dev. Disord. 24, 247\u2013257 (1994).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR21","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1017\/S0954579402002031","volume":"14","author":"J Osterling","year":"2002","unstructured":"Osterling, J., Dawson, G. & Munson, J. Early recognition of 1-year-old infants with autism spectrum disorder versus mental retardation. Dev. Psychopathol. 14, 239\u2013251 (2002).","journal-title":"Dev. Psychopathol."},{"key":"24_CR22","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1097\/00004703-200604002-00003","volume":"27","author":"R Palomo","year":"2006","unstructured":"Palomo, R., Belinchon, M. & Ozonoff, S. Autism and family home movies: a comprehensive review. J. Dev. Behav. Pediatr. 27, 59\u201368 (2006).","journal-title":"J. Dev. Behav. Pediatr."},{"key":"24_CR23","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1023\/A:1005463707029","volume":"30","author":"E Werner","year":"2000","unstructured":"Werner, E., Dawson, G., Osterling, J. & Dinno, N. Brief report: recognition of autism spectrum disorder before one year of age: a retrospective study based on home videotapes. J. Autism Dev. Disord. 30, 157\u2013162 (2000).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR24","unstructured":"Kumm, A. J., et al. Feasibility of a smartphone application to identify young children at risk for autism spectrum disorder in a low-income, community setting in South Africa, IMFAR (INSAR, San Francisco, 2017)."},{"key":"24_CR25","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1177\/1362361308094500","volume":"12","author":"J Brian","year":"2008","unstructured":"Brian, J. et al. Clinical assessment of autism in high-risk 18-month-olds. Autism 12, 433\u2013456 (2008).","journal-title":"Autism"},{"key":"24_CR26","doi-asserted-by":"publisher","first-page":"1305","DOI":"10.1007\/s10803-009-0746-z","volume":"39","author":"R Luyster","year":"2009","unstructured":"Luyster, R. et al. The Autism Diagnostic Observation Schedule-toddler module: a new module of a standardized diagnostic measure for autism spectrum disorders. J. Autism Dev. Disord. 39, 1305\u20131320 (2009).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR27","doi-asserted-by":"publisher","first-page":"1383","DOI":"10.1542\/peds.2008-1606","volume":"123","author":"L Zwaigenbaum","year":"2009","unstructured":"Zwaigenbaum, L. et al. Clinical assessment and management of toddlers with suspected autism spectrum disorder: insights from studies of high-risk infants. Pediatrics 123, 1383\u20131391 (2009).","journal-title":"Pediatrics"},{"key":"24_CR28","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1023\/B:JADD.0000037420.16169.28","volume":"34","author":"AA Bieberich","year":"2004","unstructured":"Bieberich, A. A. & Morgan, S. B. Self-regulation and affective expression during play in children with autism or Down Syndrome: a short-term longitudinal study. J. Autism Dev. Disord. 34, 439\u2013448 (2004).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR29","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1037\/0022-006X.61.3.475","volume":"61","author":"L Capps","year":"1993","unstructured":"Capps, L., Kasari, C., Yirmiya, N. & Sigman, M. Parental perception of emotional expressiveness in children with autism. J. Consult. Clin. Psychol. 61, 475\u2013484 (1993).","journal-title":"J. Consult. Clin. Psychol."},{"key":"24_CR30","doi-asserted-by":"publisher","first-page":"1150","DOI":"10.1016\/j.rasd.2012.03.006","volume":"6","author":"RB Grossman","year":"2012","unstructured":"Grossman, R. B. & Tager-Flusberg, H. Quality matters! Differences between expressive and receptive non-verbal communication skills in adolescents with ASD. Res. Autism Spectr. Disco. 6, 1150\u20131155 (2012).","journal-title":"Res. Autism Spectr. Disco."},{"key":"24_CR31","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/BF02206859","volume":"20","author":"C Kasari","year":"1990","unstructured":"Kasari, C., Sigman, M., Mundy, P. & Yirmiya, N. Affective sharing in the context of joint attention interactions of normal, autistic, and mentally retarded children. J. Autism Dev. Disord. 20, 87\u2013100 (1990).","journal-title":"J. Autism Dev. Disord."},{"key":"24_CR32","doi-asserted-by":"publisher","first-page":"865","DOI":"10.1111\/j.1469-7610.1989.tb00288.x","volume":"30","author":"H Macdonald","year":"1989","unstructured":"Macdonald, H. et al. Recognition and expression of emotional cues by autistic and normal adults. J. Child Psychol. Psychiatry 30, 865\u2013877 (1989).","journal-title":"J. Child Psychol. Psychiatry"},{"key":"24_CR33","doi-asserted-by":"publisher","first-page":"836","DOI":"10.1097\/00004583-198726060-00006","volume":"26","author":"ME Snow","year":"1987","unstructured":"Snow, M. E., Hertzig, M. E. & Shapiro, T. Expression of emotion in young autistic children. J. Am. Acad. Child Adolesc. Psychiatry 26, 836\u2013838 (1987).","journal-title":"J. Am. Acad. Child Adolesc. Psychiatry"},{"key":"24_CR34","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1111\/j.1469-7610.1989.tb00785.x","volume":"30","author":"N Yirmiya","year":"1989","unstructured":"Yirmiya, N., Kasari, C., Sigman, M. & Mundy, P. Facial expressions of affect in autistic, mentally retarded and normal children. J. Child Psychol. Psychiatry 30, 725\u2013735 (1989).","journal-title":"J. Child Psychol. Psychiatry"},{"key":"24_CR35","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1016\/j.biopsych.2012.11.022","volume":"74","author":"K Chawarska","year":"2013","unstructured":"Chawarska, K., Macari, S. & Shic, F. Decreased spontaneous attention to social scenes in 6-month-old infants later diagnosed with autism spectrum disorders. Biol. Psychiatry 74, 195\u2013203 (2013).","journal-title":"Biol. Psychiatry"},{"key":"24_CR36","doi-asserted-by":"publisher","DOI":"10.1186\/s13229-015-0019-y","volume":"6","author":"AK Halladay","year":"2015","unstructured":"Halladay, A. K. et al. Sex and gender differences in autism spectrum disorder: summarizing evidence gaps and identifying emerging areas of priority. Mol. Autism 6, 36 (2015).","journal-title":"Mol. Autism"},{"key":"24_CR37","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.jaac.2014.10.003","volume":"54","author":"MC Lai","year":"2015","unstructured":"Lai, M. C. et al. Sex\/gender differences and autism: setting the scene for future research. J. Am. Acad. Child Adolesc. Psychiatry 54, 11\u201324 (2015).","journal-title":"J. Am. Acad. Child Adolesc. Psychiatry"},{"key":"24_CR38","first-page":"1","volume":"6:24","author":"MC Lai","year":"2015","unstructured":"Lai, M. C., Baron-Cohen, S. & Buxbaum, J. D. Understanding autism in the light of sex\/gender. Mol. Autism 6:24, 1\u20135 (2015).","journal-title":"Mol. Autism"},{"key":"24_CR39","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s10882-014-9397-x","volume":"27","author":"E Rubenstein","year":"2015","unstructured":"Rubenstein, E., Wiggins, L. D. & Lee, L. C. A review of the differences in developmental, psychiatric, and medical endophenotypes between males and females with autism spectrum disorder. J. Dev. Phys. Disabil. 27, 119\u2013139 (2015).","journal-title":"J. Dev. Phys. Disabil."},{"key":"24_CR40","first-page":"1","volume":"7:10","author":"A Rynkiewicz","year":"2016","unstructured":"Rynkiewicz, A. et al. An investigation of the \u2018female camouflage effect\u2019 in autism using computerized ADOS-2 and test of sex\/gender differences. Mol. Autism 7:10, 1\u20138 (2016).","journal-title":"Mol. Autism"},{"key":"24_CR41","first-page":"276","volume":"2015","author":"Q Qiu","year":"2015","unstructured":"Qiu, Q. et al. Low-cost gaze and pulse analysis using RealSense. MobiHealth 2015, 276\u2013279 (2015).","journal-title":"MobiHealth"},{"key":"24_CR42","doi-asserted-by":"crossref","unstructured":"Chen, J., et al. Realsense\u2009=\u2009Real heart rate: illumination invariant heart rate estimation from videos. IEEE International Conference on Image Processing Theory, Tools, and Applications (IEEE, Finland, 2016).","DOI":"10.1109\/IPTA.2016.7820970"},{"key":"24_CR43","unstructured":"Sapiro, G., et al. Methods, systems, and computer readable media for automated mental health behavioral assessment. Patent Appl. (2017)."},{"key":"24_CR44","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","volume":"57","author":"Y Benjamini","year":"1995","unstructured":"Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B Methodol. 57, 289\u2013300 (1995).","journal-title":"J. R. Stat. Soc. Ser. B Methodol."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-018-0024-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-018-0024-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-018-0024-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,6]],"date-time":"2024-07-06T17:26:34Z","timestamp":1720286794000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-018-0024-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,1]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,12]]}},"alternative-id":["24"],"URL":"https:\/\/doi.org\/10.1038\/s41746-018-0024-6","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6,1]]},"assertion":[{"value":"3 December 2017","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 February 2018","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 March 2018","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 June 2018","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"G.D. is on the Scientific Advisory Boards of Janssen Research and Development, Akili, Inc., and Roche Pharmaceuticals, has received grant funding from Janssen Research and Development, L.L.C., received royalties from Guildford Press and Oxford University Press, and is a member of DASIO, L.L.C., G.S. is on the Board of Surgical Information Sciences, L.L.C., and is a member of DASIO, L.C.C., R.B. moved to Apple after the design was completed and the app was released. Duke University has IP and patent pending on some of the material reported in this work. The remaining authors declare no competing financial interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"20"}}