{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T00:22:00Z","timestamp":1785284520229,"version":"3.55.0"},"reference-count":59,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T00:00:00Z","timestamp":1713225600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["R01MH124935"],"award-info":[{"award-number":["R01MH124935"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Prevention Autism and Developmental Disabilities Monitoring"},{"name":"CDC Cooperative","award":["5UR3\/DD000680"],"award-info":[{"award-number":["5UR3\/DD000680"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,5,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Machine learning (ML) is increasingly employed to diagnose medical conditions, with algorithms trained to assign a single label using a black-box approach. We created an ML approach using deep learning that generates outcomes that are transparent and in line with clinical, diagnostic rules. We demonstrate our approach for autism spectrum disorders (ASD), a neurodevelopmental condition with increasing prevalence.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We use unstructured data from the Centers for Disease Control and Prevention (CDC) surveillance records labeled by a CDC-trained clinician with ASD A1-3 and B1-4 criterion labels per sentence and with ASD cases labels per record using Diagnostic and Statistical Manual of Mental Disorders (DSM5) rules. One rule-based and three deep ML algorithms and six ensembles were compared and evaluated using a test set with 6773 sentences (N\u2009=\u200935 cases) set aside in advance. Criterion and case labeling were evaluated for each ML algorithm and ensemble. Case labeling outcomes were compared also with seven traditional tests.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Performance for criterion labeling was highest for the hybrid BiLSTM ML model. The best case labeling was achieved by an ensemble of two BiLSTM ML models using a majority vote. It achieved 100% precision (or PPV), 83% recall (or sensitivity), 100% specificity, 91% accuracy, and 0.91 F-measure. A comparison with existing diagnostic tests shows that our best ensemble was more accurate overall.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Transparent ML is achievable even with small datasets. By focusing on intermediate steps, deep ML can provide transparent decisions. By leveraging data redundancies, ML errors at the intermediate level have a low impact on final outcomes.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae080","type":"journal-article","created":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T19:56:24Z","timestamp":1713297384000},"page":"1313-1321","source":"Crossref","is-referenced-by-count":13,"title":["Transparent deep learning to identify autism spectrum disorders (ASD) in EHR using clinical notes"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4751-6680","authenticated-orcid":false,"given":"Gondy","family":"Leroy","sequence":"first","affiliation":[{"name":"Department of Management Information Systems, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer G","family":"Andrews","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madison","family":"KeAlohi-Preece","sequence":"additional","affiliation":[{"name":"Department of Psychology, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ajay","family":"Jaswani","sequence":"additional","affiliation":[{"name":"Department of Management Information Systems, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyunju","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maureen Kelly","family":"Galindo","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sydney A","family":"Rice","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, The University of Arizona , Tucson, AZ 85621, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,16]]},"reference":[{"issue":"1","key":"2024052019463182400_ocae080-B1","doi-asserted-by":"crossref","first-page":"e17","DOI":"10.1542\/peds.2009-0958","article-title":"Randomized, controlled trial of an intervention for toddlers with autism: the early start Denver model","volume":"125","author":"Dawson","year":"2010","journal-title":"Pediatrics"},{"issue":"5","key":"2024052019463182400_ocae080-B2","doi-asserted-by":"crossref","first-page":"1683","DOI":"10.1007\/s10803-019-03927-z","article-title":"The effects of early intervention on social communication outcomes for children with autism spectrum disorder: a meta-analysis","volume":"50","author":"Fuller","year":"2020","journal-title":"J Autism Dev Disord"},{"key":"2024052019463182400_ocae080-B3","article-title":"Increase the proportion of children with autism spectrum disorder who receive special services by age 4 years \u2014 MICH-18","author":"U.S. Department of Health and Human Services"},{"issue":"11","key":"2024052019463182400_ocae080-B4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.15585\/mmwr.ss7011a1","article-title":"Prevalence and characteristics of autism spectrum disorder among children aged 8 years\u2013autism and developmental disabilities monitoring network, 11 sites, United States, 2018","volume":"70","author":"Maenner","year":"2021","journal-title":"MMWR Surveill Summ"},{"issue":"6","key":"2024052019463182400_ocae080-B5","doi-asserted-by":"crossref","first-page":"1431","DOI":"10.1016\/j.jpeds.2015.03.007","article-title":"Parental concerns, provider response, and timeliness of autism spectrum disorder diagnosis","volume":"166","author":"Zuckerman","year":"2015","journal-title":"J Pediatr"},{"issue":"4","key":"2024052019463182400_ocae080-B6","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1017\/S1041610216001599","article-title":"Aging with autism spectrum disorder: an emerging public health problem","volume":"29","author":"Hategan","year":"2016","journal-title":"Int Psychogeriatr"},{"issue":"6","key":"2024052019463182400_ocae080-B7","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1192\/bjp.bp.115.174649","article-title":"Epidemiology of autism in adults across age groups and ability levels","volume":"209","author":"Brugha","year":"2016","journal-title":"Br J Psychiatry"},{"key":"2024052019463182400_ocae080-B8","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1186\/1471-244X-9-35","article-title":"Psychiatric and psychosocial problems in adults with normal-intelligence autism spectrum disorders","volume":"9","author":"Hofvander","year":"2009","journal-title":"BMC Psychiatry"},{"issue":"10","key":"2024052019463182400_ocae080-B9","doi-asserted-by":"crossref","first-page":"1435","DOI":"10.1038\/s41380-018-0321-0","article-title":"Big data approaches to decomposing heterogeneity across the autism spectrum","volume":"24","author":"Lombardo","year":"2019","journal-title":"Mol Psychiatry"},{"issue":"7","key":"2024052019463182400_ocae080-B10","first-page":"CD009044","article-title":"Diagnostic tests for autism spectrum disorder (ASD) in preschool children","volume":"7","author":"Randall","year":"2018","journal-title":"Cochrane Database Syst Rev"},{"issue":"4","key":"2024052019463182400_ocae080-B11","doi-asserted-by":"crossref","first-page":"1391","DOI":"10.1007\/s10803-018-3831-3","article-title":"When are we sure? Predictors of clinician certainty in the diagnosis of autism spectrum disorder","volume":"49","author":"McDonnell","year":"2019","journal-title":"J Autism Dev Disord"},{"issue":"2","key":"2024052019463182400_ocae080-B12","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1542\/peds.2009-0710","article-title":"Electronic health record decision support and quality of care for children with ADHD","volume":"126","author":"Co","year":"2010","journal-title":"Pediatrics"},{"issue":"7","key":"2024052019463182400_ocae080-B13","doi-asserted-by":"crossref","first-page":"e0159621","DOI":"10.1371\/journal.pone.0159621","article-title":"Electronic health record based algorithm to identify patients with autism spectrum disorder","volume":"11","author":"Lingren","year":"2016","journal-title":"PLoS One"},{"issue":"9","key":"2024052019463182400_ocae080-B14","doi-asserted-by":"crossref","first-page":"e0222907","DOI":"10.1371\/journal.pone.0222907","article-title":"A comparison of machine learning algorithms for the surveillance of autism spectrum disorder","volume":"14","author":"Lee","year":"2019","journal-title":"PLoS One"},{"issue":"1","key":"2024052019463182400_ocae080-B15","doi-asserted-by":"crossref","first-page":"5014","DOI":"10.1038\/s41598-020-61213-w","article-title":"Multi-modular AI approach to streamline autism diagnosis in young children","volume":"10","author":"Abbas","year":"2020","journal-title":"Sci Rep"},{"issue":"8","key":"2024052019463182400_ocae080-B16","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1093\/jamia\/ocy039","article-title":"Machine learning approach for early detection of autism by combining questionnaire and home video screening","volume":"25","author":"Abbas","year":"2018","journal-title":"J Am Med Inform Assoc (JAMIA)"},{"key":"2024052019463182400_ocae080-B17","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s13637-017-0057-1","article-title":"Autism spectrum disorder detection from semi-structured and unstructured medical data","volume":"2017","author":"Yuan","year":"2017","journal-title":"EURASIP J Bioinform Syst Biol"},{"key":"2024052019463182400_ocae080-B18","author":"Omar","year":"2019"},{"issue":"6","key":"2024052019463182400_ocae080-B19","doi-asserted-by":"crossref","first-page":"3081","DOI":"10.1002\/hbm.23575","article-title":"Multi-task diagnosis for autism spectrum disorders using multi-modality features: A multi-center study","volume":"38","author":"Wang","year":"2017","journal-title":"Hum Brain Mapp"},{"issue":"235","key":"2024052019463182400_ocae080-B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jneumeth.2014.06.030","article-title":"Creating multimodal predictors using missing data: classifying and subtyping autism spectrum disorder","volume":"235","author":"Ingalhalikar","year":"2014","journal-title":"J Neurosci Methods"},{"key":"2024052019463182400_ocae080-B21","doi-asserted-by":"crossref","first-page":"949926","DOI":"10.3389\/fninf.2022.949926","article-title":"Machine learning for autism spectrum disorder diagnosis using structural magnetic resonance imaging: promising but challenging","volume":"16","author":"Bahathiq","year":"2022","journal-title":"Front Neuroinform"},{"key":"2024052019463182400_ocae080-B22","doi-asserted-by":"crossref","first-page":"107877","DOI":"10.1016\/j.chb.2023.107877","article-title":"Early prediction of autism spectrum disorders through interaction analysis in home videos and explainable artificial intelligence","volume":"148","author":"Paolucci","year":"2023","journal-title":"Computers in Human Behavior"},{"issue":"5","key":"2024052019463182400_ocae080-B23","doi-asserted-by":"crossref","first-page":"E384","DOI":"10.1016\/S2589-7500(22)00003-6","article-title":"The medical algorithm audit","volume":"4","author":"Liu","year":"2022","journal-title":"Lancet Digit Health"},{"key":"2024052019463182400_ocae080-B24","first-page":"4768","author":"Lundberg","year":"2017"},{"key":"2024052019463182400_ocae080-B25","first-page":"1135","author":"Ribeiro","year":"2016"},{"key":"2024052019463182400_ocae080-B26","author":"Baan","year":"2019"},{"key":"2024052019463182400_ocae080-B27","author":"Magboo","year":"2022"},{"issue":"1","key":"2024052019463182400_ocae080-B28","article-title":"Uncovering and correcting shortcut learning in machine learning models for skin cancer diagnosis","volume":"12","author":"Nauta","year":"2021","journal-title":"Diagnostics (Basel)"},{"issue":"12","key":"2024052019463182400_ocae080-B29","doi-asserted-by":"crossref","first-page":"949","DOI":"10.3390\/brainsci10120949","article-title":"A review of machine learning methods of feature selection and classification for autism spectrum disorder","volume":"10","author":"Rahman","year":"2020","journal-title":"Brain Sci"},{"issue":"10","key":"2024052019463182400_ocae080-B30","doi-asserted-by":"crossref","first-page":"1419","DOI":"10.1093\/jamia\/ocy068","article-title":"Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review","volume":"25","author":"Xiao","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2024052019463182400_ocae080-B31","doi-asserted-by":"crossref","DOI":"10.1007\/s11764-023-01465-3","article-title":"Machine learning (ML) techniques to predict breast cancer in imbalanced datasets: a systematic review","author":"Ghavidel","year":"2023","journal-title":"J Cancer Surviv"},{"key":"2024052019463182400_ocae080-B32","author":"Woolsey","year":"2024"},{"key":"2024052019463182400_ocae080-B33","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic Minority Over-sampling Technique","volume":"16","author":"Chawla","year":"2002","journal-title":"JAIR"},{"key":"2024052019463182400_ocae080-B34","author":"Gu","year":"2019"},{"key":"2024052019463182400_ocae080-B35","first-page":"34","author":"Leroy","year":"2017"},{"issue":"11","key":"2024052019463182400_ocae080-B36","first-page":"e10497","article-title":"Automated extraction of diagnostic criteria from electronic health records for autism spectrum disorders: development, evaluation and case study","volume":"20","author":"Leroy","year":"2018","journal-title":"J Med Inter Res (JMIR)"},{"key":"2024052019463182400_ocae080-B37","author":"Gu","year":"2018"},{"key":"2024052019463182400_ocae080-B38","author":"CDC","year":"2024"},{"key":"2024052019463182400_ocae080-B39","author":"AutismSpeaks","year":"2023"},{"issue":"6","key":"2024052019463182400_ocae080-B40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.15585\/mmwr.ss6706a1","article-title":"Prevalence of autism spectrum disorder among children aged 8 Years- Autism and developmental disabilities monitoring network, 11 Sites, United States,, 2014","volume":"67","author":"Baio","year":"2018","journal-title":"MMWR Surveill Summ"},{"key":"2024052019463182400_ocae080-B41","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1176\/appi.books.9780890423349","volume-title":"Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition\u2014Text Revision (DSMIV-TR)","author":"American Psychiatric Association","year":"2000","edition":"4th Edition ed"},{"key":"2024052019463182400_ocae080-B42","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1176\/appi.books.9780890425596","volume-title":"Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition\u2014Text Revision (DSMI5)","author":"American Psychiatric","year":"2013","edition":"5th ed"},{"issue":"12","key":"2024052019463182400_ocae080-B43","doi-asserted-by":"crossref","first-page":"5308","DOI":"10.1007\/s10803-021-05377-y","article-title":"Defining in detail and evaluating reliability of DSM-5 criteria for autism spectrum disorder (ASD) among children","volume":"52","author":"Rice","year":"2022","journal-title":"J Autism Dev Disord"},{"issue":"4","key":"2024052019463182400_ocae080-B44","first-page":"585","article-title":"Natural language processing framework to assess clinical conditions","volume":"16","author":"Ware","year":"2009","journal-title":"JAMIA"},{"issue":"4","key":"2024052019463182400_ocae080-B45","first-page":"596","article-title":"A text mining approach to the prediction of disease status from clinical discharge summaries","volume":"16","author":"Yang","year":"2009","journal-title":"JAMIA"},{"issue":"3","key":"2024052019463182400_ocae080-B46","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1093\/jamia\/ocaa218","article-title":"Deep learning approaches for extracting adverse events and indications of dietary supplements from clinical text","volume":"28","author":"Fan","year":"2021","journal-title":"J Am Med Inform Assoc (JAMIA)"},{"key":"2024052019463182400_ocae080-B47","author":"Pennington","year":"2014"},{"issue":"2","key":"2024052019463182400_ocae080-B48","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1007\/s10803-019-04272-x","article-title":"Utility of the asperger syndrome diagnostic scale in the assessment of autism spectrum disorders","volume":"50","author":"Camodeca","year":"2020","journal-title":"J Autism Dev Disord"},{"key":"2024052019463182400_ocae080-B49","volume-title":"The Childhood Autism Rating Scale (CARS) Manual","author":"Schopler","year":"2015"},{"issue":"2","key":"2024052019463182400_ocae080-B50","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1007\/s10882-006-9008-6","article-title":"Validity of the asperger syndrome diagnostic scale","volume":"18","author":"Boggs","year":"2006","journal-title":"J Dev Phys Disabil"},{"issue":"6","key":"2024052019463182400_ocae080-B51","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1177\/1362361306068505","article-title":"The discriminative ability and diagnostic utility of the ADOS-G, ADI-R, and GARS for children in a clinical setting","volume":"10","author":"Mazefsky","year":"2006","journal-title":"Autism"},{"issue":"1","key":"2024052019463182400_ocae080-B52","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1542\/peds.2013-1813","article-title":"Validation of the modified checklist for autism in toddlers, revised with follow-up (M-CHAT-R\/F)","volume":"133","author":"Robins","year":"2014","journal-title":"Pediatrics"},{"issue":"1","key":"2024052019463182400_ocae080-B53","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1177\/0734282913490916","article-title":"Test review: autism diagnostic observation schedule, (ADOS-2) manual (Part II): toddler module","volume":"32","author":"McCrimmon","year":"2014","journal-title":"J Psychoeduc Ass"},{"key":"2024052019463182400_ocae080-B54","volume-title":"Autism Diagnostic Observation Schedule (ADOS) Manual","author":"Lord","year":"2001"},{"issue":"12","key":"2024052019463182400_ocae080-B55","doi-asserted-by":"crossref","first-page":"e0168224","DOI":"10.1371\/journal.pone.0168224","article-title":"Development of a machine learning algorithm for the surveillance of autism spectrum disorder","volume":"11","author":"Maenner","year":"2016","journal-title":"PLoS One"},{"issue":"1","key":"2024052019463182400_ocae080-B56","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1177\/1460458218824711","article-title":"A new machine learning model based on induction of rules for autism detection","volume":"26","author":"Thabtah","year":"2020","journal-title":"Health Informatics J"},{"key":"2024052019463182400_ocae080-B57","author":"Gu","year":"2018"},{"issue":"1","key":"2024052019463182400_ocae080-B58","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1038\/s41398-020-0836-4","article-title":"Precision psychiatry with immunological and cognitive biomarkers: a multi-domain prediction for the diagnosis of bipolar disorder or schizophrenia using machine learning","volume":"10","author":"Fernandes","year":"2020","journal-title":"Transl Psychiatry"},{"issue":"7-8","key":"2024052019463182400_ocae080-B59","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1016\/j.ijmedinf.2005.03.013","article-title":"Effects of information and machine learning algorithms on word sense disambiguation with small datasets","volume":"74","author":"Leroy","year":"2005","journal-title":"Int J Med Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1313\/57768979\/ocae080.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1313\/57768979\/ocae080.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,20]],"date-time":"2024-05-20T19:47:40Z","timestamp":1716234460000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/6\/1313\/7646767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,16]]},"references-count":59,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,4,16]]},"published-print":{"date-parts":[[2024,5,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae080","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,6,1]]},"published":{"date-parts":[[2024,4,16]]}}}