{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T10:08:01Z","timestamp":1776334081873,"version":"3.51.2"},"reference-count":60,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2018,7,11]],"date-time":"2018-07-11T00:00:00Z","timestamp":1531267200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["APP1134919"],"award-info":[{"award-number":["APP1134919"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Centre for Research Excellence in Digital Health","award":["APP1054146"],"award-info":[{"award-number":["APP1054146"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen\u2019s kappa measured inter-coder agreement.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d\u2009=\u20090.44, p\u2009=\u2009.04). Patient safety was rarely evaluated in the included studies.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting.<\/jats:p><\/jats:sec><jats:sec><jats:title>Protocol Registration<\/jats:title><jats:p>The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocy072","type":"journal-article","created":{"date-parts":[[2018,5,22]],"date-time":"2018-05-22T03:10:55Z","timestamp":1526958655000},"page":"1248-1258","source":"Crossref","is-referenced-by-count":1076,"title":["Conversational agents in healthcare: a systematic review"],"prefix":"10.1093","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1020-3402","authenticated-orcid":false,"given":"Liliana","family":"Laranjo","sequence":"first","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Adam G","family":"Dunn","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8462-0105","authenticated-orcid":false,"given":"Huong Ly","family":"Tong","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Ahmet Baki","family":"Kocaballi","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Jessica","family":"Chen","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Rabia","family":"Bashir","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Didi","family":"Surian","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Blanca","family":"Gallego","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Farah","family":"Magrabi","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"given":"Annie Y S","family":"Lau","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6444-6584","authenticated-orcid":false,"given":"Enrico","family":"Coiera","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}]}],"member":"286","published-online":{"date-parts":[[2018,7,11]]},"reference":[{"key":"2020110612235159000_ocy072-B1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-32967-3","volume-title":"The Conversational Interface: Talking to Smart Devices","author":"McTear","year":"2016"},{"issue":"1","key":"2020110612235159000_ocy072-B2","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1145\/365153.365168","article-title":"ELIZA\u2014a computer program for the study of natural language communication between man and machine","volume":"9","author":"Weizenbaum","year":"1966","journal-title":"Commun ACM"},{"issue":"10","key":"2020110612235159000_ocy072-B3","doi-asserted-by":"crossref","first-page":"1676","DOI":"10.1111\/jgs.12449","article-title":"A randomized controlled trial of an automated exercise coach for older adults","volume":"61","author":"Bickmore","year":"2013","journal-title":"J Am Geriatr Soc"},{"issue":"2","key":"2020110612235159000_ocy072-B4","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.pec.2013.05.011","article-title":"Automated interventions for multiple health behaviors using conversational agents","volume":"92","author":"Bickmore","year":"2013","journal-title":"Patient Educ Couns"},{"issue":"1","key":"2020110612235159000_ocy072-B5","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.2196\/jmir.1629","article-title":"An internet-based virtual coach to promote physical activity adherence in overweight adults: randomized controlled trial","volume":"14","author":"Watson","year":"2012","journal-title":"J Med Internet Res"},{"issue":"10","key":"2020110612235159000_ocy072-B6","doi-asserted-by":"crossref","first-page":"1961","DOI":"10.1007\/s10995-013-1222-0","article-title":"Use of an interactive computer agent to support breastfeeding","volume":"17","author":"Edwards","year":"2013","journal-title":"Matern Child Health J"},{"key":"2020110612235159000_ocy072-B7","author":"Stone","year":"2016"},{"issue":"1","key":"2020110612235159000_ocy072-B8","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1145\/505282.505285","article-title":"Spoken dialogue technology: enabling the conversational user interface","volume":"34","author":"McTear","year":"2002","journal-title":"ACM Comput Surv"},{"key":"2020110612235159000_ocy072-B9","first-page":"1704","volume-title":"arXiv Prepr","author":"Radziwill","year":"2017"},{"key":"2020110612235159000_ocy072-B10","doi-asserted-by":"crossref","DOI":"10.1007\/978-4-431-55040-2","volume-title":"Conversational Informatics: A Data-Intensive Approach with Emphasis on Nonverbal Communication","author":"Nishida","year":"2014"},{"issue":"4","key":"2020110612235159000_ocy072-B11","doi-asserted-by":"crossref","first-page":"854","DOI":"10.1177\/1460458215593329","article-title":"Designing a spoken dialogue interface to an intelligent cognitive assistant for people with dementia","volume":"22","author":"Wolters","year":"2016","journal-title":"Health Informatics J"},{"issue":"5","key":"2020110612235159000_ocy072-B12","doi-asserted-by":"crossref","first-page":"e151","DOI":"10.2196\/jmir.6553","article-title":"Embodied conversational agents in clinical psychology: a scoping review","volume":"19","author":"Provoost","year":"2017","journal-title":"J Med Internet Res"},{"issue":"5","key":"2020110612235159000_ocy072-B13","doi-asserted-by":"crossref","first-page":"556","DOI":"10.1016\/j.jbi.2005.12.004","article-title":"Health dialog systems for patients and consumers","volume":"39","author":"Bickmore","year":"2006","journal-title":"J Biomed Inform"},{"key":"2020110612235159000_ocy072-B14","first-page":"187","year":"2007"},{"issue":"1","key":"2020110612235159000_ocy072-B15","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1186\/s40469-015-0006-9","article-title":"Evaluating embodied conversational agents in multimodal interfaces","volume":"1","author":"Weiss","year":"2015","journal-title":"Comput Cogn Sci"},{"issue":"4","key":"2020110612235159000_ocy072-B16","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1111\/j.1468-0009.2005.00397.x","article-title":"Evaluating the quality of medical care. 1966","volume":"83","author":"Donabedian","year":"2005","journal-title":"Milbank Q"},{"key":"2020110612235159000_ocy072-B17","doi-asserted-by":"crossref","DOI":"10.1002\/9780470712184","volume-title":"Cochrane Handbook for Systematic Reviews of Interventions","author":"Higgins","year":"2008"},{"issue":"7","key":"2020110612235159000_ocy072-B18","doi-asserted-by":"crossref","first-page":"e1000100","DOI":"10.1371\/journal.pmed.1000100","article-title":"The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration","volume":"6","author":"Liberati","year":"2009","journal-title":"PLoS Med"},{"issue":"2","key":"2020110612235159000_ocy072-B19","doi-asserted-by":"crossref","first-page":"e19.","DOI":"10.2196\/mental.7785","article-title":"Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial","volume":"4","author":"Fitzpatrick","year":"2017","journal-title":"JMIR Ment Health"},{"issue":"8","key":"2020110612235159000_ocy072-B20","doi-asserted-by":"crossref","first-page":"e0182151","DOI":"10.1371\/journal.pone.0182151","article-title":"Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders","volume":"12","author":"Tanaka","year":"2017","journal-title":"PLoS One"},{"key":"2020110612235159000_ocy072-B21","first-page":"55","article-title":"Hello Harlie: enabling speech monitoring through chat-bot conversations","volume":"227","author":"Ireland","year":"2016","journal-title":"Stud Health Technol Inform"},{"key":"2020110612235159000_ocy072-B22","doi-asserted-by":"crossref","first-page":"42656.","DOI":"10.1038\/srep42656","article-title":"Virtual human as a new diagnostic tool, a proof of concept study in the field of major depressive disorders","volume":"7","author":"Philip","year":"2017","journal-title":"Sci Rep"},{"issue":"4","key":"2020110612235159000_ocy072-B23","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1162\/PRES_a_00197","article-title":"Could a virtual human be used to explore excessive daytime sleepiness in patients?","volume":"23","author":"Philip","year":"2014","journal-title":"Presence Teleoperators Virtual Environ"},{"issue":"2","key":"2020110612235159000_ocy072-B24","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1016\/j.pec.2013.05.007","article-title":"Virtual training and coaching of health behavior: example from mindfulness meditation training","volume":"92","author":"Hudlicka","year":"2013","journal-title":"Patient Educ Couns"},{"issue":"5","key":"2020110612235159000_ocy072-B25","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1016\/j.jbi.2005.12.008","article-title":"Automatic generation of spoken dialogue from medical plans and ontologies","volume":"39","author":"Beveridge","year":"2006","journal-title":"J Biomed Inform"},{"issue":"5","key":"2020110612235159000_ocy072-B26","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1001\/jamainternmed.2016.0400","article-title":"Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health","volume":"176","author":"Miner","year":"2016","journal-title":"JAMA Intern Med"},{"issue":"5","key":"2020110612235159000_ocy072-B27","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1016\/j.jadohealth.2010.09.002","article-title":"An artificially intelligent chat agent that answers adolescents\u2019 questions related to sex, drugs, and alcohol: an exploratory study","volume":"48","author":"Crutzen","year":"2011","journal-title":"J Adolesc Health"},{"key":"2020110612235159000_ocy072-B28","doi-asserted-by":"crossref","first-page":"63","DOI":"10.2147\/PPA.S53504","article-title":"Mobile phone-based asthma self-management aid for adolescents (mASMAA): a feasibility study","volume":"8","author":"Rhee","year":"2014","journal-title":"Patient Prefer Adherence"},{"key":"2020110612235159000_ocy072-B29","first-page":"506","author":"Black","year":"2005"},{"key":"2020110612235159000_ocy072-B30","first-page":"219","author":"Harper"},{"issue":"9","key":"2020110612235159000_ocy072-B31","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1080\/08839514.2013.835230","article-title":"An automatic dialog simulation technique to develop and evaluate interactive conversational agents","volume":"27","author":"Griol","year":"2013","journal-title":"Appl Artif Intell"},{"issue":"4","key":"2020110612235159000_ocy072-B32","doi-asserted-by":"crossref","first-page":"e30.","DOI":"10.2196\/jmir.8.4.e30","article-title":"Evaluation of spoken dialogue technology for real-time health data collection","volume":"8","author":"Levin","year":"2006","journal-title":"J Med Internet Res"},{"issue":"2\u20134","key":"2020110612235159000_ocy072-B33","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/j.ijmedinf.2004.04.026","article-title":"Automated spoken dialogue system for hypertensive patient home management","volume":"74","author":"Giorgino","year":"2005","journal-title":"Int J Med Inform"},{"key":"2020110612235159000_ocy072-B34","first-page":"146","article-title":"Automated spoken dialog system for home care and data acquisition from chronic patients","volume":"95","author":"Azzini","year":"2003","journal-title":"Stud Health Technol Inform"},{"key":"2020110612235159000_ocy072-B35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3389\/frobt.2017.00051","article-title":"Reporting mental health symptoms: breaking down barriers to care with virtual human interviewers","volume":"4","author":"Lucas","year":"2017","journal-title":"Front Robot AI"},{"key":"2020110612235159000_ocy072-B36","doi-asserted-by":"crossref","first-page":"223","DOI":"10.4018\/978-1-60960-617-6.ch010","volume-title":"Conversational Agents and Natural Language Interaction","author":"L\u00f3pez-C\u00f3zar","year":"2011"},{"key":"2020110612235159000_ocy072-B37","first-page":"CD009921","article-title":"Automated telephone communication systems for preventive healthcare and management of long-term conditions","volume":"12","author":"Posadzki","year":"2016","journal-title":"Cochrane Database Syst Rev"},{"key":"2020110612235159000_ocy072-B38","author":"Serban"},{"key":"2020110612235159000_ocy072-B39","first-page":"1","author":"Mallios","year":"2016"},{"key":"2020110612235159000_ocy072-B40","doi-asserted-by":"crossref","first-page":"1142","DOI":"10.1109\/5.880077","article-title":"Automatic speech recognition and understanding: a first step toward natural human machine communication","volume":"88","author":"Juang","year":"2000","journal-title":"Proc IEEE"},{"issue":"5","key":"2020110612235159000_ocy072-B41","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1109\/JPROC.2012.2225812","article-title":"Pomdp-based statistical spoken dialog systems: a review","volume":"101","author":"Young","year":"2013","journal-title":"Proc IEEE"},{"key":"2020110612235159000_ocy072-B42","author":"Lamel","year":"2000"},{"key":"2020110612235159000_ocy072-B43","doi-asserted-by":"crossref","first-page":"271","DOI":"10.3115\/979617.979652","volume-title":"Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics","author":"Walker","year":"1997"},{"key":"2020110612235159000_ocy072-B44","volume-title":"Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: further Development of the RTI Item Bank","author":"Viswanathan","year":"2013"},{"key":"2020110612235159000_ocy072-B45","author":"Streiner","year":"2008","edition":"4th ed"},{"issue":"4","key":"2020110612235159000_ocy072-B46","doi-asserted-by":"crossref","first-page":"e126","DOI":"10.2196\/jmir.1923","article-title":"CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions","volume":"13","author":"Eysenbach","year":"2011","journal-title":"J Med Internet Res"},{"issue":"3","key":"2020110612235159000_ocy072-B47","doi-asserted-by":"crossref","first-page":"361","DOI":"10.2105\/AJPH.94.3.361","article-title":"Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement","volume":"94","author":"Des Jarlais","year":"2004","journal-title":"Am J Public Health"},{"key":"2020110612235159000_ocy072-B48","doi-asserted-by":"crossref","first-page":"h5527","DOI":"10.1136\/bmj.h5527","article-title":"STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies","volume":"351","author":"Bossuyt","year":"2015","journal-title":"BMJ"},{"key":"2020110612235159000_ocy072-B49","doi-asserted-by":"crossref","first-page":"g1687","DOI":"10.1136\/bmj.g1687","article-title":"Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide","volume":"348","author":"Hoffmann","year":"2014","journal-title":"BMJ"},{"key":"2020110612235159000_ocy072-B50","first-page":"1","article-title":"Does health informatics have a replication crisis?","author":"Coiera","year":"2018","journal-title":"J Am Med Informatics Assoc"},{"issue":"6","key":"2020110612235159000_ocy072-B51","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1093\/jamia\/ocx073","article-title":"Efficiency and safety of speech recognition for documentation in the electronic health record","volume":"24","author":"Hodgson","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2020110612235159000_ocy072-B52","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1197\/jamia.M1471","article-title":"Some unintended consequences of information technology in health care: the nature of patient care information system-related errors","volume":"11","author":"Ash","year":"2003","journal-title":"J Am Med Informatics Assoc"},{"issue":"1","key":"2020110612235159000_ocy072-B53","first-page":"163","article-title":"The unintended consequences of health information technology revisited","author":"Coiera","year":"2016","journal-title":"Yearb Med Inform"},{"key":"2020110612235159000_ocy072-B54","doi-asserted-by":"crossref","first-page":"ocw154","DOI":"10.1093\/jamia\/ocw154","article-title":"Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review","volume":"24","author":"Kim","year":"2016","journal-title":"J Am Med Informatics Assoc"},{"issue":"13","key":"2020110612235159000_ocy072-B55","doi-asserted-by":"crossref","first-page":"1351.","DOI":"10.1001\/jama.2013.393","article-title":"The inevitable application of big data to health care","volume":"309","author":"Murdoch","year":"2013","journal-title":"JAMA"},{"issue":"3","key":"2020110612235159000_ocy072-B56","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1109\/3468.844354","article-title":"A model for types and levels of human interaction with automation","volume":"30","author":"Parasuraman","year":"2000","journal-title":"IEEE Trans Syst Man Cybern A Syst Hum"},{"key":"2020110612235159000_ocy072-B57","article-title":"First compute no harm","author":"Coiera","year":"2017","journal-title":"BMJ Opin"},{"issue":"6","key":"2020110612235159000_ocy072-B58","doi-asserted-by":"crossref","first-page":"517.","DOI":"10.1001\/jama.2017.7797","article-title":"Unintended consequences of machine learning in medicine","volume":"318","author":"Cabitza","year":"2017","journal-title":"JAMA"},{"issue":"7625","key":"2020110612235159000_ocy072-B59","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1038\/538311a","article-title":"There is a blind spot in AI research","volume":"538","author":"Crawford","year":"2016","journal-title":"Nature"},{"key":"2020110612235159000_ocy072-B60","doi-asserted-by":"crossref","first-page":"262","DOI":"10.3115\/979617.979651","volume-title":"Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics","author":"Chu-Carroll","year":"1997"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/9\/1248\/34150600\/ocy072.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/9\/1248\/34150600\/ocy072.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,2]],"date-time":"2023-09-02T19:03:49Z","timestamp":1693681429000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/9\/1248\/5052181"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,11]]},"references-count":60,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2018,7,11]]},"published-print":{"date-parts":[[2018,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy072","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,9]]},"published":{"date-parts":[[2018,7,11]]}}}