{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T00:20:03Z","timestamp":1776385203557,"version":"3.51.2"},"reference-count":91,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,6,3]],"date-time":"2020-06-03T00:00:00Z","timestamp":1591142400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,6,3]],"date-time":"2020-06-03T00:00:00Z","timestamp":1591142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80% and positive predictive value was 83%. For clinician-identified harm-related sentences, the word error rate was 34%. These results suggest that automatic speech recognition may support understanding of language patterns and subgroup variation in existing treatments but may not be ready for individual-level safety surveillance.<\/jats:p>","DOI":"10.1038\/s41746-020-0285-8","type":"journal-article","created":{"date-parts":[[2020,6,3]],"date-time":"2020-06-03T15:01:45Z","timestamp":1591196505000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":49,"title":["Assessing the accuracy of automatic speech recognition for psychotherapy"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5125-4735","authenticated-orcid":false,"given":"Adam S.","family":"Miner","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6769-6370","authenticated-orcid":false,"given":"Albert","family":"Haque","sequence":"additional","affiliation":[]},{"given":"Jason A.","family":"Fries","sequence":"additional","affiliation":[]},{"given":"Scott L.","family":"Fleming","sequence":"additional","affiliation":[]},{"given":"Denise E.","family":"Wilfley","sequence":"additional","affiliation":[]},{"given":"G.","family":"Terence Wilson","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2794-2378","authenticated-orcid":false,"given":"Arnold","family":"Milstein","sequence":"additional","affiliation":[]},{"given":"Dan","family":"Jurafsky","sequence":"additional","affiliation":[]},{"given":"Bruce A.","family":"Arnow","sequence":"additional","affiliation":[]},{"given":"W.","family":"Stewart Agras","sequence":"additional","affiliation":[]},{"given":"Li","family":"Fei-Fei","sequence":"additional","affiliation":[]},{"given":"Nigam H.","family":"Shah","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,6,3]]},"reference":[{"key":"285_CR1","doi-asserted-by":"publisher","unstructured":"Merz, J., Schwarzer, G. & Gerger, H. Comparative efficacy and acceptability of pharmacological, psychotherapeutic, and combination treatments in adults with posttraumatic stress disorder: a network meta-analysis. JAMA Psychiatry. https:\/\/doi.org\/10.1001\/jamapsychiatry.2019.0951 (2019).","DOI":"10.1001\/jamapsychiatry.2019.0951"},{"key":"285_CR2","doi-asserted-by":"crossref","unstructured":"Castonguay, L. G. & Hill, C. E. How and why are some therapists better than others?: Understanding Therapist Effects Vol. 356 (American Psychological Association, 2017).","DOI":"10.1037\/0000034-000"},{"key":"285_CR3","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1037\/a0036841","volume":"52","author":"ZE Imel","year":"2015","unstructured":"Imel, Z. E., Steyvers, M. & Atkins, D. C. Computational psychotherapy research: scaling up the evaluation of patient-provider interactions. Psychotherapy 52, 19\u201330 (2015).","journal-title":"Psychotherapy"},{"key":"285_CR4","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/S2215-0366(17)30513-8","volume":"5","author":"EA Holmes","year":"2018","unstructured":"Holmes, E. A. et al. The Lancet Psychiatry Commission on psychological treatments research in tomorrow\u2019s science. Lancet Psychiatry 5, 237\u2013286 (2018).","journal-title":"Lancet Psychiatry"},{"key":"285_CR5","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1016\/j.brat.2016.06.004","volume":"88","author":"AE Kazdin","year":"2017","unstructured":"Kazdin, A. E. Addressing the treatment gap: a key challenge for extending evidence-based psychosocial interventions. Behav. Res. Ther. 88, 7\u201318 (2017).","journal-title":"Behav. Res. Ther."},{"key":"285_CR6","doi-asserted-by":"publisher","first-page":"746","DOI":"10.3389\/fpsyt.2019.00746","volume":"10","author":"AS Miner","year":"2019","unstructured":"Miner, A. S. et al. Key considerations for incorporating conversational AI in psychotherapy. Front. Psychiatry 10, 746 (2019).","journal-title":"Front. Psychiatry"},{"key":"285_CR7","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1037\/amp0000365","volume":"74","author":"MR Goldfried","year":"2019","unstructured":"Goldfried, M. R. Obtaining consensus in psychotherapy: what holds us back? Am. Psychol. 74, 484\u2013496 (2019).","journal-title":"Am. Psychol."},{"key":"285_CR8","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1111\/j.1939-0025.1942.tb05930.x","volume":"12","author":"CR Rogers","year":"1942","unstructured":"Rogers, C. R. The use of electrically recorded interviews in improving psychotherapeutic techniques. Am. J. Orthopsychiatry 12, 429\u2013434 (1942).","journal-title":"Am. J. Orthopsychiatry"},{"key":"285_CR9","doi-asserted-by":"crossref","unstructured":"Gelo, O., Pritz, A. & Rieken, B. Psychotherapy Research: Foundations, Process, and Outcome (Springer, 2016).","DOI":"10.1007\/978-3-7091-1382-0"},{"key":"285_CR10","doi-asserted-by":"crossref","first-page":"45","DOI":"10.4081\/ripppo.2012.144","volume":"15","author":"OCG Gelo","year":"2012","unstructured":"Gelo, O. C. G., Salcuni, S. & Colli, A. Text Analysis within quantitative and qualitative psychotherapy process research: introduction to special issue. Res. Psychother. 15, 45\u201353 (2012).","journal-title":"Res. Psychother."},{"key":"285_CR11","doi-asserted-by":"publisher","unstructured":"Ewbank, M. P. et al. Quantifying the association between psychotherapy content and clinical outcomes using deep learning. JAMA Psychiatry. https:\/\/doi.org\/10.1001\/jamapsychiatry.2019.2664 (2019).","DOI":"10.1001\/jamapsychiatry.2019.2664"},{"key":"285_CR12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0143055","volume":"10","author":"B Xiao","year":"2015","unstructured":"Xiao, B., Imel, Z. E., Georgiou, P. G., Atkins, D. C. & Narayanan, S. S. \u2018Rate My Therapist\u2019: Automated detection of empathy in drug and alcohol counseling via speech and language processing. PLOS ONE 10, e0143055 (2015).","journal-title":"PLOS ONE"},{"key":"285_CR13","doi-asserted-by":"publisher","first-page":"563","DOI":"10.1016\/j.mayocp.2018.02.016","volume":"93","author":"SY Lin","year":"2018","unstructured":"Lin, S. Y., Shanafelt, T. D. & Asch, S. M. Reimagining clinical documentation with artificial intelligence. Mayo Clin. Proc. 93, 563\u2013565 (2018).","journal-title":"Mayo Clin. Proc."},{"key":"285_CR14","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1093\/jamia\/ocy179","volume":"26","author":"SV Blackley","year":"2019","unstructured":"Blackley, S. V., Huynh, J., Wang, L., Korach, Z. & Zhou, L. Speech recognition for clinical documentation from 1990 to 2018: a systematic review. J. Am. Med. Inform. Assoc. 26, 324\u2013338 (2019).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"285_CR15","doi-asserted-by":"publisher","unstructured":"Chiu, C.-C. et al. Speech recognition for medical conversations. Interspeech. https:\/\/doi.org\/10.21437\/Interspeech.2018-40 (2018).","DOI":"10.21437\/Interspeech.2018-40"},{"key":"285_CR16","unstructured":"Labov, W. & Fanshel, D. Therapeutic Discourse: Psychotherapy as Conversation (Academic Press, 1977)."},{"key":"285_CR17","first-page":"683","volume":"2018","author":"J Kodish-Wachs","year":"2018","unstructured":"Kodish-Wachs, J., Agassi, E., Kenny, P. 3rd & Overhage, J. M. A systematic comparison of contemporary automatic speech recognition engines for conversational clinical speech. AMIA Annu. Symp. Proc. 2018, 683\u2013689 (2018).","journal-title":"AMIA Annu. Symp. Proc."},{"key":"285_CR18","doi-asserted-by":"publisher","unstructured":"Rajkomar, A. et al. Automatically charting symptoms from patient-physician conversations using machine learning. JAMA Intern. Med. https:\/\/doi.org\/10.1001\/jamainternmed.2018.8558 (2019).","DOI":"10.1001\/jamainternmed.2018.8558"},{"key":"285_CR19","doi-asserted-by":"publisher","unstructured":"Marmar, C. R. et al. Speech-based markers for posttraumatic stress disorder in US veterans. Depress. Anxiety https:\/\/doi.org\/10.1002\/da.22890 (2019).","DOI":"10.1002\/da.22890"},{"key":"285_CR20","unstructured":"Mieskes, M. & Stiegelmayr, A. Preparing data from psychotherapy for natural language processing. In International Conference on Language Resources and Evaluation (European Language Resources Association, 2018)."},{"key":"285_CR21","doi-asserted-by":"publisher","first-page":"7684","DOI":"10.1073\/pnas.1915768117","volume":"117","author":"A Koenecke","year":"2020","unstructured":"Koenecke, A. et al. Racial disparities in automated speech recognition. Proc. Natl Acad. Sci. USA 117, 7684\u20137689 (2020).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"285_CR22","doi-asserted-by":"publisher","first-page":"E167","DOI":"10.1001\/amajethics.2019.167","volume":"21","author":"IY Chen","year":"2019","unstructured":"Chen, I. Y., Szolovits, P. & Ghassemi, M. Can AI help reduce disparities in general medical and mental health care? AMA J. Ethics 21, E167\u2013E179 (2019).","journal-title":"AMA J. Ethics"},{"key":"285_CR23","doi-asserted-by":"publisher","unstructured":"Schueller, S. M., Hunter, J. F., Figueroa, C. & Aguilera, A. Use of digital mental health for marginalized and underserved populations. Curr. Treatment Opt. Psychiatry. https:\/\/doi.org\/10.1007\/s40501-019-00181-z (2019).","DOI":"10.1007\/s40501-019-00181-z"},{"key":"285_CR24","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1016\/j.cct.2018.07.002","volume":"72","author":"DE Wilfley","year":"2018","unstructured":"Wilfley, D. E. et al. Training models for implementing evidence-based psychological treatment for college mental health: a cluster randomized trial study protocol. Contemp. Clin. Trials 72, 117\u2013125 (2018).","journal-title":"Contemp. Clin. Trials"},{"key":"285_CR25","unstructured":"Google. Cloud Speech-to-Text (Google, 2020)."},{"key":"285_CR26","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","volume":"16","author":"K Kroenke","year":"2001","unstructured":"Kroenke, K., Spitzer, R. L. & Williams, J. B. W. The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 16, 606\u2013613 (2001).","journal-title":"J. Gen. Intern. Med."},{"key":"285_CR27","first-page":"3111","volume":"26","author":"T Mikolov","year":"2013","unstructured":"Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. & Dean, J. Distributed representations of words and phrases and their compositionality. Adv. Neural Inform. Process. Syst. 26, 3111\u20133119 (2013).","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"285_CR28","doi-asserted-by":"publisher","unstructured":"Lewis, C. C. et al. Implementing measurement-based care in behavioral health: a review. JAMA Psychiatry. https:\/\/doi.org\/10.1001\/jamapsychiatry.2018.3329 (2018).","DOI":"10.1001\/jamapsychiatry.2018.3329"},{"key":"285_CR29","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","volume":"25","author":"A Esteva","year":"2019","unstructured":"Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25, 24\u201329 (2019).","journal-title":"Nat. Med."},{"key":"285_CR30","unstructured":"Haque, A., Guo, M., Miner, A. S. & Fei-Fei, L. Measuring depression symptom severity from spoken language and 3D facial expressions. In: Thirty-second Conference on Neural Information Processing Systems, Machine Learning for Health workshop. Preprint at: arXiv:1811.08592 (Montreal, Canada, 2018)."},{"key":"285_CR31","doi-asserted-by":"publisher","first-page":"478","DOI":"10.1126\/science.360.6388.478","volume":"360","author":"M Hutson","year":"2018","unstructured":"Hutson, M. Has artificial intelligence become alchemy? Science 360, 478 (2018).","journal-title":"Science"},{"key":"285_CR32","doi-asserted-by":"publisher","first-page":"883","DOI":"10.7326\/M18-3297","volume":"169","author":"SN Goodman","year":"2018","unstructured":"Goodman, S. N., Goel, S. & Cullen, M. R. Machine learning, health disparities, and causal reasoning. Ann. Intern. Med. 169, 883\u2013884 (2018).","journal-title":"Ann. Intern. Med."},{"key":"285_CR33","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","volume":"366","author":"Z Obermeyer","year":"2019","unstructured":"Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447\u2013453 (2019).","journal-title":"Science"},{"key":"285_CR34","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1126\/science.aal4230","volume":"356","author":"A Caliskan","year":"2017","unstructured":"Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 356, 183\u2013186 (2017).","journal-title":"Science"},{"key":"285_CR35","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1037\/a0022161","volume":"48","author":"JC Norcross","year":"2011","unstructured":"Norcross, J. C. & Wampold, B. E. Evidence-based therapy relationships: research conclusions and clinical practices. Psychotherapy 48, 98\u2013102 (2011).","journal-title":"Psychotherapy"},{"key":"285_CR36","first-page":"10","volume":"6","author":"I Elkin","year":"1999","unstructured":"Elkin, I. A major dilemma in psychotherapy outcome research: disentangling therapists from therapies. Clin. Psychol.: Sci. Pract. 6, 10\u201332 (1999).","journal-title":"Clin. Psychol.: Sci. Pract."},{"key":"285_CR37","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1080\/10503300500264911","volume":"16","author":"D-M Kim","year":"2006","unstructured":"Kim, D.-M., Wampold, B. E. & Bolt, D. M. Therapist effects in psychotherapy: a random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychother. Res. 16, 161\u2013172 (2006).","journal-title":"Psychother. Res."},{"key":"285_CR38","unstructured":"Baldwin, S. A. & Imel, Z. E. Therapist effects: findings and methods. In: Bergin and Garfield's Handbook of Psychotherapy and Behavior Change. 258\u2013297 (Wiley, 2013)."},{"key":"285_CR39","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/j.cpr.2018.08.004","volume":"67","author":"RG Johns","year":"2019","unstructured":"Johns, R. G., Barkham, M., Kellett, S. & Saxon, D. A systematic review of therapist effects: a critical narrative update and refinement to review. Clin. Psychol. Rev. 67, 78\u201393 (2019).","journal-title":"Clin. Psychol. Rev."},{"key":"285_CR40","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1037\/cou0000149","volume":"63","author":"J Owen","year":"2016","unstructured":"Owen, J. & Imel, Z. E. Introduction to the special section \u2018Big\u2019er\u2019 Data\u2019: Scaling up psychotherapy research in counseling psychology. J. Couns. Psychol. 63, 247\u2013248 (2016).","journal-title":"J. Couns. Psychol."},{"key":"285_CR41","doi-asserted-by":"publisher","DOI":"10.1017\/gmh.2019.5","volume":"6","author":"C Cork","year":"2019","unstructured":"Cork, C., Kaiser, B. N. & White, R. G. The integration of idioms of distress into mental health assessments and interventions: a systematic review. Glob. Ment. Health 6, e7 (2019).","journal-title":"Glob. Ment. Health"},{"key":"285_CR42","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1002\/jclp.20256","volume":"62","author":"LG Castonguay","year":"2006","unstructured":"Castonguay, L. G. & Beutler, L. E. Principles of therapeutic change: a task force on participants, relationships, and techniques factors. J. Clin. Psychol. 62, 631\u2013638 (2006).","journal-title":"J. Clin. Psychol."},{"key":"285_CR43","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1200\/JCO.2005.03.1955","volume":"24","author":"HS Gordon","year":"2006","unstructured":"Gordon, H. S., Street, R. L. Jr., Sharf, B. F., Kelly, P. A. & Souchek, J. Racial differences in trust and lung cancer patients\u2019 perceptions of physician communication. J. Clin. Oncol. 24, 904\u2013909 (2006).","journal-title":"J. Clin. Oncol."},{"key":"285_CR44","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1037\/cou0000114","volume":"63","author":"JN Hook","year":"2016","unstructured":"Hook, J. N. et al. Cultural humility and racial microaggressions in counseling. J. Couns. Psychol. 63, 269\u2013277 (2016).","journal-title":"J. Couns. Psychol."},{"key":"285_CR45","doi-asserted-by":"publisher","first-page":"1147","DOI":"10.1056\/NEJMsa044464","volume":"354","author":"SM Asch","year":"2006","unstructured":"Asch, S. M. et al. Who is at greatest risk for receiving poor-quality health care? N. Engl. J. Med. 354, 1147\u20131156 (2006).","journal-title":"N. Engl. J. Med."},{"key":"285_CR46","first-page":"343","volume":"11","author":"SW Stirman","year":"2004","unstructured":"Stirman, S. W., Crits-Christoph, P. & DeRubeis, R. J. Achieving successful dissemination of empirically supported psychotherapies: A synthesis of dissemination theory. Clin. Psychol.: Sci. Pract. 11, 343\u2013359 (2004).","journal-title":"Clin. Psychol.: Sci. Pract."},{"key":"285_CR47","doi-asserted-by":"publisher","first-page":"7","DOI":"10.30770\/2572-1852-102.2.7","volume":"102","author":"J Drescher","year":"2016","unstructured":"Drescher, J. et al. The growing regulation of conversion therapy. J. Med Regul. 102, 7\u201312 (2016).","journal-title":"J. Med Regul."},{"key":"285_CR48","doi-asserted-by":"crossref","unstructured":"Vessey, J. T. & Howard, K. I. Who seeks psychotherapy? (Group Dynamics,1993).","DOI":"10.1037\/0033-3204.30.4.546"},{"key":"285_CR49","doi-asserted-by":"publisher","first-page":"1493","DOI":"10.1093\/jamia\/ocz140","volume":"26","author":"J Park","year":"2019","unstructured":"Park, J. et al. Detecting conversation topics in primary care office visits from transcripts of patient-provider interactions. J. Am. Med. Inform. Assoc. 26, 1493\u20131504 (2019).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"285_CR50","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1080\/10503307.2011.563249","volume":"21","author":"DR Kraus","year":"2011","unstructured":"Kraus, D. R., Castonguay, L., Boswell, J. F., Nordberg, S. S. & Hayes, J. A. Therapist effectiveness: implications for accountability and patient care. Psychother. Res. 21, 267\u2013276 (2011).","journal-title":"Psychother. Res."},{"key":"285_CR51","unstructured":"Institute of Medicine. Vital Signs: Core Metrics for Health and Health Care Progress (National Academies Press, 2015)."},{"key":"285_CR52","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1037\/pst0000224","volume":"56","author":"AE P\u00e9rez-Rojas","year":"2019","unstructured":"P\u00e9rez-Rojas, A. E., Brown, R., Cervantes, A., Valente, T. & Pereira, S. R. \u2018Alguien abri\u00f3 la puerta:\u201d The phenomenology of bilingual Latinx clients\u2019 use of Spanish and English in psychotherapy. Psychotherapy 56, 241\u2013253 (2019).","journal-title":"Psychotherapy"},{"key":"285_CR53","doi-asserted-by":"publisher","unstructured":"Yu, Z., Cohen, T., Wallace, B., Bernstam, E. & Johnson, T. Retrofitting word vectors of mesh terms to improve semantic similarity measures. In: Workshop on Health Text Mining and Information Analysis. 43\u201351. https:\/\/doi.org\/10.18653\/v1\/W16-6106 (2016).","DOI":"10.18653\/v1\/W16-6106"},{"key":"285_CR54","unstructured":"Aronson, A. R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. In: Proc. AMIA Symp. 17\u201321 (American Medical Informatics Association, 2001)."},{"key":"285_CR55","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova, G. K. et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J. Am. Med. Inform. Assoc. 17, 507\u2013513 (2010).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"285_CR56","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1093\/jamia\/ocx132","volume":"25","author":"E Soysal","year":"2018","unstructured":"Soysal, E. et al. CLAMP\u2013a toolkit for efficiently building customized clinical natural language processing pipelines. J. Am. Med. Inform. Assoc. 25, 331\u2013336 (2018).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"285_CR57","doi-asserted-by":"publisher","unstructured":"Rubner, Y., Tomasi, C. & Guibas, L. J. A metric for distributions with applications to image databases. In: International Conference on Computer Vision. https:\/\/doi.org\/10.1109\/ICCV.1998.710701 (IEEE, 1998).","DOI":"10.1109\/ICCV.1998.710701"},{"key":"285_CR58","first-page":"306","volume":"68","author":"S Amir","year":"2017","unstructured":"Amir, S., Coppersmith, G., Carvalho, P., Silva, M. J. & Wallace, B. C. Quantifying mental health from social media with neural user embeddings. Mach. Learn. Healthc. Conf. 68, 306\u2013321 (2017).","journal-title":"Mach. Learn. Healthc. Conf."},{"key":"285_CR59","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234\u20131240 (2020).","journal-title":"Bioinformatics"},{"key":"285_CR60","doi-asserted-by":"publisher","unstructured":"Pennington, J., Socher, R. & Manning, C. D. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing. 1532\u20131543. https:\/\/doi.org\/10.3115\/v1\/D14-1162 (2014).","DOI":"10.3115\/v1\/D14-1162"},{"key":"285_CR61","doi-asserted-by":"crossref","unstructured":"Tatman, R. Gender and dialect bias in YouTube\u2019s automatic captions. In Workshop on Ethics in Natural Language Processing 53\u201359 (ACL, 2017).","DOI":"10.18653\/v1\/W17-1606"},{"key":"285_CR62","doi-asserted-by":"publisher","first-page":"E3635","DOI":"10.1073\/pnas.1720347115","volume":"115","author":"N Garg","year":"2018","unstructured":"Garg, N., Schiebinger, L., Jurafsky, D. & Zou, J. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proc. Natl Acad. Sci. USA 115, E3635\u2013E3644 (2018).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"285_CR63","doi-asserted-by":"publisher","first-page":"2507","DOI":"10.1056\/NEJMp1702071","volume":"376","author":"JH Chen","year":"2017","unstructured":"Chen, J. H. & Asch, S. M. Machine learning and prediction in medicine - beyond the peak of inflated expectations. N. Engl. J. Med. 376, 2507\u20132509 (2017).","journal-title":"N. Engl. J. Med."},{"key":"285_CR64","doi-asserted-by":"publisher","unstructured":"Emanuel, E. J. & Wachter, R. M. Artificial intelligence in health care: will the value match the hype? JAMA. https:\/\/doi.org\/10.1001\/jama.2019.4914 (2019).","DOI":"10.1001\/jama.2019.4914"},{"key":"285_CR65","doi-asserted-by":"publisher","first-page":"101753","DOI":"10.1016\/j.artmed.2019.101753","volume":"102","author":"PM Doraiswamy","year":"2020","unstructured":"Doraiswamy, P. M., Blease, C. & Bodner, K. Artificial intelligence and the future of psychiatry: insights from a global physician survey. Artif. Intell. Med. 102, 101753 (2020).","journal-title":"Artif. Intell. Med."},{"key":"285_CR66","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1038\/s41746-018-0046-0","volume":"1","author":"H Hsin","year":"2018","unstructured":"Hsin, H. et al. Transforming psychiatry into data-driven medicine with digital measurement tools. NPJ Digit Med 1, 37 (2018).","journal-title":"NPJ Digit Med"},{"key":"285_CR67","unstructured":"Roberts, L. W. A Clinical Guide to Psychiatric Ethics (American Psychiatric Publication, 2016)."},{"key":"285_CR68","doi-asserted-by":"publisher","DOI":"10.2196\/mental.9423","volume":"5","author":"N Martinez-Martin","year":"2018","unstructured":"Martinez-Martin, N. & Kreitmair, K. Ethical issues for direct-to-consumer digital psychotherapy apps: addressing accountability, data protection, and consent. JMIR Ment. Health 5, e32 (2018).","journal-title":"JMIR Ment. Health"},{"key":"285_CR69","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1038\/s41591-018-0307-0","volume":"25","author":"J He","year":"2019","unstructured":"He, J. et al. The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25, 30\u201336 (2019).","journal-title":"Nat. Med."},{"key":"285_CR70","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-019-05035-1","author":"SY Lin","year":"2019","unstructured":"Lin, S. Y., Mahoney, M. R. & Sinsky, C. A. Ten ways artificial intelligence will transform primary care. J. Gen. Intern. Med. https:\/\/doi.org\/10.1007\/s11606-019-05035-1 (2019).","journal-title":"J. Gen. Intern. Med."},{"key":"285_CR71","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1097\/ACM.0000000000002710","volume":"94","author":"BC O\u2019Brien","year":"2019","unstructured":"O\u2019Brien, B. C. Do you see what i see? Reflections on the relationship between transparency and trust. Acad. Med. 94, 757\u2013759 (2019).","journal-title":"Acad. Med."},{"key":"285_CR72","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1177\/2167702612463566","volume":"1","author":"AE Kazdin","year":"2013","unstructured":"Kazdin, A. E. & Rabbitt, S. M. Novel models for delivering mental health services and reducing the burdens of mental illness. Clin. Psychol. Sci. 1, 170\u2013191 (2013).","journal-title":"Clin. Psychol. Sci."},{"key":"285_CR73","doi-asserted-by":"publisher","first-page":"20176","DOI":"10.1038\/s41746-017-0006-0","volume":"1","author":"LW Roberts","year":"2018","unstructured":"Roberts, L. W., Chan, S. & Torous, J. New tests, new tools: mobile and connected technologies in advancing psychiatric diagnosis. npj Dig. Med. 1, 20176 (2018).","journal-title":"npj Dig. Med."},{"key":"285_CR74","doi-asserted-by":"publisher","unstructured":"The Lancet Digital Health. Walking the tightrope of artificial intelligence guidelines in clinical practice. The Lancet Digital Health. https:\/\/doi.org\/10.1016\/S2589-7500(19)30063-9 (2019).","DOI":"10.1016\/S2589-7500(19)30063-9"},{"key":"285_CR75","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1377-7","volume":"17","author":"C Nebeker","year":"2019","unstructured":"Nebeker, C., Torous, J. & Bartlett Ellis, R. J. Building the case for actionable ethics in digital health research supported by artificial intelligence. BMC Med. 17, 137 (2019).","journal-title":"BMC Med."},{"key":"285_CR76","unstructured":"National Institute of Mental Health. Strategic Objective 3: Strive for Prevention and Cures. NIMH Strategic Plan for Research. https:\/\/www.nimh.nih.gov\/about\/strategic-planning-reports\/strategic-objective-3.shtml (2019)."},{"key":"285_CR77","doi-asserted-by":"publisher","first-page":"e180530","DOI":"10.1001\/jamanetworkopen.2018.0530","volume":"1","author":"L Zhou","year":"2018","unstructured":"Zhou, L. et al. Analysis of errors in dictated clinical documents assisted by speech recognition software and professional transcriptionists. JAMA Netw Open 1, e180530 (2018).","journal-title":"JAMA Netw Open"},{"key":"285_CR78","unstructured":"Jurafsky, D. & Martin, J. H. Speech and Language Processing. (Prentice Hall, 2008)."},{"key":"285_CR79","doi-asserted-by":"publisher","unstructured":"Nanjo, H. & Kawahara, T. A new ASR evaluation measure and minimum Bayes-risk decoding for open-domain speech understanding. In: International Conference on Acoustics, Speech, and Signal Processing. https:\/\/doi.org\/10.1109\/ICASSP.2005.1415298 (IEEE, 2005).","DOI":"10.1109\/ICASSP.2005.1415298"},{"key":"285_CR80","doi-asserted-by":"publisher","first-page":"7:1","DOI":"10.1145\/3325862","volume":"12","author":"S Kafle","year":"2019","unstructured":"Kafle, S. & Huenerfauth, M. Predicting the understandability of imperfect english captions for people who are deaf or hard of hearing. ACM Trans. Access. Comput. 12, 7:1\u20137:32 (2019).","journal-title":"ACM Trans. Access. Comput."},{"key":"285_CR81","doi-asserted-by":"publisher","unstructured":"Spiccia, C., Augello, A., Pilato, G. & Vassallo, G. Semantic word error rate for sentence similarity. In: International Conference on Semantic Computing. 266\u2013269. https:\/\/doi.org\/10.1109\/ICSC.2016.11 (2016).","DOI":"10.1109\/ICSC.2016.11"},{"key":"285_CR82","doi-asserted-by":"crossref","unstructured":"Mishra, T., Ljolje, A. & Gilbert, M. Predicting human perceived accuracy of ASR systems. In: 12th Annual Conference of the International Speech Communication Association. 1945\u20131948. https:\/\/www.iscaspeech.org\/archive\/interspeech_2011\/i11_1945.html(Florence, Italy, 2011).","DOI":"10.21437\/Interspeech.2011-364"},{"key":"285_CR83","doi-asserted-by":"publisher","unstructured":"Levit, M., Chang, S., Buntschuh, B. & Kibre, N. End-to-end speech recognition accuracy metric for voice-search tasks. In International Conference on Acoustics, Speech and Signal Processing. 5141\u20135144. https:\/\/doi.org\/10.1109\/ICASSP.2012.6289078 (2012).","DOI":"10.1109\/ICASSP.2012.6289078"},{"key":"285_CR84","first-page":"3294","volume":"28","author":"R Kiros","year":"2015","unstructured":"Kiros, R. et al. Skip-thought vectors. Adv. Neural Inform. Process. Syst. 28, 3294\u20133302 (2015).","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"285_CR85","unstructured":"Wieting, J., Bansal, M., Gimpel, K. & Livescu, K. Towards universal paraphrastic sentence embeddings. In: Proceedings of the International Conference on Learning Representations, Preprint at: arXiv:1511.08198 (San Juan, Puerto Rico, 2016)."},{"key":"285_CR86","doi-asserted-by":"publisher","unstructured":"Shen, D. et al. Baseline needs more love: on simple word-embedding-based models and associated pooling mechanisms. In Annual Meeting of the Association for Computational Linguistics. 440\u2013450. https:\/\/doi.org\/10.18653\/v1\/P18-1041 (2018).","DOI":"10.18653\/v1\/P18-1041"},{"key":"285_CR87","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","volume":"73","author":"K Kreimeyer","year":"2017","unstructured":"Kreimeyer, K. et al. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J. Biomed. Inform. 73, 14\u201329 (2017).","journal-title":"J. Biomed. Inform."},{"key":"285_CR88","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","volume":"32","author":"O Bodenreider","year":"2004","unstructured":"Bodenreider, O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32, D267\u2013D270 (2004).","journal-title":"Nucleic Acids Res."},{"key":"285_CR89","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-017-0556-8","volume":"17","author":"W-H Weng","year":"2017","unstructured":"Weng, W.-H., Wagholikar, K. B., McCray, A. T., Szolovits, P. & Chueh, H. C. Medical subdomain classification of clinical notes using a machine learning-based natural language processing approach. BMC Med. Inform. Decis. Mak. 17, 155 (2017).","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"285_CR90","unstructured":"Hill, F., Cho, K., Jean, S., Devin, C. & Bengio, Y. Embedding word similarity with neural machine translation. In: International Conference on Learning Representations, Preprint at: arXiv:1412.6448 (San Diego, CA, USA, 2015)."},{"key":"285_CR91","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","volume":"17","author":"P Virtanen","year":"2020","unstructured":"Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261\u2013272 (2020).","journal-title":"Nat. Methods"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0285-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0285-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0285-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,1]],"date-time":"2023-10-01T22:16:08Z","timestamp":1696198568000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0285-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,3]]},"references-count":91,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["285"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0285-8","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,3]]},"assertion":[{"value":"26 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 April 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 June 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"L.F-F. served as Chief Scientist at Google Cloud from 2017 to 2018. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"82"}}