{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T17:03:33Z","timestamp":1780333413036,"version":"3.54.1"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2019,9,18]],"date-time":"2019-09-18T00:00:00Z","timestamp":1568764800000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006093","name":"Patient-Centered Outcomes Research Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Amid electronic health records, laboratory tests, and other technology, office-based patient and provider communication is still the heart of primary medical care. Patients typically present multiple complaints, requiring physicians to decide how to balance competing demands. How this time is allocated has implications for patient satisfaction, payments, and quality of care. We investigate the effectiveness of machine learning methods for automated annotation of medical topics in patient-provider dialog transcripts.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We used dialog transcripts from 279 primary care visits to predict talk-turn topic labels. Different machine learning models were trained to operate on single or multiple local talk-turns (logistic classifiers, support vector machines, gated recurrent units) as well as sequential models that integrate information across talk-turn sequences (conditional random fields, hidden Markov models, and hierarchical gated recurrent units).<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Evaluation was performed using cross-validation to measure 1) classification accuracy for talk-turns and 2) precision, recall, and F1 scores at the visit level. Experimental results showed that sequential models had higher classification accuracy at the talk-turn level and higher precision at the visit level. Independent models had higher recall scores at the visit level compared with sequential models.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Incorporating sequential information across talk-turns improves the accuracy of topic prediction in patient-provider dialog by smoothing out noisy information from talk-turns. Although the results are promising, more advanced prediction techniques and larger labeled datasets will likely be required to achieve prediction performance appropriate for real-world clinical applications.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocz140","type":"journal-article","created":{"date-parts":[[2019,7,24]],"date-time":"2019-07-24T11:25:23Z","timestamp":1563967523000},"page":"1493-1504","source":"Crossref","is-referenced-by-count":30,"title":["Detecting conversation topics in primary care office visits from transcripts of patient-provider interactions"],"prefix":"10.1093","volume":"26","author":[{"given":"Jihyun","family":"Park","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, Irvine, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitrios","family":"Kotzias","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, Irvine, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patty","family":"Kuo","sequence":"additional","affiliation":[{"name":"Department of Educational Psychology, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert L","family":"Logan IV","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, Irvine, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kritzia","family":"Merced","sequence":"additional","affiliation":[{"name":"Department of Educational Psychology, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sameer","family":"Singh","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, Irvine, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Tanana","sequence":"additional","affiliation":[{"name":"Social Research Institute, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Efi","family":"Karra Taniskidou","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, Irvine, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer Elston","family":"Lafata","sequence":"additional","affiliation":[{"name":"Division of Pharmaceutical Outcomes and Policy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"},{"name":"Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit, Michigan, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David C","family":"Atkins","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Tai-Seale","sequence":"additional","affiliation":[{"name":"Department of Family Medicine and Public Health, University of California San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zac E","family":"Imel","sequence":"additional","affiliation":[{"name":"Department of Educational Psychology, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Padhraic","family":"Smyth","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, Irvine, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,9,17]]},"reference":[{"issue":"6","key":"2020110612454385800_ocz140-B1","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.amjmed.2012.11.020","article-title":"The write stuff: how good writing can enhance patient care and professional growth","volume":"126","author":"Simon","year":"2013","journal-title":"Am J Med"},{"issue":"2","key":"2020110612454385800_ocz140-B2","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1177\/0261927X08330612","article-title":"Communication in medical records: intergroup. Language and patient care","volume":"28","author":"Hewett","year":"2009","journal-title":". J Lang Soc Psychol"},{"key":"2020110612454385800_ocz140-B3","doi-asserted-by":"crossref","first-page":"S52","DOI":"10.1089\/jpm.2017.0542","article-title":"Using electronic health records for quality measurement and accountability in care of the seriously ill: opportunities and challenges","volume":"21","author":"Curtis","year":"2018","journal-title":"J Palliat Med"},{"issue":"16","key":"2020110612454385800_ocz140-B4","doi-asserted-by":"crossref","first-page":"2361","DOI":"10.1001\/jama.1988.03410160035005","article-title":"Measurement and analysis of intraservice work","volume":"260","author":"Hsiao","year":"1988","journal-title":"JAMA"},{"issue":"22","key":"2020110612454385800_ocz140-B5","doi-asserted-by":"crossref","first-page":"2327","DOI":"10.1001\/jama.2013.6030","article-title":"Enhancing patient-centered communication and collaboration by using the electronic health record in the examination room","volume":"309","author":"White","year":"2013","journal-title":"JAMA"},{"issue":"3","key":"2020110612454385800_ocz140-B6","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1377\/hlthaff.2018.05404","article-title":"Consumer-facing data, information, and tools: self-management of health in the digital age","volume":"38","author":"Singh","year":"2019","journal-title":"Health Aff (Millwood)"},{"issue":"4","key":"2020110612454385800_ocz140-B7","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1111\/j.1365-2753.2008.01065.x","article-title":"The impact of electronic medical records on patient-doctor communication during consultation: a narrative literature review","volume":"15","author":"Shachak","year":"2009","journal-title":"J Eval Clin Pract"},{"issue":"2","key":"2020110612454385800_ocz140-B8","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1370\/afm.425","article-title":"Physicians, patients, and the electronic health record: an ethnographic analysis","volume":"4","author":"Ventres","year":"2006","journal-title":"Ann Fam Med"},{"key":"2020110612454385800_ocz140-B9","author":"Sinsky"},{"issue":"4","key":"2020110612454385800_ocz140-B10","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1377\/hlthaff.2016.0811","article-title":"Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine","volume":"36","author":"Tai-Seale","year":"2017","journal-title":"Health Aff (Millwood)"},{"issue":"5","key":"2020110612454385800_ocz140-B11","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"Arndt","year":"2017","journal-title":"Ann Fam Med"},{"issue":"12","key":"2020110612454385800_ocz140-B12","doi-asserted-by":"crossref","first-page":"1600","DOI":"10.1016\/j.mayocp.2015.08.023","article-title":"Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014","volume":"90","author":"Shanafelt","year":"2015","journal-title":"Mayo Clin Proc"},{"key":"2020110612454385800_ocz140-B13","doi-asserted-by":"crossref","first-page":"1073\u201378.","DOI":"10.1377\/hlthaff.2018.05509","article-title":"Physicians\u2019 well-being linked to in-basket messages generated by algorithms in electronic health records","volume":"38:","author":"Tai-Seale","year":"2019","journal-title":"Health Aff (Millwood)"},{"key":"2020110612454385800_ocz140-B14","first-page":"1.","article-title":"Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy","volume":"3","author":"Friedberg","year":"2014","journal-title":"Rand Health Q"},{"key":"2020110612454385800_ocz140-B15","doi-asserted-by":"crossref","first-page":"S122","DOI":"10.1016\/j.ijmedinf.2006.06.004","article-title":"Copying and pasting of examinations within the electronic medical record","volume":"76 Suppl 1","author":"Thielke","year":"2007","journal-title":"Int J Med Inform"},{"key":"2020110612454385800_ocz140-B16","first-page":"269","article-title":"Are electronic medical records trustworthy? Observations on copying, pasting and duplication","volume":"2003","author":"Hammond","year":"2003","journal-title":"AMIA Annu Sympos Proc"},{"key":"2020110612454385800_ocz140-B17","doi-asserted-by":"crossref","first-page":"2972","DOI":"10.21437\/Interspeech.2018-40","article-title":"Speech recognition for medical conversations","volume":"2018","author":"Chiu","year":"2018","journal-title":"Proc Interspeech"},{"issue":"6","key":"2020110612454385800_ocz140-B18","doi-asserted-by":"crossref","first-page":"836\u20138","DOI":"10.1001\/jamainternmed.2018.8558","article-title":"Automatically charting symptoms from patient-physician conversations using machine learning","volume":"179","author":"Rajkomar","year":"2019","journal-title":"JAMA Intern Med"},{"issue":"8","key":"2020110612454385800_ocz140-B19","doi-asserted-by":"crossref","first-page":"e008566.","DOI":"10.1136\/bmjopen-2015-008566","article-title":"Patients recording clinical encounters: a path to empowerment? Assessment by mixed methods","volume":"5","author":"Elwyn","year":"2015","journal-title":"BMJ Open"},{"issue":"7","key":"2020110612454385800_ocz140-B20","doi-asserted-by":"crossref","first-page":"e121","DOI":"10.2196\/resprot.7735","article-title":"Sharing annotated audio recordings of clinic visits with patients-development of the open recording automated logging system (ORALS): study protocol","volume":"6","author":"Barr","year":"2017","journal-title":"JMIR Res Protoc"},{"issue":"11","key":"2020110612454385800_ocz140-B21","doi-asserted-by":"crossref","first-page":"1591","DOI":"10.1016\/j.ajem.2013.06.028","article-title":"4000 clicks: a productivity analysis of electronic medical records in a community hospital ED","volume":"31","author":"Hill","year":"2013","journal-title":"Am J Emerg Med"},{"issue":"1","key":"2020110612454385800_ocz140-B22","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1001\/jama.2017.19198","article-title":"What this computer needs is a physician: humanism and artificial intelligence","volume":"319","author":"Verghese","year":"2018","journal-title":"JAMA"},{"issue":"7","key":"2020110612454385800_ocz140-B23","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1097\/00005650-198807000-00002","article-title":"Meta-analysis of correlates of provider behavior in medical encounters","volume":"26","author":"Hall","year":"1988","journal-title":"Med Care"},{"key":"2020110612454385800_ocz140-B24","first-page":"25","article-title":"Physician-patient communication in the primary care office: a systematic review","volume":"15","author":"Beck","year":"2002","journal-title":"J Am Board Fam Pract"},{"key":"2020110612454385800_ocz140-B25","volume-title":"The Discourse of Medicine: Dialectics of Medical Interviews","author":"Mishler","year":"1984"},{"issue":"8","key":"2020110612454385800_ocz140-B26","doi-asserted-by":"crossref","first-page":"1482","DOI":"10.1016\/j.pec.2017.03.009","article-title":"Specifying the effects of physician\u2019s communication on patients' outcomes: a randomised controlled trial","volume":"100","author":"van Osch","year":"2017","journal-title":"Patient Educ Couns"},{"key":"2020110612454385800_ocz140-B27","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1007\/978-3-319-27625-0_8","volume-title":"Empathy in Health Professions Education and Patient Care","author":"Hojat","year":"2016"},{"issue":"5","key":"2020110612454385800_ocz140-B28","doi-asserted-by":"crossref","first-page":"1871","DOI":"10.1111\/j.1475-6773.2006.00689.x","article-title":"Time allocation in primary care office visits","volume":"42","author":"Tai-Seale","year":"2007","journal-title":"Health Serv Res"},{"issue":"3","key":"2020110612454385800_ocz140-B29","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1097\/JAC.0000000000000176","article-title":"Patient and physician race and the allocation of time and patient engagement efforts to mental health discussions in primary care","volume":"40","author":"Foo","year":"2017","journal-title":"J Ambul Care Manage"},{"issue":"12","key":"2020110612454385800_ocz140-B30","doi-asserted-by":"crossref","first-page":"1903","DOI":"10.1111\/j.1532-5415.2007.01467.x","article-title":"Two-minute mental health care for elderly patients: inside primary care visits","volume":"55","author":"Tai-Seale","year":"2007","journal-title":"J Am Geriatr Soc"},{"key":"2020110612454385800_ocz140-B31","doi-asserted-by":"crossref","first-page":"1635","DOI":"10.2147\/PPA.S145942","article-title":"Healthcare provider relational quality is associated with better self-management and less treatment burden in people with multiple chronic conditions","volume":"11","author":"Eton","year":"2017","journal-title":"Patient Prefer Adherence"},{"issue":"7","key":"2020110612454385800_ocz140-B32","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1016\/0277-9536(94)90207-0","article-title":"Multi-dimensional interaction analysis: a collaborative approach to the study of medical discourse","volume":"39","author":"Charon","year":"1994","journal-title":"Soc Sci Med"},{"key":"2020110612454385800_ocz140-B33","first-page":"e350","article-title":"Periodic health examinations and missed opportunities among patients likely needing mental health care","volume":"22","author":"Tai-Seale","year":"2016","journal-title":"Am J Manag Care"},{"issue":"1","key":"2020110612454385800_ocz140-B34","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.jsat.2004.11.001","article-title":"Assessing competence in the use of motivational interviewing","volume":"28","author":"Moyers","year":"2005","journal-title":"J Subst Abuse Treat"},{"issue":"4","key":"2020110612454385800_ocz140-B35","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1037\/adb0000359","article-title":"Rating motivational interviewing fidelity from thin slices","volume":"32","author":"Caperton","year":"2018","journal-title":"Psychol Addict Behav"},{"issue":"7","key":"2020110612454385800_ocz140-B36","doi-asserted-by":"crossref","first-page":"1310","DOI":"10.1377\/hlthaff.2009.0450","article-title":"Developing physician communication skills for patient-centered care","volume":"29","author":"Levinson","year":"2010","journal-title":"Health Aff (Millwood)"},{"issue":"1","key":"2020110612454385800_ocz140-B37","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1186\/1471-2407-13-188","article-title":"Values and options in cancer care (VOICE): study design and rationale for a patient-centered communication and decision-making intervention for physicians, patients with advanced cancer, and their caregivers","volume":"13","author":"Hoerger","year":"2013","journal-title":"BMC Cancer"},{"issue":"5","key":"2020110612454385800_ocz140-B38","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","article-title":"Natural language processing: an introduction","volume":"18","author":"Nadkarni","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020110612454385800_ocz140-B39","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1136\/amiajnl-2012-001012","article-title":"Large-scale evaluation of automated clinical note de-identification and its impact on information extraction","volume":"20","author":"Deleger","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110612454385800_ocz140-B40","doi-asserted-by":"crossref","first-page":"568","DOI":"10.1136\/amiajnl-2011-000152","article-title":"A flexible framework for deriving assertions from electronic medical records","volume":"18","author":"Roberts","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110612454385800_ocz140-B41","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1136\/jamia.2010.003970","article-title":"Extracting Rx information from clinical narrative","volume":"17","author":"Mork","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020110612454385800_ocz140-B42","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1093\/jamia\/ocw156","article-title":"De-identification of patient notes with recurrent neural networks","volume":"24","author":"Dernoncourt","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"e1","key":"2020110612454385800_ocz140-B43","doi-asserted-by":"crossref","first-page":"e122","DOI":"10.1136\/amiajnl-2013-001898","article-title":"Automating annotation of information-giving for analysis of clinical conversation","volume":"21","author":"Mayfield","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2020110612454385800_ocz140-B44","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1007\/s10461-009-9664-5","article-title":"Patient-provider communication differs for black compared with white HIV-infected patients","volume":"15","author":"Beach","year":"2011","journal-title":"AIDS Behav"},{"issue":"1","key":"2020110612454385800_ocz140-B45","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1007\/s10461-012-0143-z","article-title":"Provider-patient adherence dialogue in HIV care: results of a multisite study","volume":"17","author":"Laws","year":"2013","journal-title":"AIDS Behav"},{"key":"2020110612454385800_ocz140-B46","first-page":"785","article-title":"Interpretable probabilistic latent variable models for automatic annotation of clinical text","volume":"2015","author":"Kotov","year":"2015","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110612454385800_ocz140-B47","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.jbi.2016.05.004","article-title":"A study of the effectiveness of machine learning methods for classification of clinical interview fragments into a large number of categories","volume":"62","author":"Hasan","year":"2016","journal-title":"J Biomed Inform"},{"issue":"4","key":"2020110612454385800_ocz140-B48","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1177\/0272989X13514777","article-title":"Automatically annotating topics in transcripts of patient-provider interactions via machine learning","volume":"34","author":"Wallace","year":"2014","journal-title":"Med Decis Mak"},{"issue":"2","key":"2020110612454385800_ocz140-B49","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1109\/JBHI.2015.2503985","article-title":"Content coding of psychotherapy transcripts using labeled topic models","volume":"21","author":"Gaut","year":"2017","journal-title":"IEEE J Biomed Health Inform"},{"key":"2020110612454385800_ocz140-B50","volume-title":"Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit","author":"Bird","year":"2009"},{"key":"2020110612454385800_ocz140-B51","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: global vectors for word representation","author":"Pennington","year":"2014","journal-title":"Proceedings of Empirical Methods in Natural Language Processing (EMNLP)"},{"key":"2020110612454385800_ocz140-B52","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-02165-7","volume-title":"Neural Network Methods in Natural Language Processing (Synthesis Lectures on Human Language Technologies)","author":"Goldberg","year":"2017"},{"key":"2020110612454385800_ocz140-B53","first-page":"282","volume-title":"Proceedings of the Eighteenth International Conference on Machine Learning","author":"Lafferty","year":"2001"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1493\/34152735\/ocz140.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1493\/34152735\/ocz140.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T13:04:05Z","timestamp":1664024645000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/12\/1493\/5571446"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,17]]},"references-count":53,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,9,17]]},"published-print":{"date-parts":[[2019,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz140","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,12]]},"published":{"date-parts":[[2019,9,17]]}}}