{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T23:57:51Z","timestamp":1789516671986,"version":"build-2803163510"},"reference-count":21,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n The promise of artificial intelligence (AI) in medicine depends on its ability to learn from data that reflect what matters to patients and clinicians in the care process. Most existing models are trained on electronic health records (EHRs), which primarily capture biological measures but rarely the interactions and relationships between patients and clinicians. These relationships, central to how care is understood, negotiated, and delivered, unfold across multiple modalities, including voice, text, and video, yet remain largely absent from current datasets. As a result, AI systems trained solely on EHRs risk perpetuating a narrow biomedical view of medicine and overlooking the lived exchanges that define clinical encounters.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to design, implement, and evaluate the feasibility of a longitudinal, multimodal system for capturing patient-clinician encounters, linking 360\u00b0 video or audio recordings with postvisit surveys and EHR data, to create a foundational dataset for downstream AI research.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This single-site study was conducted in an academic outpatient specialty clinic (Division of Endocrinology, Mayo Clinic, Rochester, Minnesota, United States). Adult patients attending in-person visits with participating clinicians were invited to enroll. Encounters were recorded using a 360\u00b0 2D monocular video camera and dual-channel audio. After each visit, patients completed a brief survey assessing relational empathy, satisfaction, visit pace, and treatment burden. Demographic and clinical data were extracted from the EHR. Feasibility was assessed using 5 prespecified end points: clinician consent, patient consent, recording success, survey completion, and data linkage across modalities.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Recruitment began in January 2025. By August 2025, 35 of 36 (97%) eligible clinicians and 212 of 281 (75%) approached eligible patients had consented. Of the consented encounters, 162 (76%) resulted in a complete 360\u00b0 video recording, and the postvisit surveys were completed for 204 of 212 (96%) consented encounters, reflecting 1 survey per encounter. Data collection is ongoing as of December 2025, and further analyses will be reported in subsequent publications.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This protocol describes a longitudinal multimodal encounter capture system that links 360\u00b0 audio or video with postvisit surveys and EHR data. The study specifies operational definitions, workflows, feasibility end points, and governance procedures to support implementation and replication in other clinical settings.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/84688","type":"journal-article","created":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T18:13:43Z","timestamp":1774376023000},"page":"e84688-e84688","source":"Crossref","is-referenced-by-count":2,"title":["Longitudinal and Multimodal Recording System to Capture Real-World Patient-Clinician Conversations for AI and Encounter Research: Protocol for an Observational Study"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-1999-9351","authenticated-orcid":false,"given":"Misk","family":"Al Zahidy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1881-3571","authenticated-orcid":false,"given":"Kerly","family":"Guevara Maldonado","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5153-0904","authenticated-orcid":false,"given":"Luis","family":"Vilatuna Andrango","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2373-7116","authenticated-orcid":false,"given":"Ana Cristina","family":"Proano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3632-523X","authenticated-orcid":false,"given":"Ana Gabriela","family":"Claros","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2111-4440","authenticated-orcid":false,"given":"Maria","family":"Lizarazo Jimenez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2349-850X","authenticated-orcid":false,"given":"Esteban","family":"Gomez-Alvarez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1910-4144","authenticated-orcid":false,"given":"David","family":"Toro-Tobon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0595-2898","authenticated-orcid":false,"given":"Victor M","family":"Montori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5729-8408","authenticated-orcid":false,"given":"Oscar J","family":"Ponce-Ponte","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1840-1745","authenticated-orcid":false,"given":"Juan P","family":"Brito","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,3,24]]},"reference":[{"issue":"12","key":"R1","doi-asserted-by":"publisher","first-page":"1743","DOI":"10.1001\/jama.260.12.1743","article-title":"The quality of care. How can it be assessed?","volume":"260","author":"Donabedian","journal-title":"JAMA"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b2803","article-title":"We need minimally disruptive medicine","volume":"339","author":"May","journal-title":"BMJ"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"50","DOI":"10.7326\/M18-0139","article-title":"Physician burnout in the electronic health record era: are we ignoring the real cause?","volume":"169","author":"Downing","journal-title":"Ann Intern Med"},{"issue":"7","key":"R4","doi-asserted-by":"publisher","first-page":"836","DOI":"10.1016\/j.mayocp.2016.05.007","article-title":"Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction","volume":"91","author":"Shanafelt","journal-title":"Mayo Clin Proc"},{"issue":"831-5","key":"R5","doi-asserted-by":"publisher","first-page":"831","DOI":"10.2147\/PPA.S81975","article-title":"Measuring the quality of patient-centered care: why patient-reported measures are critical to reliable assessment","volume":"9","author":"Tzelepis","journal-title":"Patient Prefer Adherence"},{"issue":"1072-80","key":"R6","first-page":"1072","volume":"2022","author":"Tran","journal-title":"AMIA Annu Symp Proc"},{"issue":"13","key":"R7","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.adf3197","article-title":"The CANDOR corpus: insights from a large multimodal dataset of naturalistic conversation","volume":"9","author":"Reece","journal-title":"Sci Adv"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-025-04579-w","article-title":"Validation of a novel patient-reported measure of the burden of digital care in diabetes","volume":"24","author":"Al Zahidy","journal-title":"BMC Med"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/1741-7015-12-109","article-title":"Adaptation and validation of the Treatment Burden Questionnaire (TBQ) in English using an internet platform","volume":"12","author":"Tran","journal-title":"BMC Med"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s12875-015-0374-y","article-title":"Measuring empathic, person-centred communication in primary CARE nurses: validity and reliability of the Consultation and Relational Empathy (CARE) measure","volume":"16","author":"Bikker","journal-title":"BMC Fam Pract"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1016\/j.pec.2024.108237","article-title":"Developing and validating the Unhurried Conversations Assessment Tool (UCAT)","volume":"123","author":"Mandhana","journal-title":"Patient Educ Couns"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","first-page":"3099","DOI":"10.1111\/hex.12297","article-title":"The net promoter score\u2014an asset to patient experience surveys?","volume":"18","author":"Krol","journal-title":"Health Expect"},{"key":"R13","unstructured":"Srivastav V Majumdar S Koluguri N et al. Open automatic speech recognition leaderboard. Hugging Face Spaces. 2023. URL: https:\/\/huggingface.co\/spaces\/hf-audio\/open_asr_leaderboard [Accessed 09-03-2026]"},{"key":"R14","unstructured":"Kuchaiev O Li J Nguyen H et al. NeMo: a toolkit for building AI applications using neural modules. arXiv. Preprint posted online on Sep 14, 2019. [doi: 10.48550\/arXiv.1909.09577]"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.wocn.2018.07.001","article-title":"Introducing Parselmouth: a Python interface to Praat","volume":"71","author":"Jadoul","journal-title":"J Phon"},{"key":"R16","unstructured":"OpenCV \u2013 Open Computer Vision Library. 2025. URL: https:\/\/opencv.org\/ [Accessed 02-03-2026]"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"Kazemi V Sullivan J . One millisecond face alignment with an ensemble of regression trees. Presented at: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Jun 23-28, 2014; Columbus, OH. [doi: 10.1109\/CVPR.2014.241]","DOI":"10.1109\/CVPR.2014.241"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Baltrusaitis T Zadeh A Lim YC Morency LP . OpenFace 2.0: facial behavior analysis toolkit. Presented at: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018); May 15-19, 2018; Xi\u2019an, China. [doi: 10.1109\/FG.2018.00019]","DOI":"10.1109\/FG.2018.00019"},{"key":"R19","unstructured":"Huang D Li Q Yan C et al. Emotion-Qwen: a unified framework for emotion and vision understanding. arXiv. Preprint posted online on May 10, 2025. [doi: 10.48550\/arXiv.2505.06685]"},{"issue":"3","key":"R20","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2022-065477","article-title":"Effect of multimodal comprehensive communication skills training with video analysis by artificial intelligence for physicians on acute geriatric care: a mixed-methods study","volume":"13","author":"Kobayashi","journal-title":"BMJ Open"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1016\/j.jclinepi.2013.08.015","article-title":"Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects","volume":"67","author":"McCambridge","journal-title":"J Clin Epidemiol"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e84688","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:18:48Z","timestamp":1782843528000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e84688"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,24]]},"references-count":21,"URL":"https:\/\/doi.org\/10.2196\/84688","relation":{},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,24]]},"article-number":"v15i12e84688"}}