{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"Research Square"}],"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T13:55:04Z","timestamp":1781790904355,"version":"3.54.5"},"posted":{"date-parts":[[2026,6,18]]},"group-title":"In Review","reference-count":0,"publisher":"Springer Science and Business Media LLC","license":[{"start":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T00:00:00Z","timestamp":1781740800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,5,16]]},"abstract":"<title>Abstract<\/title>\n                <p>Large Language Models (LLMs) demonstrate strong capabilities in healthcare applications. However, recruiting human evaluators to assess usability and effectiveness becomes challenging in the large-scale LLM-generated outcomes. Existing metrics are often outdated for evaluating LLM-based healthcare technologies. This paper proposes a Research through Evaluation (RtE) approach that can refine metrics and improve assessments of LLM-generated outputs. We deploy RtE in a retrospective comparative study of prostate cancer patient inquiries, comparing responses from human clinical care teams with those generated by a GPT-4-based in-basket bot. Through three rounds of co-evaluations with clinical professionals and an LLM-as-graders study with four LLMs, the RtE method shows that starting with small, iterative evaluations helps human graders adjust metrics to suit their daily practices better. Additionally, in the LLM-as-graders study, LLMs exhibit similar behavior, preferring the in-basket bot's responses over the clinical care team. Our methodology and findings demonstrate that the RtE method can enhance the transparency and robustness of evaluation metrics for LLM-generated content, with the potential to generalize to evaluations in other clinical practices.<\/p>","DOI":"10.21203\/rs.3.rs-9735115\/v1","type":"posted-content","created":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T13:42:47Z","timestamp":1781790167000},"source":"Crossref","is-referenced-by-count":0,"title":["Research through Evaluation for Large Language Model in Patient-Clinician Communications"],"prefix":"10.21203","author":[{"given":"Yuexing","family":"Hao","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Holmes","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jared","family":"Hobson","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandra","family":"Bennett","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth L.","family":"McKone","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel K.","family":"Ebner","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David M.","family":"Routman","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Satomi","family":"Shiraishi","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samir H.","family":"Patel","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan Y.","family":"Yu","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris L.","family":"Hallemeier","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brooke E.","family":"Ball","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saleh","family":"Kalantari","sequence":"additional","affiliation":[{"name":"Cornell University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marzyeh","family":"Ghassemi","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark","family":"Waddle","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Liu","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","container-title":[],"original-title":[],"link":[{"URL":"https:\/\/www.researchsquare.com\/article\/rs-9735115\/v1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.researchsquare.com\/article\/rs-9735115\/v1.html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T13:44:23Z","timestamp":1781790263000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchsquare.com\/article\/rs-9735115\/v1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,18]]},"references-count":0,"URL":"https:\/\/doi.org\/10.21203\/rs.3.rs-9735115\/v1","relation":{},"subject":[],"published":{"date-parts":[[2026,6,18]]},"subtype":"preprint"}}