{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T00:14:28Z","timestamp":1775002468051,"version":"3.50.1"},"reference-count":16,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2022,6,25]],"date-time":"2022-06-25T00:00:00Z","timestamp":1656115200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100006545","name":"The National Institute on Minority Health and Health Disparities","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006545","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R21MD012693"],"award-info":[{"award-number":["R21MD012693"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,8,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Electronic consultation (eConsult) content reflects important information about referring clinician needs across an organization, but is challenging to extract. The objective of this work was to develop machine learning models for classifying eConsult questions for question type and question content. Another objective of this work was to investigate the ability to solve this task with constrained expert time resources.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Our data source is the San Francisco Health Network eConsult system, with over 700\u00a0000 deidentified questions from the years 2008\u20132017, from gastroenterology, urology, and neurology specialties. We develop classifiers based on Bidirectional Encoder Representations from Transformers, experimenting with multitask learning to learn when information can be shared across classifiers. We produce learning curves to understand when we may be able to reduce the amount of human labeling required.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Multitask learning shows benefits only in the neurology\u2013urology pair where they shared substantial similarities in the distribution of question types. Continued pretraining of models in new domains is highly effective. In the neurology\u2013urology pair, near-peak performance is achieved with only 10% of the urology training data given all of the neurology data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Sharing information across classifier types shows little benefit, whereas sharing classifier components across specialties can help if they are similar in the balance of procedural versus cognitive patient care.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>We can accurately classify eConsult content with enough labeled data, but only in special cases do methods for reducing labeling effort apply. Future work should explore new learning paradigms to further reduce labeling effort.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocac092","type":"journal-article","created":{"date-parts":[[2022,6,25]],"date-time":"2022-06-25T12:10:42Z","timestamp":1656159042000},"page":"1607-1617","source":"Crossref","is-referenced-by-count":5,"title":["Classifying unstructured electronic consult messages to understand primary care physician specialty information needs"],"prefix":"10.1093","volume":"29","author":[{"given":"Xiyu","family":"Ding","sequence":"first","affiliation":[{"name":"Biomedical Informatics & Data Science Section, Division of General Internal Medicine, The Johns Hopkins University School of Medicine , Baltimore, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Barnett","sequence":"additional","affiliation":[{"name":"Department of Health Policy and Management, Harvard T.H. Chan School of Public Health , Boston, Massachusetts, USA"},{"name":"Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women\u2019s Hospital , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2223-1582","authenticated-orcid":false,"given":"Ateev","family":"Mehrotra","sequence":"additional","affiliation":[{"name":"Department of Health Care Policy, Harvard Medical School , Boston, Massachusetts, USA"},{"name":"Division of General Medicine, Department of Medicine, Beth Israel Deaconess Medical Center , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1484-8945","authenticated-orcid":false,"given":"Delphine S","family":"Tuot","sequence":"additional","affiliation":[{"name":"Division of Nephrology, Department of Medicine, University of California, San Francisco , San Francisco, California, USA"},{"name":"UCSF Center for Innovation in Access and Quality at Zuckerberg San Francisco General Hospital , San Francisco, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Danielle S","family":"Bitterman","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Brigham and Women\u2019s Hospital\/Dana-Farber Cancer Institute , Boston, Massachusetts, USA"},{"name":"Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4513-403X","authenticated-orcid":false,"given":"Timothy A","family":"Miller","sequence":"additional","affiliation":[{"name":"Computational Health Informatics Program, Boston Children\u2019s Hospital , Boston, Massachusetts, USA"},{"name":"Department of Pediatrics, Harvard Medical School , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2022,6,25]]},"reference":[{"issue":"26","key":"2022081707574302600_ocac092-B1","doi-asserted-by":"crossref","first-page":"2450","DOI":"10.1056\/NEJMp1215594","article-title":"eReferral\u2014a new model for integrated care","volume":"368","author":"Chen","year":"2013","journal-title":"N Engl J Med"},{"issue":"5","key":"2022081707574302600_ocac092-B2","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1377\/hlthaff.2010.0027","article-title":"Practice profile. A safety-net system gains efficiencies through \u201ceReferrals\u201d to specialists","volume":"29","author":"Chen","year":"2010","journal-title":"Health Aff (Millwood)"},{"issue":"3","key":"2022081707574302600_ocac092-B3","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1377\/hlthaff.2016.1283","article-title":"Los Angeles Safety-Net Program eConsult System was rapidly adopted and decreased wait times to see specialists","volume":"36","author":"Barnett","year":"2017","journal-title":"Health Aff (Millwood)"},{"issue":"5","key":"2022081707574302600_ocac092-B4","doi-asserted-by":"crossref","first-page":"e210456","DOI":"10.1001\/jamahealthforum.2021.0456","article-title":"Association of eConsult implementation with access to specialist care in a large urban safety-net system","volume":"2","author":"Gaye","year":"2021","journal-title":"JAMA Health Forum"},{"issue":"1","key":"2022081707574302600_ocac092-B5","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1186\/s12859-018-2045-1","article-title":"Semantic annotation of consumer health questions","volume":"19","author":"Kilicoglu","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2022081707574302600_ocac092-B6","first-page":"914","article-title":"Combining open-domain and biomedical knowledge for topic recognition in consumer health questions","volume":"2016","author":"Mrabet","year":"2017","journal-title":"AMIA Annu Symp Proc"},{"key":"2022081707574302600_ocac092-B7","first-page":"1018","article-title":"Automatically classifying question types for consumer health questions","volume":"2014","author":"Roberts","year":"2014","journal-title":"AMIA Annu Symp Proc"},{"key":"2022081707574302600_ocac092-B8","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2022081707574302600_ocac092-B9","first-page":"74","author":"Ben Abacha","year":"2021"},{"key":"2022081707574302600_ocac092-B10","author":"Ruder","year":"2017"},{"key":"2022081707574302600_ocac092-B11"},{"key":"2022081707574302600_ocac092-B12","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"issue":"1","key":"2022081707574302600_ocac092-B13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM Trans Comput Healthcare"},{"issue":"1","key":"2022081707574302600_ocac092-B14","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ Psychol Measure"},{"key":"2022081707574302600_ocac092-B15","first-page":"8342","author":"Gururangan","year":"2020"},{"key":"2022081707574302600_ocac092-B16","author":"Han","year":"2019"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/9\/1607\/45444413\/ocac092.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/9\/1607\/45444413\/ocac092.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T07:58:02Z","timestamp":1660723082000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/9\/1607\/6617820"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,25]]},"references-count":16,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,6,25]]},"published-print":{"date-parts":[[2022,8,16]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac092","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,9,1]]},"published":{"date-parts":[[2022,6,25]]}}}