{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T10:55:40Z","timestamp":1782298540607,"version":"3.54.5"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2025,9,22]],"date-time":"2025-09-22T00:00:00Z","timestamp":1758499200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Clinical and Translational Science","award":["UL1TR004419"],"award-info":[{"award-number":["UL1TR004419"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Text embeddings are promising for semantic tasks, such as retrieval augmented generation (RAG). However, their application in health care is underexplored due to a lack of benchmarking methods. We introduce a scalable benchmarking method to test embeddings for health-care semantic tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We evaluated 39 embedding models across 7 medical semantic similarity tasks using diverse datasets. These datasets comprised real-world patient data (from the Mount Sinai Health System and MIMIC IV), biomedical texts from PubMed, and synthetic data generated with Llama-3-70b. We first assessed semantic textual similarity (STS) by correlating the model-generated similarity scores with noise levels using Spearman rank correlation. We then reframed the same tasks as retrieval problems, evaluated by mean reciprocal rank and recall at k.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In total, evaluating 2000 text pairs per 7 tasks for STS and retrieval yielded 3.28 million model assessments. Larger models (&amp;gt;7b parameters), such as those based on Mistral-7b and Gemma-2-9b, consistently performed well, especially in long-context tasks. The NV-Embed-v1 model (7b parameters), although top in short tasks, underperformed in long tasks. For short tasks, smaller models such as b1ade-embed (335M parameters) performed on-par to the larger models. For long retrieval tasks, the larger models significantly outperformed the smaller ones.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The proposed benchmarking framework demonstrates scalability and flexibility, offering a structured approach to guide the selection of embedding models for a wide range of health-care tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>By matching the appropriate model with the task, the framework enables more effective deployment of embedding models, enhancing critical applications such as semantic search and retrieval-augmented generation (RAG).<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf149","type":"journal-article","created":{"date-parts":[[2025,8,28]],"date-time":"2025-08-28T11:45:11Z","timestamp":1756381511000},"page":"1877-1887","source":"Crossref","is-referenced-by-count":2,"title":["A scalable framework for benchmark embedding models in semantic health-care tasks"],"prefix":"10.1093","volume":"32","author":[{"given":"Shelly","family":"Soffer","sequence":"first","affiliation":[{"name":"Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center , Petah Tikva, 49100,","place":["Israel"]},{"name":"Gray Faculty of Medical and Health Sciences, Tel Aviv University , Tel Aviv, 6997801,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0438-0827","authenticated-orcid":false,"given":"Mahmud","family":"Omar","sequence":"additional","affiliation":[{"name":"Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]},{"name":"Department of Medicine, Icahn School of Medicine at Mount Sinai, The Hasso Plattner Institute for Digital Health at Mount Sinai , New York, NY 10019,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7789-3801","authenticated-orcid":false,"given":"Moran","family":"Gendler","sequence":"additional","affiliation":[{"name":"Azrieli Faculty of Medicine, Bar-Ilan University , Safed, 1311502,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin S","family":"Glicksberg","sequence":"additional","affiliation":[{"name":"Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]},{"name":"Department of Medicine, Icahn School of Medicine at Mount Sinai, The Hasso Plattner Institute for Digital Health at Mount Sinai , New York, NY 10019,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricia","family":"Kovatch","sequence":"additional","affiliation":[{"name":"Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6024-9110","authenticated-orcid":false,"given":"Orly","family":"Efros","sequence":"additional","affiliation":[{"name":"Gray Faculty of Medical and Health Sciences, Tel Aviv University , Tel Aviv, 6997801,","place":["Israel"]},{"name":"National Hemophilia Center and Thrombosis Institute, Sheba Medical Center , Ramat Gan, 5262100,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4946-6533","authenticated-orcid":false,"given":"Robert","family":"Freeman","sequence":"additional","affiliation":[{"name":"Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander W","family":"Charney","sequence":"additional","affiliation":[{"name":"Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Girish N","family":"Nadkarni","sequence":"additional","affiliation":[{"name":"Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]},{"name":"Department of Medicine, Icahn School of Medicine at Mount Sinai, The Hasso Plattner Institute for Digital Health at Mount Sinai , New York, NY 10019,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eyal","family":"Klang","sequence":"additional","affiliation":[{"name":"Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]},{"name":"Department of Medicine, Icahn School of Medicine at Mount Sinai, The Hasso Plattner Institute for Digital Health at Mount Sinai , New York, NY 10019,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,9,22]]},"reference":[{"key":"2025112514465865300_ocaf149-B1","first-page":"145","author":"Glicksberg","year":"2018"},{"key":"2025112514465865300_ocaf149-B2","doi-asserted-by":"crossref","first-page":"1921","DOI":"10.1093\/jamia\/ocae103","article-title":"Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room","volume":"31","author":"Glicksberg","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025112514465865300_ocaf149-B3","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025112514465865300_ocaf149-B4","author":"Muennighoff","year":"2022"},{"key":"2025112514465865300_ocaf149-B5","year":"2022"},{"key":"2025112514465865300_ocaf149-B6","first-page":"716","author":"Boteva","year":"2016"},{"key":"2025112514465865300_ocaf149-B7","doi-asserted-by":"crossref","first-page":"i49","DOI":"10.1093\/bioinformatics\/btx238","article-title":"BIOSSES: a semantic sentence similarity estimation system for the biomedical domain","volume":"33","author":"Sogancioglu","year":"2017","journal-title":"Bioinformatics"},{"key":"2025112514465865300_ocaf149-B8","year":"2024"},{"key":"2025112514465865300_ocaf149-B9","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1038\/s41597-023-02136-9","article-title":"MIMIC-IV, a freely accessible electronic health record dataset","volume":"10","author":"Johnson","year":"2023","journal-title":"Sci Data"},{"key":"2025112514465865300_ocaf149-B10","author":"Kirk","year":"2016"},{"key":"2025112514465865300_ocaf149-B11","doi-asserted-by":"crossref","first-page":"e2448723","DOI":"10.1001\/jamanetworkopen.2024.48723","article-title":"Developing and evaluating large language model\u2013generated emergency medicine handoff notes","volume":"7","author":"Hartman","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2025112514465865300_ocaf149-B12","author":"Thakur","year":"2021"},{"key":"2025112514465865300_ocaf149-B13","author":"Wadden","year":"2020"},{"key":"2025112514465865300_ocaf149-B14","author":"Solatorio","year":"2024"},{"key":"2025112514465865300_ocaf149-B15","year":"2024"},{"key":"2025112514465865300_ocaf149-B16","author":"Li","year":"2023"},{"key":"2025112514465865300_ocaf149-B17","author":"Su","year":"2022"},{"key":"2025112514465865300_ocaf149-B18","author":"Junseong Kim","year":"2024"},{"key":"2025112514465865300_ocaf149-B19","author":"Rui Meng","year":"2024"},{"key":"2025112514465865300_ocaf149-B20","author":"Ni","year":"2021"},{"key":"2025112514465865300_ocaf149-B21","first-page":"3","article-title":"Sfrembedding-mistral: enhance text retrieval with transfer learning","author":"Meng","year":"2024"},{"key":"2025112514465865300_ocaf149-B22","author":"Lee","year":"2024"},{"key":"2025112514465865300_ocaf149-B23","author":"Xiao","year":"2023"},{"key":"2025112514465865300_ocaf149-B24","year":"2023"},{"key":"2025112514465865300_ocaf149-B25","author":"Li","year":"2023"},{"key":"2025112514465865300_ocaf149-B26","year":"2024"},{"key":"2025112514465865300_ocaf149-B27","author":"Chen","year":"2024"},{"key":"2025112514465865300_ocaf149-B28","year":"2022"},{"key":"2025112514465865300_ocaf149-B29","author":"Wang","year":"2023"},{"key":"2025112514465865300_ocaf149-B30","author":"Wang","year":"2024"},{"key":"2025112514465865300_ocaf149-B31","author":"Muennighoff","year":"2022"},{"key":"2025112514465865300_ocaf149-B32","year":"2021"},{"key":"2025112514465865300_ocaf149-B33","author":"Zhu","year":"2024"},{"key":"2025112514465865300_ocaf149-B34","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btad651","article-title":"MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval","volume":"39","author":"Jin","year":"2023","journal-title":"Bioinformatics"},{"key":"2025112514465865300_ocaf149-B35","author":"DeepSeek-AI","year":"2024"},{"key":"2025112514465865300_ocaf149-B36","author":"Ni","year":"2021"},{"key":"2025112514465865300_ocaf149-B37","doi-asserted-by":"publisher","first-page":"1844","DOI":"10.1093\/jamia\/ocae029","article-title":"BioLORD-2023: semantic textual representations fusing large language models and clinical knowledge graph insights","volume":"31","author":"Remy","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025112514465865300_ocaf149-B38","year":"2021"},{"key":"2025112514465865300_ocaf149-B39","author":"Zhang","year":"2024"},{"key":"2025112514465865300_ocaf149-B40","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2025112514465865300_ocaf149-B41","author":"Xie","year":"2024"},{"key":"2025112514465865300_ocaf149-B42","author":"Alsentzer","year":"2019"},{"key":"2025112514465865300_ocaf149-B43","author":"Dadas","year":"2024"},{"key":"2025112514465865300_ocaf149-B44","author":"Gururangan","year":"2020"},{"key":"2025112514465865300_ocaf149-B45","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1093\/jamia\/ocad190","article-title":"Improving model transferability for clinical note section classification models using continued pretraining","volume":"31","author":"Zhou","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2025112514465865300_ocaf149-B46","author":"Mathieu Ciancone","year":"2024"},{"key":"2025112514465865300_ocaf149-B47","first-page":"1","author":"Chen","year":"2019"},{"key":"2025112514465865300_ocaf149-B48","first-page":"167","author":"Chang","year":"2020"},{"key":"2025112514465865300_ocaf149-B49","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1038\/s41597-019-0055-0","article-title":"BioWordVec, improving biomedical word embeddings with subword information and MeSH","volume":"6","author":"Zhang","year":"2019","journal-title":"Sci Data"},{"key":"2025112514465865300_ocaf149-B50","doi-asserted-by":"crossref","DOI":"10.1056\/AIcs2300235","article-title":"GPT-4 for information retrieval and comparison of medical oncology guidelines","volume":"1","author":"Ferber","year":"2024","journal-title":"NEJM AI"},{"key":"2025112514465865300_ocaf149-B51","doi-asserted-by":"crossref","DOI":"10.1056\/AIoa2400181","article-title":"Retrieval-augmented generation\u2013enabled GPT-4 for clinical trial screening","volume":"1","author":"Unlu","year":"2024","journal-title":"NEJM AI"},{"key":"2025112514465865300_ocaf149-B52","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-L","article-title":"High agreement but low kappa:\u202fI.\u202fThe problems of two paradoxes","volume":"43","author":"Feinstein","year":"1990","journal-title":"J Clin Epidemiol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/12\/1877\/64341318\/ocaf149.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/12\/1877\/64341318\/ocaf149.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T19:47:05Z","timestamp":1764100025000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/12\/1877\/8261126"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,22]]},"references-count":52,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,9,22]]},"published-print":{"date-parts":[[2025,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf149","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,12]]},"published":{"date-parts":[[2025,9,22]]}}}