{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,11]],"date-time":"2026-11-11T03:21:15Z","timestamp":1794367275488,"version":"4.9.0"},"reference-count":33,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"12","license":[{"start":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T00:00:00Z","timestamp":1726531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["www.ahajournals.org"],"crossmark-restriction":true},"short-container-title":["Circulation"],"published-print":{"date-parts":[[2024,9,17]]},"abstract":"<jats:sec>\n            <jats:title>BACKGROUND:<\/jats:title>\n            <jats:p>Artificial intelligence, particularly deep learning (DL), has immense potential to improve the interpretation of transthoracic echocardiography (TTE). Mitral regurgitation (MR) is the most common valvular heart disease and presents unique challenges for DL, including the integration of multiple video-level assessments into a final study-level classification.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>METHODS:<\/jats:title>\n            <jats:p>A novel DL system was developed to intake complete TTEs, identify color MR Doppler videos, and determine MR severity on a 4-step ordinal scale (none\/trace, mild, moderate, and severe) using the reading cardiologist as a reference standard. This DL system was tested in internal and external test sets with performance assessed by agreement with the reading cardiologist, weighted \u03ba, and area under the receiver-operating characteristic curve for binary classification of both moderate or greater and severe MR. In addition to the primary 4-step model, a 6-step MR assessment model was studied with the addition of the intermediate MR classes of mild-moderate and moderate-severe with performance assessed by both exact agreement and \u00b11 step agreement with the clinical MR interpretation.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>RESULTS:<\/jats:title>\n            <jats:p>A total of 61\u2009689 TTEs were split into train (n=43\u2009811), validation (n=8891), and internal test (n=8987) sets with an additional external test set of 8208 TTEs. The model had high performance in MR classification in internal (exact accuracy, 82%; \u03ba=0.84; area under the receiver-operating characteristic curve, 0.98 for moderate or greater MR) and external test sets (exact accuracy, 79%; \u03ba=0.80; area under the receiver-operating characteristic curve, 0.98 for moderate or greater MR). Most (63% internal and 66% external) misclassification disagreements were between none\/trace and mild MR. MR classification accuracy was slightly higher using multiple TTE views (accuracy, 82%) than with only apical 4-chamber views (accuracy, 80%). In subset analyses, the model was accurate in the classification of both primary and secondary MR with slightly lower performance in cases of eccentric MR. In the analysis of the 6-step classification system, the exact accuracy was 80% and 76% with a \u00b11 step agreement of 99% and 98% in the internal and external test set, respectively.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>CONCLUSIONS:<\/jats:title>\n            <jats:p>This end-to-end DL system can intake entire echocardiogram studies to accurately classify MR severity and may be useful in helping clinicians refine MR assessments.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1161\/circulationaha.124.068996","type":"journal-article","created":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T09:01:40Z","timestamp":1718614900000},"page":"911-922","update-policy":"https:\/\/doi.org\/10.1161\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":71,"title":["Deep Learning for Echo Analysis, Tracking, and Evaluation of Mitral Regurgitation (DELINEATE-MR)"],"prefix":"10.1161","volume":"150","author":[{"given":"Aaron","family":"Long","sequence":"first","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."},{"name":"Departments of Biomedical Informatics (A.L., C.M.H., P.E.), Columbia University, New York, NY."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3227-4490","authenticated-orcid":true,"given":"Christopher M.","family":"Haggerty","sequence":"additional","affiliation":[{"name":"Departments of Biomedical Informatics (A.L., C.M.H., P.E.), Columbia University, New York, NY."},{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Finer","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dustin","family":"Hartzel","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3476-2882","authenticated-orcid":true,"given":"Linyuan","family":"Jing","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7197-2788","authenticated-orcid":true,"given":"Azadeh","family":"Keivani","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Kelsey","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Rocha","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8741-979X","authenticated-orcid":true,"given":"Jeffrey","family":"Ruhl","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"vanMaanen","sequence":"additional","affiliation":[{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7816-7722","authenticated-orcid":true,"given":"Gil","family":"Metser","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2354-6394","authenticated-orcid":true,"given":"Eamon","family":"Duffy","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Mawson","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5400-5008","authenticated-orcid":true,"given":"Mathew","family":"Maurer","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2583-9278","authenticated-orcid":true,"given":"Andrew J.","family":"Einstein","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."},{"name":"Radiology (A.J.E.), Columbia University, New York, NY."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1504-1169","authenticated-orcid":true,"given":"Ashley","family":"Beecy","sequence":"additional","affiliation":[{"name":"Division of Cardiology, Department of Medicine, Weill Cornell Medicine, New York, NY (A.B., D.K.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deepa","family":"Kumaraiah","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."},{"name":"Division of Cardiology, Department of Medicine, Weill Cornell Medicine, New York, NY (A.B., D.K.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shunichi","family":"Homma","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qi","family":"Liu","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5075-989X","authenticated-orcid":true,"given":"Vratika","family":"Agarwal","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark","family":"Lebehn","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Leon","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."},{"name":"Cardiovascular Research Foundation, New York, NY (M. Leon)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1342-9191","authenticated-orcid":true,"given":"Rebecca","family":"Hahn","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9643-3024","authenticated-orcid":true,"given":"Pierre","family":"Elias","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."},{"name":"Departments of Biomedical Informatics (A.L., C.M.H., P.E.), Columbia University, New York, NY."},{"name":"Information Technology Data Science, New York\u2013Presbyterian Hospital, NY (C.M.H., J.F., D.H., L.J., A.K., C.K., D.R., J.R., D.v.M., P.E.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7284-3937","authenticated-orcid":true,"given":"Timothy J.","family":"Poterucha","sequence":"additional","affiliation":[{"name":"Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center\/New York\u2013Presbyterian Hospital, NY (A.L., G.M., E.D., T.M., M.M., A.J.E., D.K., S.H., Q.L., V.A., M. Lebehn, M. Leon, R.H., P.E., T.J.P.)."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(06)69208-8"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41569-021-00570-z"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehw229"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.mayocp.2020.08.039"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(18)30473-2"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1161\/CIR.0000000000000923"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.echo.2017.01.007"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcmg.2009.09.029"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.112.121590"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacc.2018.12.054"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.118.034338"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2145-8"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcmg.2021.04.018"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41551-020-00667-9"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamacardio.2021.6059"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacc.2023.09.800"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-05947-3"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehad456"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.echo.2023.03.008"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcmg.2021.06.018"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10044-023-01181-9"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30570-5_27"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocd.2012.03.005"},{"key":"e_1_3_3_27_2","first-page":"25","article-title":"Analysis of variability and reproducibility of echocardiography measurements in valvular aortic valve stenosis.","volume":"30","author":"Moura LM","year":"2011","unstructured":"Moura LM, Ramos SF, Pinto FJ, Barros IM, Rocha-Goncalves F. Analysis of variability and reproducibility of echocardiography measurements in valvular aortic valve stenosis. Rev Port Cardiol. 2011;30:25\u201333.","journal-title":"Rev Port Cardiol"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.acvd.2020.02.004"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.euje.2007.03.025"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcmg.2020.06.049"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcmg.2018.01.009"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0894-7317(03)00335-3"},{"key":"e_1_3_3_33_2","article-title":"End to end learning for self-driving cars.","author":"Bojarski M","year":"2016","unstructured":"Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J, et al. End to end learning for self-driving cars. arXiv. Preprint posted online April 25, 2016. arXiv:1604.07316.","journal-title":"arXiv"},{"key":"e_1_3_3_34_2","article-title":"End-to-end speech recognition: a survey.","author":"Prabhavalkar R","unstructured":"Prabhavalkar R, Hori T, Sainath TN, Schl\u00fcter R, Watanabe S. End-to-end speech recognition: a survey. arXiv. Preprint posted online March 3, 2023. arXiv:2303.03329","journal-title":"arXiv"}],"container-title":["Circulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ahajournals.org\/doi\/full\/10.1161\/CIRCULATIONAHA.124.068996","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,29]],"date-time":"2025-05-29T14:24:28Z","timestamp":1748528668000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ahajournals.org\/doi\/10.1161\/CIRCULATIONAHA.124.068996"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,17]]},"references-count":33,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,9,17]]}},"alternative-id":["10.1161\/CIRCULATIONAHA.124.068996"],"URL":"https:\/\/doi.org\/10.1161\/circulationaha.124.068996","relation":{},"ISSN":["0009-7322","1524-4539"],"issn-type":[{"value":"0009-7322","type":"print"},{"value":"1524-4539","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,17]]},"assertion":[{"value":"2024-02-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-24","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}