{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T15:12:01Z","timestamp":1783350721300,"version":"3.54.6"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T00:00:00Z","timestamp":1672185600000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Tsinghua-Qingdao Institute of Data Science"},{"DOI":"10.13039\/100020721","name":"Guoqiang Institute of Tsinghua University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100020721","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61906105"],"award-info":[{"award-number":["61906105"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61872218"],"award-info":[{"award-number":["61872218"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["2019YFB1404804"],"award-info":[{"award-number":["2019YFB1404804"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["2021YFF1201303"],"award-info":[{"award-number":["2021YFF1201303"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,1,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Recent developments of deep learning methods have demonstrated their feasibility in liver malignancy diagnosis using ultrasound (US) images. However, most of these methods require manual selection and annotation of US images by radiologists, which limit their practical application. On the other hand, US videos provide more comprehensive morphological information about liver masses and their relationships with surrounding structures than US images, potentially leading to a more accurate diagnosis. Here, we developed a fully automated artificial intelligence (AI) pipeline to imitate the workflow of radiologists for detecting liver masses and diagnosing liver malignancy. In this pipeline, we designed an automated mass-guided strategy that used segmentation information to direct diagnostic models to focus on liver masses, thus increasing diagnostic accuracy. The diagnostic models based on US videos utilized bi-directional convolutional long short-term memory modules with an attention-boosted module to learn and fuse spatiotemporal information from consecutive video frames. Using a large-scale dataset of 50\u00a0063 US images and video frames from 11\u00a0468 patients, we developed and tested the AI pipeline and investigated its applications. A dataset of annotated US images is available at https:\/\/doi.org\/10.5281\/zenodo.7272660.<\/jats:p>","DOI":"10.1093\/bib\/bbac569","type":"journal-article","created":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T04:41:50Z","timestamp":1672202510000},"source":"Crossref","is-referenced-by-count":34,"title":["Improving artificial intelligence pipeline for liver malignancy diagnosis using ultrasound images and video frames"],"prefix":"10.1093","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7871-8483","authenticated-orcid":false,"given":"Yiming","family":"Xu","sequence":"first","affiliation":[{"name":"Tsinghua University Department of Computer Science and Technology & Institute of Artificial Intelligence & BNRist, , Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bowen","family":"Zheng","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaohong","family":"Liu","sequence":"additional","affiliation":[{"name":"Tsinghua University Department of Computer Science and Technology & Institute of Artificial Intelligence & BNRist, , Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Wu","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinxiu","family":"Ju","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shijie","family":"Wang","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yufan","family":"Lian","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongjun","family":"Zhang","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tong","family":"Liang","sequence":"additional","affiliation":[{"name":"Foshan Traditional Chinese Medicine Hospital , Foshan, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ye","family":"Sang","sequence":"additional","affiliation":[{"name":"China Three Gorges University & Yichang Central People's Hospital The First College of Clinical Medical Science, , Yichang 443003 , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Jiang","sequence":"additional","affiliation":[{"name":"Tsinghua University Department of Automation & BNRist, , Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guangyu","family":"Wang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications School of Information and Communication Engineering, , Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Ren","sequence":"additional","affiliation":[{"name":"The Third Affiliated Hospital of Sun Yat-Sen University , Guangzhou, Guangdong , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting","family":"Chen","sequence":"additional","affiliation":[{"name":"Tsinghua University Department of Computer Science and Technology & Institute of Artificial Intelligence & BNRist, , Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,12,27]]},"reference":[{"issue":"6","key":"2023080106461965200_ref1","doi-asserted-by":"crossref","first-page":"394","DOI":"10.3322\/caac.21492","article-title":"Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"68","author":"Bray","year":"2018","journal-title":"CA Cancer J Clin"},{"issue":"12","key":"2023080106461965200_ref2","doi-asserted-by":"crossref","first-page":"1683","DOI":"10.1001\/jamaoncol.2017.3055","article-title":"The burden of primary liver cancer and underlying etiologies from 1990 to 2015 at the global, regional, and national level: results from the global burden of disease study 2015","volume":"3","author":"Akinyemiju","year":"2017","journal-title":"JAMA Oncol"},{"issue":"1","key":"2023080106461965200_ref3","doi-asserted-by":"crossref","first-page":"21","DOI":"10.3109\/03091908109042430","article-title":"Principles of real-time two-dimensional B-scan ultrasonic imaging","volume":"5","author":"Wilkinson","year":"1981","journal-title":"J Med Eng Technol"},{"issue":"4","key":"2023080106461965200_ref4","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1097\/00013644-200412000-00002","article-title":"Ultrasound of focal liver masses","volume":"20","author":"Tchelepi","year":"2004","journal-title":"Ultrasound Q"},{"issue":"6","key":"2023080106461965200_ref5","doi-asserted-by":"crossref","first-page":"1076","DOI":"10.1016\/S0168-8278(03)00349-0","article-title":"Screening for hepatocellular carcinoma in cirrhosis","volume":"39","author":"Bolondi","year":"2003","journal-title":"J Hepatol"},{"issue":"9","key":"2023080106461965200_ref6","doi-asserted-by":"crossref","first-page":"1171","DOI":"10.1002\/lt.25202","article-title":"Predictors of ultrasound failure to detect hepatocellular carcinoma","volume":"24","author":"Samoylova","year":"2018","journal-title":"Liver Transpl"},{"issue":"11","key":"2023080106461965200_ref7","doi-asserted-by":"crossref","first-page":"3085","DOI":"10.3390\/s20113085","article-title":"Comparison of deep-learning and conventional machine-learning methods for the automatic recognition of the hepatocellular carcinoma areas from ultrasound images","volume":"20","author":"Brehar","year":"2020","journal-title":"Sensors"},{"issue":"3","key":"2023080106461965200_ref8","doi-asserted-by":"crossref","first-page":"887","DOI":"10.1148\/radiol.2017170706","article-title":"Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study","volume":"286","author":"Yasaka","year":"2018","journal-title":"Radiology"},{"issue":"10","key":"2023080106461965200_ref9","doi-asserted-by":"crossref","first-page":"2875","DOI":"10.1111\/jgh.15522","article-title":"Artificial intelligence assists identifying malignant versus benign liver lesions using contrast-enhanced ultrasound","volume":"36","author":"Hu","year":"2021","journal-title":"J Gastroenterol Hepatol"},{"issue":"5","key":"2023080106461965200_ref36","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1016\/j.gie.2020.08.024","article-title":"Application of artificial intelligence using a novel EUS-based convolutional neural network model to identify and distinguish benign and malignant hepatic masses","volume":"93","author":"Marya","year":"2021","journal-title":"Gastrointest Endosc"},{"issue":"4","key":"2023080106461965200_ref10","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1007\/s12072-019-09937-4","article-title":"Current status and perspectives for computer-aided ultrasonic diagnosis of liver lesions using deep learning technology","volume":"13","author":"Nishida","year":"2019","journal-title":"Hepatol Int"},{"issue":"2","key":"2023080106461965200_ref11","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1007\/s00261-020-02564-w","article-title":"Deep learning for differentiation of benign and malignant solid liver lesions on ultrasonography","volume":"46","author":"Xi","year":"2021","journal-title":"Abdominal Radiol"},{"issue":"8","key":"2023080106461965200_ref12","doi-asserted-by":"crossref","first-page":"3127","DOI":"10.1007\/s13369-016-2387-9","article-title":"Diagnosis of focal liver diseases based on deep learning technique for ultrasound images","volume":"42","author":"Hassan","year":"2017","journal-title":"Arabian J Sci Eng"},{"issue":"4","key":"2023080106461965200_ref13","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1016\/j.ultrasmedbio.2019.12.018","article-title":"Development of an ultrasound prediction model to discriminate between malignant and benign liver lesions","volume":"46","author":"Shen","year":"2020","journal-title":"Ultrasound Med Biol"},{"key":"2023080106461965200_ref14","doi-asserted-by":"crossref","first-page":"102777","DOI":"10.1016\/j.ebiom.2020.102777","article-title":"Improving B-mode ultrasound diagnostic performance for focal liver lesions using deep learning: a multicentre study","volume":"56","author":"Yang","year":"2020","journal-title":"EBioMedicine"},{"issue":"8","key":"2023080106461965200_ref15","doi-asserted-by":"crossref","first-page":"2659","DOI":"10.18494\/SAM.2020.2801","article-title":"Classification of hepatocellular carcinoma and liver abscess by applying neural network to ultrasound images","volume":"32","author":"Xu","year":"2020","journal-title":"Sensors Mater"},{"issue":"4","key":"2023080106461965200_ref16","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1007\/s00535-022-01849-9","article-title":"Artificial intelligence (AI) models for the ultrasonographic diagnosis of liver tumors and comparison of diagnostic accuracies between AI and human experts","volume":"57","author":"Nishida","year":"2022","journal-title":"J Gastroenterol"},{"key":"2023080106461965200_ref17","doi-asserted-by":"crossref","DOI":"10.1016\/j.ebiom.2020.102814","article-title":"Deep learning promotes B-mode ultrasound screening for focal liver lesions","volume":"56","author":"Yamada","year":"2020","journal-title":"EBioMedicine"},{"issue":"s1","key":"2023080106461965200_ref18","first-page":"S1599","article-title":"Classification of focal liver lesions on ultrasound images by extracting hybrid textural features and using an artificial neural network","volume":"26","author":"Hwang","year":"2015","journal-title":"Biomed Mater Eng"},{"issue":"4","key":"2023080106461965200_ref19","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1016\/j.diii.2019.02.009","article-title":"Diagnosis of focal liver lesions from ultrasound using deep learning","volume":"100","author":"Schmauch","year":"2019","journal-title":"Diagn Interv Imaging"},{"issue":"9","key":"2023080106461965200_ref20","doi-asserted-by":"crossref","first-page":"2439","DOI":"10.1109\/TMI.2021.3078370","article-title":"Domain knowledge powered deep learning for breast cancer diagnosis based on contrast-enhanced ultrasound videos","volume":"40","author":"Chen","year":"2021","journal-title":"IEEE Trans Med Imaging"},{"issue":"03","key":"2023080106461965200_ref21","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1055\/a-0790-8163","article-title":"Artifacts in 3D prenatal sonography","volume":"41","author":"Tesanic","year":"2020","journal-title":"Ultraschall in der Medizin\u2013Eur J Ultrasound"},{"key":"2023080106461965200_ref22","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"Song","year":"2018"},{"key":"2023080106461965200_ref23","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Long","year":"2015"},{"key":"2023080106461965200_ref24","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Ronneberger"},{"key":"2023080106461965200_ref25","author":"Chen","year":"2017"},{"key":"2023080106461965200_ref26","volume-title":"European Conference on Computer Vision","author":"Lin"},{"issue":"1","key":"2023080106461965200_ref27","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat Mach Intell"},{"key":"2023080106461965200_ref28","author":"Lundberg","year":"2017"},{"issue":"7788","key":"2023080106461965200_ref29","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","article-title":"International evaluation of an AI system for breast cancer screening","volume":"577","author":"McKinney","year":"2020","journal-title":"Nature"},{"key":"2023080106461965200_ref30","volume-title":"Advances in Neural Information Processing Systems","author":"Xingjian"},{"issue":"1","key":"2023080106461965200_ref31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-46850-0","article-title":"Deep learning using convolutional LSTM estimates biological age from physical activity","volume":"9","author":"Rahman","year":"2019","journal-title":"Sci Rep"},{"key":"2023080106461965200_ref32","doi-asserted-by":"crossref","DOI":"10.1201\/9780429246593","volume-title":"An Introduction to the Bootstrap","author":"Efron","year":"1994"},{"key":"2023080106461965200_ref33","doi-asserted-by":"crossref","DOI":"10.1002\/9781119505969","volume-title":"Mathematical Statistics with Resampling and R","author":"Chihara","year":"2018"},{"issue":"1","key":"2023080106461965200_ref34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-12-77","article-title":"pROC: an open-source package for R and S+ to analyze and compare ROC curves","volume":"12","author":"Robin","year":"2011","journal-title":"BMC Bioinform"},{"key":"2023080106461965200_ref35","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Selvaraju"},{"issue":"6","key":"2023080106461965200_ref37","doi-asserted-by":"crossref","first-page":"1423","DOI":"10.1016\/j.cell.2020.04.045","article-title":"Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography","volume":"181","author":"Zhang","year":"2020","journal-title":"Cell"},{"key":"2023080106461965200_ref38","volume-title":"Clinical Features and Diagnosis of Hepatocellular Carcinoma","author":"Schwartz","year":"2019"},{"issue":"25","key":"2023080106461965200_ref39","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.1056\/NEJM199306243282501","article-title":"Risk factors for hepatocellular carcinoma among patients with chronic liver disease","volume":"328","author":"Tsukuma","year":"1993","journal-title":"New Engl J Med"},{"issue":"10","key":"2023080106461965200_ref40","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1080\/17474124.2018.1512855","article-title":"Testing for AFP in combination with ultrasound improves early liver cancer detection","volume":"12","author":"Tzartzeva","year":"2018","journal-title":"Expert Rev Gastroenterol Hepatol"},{"issue":"3","key":"2023080106461965200_ref41","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1007\/s10278-012-9537-8","article-title":"SVM-based characterization of liver ultrasound images using wavelet packet texture descriptors","volume":"26","author":"Virmani","year":"2013","journal-title":"J Digit Imaging"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/24\/1\/bbac569\/51014628\/bbac569.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/24\/1\/bbac569\/51014628\/bbac569.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,1]],"date-time":"2023-08-01T06:47:24Z","timestamp":1690872444000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbac569\/6961609"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,27]]},"references-count":41,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1,19]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbac569","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,1]]},"published":{"date-parts":[[2022,12,27]]},"article-number":"bbac569"}}