{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T13:00:48Z","timestamp":1791464448271,"version":"4.3.4"},"reference-count":38,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T00:00:00Z","timestamp":1618790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Norwegian Financial Mechanism 2014-2021 under the project RO-NO-2019-0138, 19\/2020 \u201cIm-proving Cancer Diagnostics in Flexible Endoscopy using Artificial Intelligence and Medical Ro-botics\u201d IDEAR, Contract No. 19\/2020","award":["Contract No. 19\/2020."],"award-info":[{"award-number":["Contract No. 19\/2020."]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Medicina"],"abstract":"<jats:p>Background and Objectives: At present, thyroid disorders have a great incidence in the worldwide population, so the development of alternative methods for improving the diagnosis process is necessary. Materials and Methods: For this purpose, we developed an ensemble method that fused two deep learning models, one based on convolutional neural network and the other based on transfer learning. For the first model, called 5-CNN, we developed an efficient end-to-end trained model with five convolutional layers, while for the second model, the pre-trained VGG-19 architecture was repurposed, optimized and trained. We trained and validated our models using a dataset of ultrasound images consisting of four types of thyroidal images: autoimmune, nodular, micro-nodular, and normal. Results: Excellent results were obtained by the ensemble CNN-VGG method, which outperformed the 5-CNN and VGG-19 models: 97.35% for the overall test accuracy with an overall specificity of 98.43%, sensitivity of 95.75%, positive and negative predictive value of 95.41%, and 98.05%. The micro average areas under each receiver operating characteristic curves was 0.96. The results were also validated by two physicians: an endocrinologist and a pediatrician. Conclusions: We proposed a new deep learning study for classifying ultrasound thyroidal images to assist physicians in the diagnosis process.<\/jats:p>","DOI":"10.3390\/medicina57040395","type":"journal-article","created":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T06:35:53Z","timestamp":1618814153000},"page":"395","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Intelligent Diagnosis of Thyroid Ultrasound Imaging Using an Ensemble of Deep Learning Methods"],"prefix":"10.3390","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4684-8213","authenticated-orcid":false,"given":"Corina Maria","family":"Vasile","sequence":"first","affiliation":[{"name":"PhD School Department, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"},{"name":"Department of Pediatric Cardiology, County Clinical Emergency Hospital of Craiova, 200642 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5560-6435","authenticated-orcid":false,"given":"Anca Loredana","family":"Udri\u0219toiu","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computers and Electronics, University of Craiova, 200776 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9829-972X","authenticated-orcid":false,"given":"Alice Elena","family":"Ghenea","sequence":"additional","affiliation":[{"name":"Department of Bacteriology-Virology-Parasitology, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mihaela","family":"Popescu","sequence":"additional","affiliation":[{"name":"Department of Endocrinology, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristian","family":"Gheonea","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carmen Elena","family":"Niculescu","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anca Marilena","family":"Ungureanu","sequence":"additional","affiliation":[{"name":"Department of Bacteriology-Virology-Parasitology, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"\u0218tefan","family":"Udri\u0219toiu","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computers and Electronics, University of Craiova, 200776 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrei Ioan","family":"Droca\u015f","sequence":"additional","affiliation":[{"name":"Department of Urology, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4526-0864","authenticated-orcid":false,"given":"Lucian Gheorghe","family":"Gruionu","sequence":"additional","affiliation":[{"name":"Faculty of Mechanics, University of Craiova, 200512 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8096-2637","authenticated-orcid":false,"given":"Gabriel","family":"Gruionu","sequence":"additional","affiliation":[{"name":"Department of Medicine, Indiana University School of Medicine, Indianapolis, IN 46202, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreea Valentina","family":"Iacob","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computers and Electronics, University of Craiova, 200776 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Drago\u015f Ovidiu","family":"Alexandru","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics and Biostatistics, University of Medicine and Pharmacy of Craiova, 200349 Craiova, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,19]]},"reference":[{"key":"ref_1","unstructured":"Gardner, D.G., Shoback, D.M., and Greenspan, F.S. (2018). The Thyroid Gland. Greenspan\u2019s Basic & Clinical Endocrinology, McGraw-Hill Medical. [10th ed.]."},{"key":"ref_2","first-page":"2010","article-title":"Autoimmune thyroid diseases in children","volume":"675703","author":"Cappa","year":"2011","journal-title":"J. Thyroid Res."},{"key":"ref_3","first-page":"241","article-title":"Hashimoto\u2019s thyroiditis associated with thyroid adenoma with Hurthle cells-case report","volume":"58","author":"Stanciu","year":"2017","journal-title":"Rom. J. Morphol. Embryol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1089\/thy.2000.10.251","article-title":"The value of ultrasonography in predicting autoimmune thyroid disease","volume":"10","author":"Pedersen","year":"2000","journal-title":"Thyroid"},{"key":"ref_5","first-page":"245","article-title":"A prospective study of thyroid function, morphology and autoimmunity in young patients with type 1 diabetes","volume":"148","author":"Dorte","year":"2003","journal-title":"Eur. J. Endocrinol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"696","DOI":"10.7326\/0003-4819-133-9-200011070-00011","article-title":"Usefulness of ultrasonography in the management of nodular thyroid disease","volume":"133","author":"Marqusee","year":"2000","journal-title":"Ann. Internal. Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2360","DOI":"10.1177\/147323001103900636","article-title":"Nodule or pseudonodule? Differentiation in Hashimoto\u2019s thyroiditis with sonoelastography","volume":"39","author":"Yildirim","year":"2011","journal-title":"J. Int. Med. Res."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Nguyen, D.T., Pham, T.D., Batchuluun, G., Yoon, H.J., and Park, K.R. (2019). Artificial Intelligence-Based Thyroid Nodule Classification Using Information from Spatial and Frequency Domains. J. Clin. Med., 8.","DOI":"10.3390\/jcm8111976"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"17843","DOI":"10.1038\/s41598-019-54434-1","article-title":"Diagnosis of Thyroid Nodules: Performance of a Deep Learning Convolutional Neural Network Model vs. Radiologists","volume":"9","author":"Park","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"357","DOI":"10.4103\/eus.eus_76_19","article-title":"Artificial intelligence in pancreatic cancer: Toward precision diagnosis","volume":"8","author":"Cazacu","year":"2019","journal-title":"Endosc. Ultrasound"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Dhaka, V.S., Rani, G., Oza, M.G., Sharma, T., and Misra, A. (2021). A deep learning model for mass screening of COVID-19. Int. J. Imaging Syst. Technol., 1\u201316. status manuscript in preparation.","DOI":"10.1002\/ima.22544"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1016\/j.eng.2018.11.020","article-title":"Deep Learning in Medical Ultrasound Analysis: A Review","volume":"5","author":"Liu","year":"2019","journal-title":"Engineering"},{"key":"ref_13","first-page":"146","article-title":"Multi-instance Deep Learning with Graph Convolutional Neural Networks for Diagnosis of Kidney Diseases Using Ultrasound Imaging","volume":"Volume 11840","author":"Yin","year":"2019","journal-title":"Uncertainty for Safe Utilization of Machine Learning in Medical Imaging and Clinical Image-Based Procedures"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"15245","DOI":"10.1038\/s41598-020-72270-6","article-title":"Diagnosis of thyroid nodules on ultrasonography by a deep convolutional neural network","volume":"10","author":"Koh","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.ultras.2016.09.011","article-title":"A pre-trained convolutional neural network based method for thyroid nodule diagnosis","volume":"73","author":"Ma","year":"2017","journal-title":"Ultrasonics"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Liu, T., Xie, S., Yu, J., Niu, L., and Sun, W.D. (2017, January 5\u20139). Classification of thyroid nodules in ultrasound images using deep model based transfer learning and hybrid features. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.","DOI":"10.1109\/ICASSP.2017.7952290"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Liu, T., Xie, S., Zhang, Y., Yu, J., Niu, L., and Sun, W.D. (2017, January 18\u201321). Feature selection and thyroid nodule classification using transfer learning. Proceedings of the 2017 IEEE 14th International Symposium on Biomedical Imaging, Melbourne, Australia.","DOI":"10.1109\/ISBI.2017.7950707"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Nguyen, D., Kang, J., Pham, T., Batchuluun, G., and Park, K. (2020). Ultrasound Image-Based Diagnosis of Malignant Thyroid Nodule Using Artificial Intelligence. Sensors, 20.","DOI":"10.3390\/s20071822"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1016\/S1470-2045(18)30762-9","article-title":"Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: A retrospective, multicohort, diagnostic study","volume":"20","author":"Li","year":"2019","journal-title":"Lancet Oncol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2009","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Abadi, M. (2016, January 19\u201321). TensorFlow: Learning functions at scale. Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming, Nara, Japan.","DOI":"10.1145\/2951913.2976746"},{"key":"ref_22","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_23","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_26","unstructured":"Tan, M., and Le, Q. (2019, January 9\u201315). Efficient Net: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, PMLR, Long Beach, CA, USA."},{"key":"ref_27","unstructured":"Keras (2020, June 01). Available online: https:\/\/github.com\/keras-team\/keras."},{"key":"ref_28","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_29","unstructured":"Dietterich, T.G. (2000, January 21\u201323). Ensemble Methods in Machine Learning. Proceedings of the First International Workshop on Multiple Classifier Systems, MCS \u201900, London, UK."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Saito, T., and Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0118432"},{"key":"ref_31","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, The MIT Press."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Fu, Z., and Fei, J. (2017, January 13\u201316). An image augmentation method using convolutional network for thyroid nodule classification by transfer learning. Proceedings of the 3rd IEEE International Conference on Computer and Communication, Chengdu, China.","DOI":"10.1109\/CompComm.2017.8322853"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1007\/s10278-017-9997-y","article-title":"Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network","volume":"30","author":"Chi","year":"2017","journal-title":"J. Digit. Imaging"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"e15133","DOI":"10.1097\/MD.0000000000015133","article-title":"Ultrasound image analysis using deep learning algorithm for the diagnosis of thyroid nodules","volume":"98","author":"Song","year":"2019","journal-title":"Medicine"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.bspc.2017.08.025","article-title":"Computer aided thyroid nodule detection system using medical ultrasound images","volume":"40","author":"Koundal","year":"2018","journal-title":"Biomed. Signal. Process. Control"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.engappai.2016.12.019","article-title":"Breast tumor classification in ultrasound images using texture analysis and super-resolution methods","volume":"59","author":"Nasser","year":"2017","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1016\/j.future.2018.01.058","article-title":"Super-resolution of retinal images using multi-kernel SVR for IoT healthcare applications","volume":"83","author":"Jebadurai","year":"2018","journal-title":"Future Gener. Comput. Syst."}],"container-title":["Medicina"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1648-9144\/57\/4\/395\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:49:35Z","timestamp":1760161775000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1648-9144\/57\/4\/395"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,19]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["medicina57040395"],"URL":"https:\/\/doi.org\/10.3390\/medicina57040395","relation":{},"ISSN":["1648-9144"],"issn-type":[{"value":"1648-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,19]]}}}