{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T16:13:30Z","timestamp":1783440810984,"version":"3.54.6"},"reference-count":41,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2017,7,29]],"date-time":"2017-07-29T00:00:00Z","timestamp":1501286400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 61272373"],"award-info":[{"award-number":["No. 61272373"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 61202254"],"award-info":[{"award-number":["No. 61202254"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 71303031"],"award-info":[{"award-number":["No. 71303031"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["No. DC13010313"],"award-info":[{"award-number":["No. DC13010313"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["No.DCPY2016077"],"award-info":[{"award-number":["No.DCPY2016077"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Natural Science Foundation of Liaoning Province, China","award":["201602195"],"award-info":[{"award-number":["201602195"]}]},{"name":"Natural Science Foundation of Liaoning Province, China","award":["No. DC201502030202"],"award-info":[{"award-number":["No. DC201502030202"]}]},{"name":"Doctoral Scientific Research Foundation of Liaoning","award":["NO. 201601084"],"award-info":[{"award-number":["NO. 201601084"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Medical images are valuable for clinical diagnosis and decision making. Image modality is an important primary step, as it is capable of aiding clinicians to access required medical image in retrieval systems. Traditional methods of modality classification are dependent on the choice of hand-crafted features and demand a clear awareness of prior domain knowledge. The feature learning approach may detect efficiently visual characteristics of different modalities, but it is limited to the number of training datasets. To overcome the absence of labeled data, on the one hand, we take deep convolutional neural networks (VGGNet, ResNet) with different depths pre-trained on ImageNet, fix most of the earlier layers to reserve generic features of natural images, and only train their higher-level portion on ImageCLEF to learn domain-specific features of medical figures. Then, we train from scratch deep CNNs with only six weight layers to capture more domain-specific features. On the other hand, we employ two data augmentation methods to help CNNs to give the full scope to their potential characterizing image modality features. The final prediction is given by our voting system based on the outputs of three CNNs. After evaluating our proposed model on the subfigure classification task in ImageCLEF2015 and ImageCLEF2016, we obtain new, state-of-the-art results\u201476.87% in ImageCLEF2015 and 87.37% in ImageCLEF2016\u2014which imply that CNNs, based on our proposed transfer learning methods and data augmentation skills, can identify more efficiently modalities of medical images.<\/jats:p>","DOI":"10.3390\/info8030091","type":"journal-article","created":{"date-parts":[[2017,8,1]],"date-time":"2017-08-01T03:30:06Z","timestamp":1501558206000},"page":"91","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":127,"title":["Deep Transfer Learning for Modality Classification of Medical Images"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7827-3143","authenticated-orcid":false,"given":"Yuhai","family":"Yu","sequence":"first","affiliation":[{"name":"School of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China"},{"name":"School of Computer Science &amp; Engineering, Dalian Minzu University, Dalian 116600, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7191-2280","authenticated-orcid":false,"given":"Hongfei","family":"Lin","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiana","family":"Meng","sequence":"additional","affiliation":[{"name":"School of Computer Science &amp; Engineering, Dalian Minzu University, Dalian 116600, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3675-1020","authenticated-orcid":false,"given":"Xiaocong","family":"Wei","sequence":"additional","affiliation":[{"name":"School of Software Engineering, Dalian University of Foreign Languages, Dalian 116044, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hai","family":"Guo","sequence":"additional","affiliation":[{"name":"School of Computer Science &amp; Engineering, Dalian Minzu University, Dalian 116600, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhehuan","family":"Zhao","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,7,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Lu, Z. (2011). PubMed and beyond: A survey of web tools for searching biomedical literature. Database.","DOI":"10.1093\/database\/baq036"},{"key":"ref_2","first-page":"135","article-title":"Application of medical images for diagnosis of diseases-review article","volume":"2","author":"Khan","year":"2017","journal-title":"World J. Microbiol. Biotechnol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Shi, J., Zheng, X., Li, Y., Zhang, Q., and Ying, S. (2017). Multimodal Neuroimaging Feature Learning with Multimodal Stacked Deep Polynomial Networks for Diagnosis of Alzheimer\u2019s Disease. IEEE J. Biomed. Health Inform.","DOI":"10.1109\/JBHI.2017.2655720"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shi, J., Wu, J., Li, Y., Zhang, Q., and Ying, S. (2016). Histopathological image classification with color pattern random binary hashing based PCANet and matrix-form classifier. IEEE J. Biomed. Health Inform.","DOI":"10.1109\/JBHI.2016.2602823"},{"key":"ref_5","unstructured":"De Herrera, A.G.S., Kalpathy-Cramer, J., Fushman, D.D., Antani, S., and M\u00fcller, H. (2013, January 23\u201326). Overview of the ImageCLEF 2013 medical tasks. Proceedings of the Working Notes of CLEF, Valencia, Spain."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijmedinf.2003.11.024","article-title":"A review of content-based image retrieval systems in medical applications\u2014Clinical benefits and future directions","volume":"73","author":"Michoux","year":"2004","journal-title":"Int. J. Med. Inform."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"168","DOI":"10.5626\/JCSE.2012.6.2.168","article-title":"Design and development of a multimodal biomedical information retrieval system","volume":"6","author":"Antani","year":"2012","journal-title":"J. Comput. Sci. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tirilly, P., Lu, K., Mu, X., Zhao, T., and Cao, Y. (2011, January 13\u201315). On modality classification and its use in text-based image retrieval in medical databases. Proceedings of the 9th International Workshop on Content-Based Multimedia Indexing (CBMI) 2011, Madrid, Spain.","DOI":"10.1109\/CBMI.2011.5972530"},{"key":"ref_9","unstructured":"De Herrera, A.G.S., Markonis, D., and M\u00fcller, H. (2013). Bag-of-Colors for Biomedical Document Image Classification. Medical Content-Based Retrieval for Clinical Decision Support (MCBR-CDS) 2012, Lecture Notes in Computer Science, Springer."},{"key":"ref_10","unstructured":"Pelka, O., and Friedrich, C.M. (2015, January 8\u201311). FHDO biomedical computer science group at medical classification task of ImageCLEF 2015. Proceedings of the Working Notes of CLEF, Toulouse, France."},{"key":"ref_11","unstructured":"Cirujeda, P., and Binefa, X. (2015, January 8\u201311). Medical Image Classification via 2D color feature based Covariance Descriptors. Proceedings of the Working Notes of CLEF, Toulouse, France."},{"key":"ref_12","unstructured":"Valavanis, L., Stathopoulos, S., and Kalamboukis, T. (2016, January 5\u20138). IPL at CLEF 2016 Medical Task. Proceedings of the Working Notes of CLEF, \u00c9vora, Portugal."},{"key":"ref_13","unstructured":"Li, P., Sorensen, S., Kolagunda, A., Jiang, X., Wang, X., Kambhamettu, C., and Shatkay, H. (2016, January 5\u20138). UDEL CIS Working Notes in ImageCLEF 2016. Proceedings of the Working Notes of CLEF, \u00c9vora, Portugal."},{"key":"ref_14","unstructured":"Pelka, O., and Friedrich, C.M. (2016). Modality prediction of biomedical literature images using multimodal feature representation. GMS Med. Inform. Biom. Epidemiol., 12."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_17","unstructured":"Simonyan, K., and Zisserman, A. (Comput. Sci., 2014). Very deep convolutional networks for large-scale image recognition, Comput. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hynes Convention Center, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_22","first-page":"5403","article-title":"Modality classification for medical images using multiple deep convolutional neural networks","volume":"11","author":"Yu","year":"2015","journal-title":"J. Colloid Interface Sci."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ravishankar, H., Sudhakar, P., Venkataramani, R., Thiruvenkadam, S., Annangi, P., Babu, N., and Vaidya, V. (2016). Understanding the Mechanisms of Deep Transfer Learning for Medical Images. Deep Learning and Data Labeling for Medical Applications. Lecture Notes in Computer Science, Springer International Publishing.","DOI":"10.1007\/978-3-319-46976-8_20"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1285","DOI":"10.1109\/TMI.2016.2528162","article-title":"Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning","volume":"35","author":"Shin","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1109\/JBHI.2016.2635663","article-title":"An ensemble of fine-tuned convolutional neural networks for medical image classification","volume":"21","author":"Kumar","year":"2017","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_26","unstructured":"Zhang, J., Xia, Y., Wu, Q., and Xie, Y. (Comput. Sci., 2017). Classification of Medical Images and Illustrations in the Biomedical Literature Using Synergic Deep Learning, Comput. Sci."},{"key":"ref_27","unstructured":"Koitka, S., and Friedrich, C.M. (2016, January 5\u20138). Traditional feature engineering and deep learning approaches at medical classification task of ImageCLEF 2016. Proceedings of the Working Notes of CLEF, \u00c9vora, Portugal."},{"key":"ref_28","unstructured":"Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014, January 21\u201326). Decaf: A deep convolutional activation feature for generic visual recognition. Proceedings of the International Conference on Machine Learning, Beijing, China."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive image features from scale-invariant keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_30","unstructured":"Wengert, C., Douze, M., and J\u00e9gou, H. (December, January 28). Bag-of-colors for improved image search. Proceedings of the 19th ACM International Conference on Multimedia, New York, NY, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Yang, J., Jiang, Y.G., Hauptmann, A.G., and Ngo, C.W. (2007, January 28\u201329). Evaluating bag-of-visual-words representations in scene classification. Proceedings of the International Workshop on ACM Multimedia Information Retrieval, University of Augsburg, Augsburg, Germany.","DOI":"10.1145\/1290082.1290111"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yin, X., D\u00fcntsch, I., and Gediga, G. (2011). Quadtree representation and compression of spatial data. Trans. Rough Sets XIII, 207\u2013239.","DOI":"10.1007\/978-3-642-18302-7_12"},{"key":"ref_33","unstructured":"De Herrera, A.G.S., M\u00fcller, H., and Bromuri, S. (2015, January 8\u201311). Overview of the ImageCLEF 2015 medical classification task. Proceedings of the Working Notes of CLEF, Toulouse, France."},{"key":"ref_34","unstructured":"De Herrera, A.G.S., Schaer, R., Bromuri, S., and M\u00fcller, H. (2016, January 5\u20138). Overview of the ImageCLEF 2016 medical task. Proceedings of the Working Notes of CLEF, \u00c9vora, Portugal."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yu, Y., Lin, H., Meng, J., and Zhao, Z. (2016). Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks. Algorithms, 9.","DOI":"10.3390\/a9020041"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Yu, Y., Lin, H., Meng, J., Wei, X., and Zhao, Z. (2017). Assembling Deep Neural Networks for Medical Compound Figure Detection. Information, 8.","DOI":"10.3390\/info8020048"},{"key":"ref_37","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), Sardinia, Italy."},{"key":"ref_38","unstructured":"Kingma, D., and Ba, J. (Comput. Sci., 2014). Adam: A method for stochastic optimization, Comput. Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/s10115-012-0586-6","article-title":"A weighted voting framework for classifiers ensembles","volume":"38","author":"Kuncheva","year":"2014","journal-title":"Knowl. Inf. Syst."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, D., and Riddle, D.L. (2008). Function of the PHA-4\/FOXA transcription factor during C. elegans post-embryonic development. BMC Dev. Biol., 8.","DOI":"10.1186\/1471-213X-8-26"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"M\u00fcller, H., Kalpathy-Cramer, J., Demner-Fushman, D., and Antani, S. (2012, January 21\u201326). Creating a classification of image types in the medical literature for visual categorization. Proceedings of the SPIE Medical Imaging, San Francisco, CA, USA.","DOI":"10.1117\/12.911186"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/8\/3\/91\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:44:29Z","timestamp":1760208269000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/8\/3\/91"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,29]]},"references-count":41,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2017,9]]}},"alternative-id":["info8030091"],"URL":"https:\/\/doi.org\/10.3390\/info8030091","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,7,29]]}}}