{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T15:30:28Z","timestamp":1778167828948,"version":"3.51.4"},"reference-count":29,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2021,2,5]],"date-time":"2021-02-05T00:00:00Z","timestamp":1612483200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Newly Hired Faculty, Taipei Medical University","award":["TMU108-AE1-B26"],"award-info":[{"award-number":["TMU108-AE1-B26"]}]},{"name":"Higher Education Sprout Project, Ministry of Education, Taiwan","award":["DP2-109-21121-01-A-06"],"award-info":[{"award-number":["DP2-109-21121-01-A-06"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,9,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Recently, language representation models have drawn a lot of attention in the natural language processing field due to their remarkable results. Among them, bidirectional encoder representations from transformers (BERT) has proven to be a simple, yet powerful language model that achieved novel state-of-the-art performance. BERT adopted the concept of contextualized word embedding to capture the semantics and context of the words in which they appeared. In this study, we present a novel technique by incorporating BERT-based multilingual model in bioinformatics to represent the information of DNA sequences. We treated DNA sequences as natural sentences and then used BERT models to transform them into fixed-length numerical matrices. As a case study, we applied our method to DNA enhancer prediction, which is a well-known and challenging problem in this field. We then observed that our BERT-based features improved more than 5\u201310% in terms of sensitivity, specificity, accuracy and Matthews correlation coefficient compared to the current state-of-the-art features in bioinformatics. Moreover, advanced experiments show that deep learning (as represented by 2D convolutional neural networks; CNN) holds potential in learning BERT features better than other traditional machine learning techniques. In conclusion, we suggest that BERT and 2D CNNs could open a new avenue in biological modeling using sequence information.<\/jats:p>","DOI":"10.1093\/bib\/bbab005","type":"journal-article","created":{"date-parts":[[2021,1,4]],"date-time":"2021-01-04T12:14:25Z","timestamp":1609762465000},"source":"Crossref","is-referenced-by-count":162,"title":["A transformer architecture based on BERT and 2D convolutional neural network to identify DNA enhancers from sequence information"],"prefix":"10.1093","volume":"22","author":[{"given":"Nguyen Quoc Khanh","family":"Le","sequence":"first","affiliation":[{"name":"Professional Master Program in Artificial Intelligence in Medicine, Taipei Medical University, Taipei, Taiwan"}]},{"given":"Quang-Thai","family":"Ho","sequence":"additional","affiliation":[{"name":"College of Information and Communication Technology, Can Tho University, Vietnam"}]},{"given":"Trinh-Trung-Duong","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Yuan Ze University, Taiwan"}]},{"given":"Yu-Yen","family":"Ou","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Yuan Ze University, Taiwan"}]}],"member":"286","published-online":{"date-parts":[[2021,2,5]]},"reference":[{"key":"2021090813533464600_ref1","doi-asserted-by":"crossref","first-page":"D506","DOI":"10.1093\/nar\/gky1049","article-title":"UniProt: a worldwide hub of protein knowledge","volume":"47","author":"The UniProt C","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2021090813533464600_ref2","doi-asserted-by":"crossref","first-page":"3336","DOI":"10.1093\/bioinformatics\/btaa155","article-title":"iMRM: a platform for simultaneously identifying multiple kinds of RNA modifications","volume":"36","author":"Liu","year":"2020","journal-title":"Bioinformatics"},{"key":"2021090813533464600_ref3","doi-asserted-by":"crossref","DOI":"10.1145\/3388440.3414701","article-title":"ProLanGO2: protein function prediction with ensemble of encoder-decoder networks","volume-title":"Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics","author":"Hippe","year":"2020"},{"key":"2021090813533464600_ref4","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.neucom.2019.09.070","article-title":"DeepETC: a deep convolutional neural network architecture for investigating and classifying electron transport chain's complexes","volume":"375","author":"Le","year":"2020","journal-title":"Neurocomputing"},{"key":"2021090813533464600_ref5","doi-asserted-by":"crossref","first-page":"3012","DOI":"10.1109\/JBHI.2020.2977091","article-title":"DeepAVP: a dual-channel deep neural network for identifying variable-length antiviral peptides","volume":"24","author":"Li","year":"2020","journal-title":"IEEE J Biomed Health Inform"},{"key":"2021090813533464600_ref6","article-title":"Using deep neural networks and biological subwords to detect protein S-sulfenylation sites","author":"Do","year":"2020","journal-title":"Brief Bioinform"},{"key":"2021090813533464600_ref7","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1038\/nrg861","article-title":"Genomics and natural language processing","volume":"3","author":"Yandell","year":"2002","journal-title":"Nat Rev Genet"},{"key":"2021090813533464600_ref8","doi-asserted-by":"crossref","first-page":"e0141287","DOI":"10.1371\/journal.pone.0141287","article-title":"Continuous distributed representation of biological sequences for deep proteomics and genomics","volume":"10","author":"Asgari","year":"2015","journal-title":"PLoS One"},{"key":"2021090813533464600_ref9","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.ab.2019.02.017","article-title":"iEnhancer-5Step: identifying enhancers using hidden information of DNA sequences via Chou's 5-step rule and word embedding","volume":"571","author":"Le","year":"2019","journal-title":"Anal Biochem"},{"key":"2021090813533464600_ref10","doi-asserted-by":"crossref","first-page":"305","DOI":"10.3389\/fbioe.2019.00305","article-title":"Classifying promoters by interpreting the hidden information of DNA sequences via deep learning and combination of continuous FastText N-grams","volume":"7","author":"Le","year":"2019","journal-title":"Front Bioeng Biotechnol"},{"key":"2021090813533464600_ref11","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1186\/s12859-019-3220-8","article-title":"Modeling aspects of the language of life through transfer-learning protein sequences","volume":"20","author":"Heinzinger","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2021090813533464600_ref12","doi-asserted-by":"crossref","first-page":"e0225317","DOI":"10.1371\/journal.pone.0225317","article-title":"An improved deep learning method for predicting DNA-binding proteins based on contextual features in amino acid sequences","volume":"14","author":"Hu","year":"2019","journal-title":"PLoS One"},{"key":"2021090813533464600_ref13","first-page":"427","article-title":"Bag of tricks for efficient text classification","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers","author":"Joulin","year":"2017"},{"key":"2021090813533464600_ref14","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1162\/tacl_a_00051","article-title":"Enriching word vectors with subword information","volume":"5","author":"Bojanowski","year":"2017","journal-title":"Trans Assoc Comput Linguist"},{"key":"2021090813533464600_ref15","first-page":"4171","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2021090813533464600_ref16","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1126\/science.281.5373.60","article-title":"Going the distance: a current view of enhancer action","volume":"281","author":"Blackwood","year":"1998","journal-title":"Science"},{"key":"2021090813533464600_ref17","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1038\/nrg3458","article-title":"Enhancers: five essential questions","volume":"14","author":"Pennacchio","year":"2013","journal-title":"Nat Rev Genet"},{"key":"2021090813533464600_ref18","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1016\/j.tcb.2018.04.003","article-title":"Enhancer logic and mechanics in development and disease","volume":"28","author":"Rickels","year":"2018","journal-title":"Trends Cell Biol"},{"key":"2021090813533464600_ref19","doi-asserted-by":"crossref","first-page":"R219","DOI":"10.1093\/hmg\/ddy167","article-title":"Enhancers: bridging the gap between gene control and human disease","volume":"27","author":"Karnuta","year":"2018","journal-title":"Hum Mol Genet"},{"key":"2021090813533464600_ref20","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1093\/bioinformatics\/btv604","article-title":"iEnhancer-2L: a two-layer predictor for identifying enhancers and their strength by pseudo k-tuple nucleotide composition","volume":"32","author":"Liu","year":"2016","journal-title":"Bioinformatics"},{"key":"2021090813533464600_ref21","doi-asserted-by":"crossref","first-page":"3150","DOI":"10.1093\/bioinformatics\/bts565","article-title":"CD-HIT: accelerated for clustering the next-generation sequencing data","volume":"28","author":"Fu","year":"2012","journal-title":"Bioinformatics"},{"key":"2021090813533464600_ref22","doi-asserted-by":"crossref","first-page":"3835","DOI":"10.1093\/bioinformatics\/bty458","article-title":"iEnhancer-EL: identifying enhancers and their strength with ensemble learning approach","volume":"34","author":"Liu","year":"2018","journal-title":"Bioinformatics"},{"key":"2021090813533464600_ref23","doi-asserted-by":"crossref","first-page":"38741","DOI":"10.1038\/srep38741","article-title":"EnhancerPred: a predictor for discovering enhancers based on the combination and selection of multiple features","volume":"6","author":"Jia","year":"2016","journal-title":"Sci Rep"},{"key":"2021090813533464600_ref24","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1007\/978-3-030-59612-5_3","article-title":"Predicting the DJIA with news headlines and historic data using hybrid genetic algorithm\/support vector regression and BERT","volume-title":"Big Data\u2013BigData 2020","author":"Warner","year":"2020"},{"key":"2021090813533464600_ref25","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1109\/ICCCS49078.2020.9118434","article-title":"A commodity review sentiment analysis based on BERT-CNN model","volume-title":"2020 5th International Conference on Computer and Communication Systems (ICCCS)","author":"Dong","year":"2020"},{"key":"2021090813533464600_ref26","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1504\/IJCBDD.2020.105095","article-title":"TopQA: a topological representation for single-model protein quality assessment with machine learning","volume":"13","author":"Smith","year":"2020","journal-title":"Int J Comput Biol Drug Des"},{"key":"2021090813533464600_ref27","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1016\/j.neucom.2020.09.056","article-title":"A convolutional neural network using dinucleotide one-hot encoder for identifying DNA N6-methyladenine sites in the rice genome","volume":"422","author":"Lv","year":"2021","journal-title":"Neurocomputing"},{"key":"2021090813533464600_ref28","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.cmpb.2019.05.016","article-title":"Identification of clathrin proteins by incorporating hyperparameter optimization in deep learning and PSSM profiles","volume":"177","author":"Le","year":"2019","journal-title":"Comput Methods Programs Biomed"},{"key":"2021090813533464600_ref29","doi-asserted-by":"crossref","first-page":"2499","DOI":"10.1093\/bioinformatics\/bty140","article-title":"iFeature: a python package and web server for features extraction and selection from protein and peptide sequences","volume":"34","author":"Chen","year":"2018","journal-title":"Bioinformatics"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/5\/bbab005\/40327912\/bbab005.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/5\/bbab005\/40327912\/bbab005.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,8]],"date-time":"2021-09-08T14:58:35Z","timestamp":1631113115000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbab005\/6128847"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,5]]},"references-count":29,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2021,9,2]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbab005","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,9]]},"published":{"date-parts":[[2021,2,5]]},"article-number":"bbab005"}}