{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T11:52:06Z","timestamp":1783597926993,"version":"3.55.0"},"reference-count":12,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2023,11,16]],"date-time":"2023-11-16T00:00:00Z","timestamp":1700092800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,16]],"date-time":"2023-11-16T00:00:00Z","timestamp":1700092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004052","name":"King Abdullah University of Science and Technology","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004052","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004047","name":"Karolinska Institute","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004047","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The rise of single-cell genomics is an attractive opportunity for data-hungry machine learning algorithms. The scBERT method, inspired by the success of BERT (\u2018bidirectional encoder representations from transformers\u2019) in natural language processing, was recently introduced by Yang et al. as a data-driven tool to annotate cell types in single-cell genomics data. Analogous to contextual embedding in BERT, scBERT leverages pretraining and self-attention mechanisms to learn the \u2018transcriptional grammar\u2019 of cells. Here we investigate the reusability beyond the original datasets, assessing the generalizability of natural language techniques in single-cell genomics. The degree of imbalance in the cell-type distribution substantially influences the performance of scBERT. Anticipating an increased utilization of transformers, we highlight the necessity to consider data distribution carefully and introduce a subsampling technique to mitigate the influence of an imbalanced distribution. Our analysis serves as a stepping stone towards understanding and optimizing the use of transformers in single-cell genomics.<\/jats:p>","DOI":"10.1038\/s42256-023-00757-8","type":"journal-article","created":{"date-parts":[[2023,11,16]],"date-time":"2023-11-16T17:01:58Z","timestamp":1700154118000},"page":"1437-1446","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Reusability report: Learning the transcriptional grammar in single-cell RNA-sequencing data using transformers"],"prefix":"10.1038","volume":"5","author":[{"given":"Sumeer Ahmad","family":"Khan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alberto","family":"Maillo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6552-6076","authenticated-orcid":false,"given":"Vincenzo","family":"Lagani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7071-4226","authenticated-orcid":false,"given":"Robert","family":"Lehmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0949-046X","authenticated-orcid":false,"given":"Narsis A.","family":"Kiani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Gomez-Cabrero","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9568-5588","authenticated-orcid":false,"given":"Jesper","family":"Tegner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,11,16]]},"reference":[{"key":"757_CR1","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1038\/s41580-022-00466-x","volume":"23","author":"Q Ma","year":"2022","unstructured":"Ma, Q. & Xu, D. Deep learning shapes single-cell data analysis. Nat. Rev. Mol. Cell Biol. 23, 303\u2013304 (2022).","journal-title":"Nat. Rev. Mol. Cell Biol."},{"key":"757_CR2","doi-asserted-by":"publisher","first-page":"829","DOI":"10.1038\/nbt.4233","volume":"36","author":"M Wainberg","year":"2018","unstructured":"Wainberg, M., Merico, D., Delong, A. & Frey, B. J. Deep learning in biomedicine. Nat. Biotechnol. 36, 829\u2013838 (2018).","journal-title":"Nat. Biotechnol."},{"key":"757_CR3","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1038\/s42256-022-00534-z","volume":"4","author":"F Yang","year":"2022","unstructured":"Yang, F. et al. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nat. Mach. Intell. 4, 852\u2013866 (2022).","journal-title":"Nat. Mach. Intell."},{"key":"757_CR4","doi-asserted-by":"publisher","unstructured":"Cui, H., Wang, C., Maan, H., Duan, N. & Wang, B. scFormer: a universal representation learning approach for single-cell data using transformers. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2022.11.20.517285 (2022).","DOI":"10.1101\/2022.11.20.517285"},{"key":"757_CR5","doi-asserted-by":"publisher","DOI":"10.1186\/s12864-018-5370-x","volume":"20","author":"J Du","year":"2019","unstructured":"Du, J. et al. Gene2vec: distributed representation of genes based on co-expression. BMC Genomics 20, 82 (2019).","journal-title":"BMC Genomics"},{"key":"757_CR6","doi-asserted-by":"publisher","first-page":"baz046","DOI":"10.1093\/database\/baz046","volume":"2019","author":"O Franz\u00e9n","year":"2019","unstructured":"Franz\u00e9n, O., Gan, L.-M. & Bj\u00f6rkegren, J. L. M. PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data. Database 2019, baz046 (2019).","journal-title":"Database"},{"key":"757_CR7","doi-asserted-by":"publisher","DOI":"10.1038\/ncomms14049","volume":"8","author":"GXY Zheng","year":"2017","unstructured":"Zheng, G. X. Y. et al. Massively parallel digital transcriptional profiling of single cells. Nat. Commun. 8, 14049 (2017).","journal-title":"Nat. Commun."},{"key":"757_CR8","doi-asserted-by":"publisher","first-page":"4383","DOI":"10.1038\/s41467-018-06318-7","volume":"9","author":"SA MacParland","year":"2018","unstructured":"MacParland, S. A. et al. Single cell RNA sequencing of human liver reveals distinct intrahepatic macrophage populations. Nat. Commun. 9, 4383 (2018).","journal-title":"Nat. Commun."},{"key":"757_CR9","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-017-1382-0","volume":"19","author":"FA Wolf","year":"2018","unstructured":"Wolf, F. A., Angerer, P. & Theis, F. J. SCANPY: large-scale single-cell gene expression data analysis. Genome Biol. 19, 15 (2018).","journal-title":"Genome Biol."},{"key":"757_CR10","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321\u2013357 (2002).","journal-title":"J. Artif. Intell. Res."},{"key":"757_CR11","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","volume":"42","author":"TY Lin","year":"2020","unstructured":"Lin, T. Y., Goyal, P., Girshick, R., He, K. & Dollar, P. Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42, 318\u2013327 (2020).","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"757_CR12","doi-asserted-by":"publisher","unstructured":"Khan, S. A. et al. Translational bioinformatics unit\/scBERT-reusability: 2.0.0. Zenodo https:\/\/doi.org\/10.5281\/zenodo.8191571 (2023).","DOI":"10.5281\/zenodo.8191571"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00757-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00757-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00757-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T20:08:41Z","timestamp":1702930121000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00757-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,16]]},"references-count":12,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["757"],"URL":"https:\/\/doi.org\/10.1038\/s42256-023-00757-8","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,16]]},"assertion":[{"value":"24 January 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 October 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}