{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T11:55:17Z","timestamp":1780055717222,"version":"3.54.0"},"reference-count":58,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T00:00:00Z","timestamp":1672876800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T00:00:00Z","timestamp":1672876800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100008899","name":"University of South Carolina","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100008899","id-type":"DOI","asserted-by":"crossref"}]},{"name":"South Carolina Honors College Research Program"},{"name":"Magellan Scholar Program"},{"name":"NSF","award":["1905775"],"award-info":[{"award-number":["1905775"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2023,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Pre-trained transformer language models (LMs) on large unlabeled corpus have produced state-of-the-art results in natural language processing, organic molecule design, and protein sequence generation. However, no such models have been applied to learn the composition patterns for the generative design of material compositions. Here we train a series of seven modern transformer models (GPT, GPT-2, GPT-Neo, GPT-J, BLMM, BART, and RoBERTa) for materials design using the expanded formulas of the ICSD, OQMD, and Materials Projects databases. Six different datasets with\/out non-charge-neutral or EB samples are used to benchmark the generative design performances and uncover the biases of modern transformer models for the generative design of materials compositions. Our experiments show that the materials transformers based on causal LMs can generate chemically valid material compositions with as high as 97.61% to be charge neutral and 91.22% to be electronegativity balanced, which has more than six times higher enrichment compared to the baseline pseudo-random sampling algorithm. Our LMs also demonstrate high generation novelty and their potential in new materials discovery is proved by their capability to recover the leave-out materials. We also find that the properties of the generated compositions can be tailored by training the models with selected training sets such as high-bandgap samples. Our experiments also show that different models each have their own preference in terms of the properties of the generated samples and their running time complexity varies a lot. We have applied our materials transformers to discover a set of new materials as validated using density functional theory calculations. All our trained materials transformer models and code can be accessed freely at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"http:\/\/www.github.com\/usccolumbia\/MTransformer\" xlink:type=\"simple\">http:\/\/www.github.com\/usccolumbia\/MTransformer<\/jats:ext-link>.<\/jats:p>","DOI":"10.1088\/2632-2153\/acadcd","type":"journal-article","created":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T22:43:25Z","timestamp":1671662605000},"page":"015001","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Material transformers: deep learning language models for generative materials design"],"prefix":"10.1088","volume":"4","author":[{"given":"Nihang","family":"Fu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lai","family":"Wei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuqi","family":"Song","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qinyang","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Xin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sadman Sadeed","family":"Omee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rongzhi","family":"Dong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8960-5273","authenticated-orcid":true,"given":"Edirisuriya M Dilanga","family":"Siriwardane","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8725-6660","authenticated-orcid":true,"given":"Jianjun","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2023,1,5]]},"reference":[{"key":"mlstacadcdbib1","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1107\/S160057671900997X","article-title":"Recent developments in the inorganic crystal structure database: theoretical crystal structure data and related features","volume":"52","author":"Zagorac","year":"2019","journal-title":"J. Appl. Crystallogr."},{"key":"mlstacadcdbib2","doi-asserted-by":"publisher","first-page":"3031","DOI":"10.1021\/acs.chemrev.0c00608","article-title":"Understanding doping of quantum materials","volume":"121","author":"Zunger","year":"2021","journal-title":"Chem. Rev."},{"key":"mlstacadcdbib3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41524-020-00352-0","article-title":"Generative adversarial networks (GAN) based efficient sampling of chemical composition space for inverse design of inorganic materials","volume":"6","author":"Dan","year":"2020","journal-title":"npj Comput. Mater."},{"key":"mlstacadcdbib4","doi-asserted-by":"publisher","DOI":"10.1002\/advs.202100566","article-title":"High-throughput discovery of novel cubic crystal materials using deep generative neural networks","volume":"8","author":"Zhao","year":"2021","journal-title":"Adv. Sci."},{"key":"mlstacadcdbib5","doi-asserted-by":"publisher","first-page":"8431","DOI":"10.1021\/acs.inorgchem.1c03879","article-title":"TCSP: a template based crystal structure prediction algorithm and web server for materials discovery","volume":"22","author":"Wei","year":"2022","journal-title":"Inorg. Chem."},{"key":"mlstacadcdbib6","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018"},{"key":"mlstacadcdbib7","first-page":"pp 1877","article-title":"Language models are few-shot learners","volume":"vol 33","author":"Brown","year":"2020"},{"key":"mlstacadcdbib8","first-page":"pp 932","article-title":"Frequency effects on syntactic rule learning in transformers","author":"Wei","year":"2021"},{"key":"mlstacadcdbib9","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1162\/tacl_a_00313","article-title":"Leveraging pre-trained checkpoints for sequence generation tasks","volume":"8","author":"Rothe","year":"2020","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"mlstacadcdbib10","doi-asserted-by":"crossref","DOI":"10.24963\/ijcai.2021\/612","article-title":"Pretrained language models for text generation: a survey","author":"Li","year":"2021"},{"key":"mlstacadcdbib11","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.naacl-main.257","article-title":"Learning to transfer prompts for text generation","author":"Li","year":"2022"},{"key":"mlstacadcdbib12","article-title":"Improving language understanding by generative pre-training","author":"Radford","year":"2018","journal-title":"OpenAI Blog"},{"key":"mlstacadcdbib13","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"mlstacadcdbib14","article-title":"Unified language model pre-training for natural language understanding and generation","volume":"vol 32","author":"Dong","year":"2019"},{"key":"mlstacadcdbib15","article-title":"Xlnet: generalized autoregressive pretraining for language understanding","volume":"vol 32","author":"Yang","year":"2019"},{"key":"mlstacadcdbib16","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019"},{"key":"mlstacadcdbib17","article-title":"BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis","year":"2019"},{"key":"mlstacadcdbib18","doi-asserted-by":"publisher","first-page":"2064","DOI":"10.1021\/acs.jcim.1c00600","article-title":"MolGPT: molecular generation using a transformer-decoder model","volume":"62","author":"Bagal","year":"2022","journal-title":"J. Chem. Inf. Model."},{"key":"mlstacadcdbib19","article-title":"C5t5: controllable generation of organic molecules with transformers","author":"Rothchild","year":"2021"},{"key":"mlstacadcdbib20","doi-asserted-by":"publisher","first-page":"5804","DOI":"10.1021\/acs.jcim.1c01289","article-title":"Generative chemical transformer: neural machine learning of molecular geometric structures from chemical language via attention","volume":"61","author":"Kim","year":"2021","journal-title":"J. Chem. Inf. Model."},{"key":"mlstacadcdbib21","doi-asserted-by":"publisher","first-page":"8362","DOI":"10.1039\/D1SC01050F","article-title":"Attention-based generative models for de novo molecular design","volume":"12","author":"Dollar","year":"2021","journal-title":"Chem. Sci."},{"key":"mlstacadcdbib22","article-title":"Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models","author":"Guimaraes","year":"2017"},{"key":"mlstacadcdbib23","article-title":"MolGAN: an implicit generative model for small molecular graphs","author":"De Cao","year":"2018"},{"key":"mlstacadcdbib24","doi-asserted-by":"crossref","DOI":"10.1101\/2020.03.07.982272","article-title":"ProGen: language modeling for protein generation","author":"Madani","year":"2020"},{"key":"mlstacadcdbib25","doi-asserted-by":"publisher","first-page":"2154","DOI":"10.1021\/acssynbio.0c00219","article-title":"Signal peptides generated by attention-based neural networks","volume":"9","author":"Wu","year":"2020","journal-title":"ACS Synth. Biol."},{"key":"mlstacadcdbib26","article-title":"RITA: a study on scaling up generative protein sequence models","author":"Hesslow","year":"2022"},{"key":"mlstacadcdbib27","article-title":"Generative models for graph-based protein design","volume":"vol 32","author":"Ingraham","year":"2019"},{"key":"mlstacadcdbib28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-022-32007-7","article-title":"ProtGPT2 is a deep unsupervised language model for protein design","volume":"13","author":"Ferruz","year":"2022","journal-title":"Nat. Commun."},{"key":"mlstacadcdbib29","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.cels.2020.05.007","article-title":"A generative neural network for maximizing fitness and diversity of synthetic DNA and protein sequences","volume":"11","author":"Linder","year":"2020","journal-title":"Cell Syst."},{"key":"mlstacadcdbib30","doi-asserted-by":"publisher","first-page":"6440","DOI":"10.1021\/acs.jpcb.1c02449","article-title":"How deep learning tools can help protein engineers find good sequences","volume":"125","author":"Osadchy","year":"2021","journal-title":"J. Phys. Chem. B"},{"key":"mlstacadcdbib31","article-title":"Crystal transformer: self-learning neural language model for generative and tinkering design of materials","author":"Wei","year":"2022"},{"key":"mlstacadcdbib32","doi-asserted-by":"publisher","first-page":"656","DOI":"10.1021\/ic102031h","article-title":"Data mined ionic substitutions for the discovery of new compounds","volume":"50","author":"Hautier","year":"2011","journal-title":"Inorg. Chem."},{"key":"mlstacadcdbib33","doi-asserted-by":"publisher","first-page":"732","DOI":"10.1038\/s41563-019-0396-2","article-title":"A map of the inorganic ternary metal nitrides","volume":"18","author":"Sun","year":"2019","journal-title":"Nat. Mater."},{"key":"mlstacadcdbib34","doi-asserted-by":"publisher","first-page":"558","DOI":"10.1103\/PhysRevB.47.558","article-title":"Ab initio molecular dynamics for liquid metals","volume":"47","author":"Kresse","year":"1993","journal-title":"Phys. Rev. B"},{"key":"mlstacadcdbib35","doi-asserted-by":"publisher","first-page":"14251","DOI":"10.1103\/PhysRevB.49.14251","article-title":"Ab initio molecular-dynamics simulation of the liquid-metal\u2013amorphous-semiconductor transition in germanium","volume":"49","author":"Kresse","year":"1994","journal-title":"Phys. Rev. B"},{"key":"mlstacadcdbib36","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/0927-0256(96)00008-0","article-title":"Efficiency of ab initio total energy calculations for metals and semiconductors using a plane-wave basis set","volume":"6","author":"Kresse","year":"1996","journal-title":"Comput. Mater. Sci."},{"key":"mlstacadcdbib37","doi-asserted-by":"publisher","first-page":"11169","DOI":"10.1103\/PhysRevB.54.11169","article-title":"Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set","volume":"54","author":"Kresse","year":"1996","journal-title":"Phys. Rev. B"},{"key":"mlstacadcdbib38","doi-asserted-by":"publisher","first-page":"17953","DOI":"10.1103\/PhysRevB.50.17953","article-title":"Projector augmented-wave method","volume":"50","author":"Bl\u00f6chl","year":"1994","journal-title":"Phys. Rev. B"},{"key":"mlstacadcdbib39","doi-asserted-by":"publisher","first-page":"1758","DOI":"10.1103\/PhysRevB.59.1758","article-title":"From ultrasoft pseudopotentials to the projector augmented-wave method","volume":"59","author":"Kresse","year":"1999","journal-title":"Phys. Rev. B"},{"key":"mlstacadcdbib40","doi-asserted-by":"publisher","first-page":"3865","DOI":"10.1103\/PhysRevLett.77.3865","article-title":"Generalized gradient approximation made simple","volume":"77","author":"Perdew","year":"1996","journal-title":"Phys. Rev. Lett."},{"key":"mlstacadcdbib41","doi-asserted-by":"publisher","first-page":"1396","DOI":"10.1103\/PhysRevLett.78.1396","article-title":"Generalized gradient approximation made simple [Phys. Rev. Lett. 77, 3865 (1996)]","volume":"78","author":"Perdew","year":"1997","journal-title":"Phys. Rev. Lett."},{"key":"mlstacadcdbib42","doi-asserted-by":"publisher","first-page":"1361","DOI":"10.21105\/joss.01361","article-title":"SMACT: semiconducting materials by analogy and chemical theory","volume":"4","author":"Davies","year":"2019","journal-title":"J. Open Source Softw."},{"key":"mlstacadcdbib43","doi-asserted-by":"publisher","DOI":"10.1063\/1.4812323","article-title":"Commentary: the materials project: a materials genome approach to accelerating materials innovation","volume":"1","author":"Jain","year":"2013","journal-title":"APL Mater."},{"key":"mlstacadcdbib44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-020-19964-7","article-title":"Predicting materials properties without crystal structure: deep representation learning from stoichiometry","volume":"11","author":"Goodall","year":"2020","journal-title":"Nat. Commun."},{"key":"mlstacadcdbib45","first-page":"pp 5186","article-title":"Blank language models","author":"Shen","year":"2020"},{"key":"mlstacadcdbib46","article-title":"RoBERTa: a robustly optimized BERT pretraining approach","author":"Liu","year":"2019"},{"key":"mlstacadcdbib47","article-title":"GPT-J-6B: a 6 billion parameter autoregressive language model","author":"Wang","year":"2021"},{"key":"mlstacadcdbib48","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5297715","article-title":"GPT-Neo: large scale autoregressive language modeling with mesh-tensorflow","author":"Black","year":"2021"},{"key":"mlstacadcdbib49","article-title":"The pile: an 800GB dataset of diverse text for language modeling","author":"Gao","year":"2020"},{"key":"mlstacadcdbib50","article-title":"Generative adversarial networks (GAN) based efficient sampling of chemical space for inverse design of inorganic materials","author":"Dan","year":"2019"},{"key":"mlstacadcdbib51","doi-asserted-by":"publisher","first-page":"1931","DOI":"10.3389\/fphar.2020.565644","article-title":"Molecular sets (MOSES): a benchmarking platform for molecular generation models","volume":"11","author":"Polykovskiy","year":"2020","journal-title":"Front. Pharmacol."},{"key":"mlstacadcdbib52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-022-30839-x","article-title":"Language models can learn complex molecular distributions","volume":"13","author":"Flam-Shepherd","year":"2022","journal-title":"Nat. Commun."},{"key":"mlstacadcdbib53","doi-asserted-by":"publisher","first-page":"18836","DOI":"10.1021\/jacs.0c07384","article-title":"Structure-based synthesizability prediction of crystals using partially supervised learning","volume":"142","author":"Jang","year":"2020","journal-title":"J. Am. Chem. Soc."},{"key":"mlstacadcdbib54","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2022.100491","article-title":"Scalable deeper graph neural networks for high-performance materials property prediction","volume":"3","author":"Omee","year":"2022","journal-title":"Patterns"},{"key":"mlstacadcdbib55","doi-asserted-by":"crossref","DOI":"10.1016\/j.commatsci.2022.111496","article-title":"Crystal structure prediction with machine learning-based element substitution","author":"Kusaba","year":"2022"},{"key":"mlstacadcdbib56","article-title":"Alphacrystal: contact map based crystal structure prediction using deep learning","author":"Hu","year":"2021"},{"key":"mlstacadcdbib57","first-page":"pp 22","article-title":"Crystal structure prediction using the USPEX code","author":"Oganov","year":"2012"},{"key":"mlstacadcdbib58","doi-asserted-by":"publisher","DOI":"10.1063\/5.0074677","article-title":"A symmetry-orientated divide-and-conquer method for crystal structure prediction","volume":"156","author":"Shao","year":"2022","journal-title":"J. Chem. Phys."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T12:39:55Z","timestamp":1672922395000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/acadcd"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,5]]},"references-count":58,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1,5]]},"published-print":{"date-parts":[[2023,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/acadcd","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,5]]},"assertion":[{"value":"Material transformers: deep learning language models for generative materials design","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2023 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2022-09-15","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-12-21","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-01-05","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}