{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T16:57:03Z","timestamp":1777568223074,"version":"3.51.4"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T00:00:00Z","timestamp":1665964800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["81825020"],"award-info":[{"award-number":["81825020"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82150208"],"award-info":[{"award-number":["82150208"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Lingang Laboratory","award":["LG-QS-202206-02"],"award-info":[{"award-number":["LG-QS-202206-02"]}]},{"name":"National Program for Special Supports of Eminent Professionals"},{"DOI":"10.13039\/501100018619","name":"National Program for Support of Top-notch Young Professionals","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100018619","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Identification of new chemical compounds with desired structural diversity and biological properties plays an essential role in drug discovery, yet the construction of such a potential space with elements of \u2018near-drug\u2019 properties is still a challenging task. In this work, we proposed a multimodal chemical information reconstruction system to automatically process, extract and align heterogeneous information from the text descriptions and structural images of chemical patents. Our key innovation lies in a heterogeneous data generator that produces cross-modality training data in the form of text descriptions and Markush structure images, from which a two-branch model with image- and text-processing units can then learn to both recognize heterogeneous chemical entities and simultaneously capture their correspondence. In particular, we have collected chemical structures from ChEMBL database and chemical patents from the European Patent Office and the US Patent and Trademark Office using keywords \u2018A61P, compound, structure\u2019 in the years from 2010 to 2020, and generated heterogeneous chemical information datasets with 210K structural images and 7818 annotated text snippets. Based on the reconstructed results and substituent replacement rules, structural libraries of a huge number of near-drug compounds can be generated automatically. In quantitative evaluations, our model can correctly reconstruct 97% of the molecular images into structured format and achieve an F1-score around 97\u201398% in the recognition of chemical entities, which demonstrated the effectiveness of our model in automatic information extraction from chemical patents, and hopefully transforming them to a user-friendly, structured molecular database enriching the near-drug space to realize the intelligent retrieval technology of chemical knowledge.<\/jats:p>","DOI":"10.1093\/bib\/bbac461","type":"journal-article","created":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T23:26:32Z","timestamp":1666049192000},"source":"Crossref","is-referenced-by-count":15,"title":["Multi-modal chemical information reconstruction from images and texts for exploring the near-drug space"],"prefix":"10.1093","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2059-0629","authenticated-orcid":false,"given":"Jie","family":"Wang","sequence":"first","affiliation":[{"name":"Shanghai Key Laboratory of New Drug Design, School of Pharmacy, East China University of Science & Technology , Shanghai 200237 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zihao","family":"Shen","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of New Drug Design, School of Pharmacy, East China University of Science & Technology , Shanghai 200237 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yichen","family":"Liao","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of New Drug Design, School of Pharmacy, East China University of Science & Technology , Shanghai 200237 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhen","family":"Yuan","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of New Drug Design, School of Pharmacy, East China University of Science & Technology , Shanghai 200237 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4414-237X","authenticated-orcid":false,"given":"Shiliang","family":"Li","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of New Drug Design, School of Pharmacy, East China University of Science & Technology , Shanghai 200237 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gaoqi","family":"He","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University , Shanghai 200062 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Man","family":"Lan","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University , Shanghai 200062 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuhong","family":"Qian","sequence":"additional","affiliation":[{"name":"Innovation Center for AI and Drug Discovery, East China Normal University , Shanghai 200062 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kai","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University , Shanghai 200062 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Honglin","family":"Li","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of New Drug Design, School of Pharmacy, East China University of Science & Technology , Shanghai 200237 , China"},{"name":"Innovation Center for AI and Drug Discovery, East China Normal University , Shanghai 200062 , China"},{"name":"Lingang Laboratory , Shanghai 200031 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2022,10,17]]},"reference":[{"issue":"5460","key":"2022112111201386900_ref1","doi-asserted-by":"crossref","first-page":"1960","DOI":"10.1126\/science.287.5460.1960","article-title":"Drug discovery: a historical perspective","volume":"287","author":"Drews","year":"2000","journal-title":"Science"},{"key":"2022112111201386900_ref2","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1016\/j.drudis.2020.01.012","article-title":"Flexibility in early drug discovery: focus on the beyond-Rule-of-5 chemical space","volume":"25","author":"Caron","year":"2020","journal-title":"Drug Discov Today"},{"key":"2022112111201386900_ref3","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1038\/nature03192","article-title":"Chemical space and biology","volume":"432","author":"Dobson","year":"2005","journal-title":"Nature"},{"key":"2022112111201386900_ref4","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1002\/(SICI)1098-1128(199601)16:1<3::AID-MED1>3.0.CO;2-6","article-title":"The art and practice of structure-based drug design: a molecular modeling perspective","volume":"16","author":"Bohacek","year":"1996","journal-title":"Med Res Rev"},{"key":"2022112111201386900_ref5","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1016\/j.trechm.2020.11.004","article-title":"Defining and exploring chemical spaces","volume":"3","author":"Coley","year":"2021","journal-title":"Trends Chem"},{"key":"2022112111201386900_ref6","doi-asserted-by":"crossref","first-page":"D1074","DOI":"10.1093\/nar\/gkx1037","article-title":"DrugBank 5.0: a major update to the DrugBank database for 2018","volume":"46","author":"Wishart","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2022112111201386900_ref7","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1007\/s10822-013-9672-4","article-title":"Estimation of the size of drug-like chemical space based on GDB-17 data","volume":"27","author":"Polishchuk","year":"2013","journal-title":"J Comput Aided Mol Des"},{"key":"2022112111201386900_ref8","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1038\/nchembio.180","article-title":"Quantifying biogenic bias in screening libraries","volume":"5","author":"Hert","year":"2009","journal-title":"Nat Chem Biol"},{"key":"2022112111201386900_ref9","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1038\/s41570-018-0066-y","article-title":"How to explore chemical space using algorithms and automation","volume":"3","author":"Gromski","year":"2019","journal-title":"Nat Rev Chem"},{"key":"2022112111201386900_ref10","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1016\/j.drudis.2019.02.013","article-title":"The next level in chemical space navigation: going far beyond enumerable compound libraries","volume":"24","author":"Hoffmann","year":"2019","journal-title":"Drug Discov Today"},{"key":"2022112111201386900_ref11","doi-asserted-by":"crossref","first-page":"2418","DOI":"10.1038\/nprot.2007.355","article-title":"Patent searches as a complement to literature searches in the life sciences\u2014a how-to\u2019 tutorial","volume":"2","author":"Seeber","year":"2007","journal-title":"Nat Protoc"},{"key":"2022112111201386900_ref12","doi-asserted-by":"crossref","first-page":"54153","DOI":"10.1039\/C7RA11831G","article-title":"Open chemoinformatic resources to explore the structure, properties and chemical space of molecules","volume":"7","author":"Gonz\u00e1lez-Medina","year":"2017","journal-title":"RSC Adv"},{"key":"2022112111201386900_ref13","article-title":"Automatic identification of relevant chemical compounds from patents","volume":"2019","author":"Saber","year":"2019","journal-title":"Database (Oxford)"},{"key":"2022112111201386900_ref14","first-page":"1","article-title":"OSCAR4: a flexible architecture for chemical text-mining","volume":"3","author":"Jessop","year":"2011","journal-title":"J Chem"},{"key":"2022112111201386900_ref15","doi-asserted-by":"crossref","first-page":"1633","DOI":"10.1093\/bioinformatics\/bts183","article-title":"ChemSpot: a hybrid system for chemical named entity recognition","volume":"28","author":"Rockt\u00e4schel","year":"2012","journal-title":"Bioinformatics"},{"key":"2022112111201386900_ref16","first-page":"S1","article-title":"LeadMine: a grammar and dictionary driven approach to entity recognition","volume":"7","author":"Lowe","year":"2015","journal-title":"J Chem"},{"key":"2022112111201386900_ref17","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1126\/science.abc2986","article-title":"A universal system for digitization and automatic execution of the chemical synthesis literature","volume":"370","author":"","year":"2020","journal-title":"Science"},{"key":"2022112111201386900_ref18","doi-asserted-by":"crossref","first-page":"3601","DOI":"10.1038\/s41467-020-17266-6","article-title":"Automated extraction of chemical synthesis actions from experimental procedures","volume":"11","author":"Vaucher","year":"2020","journal-title":"Nat Commun"},{"key":"2022112111201386900_ref19","doi-asserted-by":"crossref","first-page":"1894","DOI":"10.1021\/acs.jcim.6b00207","article-title":"ChemDataExtractor: a toolkit for automated extraction of chemical information from the scientific literature","volume":"56","author":"Swain","year":"2016","journal-title":"J Chem Inf Model"},{"issue":"6423","key":"2022112111201386900_ref20","doi-asserted-by":"crossref","first-page":"eaav2211","DOI":"10.1126\/science.aav2211","article-title":"Organic synthesis in a modular robotic system driven by a chemical programming language","volume":"363","author":"Steiner","year":"2019","journal-title":"Science"},{"key":"2022112111201386900_ref21","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1038\/nature25978","article-title":"Planning chemical syntheses with deep neural networks and symbolic AI","volume":"555","author":"Segler","year":"2018","journal-title":"Nature"},{"key":"2022112111201386900_ref22","doi-asserted-by":"crossref","first-page":"e107477","DOI":"10.1371\/journal.pone.0107477","article-title":"Annotated chemical patent corpus: a gold standard for text mining","volume":"9","author":"Akhondi","year":"2014","journal-title":"PLoS One"},{"key":"2022112111201386900_ref23","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1021\/ci800067r","article-title":"Optical structure recognition software to recover chemical information: OSRA, an open source solution","volume":"49","author":"Filippov","year":"2009","journal-title":"J Chem Inf Model"},{"key":"2022112111201386900_ref24","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1186\/s13321-020-00469-w","article-title":"DECIMER: towards deep learning for chemical image recognition","volume":"12","author":"Rajan","year":"2020","journal-title":"J Chem"},{"key":"2022112111201386900_ref25","doi-asserted-by":"crossref","first-page":"e202100069","DOI":"10.1002\/cmtd.202100069","article-title":"Image2SMILES: transformer-based molecular optical recognition engine","volume":"2","author":"Khokhlov","year":"2022","journal-title":"Chem Methods"},{"key":"2022112111201386900_ref26","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1186\/s13321-021-00538-8","article-title":"DECIMER 1.0: deep learning for chemical image recognition using transformers","volume":"13","author":"Rajan","year":"2021","journal-title":"J Chem"},{"key":"2022112111201386900_ref27","doi-asserted-by":"crossref","first-page":"10622","DOI":"10.1039\/D1SC02957F","article-title":"ChemPix: automated recognition of hand-drawn hydrocarbon structures using deep learning","volume":"12","author":"Weir","year":"2021","journal-title":"Chem Sci"},{"key":"2022112111201386900_ref28","first-page":"2949","article-title":"Multimodal learning with deep Boltzmann machines","volume":"15","author":"Srivastava","year":"2012","journal-title":"J Mach Learn Res"},{"key":"2022112111201386900_ref29","first-page":"1102","article-title":"Explainable multimodal machine learning model for classifying pregnancy drug safety","volume":"38","author":"Guy","year":"2021","journal-title":"Bioinformatics"},{"key":"2022112111201386900_ref30","doi-asserted-by":"crossref","first-page":"862","DOI":"10.1038\/s41467-022-28494-3","article-title":"A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals","volume":"13","author":"Zeng","year":"2022","journal-title":"Nat Commun"},{"key":"2022112111201386900_ref31","first-page":"5","article-title":"RDKit: open-source cheminformatics","volume":"03","author":"Landrum","year":"2022","journal-title":"Release"},{"key":"2022112111201386900_ref32","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/ICDAR.2007.4376991","volume-title":"Proceedings of the Ninth International Conference on Document Analysis & Recognition (ICDAR 2007)","author":"Smith","year":"2007"},{"key":"2022112111201386900_ref33","volume-title":"YEDDA: A Lightweight Collaborative Text Span Annotation Tool","author":"Yang","year":"2018"},{"key":"2022112111201386900_ref34","volume-title":"Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks","author":"Reimers","year":"2017"},{"key":"2022112111201386900_ref35","author":"Carpenter","year":"2009"},{"key":"2022112111201386900_ref36","volume-title":"UNET 3+: A Full-Scale Connected UNet for Medical Image Segmentation","author":"Huang","year":"2020"},{"key":"2022112111201386900_ref37","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal loss for dense object detection","volume":"42","author":"Lin","year":"2020","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2022112111201386900_ref38","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P16-1101","volume-title":"End-to-End Sequence Labeling via Bi-directional LSTM-CNNS-CRF","author":"Ma","year":"2016"},{"key":"2022112111201386900_ref39","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1016\/j.procs.2017.08.009","article-title":"Comparative study of word embedding methods in topic segmentation","volume":"112","author":"Naili","year":"2017","journal-title":"Proc Comput Sci"},{"key":"2022112111201386900_ref40","first-page":"282","volume-title":"Proceedings of the 18th International Conference on Machine Learning, Proceedings of Machine Learning Research","author":"Lafferty","year":"2001"},{"key":"2022112111201386900_ref41","volume-title":"Adam: A Method for Stochastic Optimization","author":"Kingma","year":"2014"},{"key":"2022112111201386900_ref42","first-page":"32","article-title":"Chemical structure recognition: a rule-based approach","volume":"8297","author":"Sadawi","year":"2012","journal-title":"Proc SPIE"},{"key":"2022112111201386900_ref43","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"2022112111201386900_ref44","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1145\/2939672.2939778","volume-title":"The 22nd ACM SIGKDD International Conference, Proceeding of Knowledge Discovery and Data Mining","author":"Ribeiro","year":"2016"},{"key":"2022112111201386900_ref45","volume-title":"Named Entity Recognition in Chemical Patents Using Ensemble of Contextual Language Models","author":"Copara","year":"2020"},{"key":"2022112111201386900_ref46","first-page":"185","volume-title":"The 2012 Workshop on BioNLP, Proceeding of Biomedical Natural Language Processing","author":"Kim","year":"2012"},{"key":"2022112111201386900_ref47","article-title":"Derivatives of pyridoxine for inhibiting HIV integrase","author":"Stranix","year":"2014","journal-title":"US08664248B2"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/6\/bbac461\/47144387\/bbac461.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/6\/bbac461\/47144387\/bbac461.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T11:26:19Z","timestamp":1669029979000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbac461\/6761958"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,17]]},"references-count":47,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11,19]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbac461","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,11]]},"published":{"date-parts":[[2022,10,17]]},"article-number":"bbac461"}}