{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T02:24:38Z","timestamp":1776651878527,"version":"3.51.2"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T00:00:00Z","timestamp":1576540800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T00:00:00Z","timestamp":1576540800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR002062"],"award-info":[{"award-number":["U01TR002062"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Data is foundational to high-quality artificial intelligence (AI). Given that a substantial amount of clinically relevant information is embedded in unstructured data, natural language processing (NLP) plays an essential role in extracting valuable information that can benefit decision making, administration reporting, and research. Here, we share several desiderata pertaining to development and usage of NLP systems, derived from two decades of experience implementing clinical NLP at the Mayo Clinic, to inform the healthcare AI community. Using a framework, we developed as an example implementation, the desiderata emphasize the importance of a user-friendly platform, efficient collection of domain expert inputs, seamless integration with clinical data, and a highly scalable computing infrastructure.<\/jats:p>","DOI":"10.1038\/s41746-019-0208-8","type":"journal-article","created":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T11:02:49Z","timestamp":1576580569000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":106,"title":["Desiderata for delivering NLP to accelerate healthcare AI advancement and a Mayo Clinic NLP-as-a-service implementation"],"prefix":"10.1038","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9090-8028","authenticated-orcid":false,"given":"Andrew","family":"Wen","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1691-5179","authenticated-orcid":false,"given":"Sunyang","family":"Fu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9191-3897","authenticated-orcid":false,"given":"Sungrim","family":"Moon","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9168-5855","authenticated-orcid":false,"given":"Mohamed","family":"El Wazir","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9202-0233","authenticated-orcid":false,"given":"Andrew","family":"Rosenbaum","sequence":"additional","affiliation":[]},{"given":"Vinod C.","family":"Kaggal","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9763-1164","authenticated-orcid":false,"given":"Sijia","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Sunghwan","family":"Sohn","sequence":"additional","affiliation":[]},{"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Jungwei","family":"Fan","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,12,17]]},"reference":[{"key":"208_CR1","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","volume":"46","author":"S Nemati","year":"2018","unstructured":"Nemati, S. et al. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit. Care Med. 46, 547\u2013553 (2018).","journal-title":"Crit. Care Med."},{"key":"208_CR2","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/j.cmpb.2019.01.013","volume":"173","author":"CC Wu","year":"2019","unstructured":"Wu, C. C. et al. An artificial intelligence approach to early predict non-ST-elevation myocardial infarction patients with chest pain. Comput. Methods Prog. Biomed. 173, 109\u2013117 (2019).","journal-title":"Comput. Methods Prog. Biomed."},{"key":"208_CR3","doi-asserted-by":"publisher","first-page":"277","DOI":"10.5853\/jos.2017.02054","volume":"19","author":"EJ Lee","year":"2017","unstructured":"Lee, E. J., Kim, Y. H., Kim, N. & Kang, D. W. Deep into the brain: artificial intelligence in stroke imaging. J. Stroke 19, 277\u2013285 (2017).","journal-title":"J. Stroke"},{"key":"208_CR4","doi-asserted-by":"publisher","first-page":"3970","DOI":"10.1245\/s10434-015-4475-6","volume":"22","author":"A Enshaei","year":"2015","unstructured":"Enshaei, A., Robson, C. N. & Edmondson, R. J. Artificial intelligence systems as prognostic and predictive tools in ovarian cancer. Ann. Surg. Oncol. 22, 3970\u20133975 (2015).","journal-title":"Ann. Surg. Oncol."},{"key":"208_CR5","doi-asserted-by":"publisher","first-page":"2366","DOI":"10.1001\/jama.2016.17563","volume":"316","author":"TY Wong","year":"2016","unstructured":"Wong, T. Y. & Bressler, N. M. Artificial intelligence with deep learning technology looks into diabetic retinopathy screening. JAMA 316, 2366\u20132367 (2016).","journal-title":"JAMA"},{"key":"208_CR6","first-page":"14","volume":"9","author":"F Martin-Sanchez","year":"2014","unstructured":"Martin-Sanchez, F. & Verspoor, K. Big data in medicine is driving big changes. Yearb. Med Inf. 9, 14\u201320 (2014).","journal-title":"Yearb. Med Inf."},{"key":"208_CR7","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","volume":"77","author":"Y Wang","year":"2018","unstructured":"Wang, Y. et al. Clinical information extraction applications: a literature review. J. Biomed. Inform. 77, 34\u201349 (2018).","journal-title":"J. Biomed. Inform."},{"key":"208_CR8","doi-asserted-by":"publisher","first-page":"1753","DOI":"10.1016\/j.jvs.2016.11.031","volume":"65","author":"N Afzal","year":"2017","unstructured":"Afzal, N. et al. Mining peripheral arterial disease cases from narrative clinical notes using natural language processing. J. Vasc. Surg. 65, 1753\u20131761 (2017).","journal-title":"J. Vasc. Surg."},{"key":"208_CR9","doi-asserted-by":"publisher","first-page":"567","DOI":"10.1007\/s10278-014-9762-4","volume":"28","author":"R Lacson","year":"2015","unstructured":"Lacson, R. et al. Evaluation of an automated information extraction tool for imaging data elements to populate a breast cancer screening registry. J. Digit. Imaging 28, 567\u2013575 (2015).","journal-title":"J. Digit. Imaging"},{"key":"208_CR10","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1136\/svn-2017-000101","volume":"2","author":"F Jiang","year":"2017","unstructured":"Jiang, F. et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc. Neurol. 2, 230\u2013243 (2017).","journal-title":"Stroke Vasc. Neurol."},{"key":"208_CR11","doi-asserted-by":"publisher","first-page":"124","DOI":"10.4338\/ACI-2016-07-RA-0114","volume":"26","author":"M Scheitel","year":"2017","unstructured":"Scheitel, M. et al. Effect of a novel clinical decision support tool on the efficiency and accuracy of treatment recommendations for cholesterol management. Appl. Clin. Inform. 26, 124\u2013136 (2017).","journal-title":"Appl. Clin. Inform."},{"issue":"3","key":"208_CR12","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1093\/jamia\/ocx138","volume":"25","author":"Sunghwan Sohn","year":"2017","unstructured":"Sohn, S. et al. Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions. J. Am. Med. Inform. Assoc. https:\/\/doi.org\/10.1093\/jamia\/ocx138 (2017).","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"Suppl","key":"208_CR13","doi-asserted-by":"publisher","first-page":"S189","DOI":"10.1016\/j.jbi.2015.07.008","volume":"58","author":"K Zheng","year":"2015","unstructured":"Zheng, K. et al. Ease of adoption of clinical natural language processing software: an evaluation of five systems. J. Biomed. Inf. 58(Suppl), S189\u2013S196 (2015).","journal-title":"J. Biomed. Inf."},{"key":"208_CR14","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1016\/j.ijmedinf.2017.12.024","volume":"111","author":"N Afzal","year":"2018","unstructured":"Afzal, N. et al. Natural language processing of clinical notes for identification of critical limb ischemia. Int. J. Med. Inf. 111, 83\u201389 (2018).","journal-title":"Int. J. Med. Inf."},{"key":"208_CR15","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1016\/j.surg.2018.05.043","volume":"164","author":"D Chen","year":"2018","unstructured":"Chen, D. et al. Postoperative bleeding risk prediction for patients undergoing colorectal surgery. Surgery 164, 1209\u20131216 (2018).","journal-title":"Surgery"},{"key":"208_CR16","doi-asserted-by":"publisher","DOI":"10.2196\/12109","volume":"7","author":"S Fu","year":"2019","unstructured":"Fu, S. et al. Natural language processing for the identification of silent brain infarcts from neuroimaging reports. JMIR Med. Inf. 7, e12109 (2019).","journal-title":"JMIR Med. Inf."},{"key":"208_CR17","first-page":"13","volume":"8","author":"VC Kaggal","year":"2016","unstructured":"Kaggal, V. C. et al. Toward a learning health-care system - knowledge delivery at the point of care empowered by big data and NLP. Biomed. Inf. Insights 8, 13\u201322 (2016).","journal-title":"Biomed. Inf. Insights"},{"key":"208_CR18","first-page":"522","volume":"2017","author":"F Shen","year":"2017","unstructured":"Shen, F. et al. Populating physician biographical pages based on EMR data. AMIA Jt. Summits Transl. Sci. Proc. 2017, 522\u2013530 (2017).","journal-title":"AMIA Jt. Summits Transl. Sci. Proc."},{"key":"208_CR19","doi-asserted-by":"publisher","DOI":"10.2196\/13043","volume":"21","author":"J McPadden","year":"2019","unstructured":"McPadden, J. et al. Health care and precision medicine research: analysis of a scalable data science platform. J. Med. Internet Res. 21, e13043 (2019).","journal-title":"J. Med. Internet Res."},{"key":"208_CR20","doi-asserted-by":"publisher","first-page":"6120820","DOI":"10.1155\/2017\/6120820","volume":"2017","author":"D Chrimes","year":"2017","unstructured":"Chrimes, D. & Zamani, H. Using distributed data over HBase in big data analytics pfor clinical services. Comput Math. Methods Med. 2017, 6120820 (2017).","journal-title":"Comput Math. Methods Med."},{"key":"208_CR21","first-page":"196858","volume":"2014","author":"Y Sun","year":"2014","unstructured":"Sun, Y., Xiong, Y., Xu, Q. & Wei, D. A hadoop-based method to predict potential effective drug combination. Biomed. Res. Int. 2014, 196858 (2014).","journal-title":"Biomed. Res. Int."},{"key":"208_CR22","first-page":"384","volume":"2017","author":"M Adibuzzaman","year":"2017","unstructured":"Adibuzzaman, M., DeLaurentis, P., Hill, J. & Benneyworth, B. D. Big data in healthcare \u2013 the promises, challenges and opportunities from a research perspective: A case study with a model database. AMIA Annu. Symp. Proc. 2017, 384\u2013392 (2017).","journal-title":"AMIA Annu. Symp. Proc."},{"key":"208_CR23","unstructured":"Apache Lucene (The Apache Software Foundation)."},{"key":"208_CR24","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1145\/1132956.1132959","volume":"38","author":"J Zobel","year":"2006","unstructured":"Zobel, J. & Moffat, A. Inverted files for text search engines. ACM Comput. Surv. (CSUR) 38, 6 (2006).","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"208_CR25","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1145\/2934664","volume":"59","author":"M Zaharia","year":"2016","unstructured":"Zaharia, M. et al. Apache spark. Commun. ACM 59, 56\u201365 (2016).","journal-title":"Commun. ACM"},{"key":"208_CR26","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1197\/jamia.M2844","volume":"16","author":"M Torii","year":"2009","unstructured":"Torii, M., Hu, Z., Wu, C. H. & Liu, H. BioTagger-GM: a gene\/protein name recognition system. J. Am. Med. Inf. Assoc. 16, 247\u2013255 (2009).","journal-title":"J. Am. Med. Inf. Assoc."},{"key":"208_CR27","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1017\/S1351324904003523","volume":"10","author":"D Ferrucci","year":"2004","unstructured":"Ferrucci, D. & Lally, A. UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat. Lang. Eng. 10, 327\u2013348 (2004).","journal-title":"Nat. Lang. Eng."},{"key":"208_CR28","doi-asserted-by":"publisher","first-page":"1626","DOI":"10.14778\/1687553.1687609","volume":"2","author":"A Thusoo","year":"2009","unstructured":"Thusoo, A. et al. Hive. Proc. VLDB Endow. 2, 1626\u20131629 (2009).","journal-title":"Proc. VLDB Endow."},{"key":"208_CR29","doi-asserted-by":"publisher","unstructured":"Vavilapalli, V. K. et al. Apache Hadoop YARN: Yet Another Resource Negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing, https:\/\/doi.org\/10.1145\/2523616.2523633 (2013).","DOI":"10.1145\/2523616.2523633"},{"key":"208_CR30","unstructured":"Wood, D., Loy, M. & Eckstein, R. Java Swing (O\u2019Reilly Media, Inc, 1998)."},{"key":"208_CR31","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1016\/j.jbi.2009.05.002","volume":"42","author":"H Harkema","year":"2009","unstructured":"Harkema, H., Dowling, J. N., Thornblade, T. & Chapman, W. W. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J. Biomed. Inf. 42, 839\u2013851 (2009).","journal-title":"J. Biomed. Inf."},{"key":"208_CR32","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-014-0119-0","volume":"12","author":"JP Fanning","year":"2014","unstructured":"Fanning, J. P., Wong, A. A. & Fraser, J. F. The epidemiology of silent brain infarction: a systematic review of population-based cohorts. BMC Med. 12, 119 (2014).","journal-title":"BMC Med."},{"key":"208_CR33","doi-asserted-by":"publisher","first-page":"3461","DOI":"10.1161\/STROKEAHA.114.005919","volume":"45","author":"JP Fanning","year":"2014","unstructured":"Fanning, J. P., Wesley, A. J., Wong, A. A. & Fraser, J. F. Emerging spectra of silent brain infarction. Stroke 45, 3461\u20133471 (2014).","journal-title":"Stroke"},{"key":"208_CR34","doi-asserted-by":"publisher","first-page":"611","DOI":"10.1016\/S1474-4422(07)70170-9","volume":"6","author":"SE Vermeer","year":"2007","unstructured":"Vermeer, S. E., Longstreth, W. T. Jr & Koudstaal, P. J. Silent brain infarcts: a systematic review. Lancet Neurol. 6, 611\u2013619 (2007).","journal-title":"Lancet Neurol."},{"key":"208_CR35","first-page":"1243","volume":"2017","author":"S Malmasi","year":"2018","unstructured":"Malmasi, S. et al. Extracting healthcare quality information from unstructured data. American Medical Informatics Association Annual Symposium proceedings. AMIA Symp. 2017, 1243\u20131252 (2018).","journal-title":"AMIA Symp."},{"key":"208_CR36","doi-asserted-by":"publisher","first-page":"1364","DOI":"10.1093\/jamia\/ocz068","volume":"26","author":"M Afshar","year":"2019","unstructured":"Afshar, M. et al. Development and application of a high throughput natural language processing architecture to convert all clinical documents in a clinical data warehouse into standardized medical vocabularies. J. Am. Med. Inform. Assoc. 26, 1364\u20131369 (2019).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"208_CR37","first-page":"1372","volume":"2017","author":"KJ Peterson","year":"2018","unstructured":"Peterson, K. J., Jiang, G., Brue, S. M., Shen, F. & Liu, H. Mining hierarchies and similarity clusters from value set repositories. American Medical Informatics Association Annual Symposium proceedings. AMIA Symp. 2017, 1372\u20131381 (2018).","journal-title":"AMIA Symp."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0208-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0208-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0208-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T18:36:50Z","timestamp":1671302210000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0208-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,17]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["208"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0208-8","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12,17]]},"assertion":[{"value":"5 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 November 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 December 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"130"}}