{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,19]],"date-time":"2026-04-19T09:00:11Z","timestamp":1776589211898,"version":"3.51.2"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T00:00:00Z","timestamp":1734566400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T00:00:00Z","timestamp":1734566400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009187","name":"RCUK | MRC | Medical Research Foundation","doi-asserted-by":"publisher","award":["MR\/X030075\/1"],"award-info":[{"award-number":["MR\/X030075\/1"]}],"id":[{"id":"10.13039\/501100009187","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000272","name":"DH | National Institute for Health Research","doi-asserted-by":"publisher","award":["NIHR202639"],"award-info":[{"award-number":["NIHR202639"]}],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["72274022"],"award-info":[{"award-number":["72274022"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Automated clinical coding (ACC) has emerged as a promising alternative to manual coding. This study proposes a novel human-in-the-loop (HITL) framework, CliniCoCo. Using deep learning capacities, CliniCoCo focuses on how such ACC systems and human coders can work effectively and efficiently together in real-world settings. Specifically, it implements a series of collaborative strategies at annotation, training and user interaction stages. Extensive experiments are conducted using real-world EMR datasets from Chinese hospitals. With automatically optimised annotation workloads, the model can achieve F1 scores around 0.80\u20130.84. For an EMR with 30% mistaken codes, CliniCoCo can suggest halving the annotations from 3000 admissions with an ignorable 0.01 F1 decrease. In human evaluations, compared to manual coding, CliniCoCo reduces coding time by 40% on average and significantly improves the correction rates on EMR mistakes (e.g., three times better on missing codes). Senior professional coders\u2019 performances can be boosted to more than 0.93 F1 score from 0.72.<\/jats:p>","DOI":"10.1038\/s41746-024-01363-7","type":"journal-article","created":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T08:20:40Z","timestamp":1734596440000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Optimising the paradigms of human AI collaborative clinical coding"],"prefix":"10.1038","volume":"7","author":[{"given":"Yue","family":"Gao","sequence":"first","affiliation":[]},{"given":"Yuepeng","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Minghao","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Jinge","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Yunsoo","family":"Kim","sequence":"additional","affiliation":[]},{"given":"Kaiyin","family":"Zhou","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5672-7448","authenticated-orcid":false,"given":"Miao","family":"Li","sequence":"additional","affiliation":[]},{"given":"Xien","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Xiangling","family":"Fu","sequence":"additional","affiliation":[]},{"given":"Ji","family":"Wu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0213-5668","authenticated-orcid":false,"given":"Honghan","family":"Wu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,12,19]]},"reference":[{"key":"1363_CR1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00705-7","volume":"5","author":"H Dong","year":"2022","unstructured":"Dong, H. et al. Automated clinical coding: what, why, and where we are? NPJ Digit. Med. 5, 159 (2022).","journal-title":"NPJ Digit. Med."},{"key":"1363_CR2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00768-0","volume":"6","author":"KP Venkatesh","year":"2023","unstructured":"Venkatesh, K. P., Raza, M. M. & Kvedar, J. C. Automating the overburdened clinical coding system: challenges and next steps. NPJ Digit. Med. 6, 16 (2023).","journal-title":"NPJ Digit. Med."},{"key":"1363_CR3","unstructured":"National Center for Health Statistics. International classification of diseases (ICD-10-cm\/pcs) transition\u2014background. https:\/\/www.cdc.gov\/nchs\/icd\/icd10cm_pcs_background.htm (2015)."},{"key":"1363_CR4","first-page":"4357","volume":"35","author":"F Teng","year":"2023","unstructured":"Teng, F. et al. A review on deep neural networks for ICD coding. IEEE Trans. Knowl. Data Eng. 35, 4357\u20134375 (2023).","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"1363_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10916-020-1532-x","volume":"44","author":"V Alonso","year":"2020","unstructured":"Alonso, V. et al. Problems and barriers during the process of clinical coding: a focus group study of coders\u2019 perceptions. J. Med. Syst. 44, 1\u20138 (2020).","journal-title":"J. Med. Syst."},{"key":"1363_CR6","unstructured":"National Health Commission, P. R. C. The 2021 National Report on the Services, Quality and Safety in Medical Care System (Scientific and Technical Documentation Press, 2022)."},{"key":"1363_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12913-020-05273-8","volume":"20","author":"L Mo","year":"2020","unstructured":"Mo, L. et al. Feasibility of coding-based Charlson comorbidity index for hospitalized patients in China, a representative developing country. BMC Health Serv. Res. 20, 1\u20137 (2020).","journal-title":"BMC Health Serv. Res."},{"key":"1363_CR8","doi-asserted-by":"publisher","unstructured":"Ji, S. et al. A unified review of deep learning for automated medical coding. ACM Comput. Surv. https:\/\/doi.org\/10.1145\/3664615 (2024).","DOI":"10.1145\/3664615"},{"key":"1363_CR9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00474-9","volume":"4","author":"J Liu","year":"2021","unstructured":"Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. M. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digit. Med. 4, 103 (2021).","journal-title":"NPJ Digit. Med."},{"key":"1363_CR10","unstructured":"Boyle, J. S., Kascenas, A., Lok, P., Liakata, M. & O\u2019Neil, A. Q. Automated clinical coding using off-the-shelf large language models. In Deep Generative Models for Health Workshop NeurIPS 2023 https:\/\/openreview.net\/forum?id=mqnR8rGWkn (2023)."},{"key":"1363_CR11","doi-asserted-by":"publisher","unstructured":"Tsai, S.-C., Huang, C.-W. & Chen, Y.-N. Modeling diagnostic label correlation for automatic ICD coding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Anna, R. et al.) 4043\u20134052 (Association for Computational Linguistics, 2021). https:\/\/doi.org\/10.18653\/v1\/2021.naacl-main.318.","DOI":"10.18653\/v1\/2021.naacl-main.318"},{"key":"1363_CR12","doi-asserted-by":"publisher","first-page":"103728","DOI":"10.1016\/j.jbi.2021.103728","volume":"116","author":"H Dong","year":"2021","unstructured":"Dong, H., Su\u00e1rez-Paniagua, V., Whiteley, W. & Wu, H. Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation. J. Biomed. Inform. 116, 103728 (2021).","journal-title":"J. Biomed. Inform."},{"key":"1363_CR13","doi-asserted-by":"publisher","unstructured":"Vu, T., Nguyen, D. Q. & Nguyen, A. N. A label attention model for ICD coding from clinical text. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, {IJCAI-20}, 3335\u20133341 (2020). https:\/\/doi.org\/10.24963\/ijcai.2020\/461.","DOI":"10.24963\/ijcai.2020\/461"},{"key":"1363_CR14","doi-asserted-by":"publisher","unstructured":"Cao, P. et al. Hypercore: hyperbolic and co-graph representation for automatic ICD coding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (eds Dan, J., Joyce, C., Natalie, S. & Joel, T.) 3105\u20133114 (Association for Computational Linguistics, 2020). https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.282.","DOI":"10.18653\/v1\/2020.acl-main.282"},{"key":"1363_CR15","doi-asserted-by":"crossref","unstructured":"Gao, Y., Fu, X., Liu, X., Zhou, K. & Wu, J. Smp-graph: structure-enhanced unsupervised semantic graph representation for precise medical procedure coding on emrs. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 1303\u20131308 (IEEE, 2021).","DOI":"10.1109\/BIBM52615.2021.9669522"},{"key":"1363_CR16","doi-asserted-by":"publisher","unstructured":"Mullenbach, J., Wiegreffe, S., Duke, J. D., Sun, J. & Eisenstein, J. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Marilyn, W., Heng, J. & Amanda, S.) Vol. 1, 1101\u20131111 (Association for Computational Linguistics, New Orleans, Louisiana, 2018). https:\/\/doi.org\/10.18653\/v1\/N18-1100.","DOI":"10.18653\/v1\/N18-1100"},{"key":"1363_CR17","doi-asserted-by":"publisher","unstructured":"Lu, C., Reddy, C. K., Chakraborty, P., Kleinberg, S. & Ning, Y. Collaborative graph learning with auxiliary text for temporal event prediction in healthcare. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, {IJCAI-21} (ed Zhou, Z-H.) 3529\u20133535 (2021). https:\/\/doi.org\/10.24963\/ijcai.2021\/486.","DOI":"10.24963\/ijcai.2021\/486"},{"key":"1363_CR18","doi-asserted-by":"crossref","unstructured":"Yuan, Z., Tan, C. & Huang, S. Code synonyms do matter: Multiple synonyms matching network for automatic ICD coding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Vol. 2, 808\u2013814 (Association for Computational Linguistics, Dublin, Ireland, 2022). https:\/\/aclanthology.org\/2022.acl-short.91.","DOI":"10.18653\/v1\/2022.acl-short.91"},{"key":"1363_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AE Johnson","year":"2016","unstructured":"Johnson, A. E. et al. Mimic-iii, a freely accessible critical care database. Sci. Data 3, 1\u20139 (2016).","journal-title":"Sci. Data"},{"key":"1363_CR20","doi-asserted-by":"crossref","unstructured":"Gupta, A., Sabirsh, A., W\u00e4hlby, C. & Sintorn, I.-M. Simsearch: a human-in-the-loop learning framework for fast detection of regions of interest in microscopy images. IEEE J. Biomed. Health Inform. 26, 4079\u20134089 (2022).","DOI":"10.1109\/JBHI.2022.3177602"},{"key":"1363_CR21","doi-asserted-by":"publisher","unstructured":"Searle, T., Kraljevic, Z., Bendayan, R., Bean, D. M. & Dobson, R. J. B. Medcattrainer: a biomedical free text annotation interface with active learning and research use case specific customisation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations (eds Sebastian, P. & Ruihong, H.) 139\u2013144 (Association for Computational Linguistics, Hong Kong, China, 2019). https:\/\/doi.org\/10.18653\/v1\/D19-3024.","DOI":"10.18653\/v1\/D19-3024"},{"key":"1363_CR22","doi-asserted-by":"publisher","first-page":"102285","DOI":"10.1016\/j.artmed.2022.102285","volume":"127","author":"FM Calisto","year":"2022","unstructured":"Calisto, F. M., Santiago, C., Nunes, N. J. & Nascimento, J. C. Breastscreening-AI: evaluating medical intelligent agents for human-AI interactions. Artif. Intell. Med. 127, 102285 (2022).","journal-title":"Artif. Intell. Med."},{"key":"1363_CR23","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1093\/jamia\/ocx160","volume":"25","author":"H Wu","year":"2018","unstructured":"Wu, H. et al. Semehr: a general-purpose semantic search system to surface semantic data from clinical notes for tailored care, trial recruitment, and clinical research. J. Am. Med. Inform. Assoc. 25, 530\u2013537 (2018).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"1363_CR24","doi-asserted-by":"publisher","first-page":"103436","DOI":"10.1016\/j.jbi.2020.103436","volume":"107","author":"A Feder","year":"2020","unstructured":"Feder, A. et al. Active deep learning to detect demographic traits in free-form clinical notes. J. Biomed. Inform. 107, 103436 (2020).","journal-title":"J. Biomed. Inform."},{"key":"1363_CR25","doi-asserted-by":"publisher","first-page":"109947","DOI":"10.1016\/j.asoc.2022.109947","volume":"133","author":"Y Gao","year":"2023","unstructured":"Gao, Y., Fu, X., Chen, Y., Guo, C. & Wu, J. Post-pandemic healthcare for covid-19 vaccine: tissue-aware diagnosis of cervical lymphadenopathy via multi-modal ultrasound semantic segmentation. Appl. Soft Comput. 133, 109947 (2023).","journal-title":"Appl. Soft Comput."},{"key":"1363_CR26","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00773-3","volume":"6","author":"A Sylolypavan","year":"2023","unstructured":"Sylolypavan, A., Sleeman, D., Wu, H. & Sim, M. The impact of inconsistent human annotations on AI driven clinical decision making. NPJ Digit. Med. 6, 26 (2023).","journal-title":"NPJ Digit. Med."},{"key":"1363_CR27","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0189-7","volume":"2","author":"BN Patel","year":"2019","unstructured":"Patel, B. N. et al. Human\u2013machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digit. Med. 2, 111 (2019).","journal-title":"NPJ Digit. Med."},{"key":"1363_CR28","unstructured":"Wang, Z. J., Choi, D., Xu, S. & Yang, D. Putting humans in the natural language processing loop: a survey. In Proceedings of the First Workshop on Bridging Human{--}Computer Interaction and Natural Language Processing (eds Su Lin, B., Michael, M., Brendan, O., Hanna, W. & Qian, Y.) 47\u201352 (Association for Computational Linguistics, 2021). https:\/\/aclanthology.org\/2021.hcinlp-1.8."},{"key":"1363_CR29","doi-asserted-by":"publisher","first-page":"102506","DOI":"10.1016\/j.artmed.2023.102506","volume":"138","author":"F Cabitza","year":"2023","unstructured":"Cabitza, F. et al. Rams, hounds and white boxes: investigating human-AI collaboration protocols in medical diagnosis. Artif. Intell. Med. 138, 102506 (2023).","journal-title":"Artif. Intell. Med."},{"key":"1363_CR30","doi-asserted-by":"publisher","first-page":"e13440","DOI":"10.2196\/13440","volume":"21","author":"NT Bott","year":"2019","unstructured":"Bott, N. T. et al. A protocol-driven, bedside digital conversational agent to support nurse teams and mitigate risks of hospitalization in older adults: case control pre-post study. J. Med. Internet Res. 21, e13440 (2019).","journal-title":"J. Med. Internet Res."},{"key":"1363_CR31","unstructured":"National Health Commission, P. R. C. Basic guidelines for medical record documentation. http:\/\/www.nhc.gov.cn\/yzygj\/s3585u\/200904\/ebe63919d67b4c65a76b3f61d1c80cd6.shtml (2010)."},{"key":"1363_CR32","doi-asserted-by":"publisher","first-page":"e23230","DOI":"10.2196\/23230","volume":"9","author":"P-F Chen","year":"2020","unstructured":"Chen, P.-F. et al. Automatic ICD-10 coding and training system: deep neural network based on supervised learning. JMIR Med. Inform. 9, e23230 (2020).","journal-title":"JMIR Med. Inform."},{"key":"1363_CR33","first-page":"19","volume":"49","author":"C Doktorchik","year":"2020","unstructured":"Doktorchik, C., Lu, M., Quan, H., Ringham, C. & Eastwood, C. A qualitative evaluation of clinically coded data quality from health information manager perspectives. Health Inf. Manag. J. 49, 19\u201327 (2020).","journal-title":"Health Inf. Manag. J."},{"key":"1363_CR34","first-page":"5","volume":"49","author":"S Campbell","year":"2020","unstructured":"Campbell, S. & Giadresco, K. Computer-assisted clinical coding: a narrative review of the literature on its benefits, limitations, implementation and impact on clinical coding professionals. Health Inf. Manag. J. 49, 5\u201318 (2020).","journal-title":"Health Inf. Manag. J."},{"key":"1363_CR35","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1177\/1833358319851305","volume":"49","author":"S Campbell","year":"2020","unstructured":"Campbell, S. & Giadresco, K. Computer-assisted clinical coding: a narrative review of the literature on its benefits, limitations, implementation and impact on clinical coding professionals. HIM J. 49, 5\u201318 (2020).","journal-title":"HIM J."},{"key":"1363_CR36","first-page":"144","volume":"52","author":"M Jebraeily","year":"2023","unstructured":"Jebraeily, M., Farzi, J., Fozoonkhah, S. & Sheikhtaheri, A. Identification of root causes of clinical coding problems in Iranian hospitals. Health Inf. Manag. J. 52, 144\u2013150 (2023).","journal-title":"Health Inf. Manag. J."},{"key":"1363_CR37","doi-asserted-by":"crossref","unstructured":"Shepheard, J. Clinical coding and the quality and integrity of health data. Health Information Management Journal Vol. 49, 3\u20134 (SAGE Publications Sage UK: London, England, 2020).","DOI":"10.1177\/1833358319874008"},{"key":"1363_CR38","first-page":"69","volume":"49","author":"P Hay","year":"2020","unstructured":"Hay, P., Wilton, K., Barker, J., Mortley, J. & Cumerlato, M. The importance of clinical documentation improvement for Australian hospitals. Health Inf. Manag. J. 49, 69\u201373 (2020).","journal-title":"Health Inf. Manag. J."},{"key":"1363_CR39","doi-asserted-by":"publisher","first-page":"1096","DOI":"10.1097\/SLA.0000000000000851","volume":"261","author":"S Nouraei","year":"2015","unstructured":"Nouraei, S. et al. A study of clinical coding accuracy in surgery: implications for the use of administrative big data for outcomes management. Ann. Surg. 261, 1096\u20131107 (2015).","journal-title":"Ann. Surg."},{"key":"1363_CR40","doi-asserted-by":"publisher","unstructured":"Gao, T., Yao, X. & Chen, D. Simcse: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (eds Marie-Francine, M., Xuanjing, H., Lucia, S. & Scott Wen-tau, Y.) 6894\u20136910 (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021). https:\/\/doi.org\/10.18653\/v1\/2021.emnlp-main.552.","DOI":"10.18653\/v1\/2021.emnlp-main.552"},{"key":"1363_CR41","unstructured":"Li, Y., Tarlow, D., Brockschmidt, M. & Zemel, R. S. Gated Graph Sequence Neural Networks. InternationalConference on Learning Representations (2016)."},{"key":"1363_CR42","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein, J. et al.) 4171\u20134186 (Association for Computational Linguistics, Minneapolis, Minnesota, 2019)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01363-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01363-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01363-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T09:03:34Z","timestamp":1734599014000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01363-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,19]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["1363"],"URL":"https:\/\/doi.org\/10.1038\/s41746-024-01363-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,19]]},"assertion":[{"value":"20 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 November 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"368"}}