{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T01:52:42Z","timestamp":1771379562605,"version":"3.50.1"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,7,27]],"date-time":"2021-07-27T00:00:00Z","timestamp":1627344000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,27]],"date-time":"2021-07-27T00:00:00Z","timestamp":1627344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Technology assistance of pharmacist verification tasks through the use of machine intelligence has the potential to detect dangerous and costly pharmacy dispensing errors. National Drug Codes (NDC) are unique numeric identifiers of prescription drug products for the United States Food and Drug Administration. The physical form of the medication, often tablets and capsules, captures the unique features of the NDC product to help ensure patients receive the same medication product inside their prescription bottle as is found on the label from a pharmacy. We report and evaluate using an automated check to predict the shape, color, and NDC for images showing a pile of pills inside a prescription bottle. In a test set containing 65,274 images of 345 NDC classes, overall macro-average precision was 98.5%. Patterns of incorrect NDC predictions based on similar colors, shapes, and imprints of pills were identified and recommendations to improve the model are provided.<\/jats:p>","DOI":"10.1038\/s41746-021-00483-8","type":"journal-article","created":{"date-parts":[[2021,7,27]],"date-time":"2021-07-27T10:02:50Z","timestamp":1627380170000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Performance evaluation of a prescription medication image classification model: an observational cohort"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8774-793X","authenticated-orcid":false,"given":"Corey A.","family":"Lester","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8219-5638","authenticated-orcid":false,"given":"Jiazhao","family":"Li","sequence":"additional","affiliation":[]},{"given":"Yuting","family":"Ding","sequence":"additional","affiliation":[]},{"given":"Brigid","family":"Rowell","sequence":"additional","affiliation":[]},{"given":"Jessie \u2018Xi\u2019","family":"Yang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4546-324X","authenticated-orcid":false,"given":"Raed Al","family":"Kontar","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,7,27]]},"reference":[{"key":"483_CR1","doi-asserted-by":"publisher","first-page":"864","DOI":"10.1016\/j.sapharm.2018.11.013","volume":"15","author":"A-R Holmstr\u00f6m","year":"2019","unstructured":"Holmstr\u00f6m, A.-R. et al. Inter-rater reliability of medication error classification in a voluntary patient safety incident reporting system HaiPro in Finland. Res. Soc. Adm. Pharm. 15, 864\u2013872 (2019).","journal-title":"Res. Soc. Adm. Pharm."},{"key":"483_CR2","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1016\/j.sapharm.2018.09.013","volume":"15","author":"CA Lester","year":"2019","unstructured":"Lester, C. A., Kessler, J. M., Modisett, T. & Chui, M. A. A text mining analysis of medication quality related event reports from community pharmacies. Res. Soc. Adm. Pharm. 15, 845\u2013851 (2019).","journal-title":"Res. Soc. Adm. Pharm."},{"key":"483_CR3","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1211\/ijpp.17.1.0004","volume":"17","author":"KL James","year":"2009","unstructured":"James, K. L. et al. Incidence, type and causes of dispensing errors: a review of the literature. Int. J. Pharm. Pract. 17, 9\u201330 (2009).","journal-title":"Int. J. Pharm. Pract."},{"key":"483_CR4","doi-asserted-by":"publisher","unstructured":"Reiner, G., Pierce, S. L. & Flynn, J. Wrong drug and wrong dose dispensing errors identified in pharmacist professional liability claims. J. Am. Pharm. Assoc. 60, e50\u2013e56 https:\/\/doi.org\/10.1016\/j.japh.2020.02.027 (2020).","DOI":"10.1016\/j.japh.2020.02.027"},{"key":"483_CR5","doi-asserted-by":"publisher","first-page":"e97","DOI":"10.1331\/JAPhA.2012.11119","volume":"52","author":"DM Scott","year":"2012","unstructured":"Scott, D. M., Friesner, D. L., Rathke, A. M., Peterson, C. D. & Anderson, H. C. Differences in medication errors between central and remote site telepharmacies. J. Am. Pharm. Assoc. 52, e97\u2013e104 (2012).","journal-title":"J. Am. Pharm. Assoc."},{"key":"483_CR6","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1016\/j.sapharm.2013.05.007","volume":"10","author":"MA Chui","year":"2014","unstructured":"Chui, M. A., Look, K. A. & Mott, D. A. The association of subjective workload dimensions on quality of care and pharmacist quality of work life. Res. Soc. Adm. Pharm. 10, 328\u2013340 (2014).","journal-title":"Res. Soc. Adm. Pharm."},{"key":"483_CR7","doi-asserted-by":"publisher","first-page":"e153","DOI":"10.1331\/JAPhA.2012.11135","volume":"52","author":"MA Chui","year":"2012","unstructured":"Chui, M. A. & Mott, D. A. Community pharmacists\u2019 subjective workload and perceived task performance: a human factors approach. J. Am. Pharm. Assoc. 52, e153\u2013e160 (2012).","journal-title":"J. Am. Pharm. Assoc."},{"key":"483_CR8","doi-asserted-by":"crossref","unstructured":"Reason, J. Human Error (Cambridge University Press, 1990).","DOI":"10.1017\/CBO9781139062367"},{"key":"483_CR9","unstructured":"Center for Drug Evaluation & Research. National Drug Code Directory. Food and Drug Administration. Available at: https:\/\/www.fda.gov\/drugs\/drug-approvals-and-databases\/national-drug-code-directory. Accessed: 20 Jan 2021."},{"key":"483_CR10","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6963-13-175","volume":"13","author":"DS Debono","year":"2013","unstructured":"Debono, D. S. et al. Nurses\u2019 workarounds in acute healthcare settings: a scoping review. BMC Health Serv. Res. 13, 175 (2013).","journal-title":"BMC Health Serv. Res."},{"key":"483_CR11","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1197\/jamia.M2616","volume":"15","author":"R Koppel","year":"2008","unstructured":"Koppel, R., Wetterneck, T., Telles, J. L. & Karsh, B.-T. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J. Am. Med. Inform. Assoc. 15, 408\u2013423 (2008).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"483_CR12","unstructured":"Gabler, E. How chaos at chain pharmacies is putting patients at risk. The New York Times (2020). Available at: https:\/\/www.nytimes.com\/2020\/01\/31\/health\/pharmacists-medication-errors.html. Accessed: 20 Jan 2021."},{"key":"483_CR13","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1310\/hpj5001-34","volume":"50","author":"AR Oldland","year":"2015","unstructured":"Oldland, A. R., Golightly, L. K., May, S. K., Barber, G. R. & Stolpman, N. M. Electronic inventory systems and barcode technology: impact on pharmacy technical accuracy and error liability. Hosp. Pharm. 50, 34\u201341 (2015).","journal-title":"Hosp. Pharm."},{"key":"483_CR14","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1331\/1544-3191.46.5.613.Flynn","volume":"46","author":"EA Flynn","year":"2006","unstructured":"Flynn, E. A. & Barker, K. N. Effect of an automated dispensing system on errors in two pharmacies. J. Am. Pharm. Assoc. 46, 613\u2013615 (2006).","journal-title":"J. Am. Pharm. Assoc."},{"key":"483_CR15","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0086-0","volume":"2","author":"N Larios Delgado","year":"2019","unstructured":"Larios Delgado, N. et al. Fast and accurate medication identification. npj Digital Med. 2, 10 (2019).","journal-title":"npj Digital Med."},{"key":"483_CR16","unstructured":"Zeng, X., Cao, K. & Zhang, M. MobileDeepPill: a small-footprint mobile deep learning system for recognizing unconstrained pill images. In Proceedings of the 15th Annual International (2017)."},{"key":"483_CR17","doi-asserted-by":"crossref","unstructured":"Usuyama, N., Larios Delgado, N., Hall, A. K. & Lundin, J. ePillID dataset: a low-shot fine-grained benchmark for pill identification. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops 910\u2013911 Available at: openaccess.thecvf.com. (2020).","DOI":"10.1109\/CVPRW50498.2020.00463"},{"key":"483_CR18","unstructured":"Madsen, D. E. et al. Automatic Pill Identification from Pillbox Images. In VISAPP (1) 378\u2013384 (2013)."},{"key":"483_CR19","doi-asserted-by":"crossref","unstructured":"Deng, J. et al. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition 248\u2013255 (2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"483_CR20","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. arXiv [cs.CV] (2015).","DOI":"10.1109\/CVPR.2016.90"},{"key":"483_CR21","unstructured":"Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems. 1097\u20131105 (2012)."},{"key":"483_CR22","doi-asserted-by":"publisher","first-page":"e0118432","DOI":"10.1371\/journal.pone.0118432","volume":"10","author":"T Saito","year":"2015","unstructured":"Saito, T. & Rehmsmeier, M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One 10, e0118432 (2015).","journal-title":"PLoS One"},{"key":"483_CR23","doi-asserted-by":"crossref","unstructured":"Ding, B., Qian, H. & Zhou, J. Activation functions and their characteristics in deep neural networks. 2018 Chinese Control And Decision Conference. 1836\u20131841 (2018).","DOI":"10.1109\/CCDC.2018.8407425"},{"key":"483_CR24","unstructured":"Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning. 70, 1321\u20131330 (2017)."},{"key":"483_CR25","first-page":"191","volume":"43","author":"EA Flynn","year":"2003","unstructured":"Flynn, E. A., Barker, K. N. & Carnahan, B. J. National observational study of prescription dispensing accuracy and safety in 50 pharmacies. J. Am. Pharm. Assoc. 43, 191\u2013200 (2003).","journal-title":"J. Am. Pharm. Assoc."},{"key":"483_CR26","doi-asserted-by":"publisher","first-page":"27","DOI":"10.2466\/pms.2002.95.1.27","volume":"95","author":"S Reilley","year":"2002","unstructured":"Reilley, S., Grasha, A. F. & Schafer, J. Workload, error detection, and experienced stress in a simulated pharmacy verification task. Percept. Mot. Skills 95, 27\u201346 (2002).","journal-title":"Percept. Mot. Skills"},{"key":"483_CR27","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1109\/3468.844354","volume":"30","author":"R Parasuraman","year":"2000","unstructured":"Parasuraman, R., Sheridan, T. B. & Wickens, C. D. A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 30, 286\u2013297 (2000).","journal-title":"IEEE Trans. Syst. Man Cybern. A Syst. Hum."},{"key":"483_CR28","doi-asserted-by":"crossref","unstructured":"Yang, X. J., Unhelkar, V. V., Li, K. & Shah, J. A. Evaluating effects of user experience and system transparency on trust in automation. 12th ACM\/IEEE International Conference on Human-Robot Interaction. 408\u2013416 (2017).","DOI":"10.1145\/2909824.3020230"},{"key":"483_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0010-0277(95)00664-8","volume":"58","author":"L Cosmides","year":"1996","unstructured":"Cosmides, L. & Tooby, J. Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgment under uncertainty. Cognition 58, 1\u201373 (1996).","journal-title":"Cognition"},{"key":"483_CR30","unstructured":"Food, Drug, and Cosmetics Act. Code of Federal Regulations vol. 21 (1993)."},{"key":"483_CR31","unstructured":"Markson, C. R., Shah, P. J. & Lehmuth, C. G. Computer imaging pre-processing for automated medication dispensing analysis. US Patent (2019)."},{"key":"483_CR32","unstructured":"National Library of Medicine. PillBox API. PillBox https:\/\/datadiscovery.nlm.nih.gov\/resource\/crzr-uvwg.json."},{"key":"483_CR33","unstructured":"Paszke, A. et al. Automatic differentiation in PyTorch (2017)."},{"key":"483_CR34","doi-asserted-by":"publisher","first-page":"855","DOI":"10.1016\/j.jclinepi.2015.02.010","volume":"68","author":"B Ozenne","year":"2015","unstructured":"Ozenne, B., Subtil, F. & Maucort-Boulch, D. The precision\u2013recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J. Clin. Epidemiol. 68, 855\u2013859 (2015).","journal-title":"J. Clin. Epidemiol."},{"key":"483_CR35","doi-asserted-by":"crossref","unstructured":"Manning, C. D., Raghavan, P. & Sch\u00fctze, H. Introduction to Information Retrieval. (Cambridge University Press, 2008).","DOI":"10.1017\/CBO9780511809071"},{"key":"483_CR36","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"483_CR37","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","volume":"9","author":"JD Hunter","year":"2007","unstructured":"Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90\u201395 (2007).","journal-title":"Comput. Sci. Eng."},{"key":"483_CR38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v028.i05","volume":"28","author":"M Kuhn","year":"2008","unstructured":"Kuhn, M., Others. Building predictive models in R using the caret package. J. Stat. Softw. 28, 1\u201326 (2008).","journal-title":"J. Stat. Softw."},{"key":"483_CR39","doi-asserted-by":"publisher","first-page":"1686","DOI":"10.21105\/joss.01686","volume":"4","author":"H Wickham","year":"2019","unstructured":"Wickham, H. et al. Welcome to the Tidyverse. J. Open Source Softw. 4, 1686 (2019).","journal-title":"J. Open Source Softw."}],"updated-by":[{"DOI":"10.1038\/s41746-022-00564-2","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2022,2,16]],"date-time":"2022-02-16T00:00:00Z","timestamp":1644969600000}}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00483-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00483-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00483-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T18:57:15Z","timestamp":1670093835000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00483-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,27]]},"references-count":39,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["483"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00483-8","relation":{"correction":[{"id-type":"doi","id":"10.1038\/s41746-022-00564-2","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,27]]},"assertion":[{"value":"8 September 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 June 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 July 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 February 2022","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1038\/s41746-022-00564-2","URL":"https:\/\/doi.org\/10.1038\/s41746-022-00564-2","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"118"}}