{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T15:28:41Z","timestamp":1776698921654,"version":"3.51.2"},"reference-count":40,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T00:00:00Z","timestamp":1664236800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Patient compliance with prescribed medication regimens is critical for maintaining health and managing disease and illness. To encourage patient compliance, multiple aids, like automatic pill dispensers, pill organizers, and various reminder applications, have been developed to help people adhere to their medication regimens. However, when utilizing these aids, the user or patient must manually enter their medication information and schedule. This process is time-consuming and often prone to error. For example, elderly patients may have difficulty reading medication information on the bottle due to decreased eyesight, leading them to enter medication information incorrectly. This study explored methods for extracting pertinent information from cylindrically distorted prescription drug labels using Machine Learning and Computer Vision techniques. This study found that Deep Convolutional Neural Networks (DCNN) performed better than other techniques in identifying label key points under different lighting conditions and various backgrounds. This method achieved a percentage of Correct Key points PCK @ 0.03 of 97%. These key points were then used to correct the cylindrical distortion. Next, the multiple dewarped label images were stitched together and processed by an Optical Character Recognition (OCR) engine. Pertinent information, such as patient name, drug name, drug strength, and directions of use, were extracted from the recognized text using Natural Language Processing (NLP) techniques. The system created in this study can be used to improve patient health and compliance by creating an accurate medication schedule.<\/jats:p>","DOI":"10.3390\/make4040043","type":"journal-article","created":{"date-parts":[[2022,9,28]],"date-time":"2022-09-28T01:51:49Z","timestamp":1664329909000},"page":"852-864","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Automatic Extraction of Medication Information from Cylindrically Distorted Pill Bottle Labels"],"prefix":"10.3390","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4948-2103","authenticated-orcid":false,"given":"Kseniia","family":"Gromova","sequence":"first","affiliation":[{"name":"Computer Science Program, Division of Science and Engineering, Penn State Abington, Abington, PA 19001, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vinayak","family":"Elangovan","sequence":"additional","affiliation":[{"name":"Computer Science Program, Division of Science and Engineering, Penn State Abington, Abington, PA 19001, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,27]]},"reference":[{"key":"ref_1","first-page":"269","article-title":"Factors affecting therapeutic compliance: A review from the patient\u2019s perspective","volume":"4","author":"Jin","year":"2018","journal-title":"Ther. Clin. Risk Manag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"155","DOI":"10.5001\/omj.2011.38","article-title":"Patient medication adherence: Measures in daily practice","volume":"26","author":"Jimmy","year":"2011","journal-title":"Oman Med. J."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1303","DOI":"10.2147\/PPA.S87551","article-title":"Interventional tools to improve medication adherence: Review of literature","volume":"9","author":"Costa","year":"2015","journal-title":"Patient Prefer. Adherence"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Arain, M., Ahmad, A., Chiu, V., and Kembel, L. (2021). Medication adherence support of an in-home electronic medication dispensing system for individuals living with chronic conditions: A pilot randomized controlled trial. BMC Geriatr., 21.","DOI":"10.1186\/s12877-020-01979-w"},{"key":"ref_5","unstructured":"(2021, August 01). Prescription Per Capita in the United States by Age Group. Statista Research Department. Available online: https:\/\/www.statista.com\/statistics\/315476\/prescriptions-in-us-per-capita-by-age-group\/."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Liu, X., Meehan, J., Tong, W., Wu, L., Xu, X., and Xu, J. (2021). DLI-IT: A deep learning approach to drug label identification through image and text embedding. BMC Med. Inform. Decis. Mak., 20.","DOI":"10.1186\/s12911-020-1078-3"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Gundimeda, V., Murali, R.S., Joseph, R., and Babu, N.N. (2019). An automated computer vision system for extraction of retail food product metadata. First International Conference on Artificial Intelligence and Cognitive Computing, Springer Nature. [1st ed.].","DOI":"10.1007\/978-981-13-1580-0_20"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tangtisanon, P. (2016, January 16\u201319). Healthcare system for elders with automatic drug label detection. Proceedings of the 16th International Conference on Control, Automation and Systems (ICCAS), Gyeongju, Korea.","DOI":"10.1109\/ICCAS.2016.7832390"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Xu, J., Chen, C., Xie, H., and Lu, F. (2017, January 8\u201311). Cylindrical product label image stitching method. Proceedings of the 2017 2nd IEEE International Conference on Computational Intelligence and Applications (ICCIA), Beijing, China.","DOI":"10.1109\/CIAPP.2017.8167233"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ye, Z., Yi, C., and Tian, Y. (2013, January 15\u201319). Reading labels of cylinder objects for blind persons. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo (ICME), San Jose, CA, USA.","DOI":"10.1109\/ICME.2013.6607632"},{"key":"ref_11","unstructured":"Zankevich, A. (2021, June 15). How to Unwrap Wine Labels Programmatically, Medium. Available online: https:\/\/medium.com\/hackernoon\/how-to-unwrap-wine-labels-programmatically-31c8c62b30ce."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Su, Y.-H., Chao, C.-P., Hung, L.-C., Sung, S.-F., and Lee, P.-J. (2020). A Natural Language Processing Approach to Automated Highlighting of New Information in Clinical Notes. Appl. Sci., 10.","DOI":"10.3390\/app10082824"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1093\/jamia\/ocz101","article-title":"Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods","volume":"27","author":"Christopoulou","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1093\/jamia\/ocz063","article-title":"A study of deep learning approaches for medication and adverse drug event extraction from clinical text","volume":"27","author":"Wei","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_15","first-page":"15","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_16","first-page":"6631837","article-title":"A BERT0-BiGRU-CRF Model for Entity Recognition of Chinese Electonic Medical Records","volume":"2021","author":"Qin","year":"2021","journal-title":"Artif. Intell. Smart Syst. Simul."},{"key":"ref_17","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv."},{"key":"ref_18","unstructured":"Zhu, W., Zhang, W., Li, G.-W., He, C., and Zhang, L. (2016, January 15\u201318). A Study of Damp-Heat Syndrome Classification Using Word2vec and TF-IDF. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Shenzhen, China."},{"key":"ref_19","unstructured":"Halir, R., and Flusser, J. (1998, January 9\u201313). Numerically stable direct least squares fitting of ellipses. Proceedings of the 6th International Conference in Central Europe on Computer Graphics and Visualization, Plzen-Bory, Czech Republic."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1109\/TPAMI.1986.4767851","article-title":"A Computational Approach to Edge Detection","volume":"8","author":"Canny","year":"1986","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1145\/361237.361242","article-title":"Use of the Hough transformation to detect lines and curves in pictures","volume":"15","author":"Duda","year":"1972","journal-title":"Commun. ACM"},{"key":"ref_22","unstructured":"Xie, Y., and Ji, Q. (2002, January 11\u201315). A new efficient ellipse detection method. Proceedings of the 2002 International Conference on Pattern Recognition, Quebec City, QC, Canada."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"ref_24","unstructured":"Tan, M., and Le, Q. (2019, January 9\u201315). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Xiao, B., Wu, H., and Wei, Y. (2018, January 8\u201314). Simple baselines for human pose estimation and tracking. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01231-1_29"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Newell, A., Yang, K., and Deng, J. (2016, January 8\u201316). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46484-8_29"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15\u201320). Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00584"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wei, S.E., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016, January 27\u201330). Convolutional pose machines. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.511"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Carreira, J., Agrawal, P., Fragkiadaki, K., and Malik, J. (2016, January 27\u201330). Human pose estimation with iterative error feedback. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.512"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201323). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/s11263-006-0002-3","article-title":"Automatic panoramic image stitching using invariant features","volume":"74","author":"Brown","year":"2007","journal-title":"Int. J. Comput. Vis."},{"key":"ref_33","unstructured":"Liu, R., Li, Z., and Jia, J. (2008, January 23\u201328). Image partial blur detection and classification. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA."},{"key":"ref_34","unstructured":"(2022, August 01). Tesseract Documentation. Available online: https:\/\/tesseract-ocr.github.io\/tessdoc\/ImproveQuality."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1016\/j.compeleceng.2016.11.034","article-title":"An adaptive image registration method based on SIFT features and RANSAC transform","volume":"62","author":"Nasri","year":"2017","journal-title":"Comput. Electr. Eng."},{"key":"ref_36","first-page":"691","article-title":"Quality and variability of patient directions in electronic prescriptions in the ambulatory care setting","volume":"24","author":"Yang","year":"2018","journal-title":"J. Manag. Care Spec. Pharm."},{"key":"ref_37","unstructured":"Bird, S., Loper, E., and Klein, E. (2009). Natural Language Processing with Python, O\u2019Reilly Media Inc."},{"key":"ref_38","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_39","unstructured":"(2021, August 01). U.S. Food and Drug Administration (FDA) API Basics, Available online: https:\/\/open.fda.gov\/api\/reference."},{"key":"ref_40","unstructured":"Liu, P., Qiu, X., and Huang, X. (2016). Recurrent neural network for text classification with multi-task learning. arXiv."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/4\/4\/43\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:40:17Z","timestamp":1760143217000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/4\/4\/43"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,27]]},"references-count":40,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["make4040043"],"URL":"https:\/\/doi.org\/10.3390\/make4040043","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,27]]}}}