{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T14:06:49Z","timestamp":1762956409995,"version":"build-2065373602"},"reference-count":63,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2020,8,5]],"date-time":"2020-08-05T00:00:00Z","timestamp":1596585600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Data charts are widely used in our daily lives, being present in regular media, such as newspapers, magazines, web pages, books, and many others. In general, a well-constructed data chart leads to an intuitive understanding of its underlying data. In the same way, when data charts have wrong design choices, a redesign of these representations might be needed. However, in most cases, these charts are shown as a static image, which means that the original data are not usually available. Therefore, automatic methods could be applied to extract the underlying data from the chart images to allow these changes. The task of recognizing charts and extracting data from them is complex, largely due to the variety of chart types and their visual characteristics. Other features in real-world images that can make this task difficult are photo distortions, noise, alignment, etc. Two computer vision techniques that can assist this task and have been little explored in this context are perspective detection and correction. These methods transform a distorted and noisy chart in a clear chart, with its type ready for data extraction or other uses. This paper proposes a classification, detection, and perspective correction process that is suitable for real-world usage, when considering the data used for training a state-of-the-art model for the extraction of a chart in real-world photography. The results showed that, with slight changes, chart recognition methods are now ready for real-world charts, when taking time and accuracy into consideration.<\/jats:p>","DOI":"10.3390\/s20164370","type":"journal-article","created":{"date-parts":[[2020,8,5]],"date-time":"2020-08-05T15:13:18Z","timestamp":1596640398000},"page":"4370","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["A Real-World Approach on the Problem of Chart Recognition Using Classification, Detection and Perspective Correction"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4971-9951","authenticated-orcid":false,"given":"Tiago","family":"Ara\u00fajo","sequence":"first","affiliation":[{"name":"Computer Science Graduate Program (PPGCC), Federal University of Par\u00e1 (UFPA), 66075-110 Bel\u00e9m, Brazil"},{"name":"Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Department of Electronics, Telecommunications e Informatics (DETI), University of Aveiro (UA), 3810-193 Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8120-3018","authenticated-orcid":false,"given":"Paulo","family":"Chagas","sequence":"additional","affiliation":[{"name":"Computer Science Graduate Program (PGCOMP), Federal University of Bahia (UFBA), 40210-630 Salvador, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3430-5211","authenticated-orcid":false,"given":"Jo\u00e3o","family":"Alves","sequence":"additional","affiliation":[{"name":"Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Department of Electronics, Telecommunications e Informatics (DETI), University of Aveiro (UA), 3810-193 Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2193-5783","authenticated-orcid":false,"given":"Carlos","family":"Santos","sequence":"additional","affiliation":[{"name":"Computer Science Graduate Program (PPGCC), Federal University of Par\u00e1 (UFPA), 66075-110 Bel\u00e9m, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2219-2731","authenticated-orcid":false,"given":"Beatriz","family":"Sousa Santos","sequence":"additional","affiliation":[{"name":"Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Department of Electronics, Telecommunications e Informatics (DETI), University of Aveiro (UA), 3810-193 Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5872-4827","authenticated-orcid":false,"given":"Bianchi","family":"Serique Meiguins","sequence":"additional","affiliation":[{"name":"Computer Science Graduate Program (PPGCC), Federal University of Par\u00e1 (UFPA), 66075-110 Bel\u00e9m, Brazil"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,5]]},"reference":[{"key":"ref_1","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., and Heer, J. (2011, January 16\u201319). Revision: Automated classification, analysis and redesign of chart images. Proceedings of the 24th annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA.","DOI":"10.1145\/2047196.2047247"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.jvlc.2018.08.005","article-title":"Chart decoder: Generating textual and numeric information from chart images automatically","volume":"48","author":"Dai","year":"2018","journal-title":"J. Vis. Lang. Comput."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Battle, L., Duan, P., Miranda, Z., Mukusheva, D., Chang, R., and Stonebraker, M. (2018, January 21\u201326). Beagle: Automated extraction and interpretation of visualizations from the web. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174168"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Jung, D., Kim, W., Song, H., Hwang, J.i., Lee, B., Kim, B., and Seo, J. (, January May). ChartSense: Interactive data extraction from chart images. Proceedings of the 2017 chi Conference on Human Factors in Computing Systems, Denver, CO, USA.","DOI":"10.1145\/3025453.3025957"},{"key":"ref_6","unstructured":"Tummers, B. (2020, July 14). Datathief iii. Available online: https:\/\/datathief.org\/."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mishchenko, A., and Vassilieva, N. (2011, January 26\u201328). Chart image understanding and numerical data extraction. Proceedings of the 2011 Sixth International Conference on Digital Information Management. IEEE, Melbourn, QLD, Australia.","DOI":"10.1109\/ICDIM.2011.6093320"},{"key":"ref_8","unstructured":"Al-Zaidy, R.A., Choudhury, S.R., and Giles, C.L. (2016, January 12\u201313). Automatic summary generation for scientific data charts. Proceedings of the Workshops at the thirtieth aaai Conference on Artificial Intelligence, Phoenix, AZ, USA."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Chagas, P., Akiyama, R., Meiguins, A., Santos, C., Saraiva, F., Meiguins, B., and Morais, J. (2018, January 8\u201313). Evaluation of convolutional neural network architectures for chart image classification. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.","DOI":"10.1109\/IJCNN.2018.8489315"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kavasidis, I., Pino, C., Palazzo, S., Rundo, F., Giordano, D., Messina, P., and Spampinato, C. (2019, January 9\u201313). A saliency-based convolutional neural network for table and chart detection in digitized documents. Proceedings of the International Conference on Image Analysis and Processing, Trento, Italy.","DOI":"10.1007\/978-3-030-30645-8_27"},{"key":"ref_11","unstructured":"Svendsen, J.P. (2015). Chart Detection and Recognition in Graphics Intensive Business Documents. [Ph.D. Thesis, University of Victoria]."},{"key":"ref_12","unstructured":"He, Y., Yu, X., Gan, Y., Zhu, T., Xiong, S., Peng, J., Hu, L., Xu, G., and Yuan, X. (2017, January 4\u20138). Bar charts detection and analysis in biomedical literature of PubMed Central. Proceedings of the AMIA Annual Symposium Proceedings. American Medical Informatics Association, Washington, DC, USA."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1007\/s001380050120","article-title":"A compact algorithm for rectification of stereo pairs","volume":"12","author":"Fusiello","year":"2000","journal-title":"Mach. Vis. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Chaudhury, K., DiVerdi, S., and Ioffe, S. (2014, January 27\u201330). Auto-rectification of user photos. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7025706"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Takezawa, Y., Hasegawa, M., and Tabbone, S. (2017, January 9\u201315). Robust perspective rectification of camera-captured document images. Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan.","DOI":"10.1109\/ICDAR.2017.345"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Shemiakina, J., Konovalenko, I., Tropin, D., and Faradjev, I. (2019, January 6\u201318). Fast projective image rectification for planar objects with Manhattan structure. Proceedings of the Twelfth International Conference on Machine Vision (ICMV 2019), Amsterdam, The Netherlands.","DOI":"10.1117\/12.2559630"},{"key":"ref_17","first-page":"1","article-title":"Data and information visualization methods, and interactive mechanisms: A survey","volume":"34","author":"Khan","year":"2011","journal-title":"Int. J. Comput. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.sigpro.2015.09.027","article-title":"Deepchart: Combining deep convolutional networks and deep belief networks in chart classification","volume":"124","author":"Tang","year":"2016","journal-title":"Signal Process."},{"key":"ref_19","unstructured":"Junior, P.R.S.C., De Freitas, A.A., Akiyama, R.D., Miranda, B.P., De Ara\u00fajo, T.D.O., Dos Santos, C.G.R., Meiguins, B.S., and De Morais, J.M. (2017, January 11\u201314). Architecture proposal for data extraction of chart images using Convolutional Neural Network. Proceedings of the 2017 21st International Conference Information Visualisation (IV), London, UK."},{"key":"ref_20","unstructured":"Linowes, J., and Babilinski, K. (2017). Augmented Reality for Developers: Build Practical Augmented Reality Applications with Unity, ARCore, ARKit, and Vuforia, Packt Publishing Ltd."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Passian, A., and Imam, N. (2019). Nanosystems, Edge Computing, and the Next Generation Computing Systems. Sensors, 19.","DOI":"10.3390\/s19184048"},{"key":"ref_22","unstructured":"Parker, J.R. (2010). Algorithms for Image Processing and Computer Vision, John Wiley & Sons."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_25","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_26","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 11\u201314). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the Thirty-first AAAI conference on artificial intelligence, Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1109\/TPAMI.2015.2437384","article-title":"Region-based convolutional networks for accurate object detection and segmentation","volume":"38","author":"Girshick","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1109\/TPAMI.2016.2587640","article-title":"Show and tell: Lessons learned from the 2015 mscoco image captioning challenge","volume":"39","author":"Vinyals","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s11263-009-0275-4","article-title":"The pascal visual object classes (voc) challenge","volume":"88","author":"Everingham","year":"2010","journal-title":"Int. J. Comput. Vis."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017, January 22\u201329). Focal loss for dense object detection. Proceedings of the IEEE international Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 11\u201318). Fast r-cnn. Proceedings of the IEEE international Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_35","unstructured":"Jagannathan, L., and Jawahar, C. (September, January 29). Perspective correction methods for camera based document analysis. Proceedings of the First Int. Workshop on Camera-based Document Analysis and Recognition, Seoul, Korea."},{"key":"ref_36","first-page":"042099","article-title":"Camera model and parameter calibration","volume":"440","author":"Li","year":"2020","journal-title":"E&ES"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Sheshkus, A., Ingacheva, A., Arlazarov, V., and Nikolaev, D. (2019). HoughNet: Neural network architecture for vanishing points detection. arXiv.","DOI":"10.1109\/ICDAR.2019.00140"},{"key":"ref_38","unstructured":"Arlazarov, V.V., Bulatov, K.B., Chernov, T.S., and Arlazarov, V.L. MIDV-500: A dataset for identity document analysis and recognition on mobile devices in video stream. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"El Abed, H., Wenyin, L., and Margner, V. (2011, January 18\u201321). International conference on document analysis and recognition (ICDAR 2011)-competitions overview. Proceedings of the 2011 International Conference on Document Analysis and Recognition, Beijing, China.","DOI":"10.1109\/ICDAR.2011.286"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"G\u00f6bel, M., Hassan, T., Oro, E., and Orsi, G. (2013, January 25\u201328). ICDAR 2013 table competition. Proceedings of the 2013 12th International Conference on Document Analysis and Recognition, Washington, DC, USA.","DOI":"10.1109\/ICDAR.2013.292"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Gatos, B., Danatsas, D., Pratikakis, I., and Perantonis, S.J. (2005, January 7\u201311). Automatic table detection in document images. Proceedings of the International Conference on Pattern Recognition and Image Analysis, Genoa, Italy.","DOI":"10.1007\/11551188_67"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Schreiber, S., Agne, S., Wolf, I., Dengel, A., and Ahmed, S. (2017, January 9\u201315). Deepdesrt: Deep learning for detection and structure recognition of tables in document images. Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan.","DOI":"10.1109\/ICDAR.2017.192"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Huang, W., and Tan, C.L. (2007, January 23\u201326). Locating charts from scanned document pages. Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), Parana, Brazil.","DOI":"10.1109\/ICDAR.2007.4378722"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1111\/cgf.13193","article-title":"Reverse-engineering visualizations: Recovering visual encodings from chart images","volume":"Volume 36","author":"Poco","year":"2017","journal-title":"Computer Graphics Forum"},{"key":"ref_45","unstructured":"Bylinskii, Z., and Borkin, M. (2015). Eye fixation metrics for large scale analysis of information visualizations. ETVIS Work. Eye Track. Vis."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/j.compag.2017.11.040","article-title":"Synthetic bootstrapping of convolutional neural networks for semantic plant part segmentation","volume":"161","author":"Barth","year":"2019","journal-title":"Comput. Electron. Agric."},{"key":"ref_47","first-page":"1","article-title":"Improving handwritten arabic character recognition by modeling human handwriting distortions","volume":"15","author":"Shatnawi","year":"2015","journal-title":"ACM Trans. Asian Low-Resource Lang. Inf. Proc."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Eggert, C., Winschel, A., and Lienhart, R. (2015, January 23\u201327). On the benefit of synthetic data for company logo detection. Proceedings of the 23rd ACM international conference on Multimedia, Mountain View, CA, USA.","DOI":"10.1145\/2733373.2806407"},{"key":"ref_49","unstructured":"(2020, April 08). CVPR2020 Workshop on Text and Documents in the Deep Learning Era. Available online: https:\/\/cvpr2020text.wordpress.com\/."},{"key":"ref_50","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_51","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2016). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. arXiv."},{"key":"ref_52","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_54","unstructured":"Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z. (2015). Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv."},{"key":"ref_55","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7\u201312). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/s11263-019-01247-4","article-title":"Deep learning for generic object detection: A survey","volume":"128","author":"Liu","year":"2020","journal-title":"Int. J. Comput. Vis."},{"key":"ref_57","unstructured":"Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., and Girshick, R. (2020, April 08). Detectron2. Available online: https:\/\/github.com\/facebookresearch\/detectron2."},{"key":"ref_58","unstructured":"(2020, April 08). Image-Rectification. Available online: https:\/\/github.com\/chsasank\/Image-Rectification."},{"key":"ref_59","unstructured":"Mordvintsev, A., Olah, C., and Tyka, M. (2020, April 08). Deepdream-a code example for visualizing neural networks. Google Research, Available online: https:\/\/ai.googleblog.com\/2015\/07\/deepdream-code-example-for-visualizing.html."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Ledig, C., Theis, L., Husz\u00e1r, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21\u201326). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.19"},{"key":"ref_61","unstructured":"(2020, April 08). Get Office Lens\u2014Microsoft Store. Available online: https:\/\/www.microsoft.com\/en-us\/p\/office-lens\/9wzdncrfj3t8\/."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1016\/j.vlsi.2019.07.005","article-title":"Computer vision algorithms and hardware implementations: A survey","volume":"69","author":"Feng","year":"2019","journal-title":"Integration"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Raaen, K., and Kjellmo, I. (2015, January 18\u201321). Measuring latency in virtual reality systems. Proceedings of the International Conference on Entertainment Computing, Tsukuba City, Japan.","DOI":"10.1007\/978-3-319-24589-8_40"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/16\/4370\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:54:32Z","timestamp":1760176472000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/16\/4370"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,5]]},"references-count":63,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2020,8]]}},"alternative-id":["s20164370"],"URL":"https:\/\/doi.org\/10.3390\/s20164370","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,8,5]]}}}