{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T15:35:49Z","timestamp":1777390549701,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":38,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,9,5]],"date-time":"2021-09-05T00:00:00Z","timestamp":1630800000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,9,5]]},"DOI":"10.1145\/3476887.3476888","type":"proceedings-article","created":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T04:05:14Z","timestamp":1635739514000},"page":"13-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":59,"title":["A survey of OCR evaluation tools and metrics"],"prefix":"10.1145","author":[{"given":"Clemens","family":"Neudecker","sequence":"first","affiliation":[{"name":"Berlin State Library, Germany"}]},{"given":"Konstantin","family":"Baierer","sequence":"additional","affiliation":[{"name":"Berlin State Library, Germany"}]},{"given":"Mike","family":"Gerber","sequence":"additional","affiliation":[{"name":"Berlin State Library, Germany"}]},{"given":"Christian","family":"Clausner","sequence":"additional","affiliation":[{"name":"University of Salford, United Kingdom"}]},{"given":"Apostolos","family":"Antonacopoulos","sequence":"additional","affiliation":[{"name":"University of Salford, United Kingdom"}]},{"given":"Stefan","family":"Pletschacher","sequence":"additional","affiliation":[{"name":"University of Salford, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2021,10,31]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2595188.2595214"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2037342.2037369"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3322905.3322916"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2015.7333898"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2011.282"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2013.141"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/DAS.2016.82"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2020.02.003"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141753.1141759"},{"key":"e_1_3_2_1_10_1","volume-title":"CLEF 2020 Working Notes. Conference and Labs of the Evaluation Forum, Vol.\u00a02696","author":"Ehrmann Maud","year":"2020","unstructured":"Maud Ehrmann , Matteo Romanello , Alex Fl\u00fcckiger , and Simon Clematide . 2020 . Extended overview of CLEF HIPE 2020: named entity processing on historical newspapers . In CLEF 2020 Working Notes. Conference and Labs of the Evaluation Forum, Vol.\u00a02696 . CEUR, Aachen, Germany, 1\u201338. Maud Ehrmann, Matteo Romanello, Alex Fl\u00fcckiger, and Simon Clematide. 2020. Extended overview of CLEF HIPE 2020: named entity processing on historical newspapers. In CLEF 2020 Working Notes. Conference and Labs of the Evaluation Forum, Vol.\u00a02696. CEUR, Aachen, Germany, 1\u201338."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/JCDL.2019.00057"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1093\/llc\/fqz024"},{"key":"e_1_3_2_1_13_1","article-title":"How good can it get? Analysing and improving OCR accuracy in large scale historic newspaper digitisation programs","volume":"15","author":"Holley Rose","year":"2009","unstructured":"Rose Holley . 2009 . How good can it get? Analysing and improving OCR accuracy in large scale historic newspaper digitisation programs . D-Lib Magazine 15 , 3\/4 (2009), Unpaginated. Rose Holley. 2009. How good can it get? Analysing and improving OCR accuracy in large scale historic newspaper digitisation programs. D-Lib Magazine 15, 3\/4 (2009), Unpaginated.","journal-title":"D-Lib Magazine"},{"key":"e_1_3_2_1_14_1","first-page":"24","article-title":"Old content and modern tools-searching named entities in a Finnish OCRed historical newspaper collection 1771-1910","volume":"11","author":"Kettunen Kimmo","year":"2017","unstructured":"Kimmo Kettunen , Eetu M\u00e4kel\u00e4 , Teemu Ruokolainen , Juha Kuokkala , and Laura L\u00f6fberg . 2017 . Old content and modern tools-searching named entities in a Finnish OCRed historical newspaper collection 1771-1910 . Digital Humanities Quarterly 11 (2017), 24 . Issue 3. Kimmo Kettunen, Eetu M\u00e4kel\u00e4, Teemu Ruokolainen, Juha Kuokkala, and Laura L\u00f6fberg. 2017. Old content and modern tools-searching named entities in a Finnish OCRed historical newspaper collection 1771-1910. Digital Humanities Quarterly 11 (2017), 24. Issue 3.","journal-title":"Digital Humanities Quarterly"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2009.133"},{"key":"e_1_3_2_1_16_1","volume-title":"End-To-End Measure for Text Recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, NY, USA, 1424\u20131431","author":"Leifert Gundram","year":"2019","unstructured":"Gundram Leifert , Roger Labahn , Tobias Gr\u00fcning , and Svenja Leifert . 2019 . End-To-End Measure for Text Recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, NY, USA, 1424\u20131431 . Gundram Leifert, Roger Labahn, Tobias Gr\u00fcning, and Svenja Leifert. 2019. End-To-End Measure for Text Recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, NY, USA, 1424\u20131431."},{"key":"e_1_3_2_1_17_1","volume-title":"Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady 10, 8","author":"Levenshtein I","year":"1966","unstructured":"Vladimir\u00a0 I Levenshtein . 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady 10, 8 ( 1966 ), 707\u2013710. Vladimir\u00a0I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady 10, 8 (1966), 707\u2013710."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10032-009-0094-8"},{"key":"e_1_3_2_1_19_1","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","author":"Mieskes Margot","year":"2019","unstructured":"Margot Mieskes and Stefan Schmunk . 2019 . OCR Quality and NLP Preprocessing . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics , Florence, Italy. ACL, Stroudsburg PA, USA, 102\u2013105. Margot Mieskes and Stefan Schmunk. 2019. OCR Quality and NLP Preprocessing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. ACL, Stroudsburg PA, USA, 102\u2013105."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501115.2501130"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2010.72"},{"key":"e_1_3_2_1_22_1","volume-title":"International Conference on Asian Digital Libraries","author":"Pontes Elvys\u00a0Linhares","year":"2019","unstructured":"Elvys\u00a0Linhares Pontes , Ahmed Hamdi , Nicolas Sidere , and Antoine Doucet . 2019 . Impact of OCR quality on named entity linking . In International Conference on Asian Digital Libraries . Springer, NY, USA, 102\u2013115. Elvys\u00a0Linhares Pontes, Ahmed Hamdi, Nicolas Sidere, and Antoine Doucet. 2019. Impact of OCR quality on named entity linking. In International Conference on Asian Digital Libraries. Springer, NY, USA, 102\u2013115."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2012.10.002"},{"key":"e_1_3_2_1_24_1","first-page":"15","article-title":"QURATOR: Innovative Technologies for Content and Data Curation","volume":"2535","author":"Rehm Georg","year":"2020","unstructured":"Georg Rehm , Peter Bourgonje , Stefanie Hegele , Florian Kintzel , Juli\u00e1n\u00a0Moreno Schneider , Malte Ostendorff , Karolina Zaczynska , Armin Berger , Stefan Grill , and S\u00f6ren et \u00a0al . R\u00e4uchle. 2020 . QURATOR: Innovative Technologies for Content and Data Curation . CEUR-WS 2535 , 1 (2020), 15 . Georg Rehm, Peter Bourgonje, Stefanie Hegele, Florian Kintzel, Juli\u00e1n\u00a0Moreno Schneider, Malte Ostendorff, Karolina Zaczynska, Armin Berger, Stefan Grill, and S\u00f6ren et\u00a0al. R\u00e4uchle. 2020. QURATOR: Innovative Technologies for Content and Data Curation. CEUR-WS 2535, 1 (2020), 15.","journal-title":"CEUR-WS"},{"key":"e_1_3_2_1_25_1","volume-title":"Measuring the accuracy of page-reading systems","author":"Rice Stephen\u00a0Vincent","unstructured":"Stephen\u00a0Vincent Rice . 1996. Measuring the accuracy of page-reading systems . UNLV , Las Vega, NV . Stephen\u00a0Vincent Rice. 1996. Measuring the accuracy of page-reading systems. UNLV, Las Vega, NV."},{"key":"e_1_3_2_1_26_1","volume-title":"The ISRI analytic tools for OCR evaluation","author":"Rice V","year":"1996","unstructured":"Stephen\u00a0 V Rice and Thomas\u00a0 A Nartker . 1996. The ISRI analytic tools for OCR evaluation . UNLV\/Information Science Research Institute , TR-96 2 ( 1996 ), 45. Stephen\u00a0V Rice and Thomas\u00a0A Nartker. 1996. The ISRI analytic tools for OCR evaluation. UNLV\/Information Science Research Institute, TR-96 2 (1996), 45."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2015.7333823"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.33011\/computel.v1i.345"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/DAS.2018.14"},{"key":"e_1_3_2_1_30_1","volume-title":"A research agenda for historical and multilingual optical character recognition","author":"Smith A","unstructured":"David\u00a0 A Smith and Ryan Cordell . 2018. A research agenda for historical and multilingual optical character recognition . Northeastern University , Boston, MA . David\u00a0A Smith and Ryan Cordell. 2018. A research agenda for historical and multilingual optical character recognition. Northeastern University, Boston, MA."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2011.114"},{"key":"e_1_3_2_1_32_1","volume-title":"Automatic quality evaluation and (semi-) automatic improvement of OCR models for historical printings. arXiv preprint arXiv:1606.05157","author":"Springmann Uwe","year":"2016","unstructured":"Uwe Springmann , Florian Fink , and Klaus\u00a0 U Schulz . 2016. Automatic quality evaluation and (semi-) automatic improvement of OCR models for historical printings. arXiv preprint arXiv:1606.05157 ( 2016 ), 8. Uwe Springmann, Florian Fink, and Klaus\u00a0U Schulz. 2016. Automatic quality evaluation and (semi-) automatic improvement of OCR models for historical printings. arXiv preprint arXiv:1606.05157 (2016), 8."},{"key":"e_1_3_2_1_33_1","volume-title":"Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin. arXiv preprint arXiv:1809.05501","author":"Springmann Uwe","year":"2018","unstructured":"Uwe Springmann , Christian Reul , Stefanie Dipper , and Johannes Baiter . 2018. Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin. arXiv preprint arXiv:1809.05501 ( 2018 ), 8. Uwe Springmann, Christian Reul, Stefanie Dipper, and Johannes Baiter. 2018. Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin. arXiv preprint arXiv:1809.05501 (2018), 8."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1045\/july2009-munoz"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24592-8_19"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01206331"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Daniel van Strien Kaspar Beelen Mariona\u00a0Coll Ardanuy Kasra Hosseini Barbara McGillivray and Giovanni Colavizza. 2020. Assessing the Impact of OCR Quality on Downstream NLP Tasks. In ICAART (1). SCITEPRESS Set\u00fabal Portugal 484\u2013496.  Daniel van Strien Kaspar Beelen Mariona\u00a0Coll Ardanuy Kasra Hosseini Barbara McGillivray and Giovanni Colavizza. 2020. Assessing the Impact of OCR Quality on Downstream NLP Tasks. In ICAART (1). SCITEPRESS Set\u00fabal Portugal 484\u2013496.","DOI":"10.5220\/0009169004840496"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1515\/abitech-2015-0014"}],"event":{"name":"HIP '21: The 6th International Workshop on Historical Document Imaging and Processing","location":"Lausanne Switzerland","acronym":"HIP '21"},"container-title":["The 6th International Workshop on Historical Document Imaging and Processing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3476887.3476888","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3476887.3476888","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:45Z","timestamp":1750188645000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3476887.3476888"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,5]]},"references-count":38,"alternative-id":["10.1145\/3476887.3476888","10.1145\/3476887"],"URL":"https:\/\/doi.org\/10.1145\/3476887.3476888","relation":{},"subject":[],"published":{"date-parts":[[2021,9,5]]},"assertion":[{"value":"2021-10-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}