{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T20:37:15Z","timestamp":1761597435640,"version":"3.37.3"},"reference-count":19,"publisher":"Wiley","license":[{"start":{"date-parts":[[2013,1,1]],"date-time":"2013-01-01T00:00:00Z","timestamp":1356998400000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"funder":[{"name":"Finnish Funding Agency for Technology and Innovation","award":["40430\/05","40039\/07"],"award-info":[{"award-number":["40430\/05","40039\/07"]}]},{"name":"Finnish Funding Agency for Technology and Innovation","award":["40430\/05","40039\/07"],"award-info":[{"award-number":["40430\/05","40039\/07"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational and Mathematical Methods in Medicine"],"published-print":{"date-parts":[[2013]]},"abstract":"<jats:p>We address the performance evaluation practices for developing medical image analysis methods, in particular, how to establish and share databases of medical images with verified ground truth and solid evaluation protocols. Such databases support the development of better algorithms, execution of profound method comparisons, and, consequently, technology transfer from research laboratories to clinical practice. For this purpose, we propose a framework consisting of reusable methods and tools for the laborious task of constructing a benchmark database. We provide a software tool for medical image annotation helping to collect class label, spatial span, and expert's confidence on lesions and a method to appropriately combine the manual segmentations from multiple experts. The tool and all necessary functionality for method evaluation are provided as public software packages. As a case study, we utilized the framework and tools to establish the DiaRetDB1 V2.1 database for benchmarking diabetic retinopathy detection algorithms. The database contains a set of retinal images, ground truth based on information from multiple experts, and a baseline algorithm for the detection of retinopathy lesions.<\/jats:p>","DOI":"10.1155\/2013\/368514","type":"journal-article","created":{"date-parts":[[2013,6,19]],"date-time":"2013-06-19T17:03:49Z","timestamp":1371661429000},"page":"1-15","source":"Crossref","is-referenced-by-count":24,"title":["Constructing Benchmark Databases and Protocols for Medical Image Analysis: Diabetic Retinopathy"],"prefix":"10.1155","volume":"2013","author":[{"given":"Tomi","family":"Kauppi","sequence":"first","affiliation":[{"name":"Machine Vision and Pattern Recognition Laboratory, Department of Mathematics and Physics, Lappeenranta University of Technology (LUT), Skinnarilankatu 34, FI-53850 Lappeenranta, Finland"}]},{"given":"Joni-Kristian","family":"K\u00e4m\u00e4r\u00e4inen","sequence":"additional","affiliation":[{"name":"Department of Signal Processing, Tampere University of Technology, Korkeakoulunkatu 10, FI-33720 Tampere, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7691-121X","authenticated-orcid":true,"given":"Lasse","family":"Lensu","sequence":"additional","affiliation":[{"name":"Machine Vision and Pattern Recognition Laboratory, Department of Mathematics and Physics, Lappeenranta University of Technology (LUT), Skinnarilankatu 34, FI-53850 Lappeenranta, Finland"}]},{"given":"Valentina","family":"Kalesnykiene","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Eastern Finland, Yliopistonranta 1, FI-70211 Kuopio, Finland"}]},{"given":"Iiris","family":"Sorri","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Eastern Finland, Yliopistonranta 1, FI-70211 Kuopio, Finland"}]},{"given":"Hannu","family":"Uusitalo","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Tampere, Biokatu 14, FI-33520 Tampere, Finland"}]},{"given":"Heikki","family":"K\u00e4lvi\u00e4inen","sequence":"additional","affiliation":[{"name":"Machine Vision and Pattern Recognition Laboratory, Department of Mathematics and Physics, Lappeenranta University of Technology (LUT), Skinnarilankatu 34, FI-53850 Lappeenranta, Finland"}]}],"member":"311","reference":[{"year":"2004","key":"37"},{"key":"22","series-title":"Lecture Notes in Computer Science","first-page":"832","volume-title":"A framework for constructing benchmark databases and protocols for retinopathy in medical image analysis","volume":"7751","year":"2012"},{"key":"17","series-title":"Lecture Notes in Computer Science","first-page":"193","volume-title":"Combining multiple image segmentations by maximizing expert agreement","year":"2012"},{"issue":"3","key":"33","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1016\/j.cviu.2007.04.006","volume":"109","year":"2008","journal-title":"Computer Vision and Image Understanding"},{"issue":"8","key":"13","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","year":"2006","journal-title":"Pattern Recognition Letters"},{"issue":"3","key":"14","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1109\/34.990138","volume":"24","year":"2002","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"7","key":"27","doi-asserted-by":"crossref","first-page":"1346","DOI":"10.1016\/j.patcog.2006.01.005","volume":"39","year":"2006","journal-title":"Pattern Recognition"},{"issue":"3","key":"16","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1109\/42.845178","volume":"19","year":"2000","journal-title":"IEEE Transactions on Medical Imaging"},{"issue":"8","key":"15","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1109\/TMI.2003.815900","volume":"22","year":"2003","journal-title":"IEEE Transactions on Medical Imaging"},{"issue":"4","key":"31","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1109\/TMI.2004.825627","volume":"23","year":"2004","journal-title":"IEEE Transactions on Medical Imaging"},{"first-page":"648","volume-title":"Comparative study of retinal vessel segmentation a new publicly available database","year":"2004","key":"25"},{"issue":"4","key":"32","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1016\/j.media.2006.05.007","volume":"10","year":"2006","journal-title":"Medical Image Analysis"},{"issue":"1","key":"26","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TMI.2009.2033909","volume":"29","year":"2010","journal-title":"IEEE Transactions on Medical Imaging"},{"issue":"1\u20133","key":"30","first-page":"7","volume":"47","year":"2002","journal-title":"International Journal of Computer Vision"},{"issue":"1\u20133","key":"29","first-page":"157","volume":"77","year":"2008","journal-title":"International Journal of Computer Vision"},{"issue":"3","key":"23","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1109\/34.667881","volume":"20","year":"1998","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"7","key":"35","doi-asserted-by":"crossref","first-page":"903","DOI":"10.1109\/TMI.2004.828354","volume":"23","year":"2004","journal-title":"IEEE Transactions on Medical Imaging"},{"issue":"5","key":"24","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1016\/j.jbi.2005.02.008","volume":"38","year":"2005","journal-title":"Journal of Biomedical Informatics"},{"issue":"10","key":"28","doi-asserted-by":"crossref","first-page":"1090","DOI":"10.1109\/34.879790","volume":"22","year":"2000","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"}],"container-title":["Computational and Mathematical Methods in Medicine"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/cmmm\/2013\/368514.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/cmmm\/2013\/368514.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/cmmm\/2013\/368514.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2017,6,21]],"date-time":"2017-06-21T10:55:49Z","timestamp":1498042549000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.hindawi.com\/journals\/cmmm\/2013\/368514\/"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013]]},"references-count":19,"alternative-id":["368514","368514"],"URL":"https:\/\/doi.org\/10.1155\/2013\/368514","relation":{},"ISSN":["1748-670X","1748-6718"],"issn-type":[{"type":"print","value":"1748-670X"},{"type":"electronic","value":"1748-6718"}],"subject":[],"published":{"date-parts":[[2013]]}}}