{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:21:58Z","timestamp":1760242918886,"version":"build-2065373602"},"reference-count":56,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2016,11,9]],"date-time":"2016-11-09T00:00:00Z","timestamp":1478649600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Since local feature detection has been one of the most active research areas in computer vision during the last decade and has found wide range of applications (such as matching and registration of remotely sensed image data), a large number of detectors have been proposed. The interest in feature-based applications continues to grow and has thus rendered the task of characterizing the performance of various feature detection methods an important issue in vision research. Inspired by the good practices of electronic system design, a generic framework based on the repeatability measure is presented in this paper that allows assessment of the upper and lower bounds of detector performance and finds statistically significant performance differences between detectors as a function of image transformation amount by introducing a new variant of McNemar\u2019s test in an effort to design more reliable and effective vision systems. The proposed framework is then employed to establish operating and guarantee regions for several state-of-the art detectors and to identify their statistical performance differences for three specific image transformations: JPEG compression, uniform light changes and blurring. The results are obtained using a newly acquired, large image database (20,482 images) with 539 different scenes. These results provide new insights into the behavior of detectors and are also useful from the vision systems design perspective. Finally, results for some local feature detectors are presented for a set of remote sensing images to showcase the potential and utility of this framework for remote sensing applications in general.<\/jats:p>","DOI":"10.3390\/rs8110928","type":"journal-article","created":{"date-parts":[[2016,11,9]],"date-time":"2016-11-09T10:09:39Z","timestamp":1478686179000},"page":"928","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["A Generic Framework for Assessing the Performance Bounds of Image Feature Detectors"],"prefix":"10.3390","volume":"8","author":[{"given":"Shoaib","family":"Ehsan","sequence":"first","affiliation":[{"name":"School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adrian","family":"Clark","sequence":"additional","affiliation":[{"name":"School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ales","family":"Leonardis","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Birmingham, Birmingham B15 2TT, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naveed","family":"Ur Rehman","sequence":"additional","affiliation":[{"name":"Electrical Engineering Department, COMSATS Institute of Information Technology, Islamabad 44000, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmad","family":"Khaliq","sequence":"additional","affiliation":[{"name":"College of Electrical and Mechanical Engineering, National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Fasli","sequence":"additional","affiliation":[{"name":"School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Klaus","family":"McDonald-Maier","sequence":"additional","affiliation":[{"name":"School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,11,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/s11263-005-3848-x","article-title":"A Comparison of Affine Region Detectors","volume":"65","author":"Mikolajczyk","year":"2005","journal-title":"Int. J. Comput. Vis."},{"key":"ref_2","unstructured":"Oxford Data Sets. Available online: http:\/\/www.robots.ox.ac.uk\/~vgg\/research\/affine\/."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Matas, J., Chum, O., Urban, M., and Pajdla, T. (2002, January 2\u20135). Robust Wide Baseline Stereo from Maximally Stable Extremal Regions. Proceedings of the British Machine Vision Conference, Cardiff, UK.","DOI":"10.5244\/C.16.36"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Torralba, A., and Efros, A.A. (2011, January 20\u201325). Unbiased Look at Dataset Bias. Proceedings of the Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995347"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1049\/el.2010.1442","article-title":"Improved Repeatability Measures for Evaluating Performance of Feature Detectors","volume":"46","author":"Ehsan","year":"2010","journal-title":"Electron. Lett."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the Sampling Error of the Difference between Correlated Proportions or Percentages","volume":"12","author":"McNemar","year":"1947","journal-title":"Psychometrika"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Fleiss, J.L., Levin, B., and Paik, M.C. (2003). Statistical Methods for Rates and Proportions, John Wiley & Sons. [3rd ed.].","DOI":"10.1002\/0471445428"},{"key":"ref_8","unstructured":"Rutkowski, W.S., and Rosenfeld, A. (1978). A Comparison of Corner Detection Techniques for Chain Coded Curves, University of Maryland. Technical Report."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/0167-8655(82)90020-4","article-title":"Gray-Level Corner Detection","volume":"1","author":"Kitchen","year":"1982","journal-title":"Pattern Recognit. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1109\/34.31447","article-title":"On the Detection of Dominant Points on Digital Curves","volume":"11","author":"Teh","year":"1989","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","unstructured":"Coelho, C., Heller, A., Mundy, J.L., Forsyth, D.A., and Zisserman, A. (1991, January 25\u201328). An Experimental Evaluation of Projective Invariants. Proceedings of the DARPA-ESPRIT Workshop on Applications of Invariants in Computer Vision, Reykjavik, Iceland."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/BF01420733","article-title":"A Computational Approach for Corner and Vertex Detection","volume":"10","author":"Deriche","year":"1993","journal-title":"Int. J. Comput. Vis."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1007\/BF01249893","article-title":"Localization Properties of Direct Corner Detectors","volume":"4","author":"Rohr","year":"1994","journal-title":"J. Math. Imaging Vis."},{"key":"ref_14","unstructured":"Brand, P., and Mohr, R. (1994, January 6). Accuracy in Image Measure. Proceedings of the SPIE Conference on Videometrics III, Boston, MA, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Heyden, A., and Rohr, K. (1996, January 25\u201329). Evaluation of Corner Extraction Schemes using Invariance Methods. Proceedings of the 13th International Conference on Pattern Recognition, Vienna, Austria.","DOI":"10.1109\/ICPR.1996.546153"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1338","DOI":"10.1109\/34.643893","article-title":"A Robust Visual Method for Assessing the Relative Performance of Edge-Detection Algorithms","volume":"19","author":"Heath","year":"1997","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1109\/TPAMI.1986.4767851","article-title":"A Computational Approach to Edge Detection","volume":"8","author":"Canny","year":"1986","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1199","DOI":"10.1109\/34.632980","article-title":"A Discrete Expression of Canny\u2019s Criteria for Step Edge Detector Performances Evaluation","volume":"19","author":"Demigny","year":"1997","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","unstructured":"Shin, M.C., Goldgof, D., and Bowyer, K.W. (1998, January 25). An Objective Comparison Methodology of Edge Detection Algorithms using a Structure from Motion Task. Proceedings of the Conference on Computer Vision and Pattern Recognition, Santa Barbara, CA, USA."},{"key":"ref_20","unstructured":"Shin, M.C., Goldgof, D., and Bowyer, K.W. (1999, January 23\u201325). Comparison of Edge Detectors using an Object Recognition Task. Proceedings of the Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, USA."},{"key":"ref_21","unstructured":"Bowyer, K., Kranenburg, C., and Dougherty, S. (1999, January 23\u201325). Edge Detector Evaluation using Empirical ROC Curves. Proceedings of the Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, USA."},{"key":"ref_22","unstructured":"Baker, S., and Nayar, S.K. (1999, January 23\u201325). Global Measures of Coherence for Edge Detector Evaluation. Proceedings of the Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/34.761263","article-title":"Evaluation of Methods for Ridge and Valley Detection","volume":"21","author":"Lopez","year":"1999","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1023\/A:1008199403446","article-title":"Evaluation of Interest Point Detectors","volume":"37","author":"Schmid","year":"2000","journal-title":"Int. J. Comput. Vis."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1016\/j.imavis.2003.08.012","article-title":"Evaluation of Salient Point Techniques","volume":"21","author":"Sebe","year":"2003","journal-title":"Image Vis. Comput."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1023\/B:VISI.0000027790.02288.f2","article-title":"Scale & Affine Invariant Interest Point Detectors","volume":"60","author":"Mikolajczyk","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Mohanna, F., and Mokhtarian, F. (2001, January 10\u201313). Performance Evaluation of Corner Detection Algorithms under Similarity and Affine Transforms. Proceedings of the British Machine Vision Conference, Manchester, UK.","DOI":"10.5244\/C.15.37"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.cviu.2005.11.001","article-title":"Performance Evaluation of Corner Detectors using Consistency and Accuracy Measures","volume":"102","author":"Mokhtarian","year":"2006","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/S0262-8856(97)00056-5","article-title":"Fast Corner Detection","volume":"16","author":"Hedley","year":"1998","journal-title":"Image Vis. Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1016\/j.imavis.2004.02.001","article-title":"Assessing the Performance of Corner Detectors for Point Feature Tracking Applications","volume":"22","author":"Tissainayagam","year":"2004","journal-title":"Image Vis. Comput."},{"key":"ref_31","unstructured":"Fraundorfer, F., and Bischof, H. (, January June). Evaluation of Local Detectors on Non-Planar Scenes. Proceedings of the Austrian Association for Pattern Recognition Workshop, Hagenberg, Austria."},{"key":"ref_32","unstructured":"Fraundorfer, F., and Bischof, H. (2005, January 20\u201325). A Novel Performance Evaluation Method of Local Detectors on Non-Planar Scenes. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ehsan, S., Kanwal, N., Clark, A.F., and McDonald-Maier, K.D. (2011, January 22\u201324). Measuring the Coverage of Interest Point Detectors. Proceedings of the 8th International Conference on Image Analysis and Recognition (ICIAR), Burnaby, BC, Canada.","DOI":"10.1007\/978-3-642-21593-3_26"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"10876","DOI":"10.3390\/s130810876","article-title":"Rapid Online Analysis of Local Feature Detectors and their Complementarity","volume":"13","author":"Ehsan","year":"2013","journal-title":"Sensors"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Moreels, P., and Perona, P. (2005, January 17\u201325). Evaluation of Features Detectors and Descriptors Based on 3D Objects. Proceedings of the International Conference on Computer Vision (ICCV), Beijing, China.","DOI":"10.1109\/ICCV.2005.89"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1007\/s11263-006-9967-1","article-title":"Evaluation of features detectors and descriptors based on 3D objects","volume":"73","author":"Moreels","year":"2007","journal-title":"Int. J. Comput. Vis."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Mikolajczyk, K., Leibe, B., and Schiele, B. (2005, January 17\u201325). Local Features for Object Class Recognition. Proceedings of the International Conference on Computer Vision (ICCV), Beijing, China.","DOI":"10.1109\/ICCV.2005.146"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Stark, M., and Schiele, B. (2007, January 14\u201321). How Good are Local Features for Classes of Geometric Objects. Proceedings of the International Conference on Computer Vision, Rio de Janeiro, Brazil.","DOI":"10.1109\/ICCV.2007.4408878"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Seemann, E., Leibe, B., Mikolajczyk, K., and Schiele, B. (2005, January 5\u20138). An Evaluation of Local Shape-based Features for Pedestrian Detection. Proceedings of the British Machine Vision Conference, Oxford, UK.","DOI":"10.5244\/C.19.5"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Dickscheid, T., and F\u00f6rstner, W. (2009, January 13\u201315). Evaluating the Suitability of Feature Detectors for Automatic Image Orientation Systems. Proceedings of the ICVS, Liege, Belgium.","DOI":"10.1007\/978-3-642-04667-4_31"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Haja, A., Jahne, B., and Abraham, S. (2008, January 23\u201328). Localization Accuracy of Region Detectors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587829"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Zeisl, B., Georgel, P., Schweiger, F., Steinbach, E., Navab, N., and Munich, G. (2009, January 7\u201310). Estimation of Location Uncertainty for Scale Invariant Feature Points. Proceedings of the British Machine Vision Conference, London, UK.","DOI":"10.5244\/C.23.57"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1007\/s11263-010-0340-z","article-title":"Coding Images with Local Features","volume":"94","author":"Dickscheid","year":"2010","journal-title":"Int. J. Comput. Vis."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"F\u00f6rstner, W., Dickscheid, T., and Schindler, F. (2009, January 7\u201310). On the Completeness of Coding with Image Features. Proceedings of the British Machine Vision Conference, London, UK.","DOI":"10.5244\/C.23.1"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1007\/s11263-011-0473-8","article-title":"Interesting Interest Points-A Comparative Study of Interest Point Performance on a Unique Data Set","volume":"97","author":"Dahl","year":"2012","journal-title":"Int. J. Comput. Vis."},{"key":"ref_46","unstructured":"Lowe, D.G. (October, January 30). Object Recognition from Local Scale-Invariant Features. Proceedings of the International Conference on Computer Vision, Corfu, Greece."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive Image Features from Scale-Invariant Keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1007\/s00138-009-0195-x","article-title":"A Comparative Evaluation of Interest Point Detectors and Local Descriptors for Visual SLAM","volume":"21","author":"Gil","year":"2010","journal-title":"Mach. Vis. Appl."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Asbach, M., Hosten, P., and Unger, M. (2008, January 7\u20139). An Evaluation of Local Features for Face Detection and Localization. Proceedings of the Ninth International Workshop on Image Analysis for Multimedia Interactive Services, Klagenfurt, Austria.","DOI":"10.1109\/WIAMIS.2008.58"},{"key":"ref_50","unstructured":"JPEG, Blurring and Uniform Light Changes Image Database. Available online: http:\/\/vase.essex.ac.uk\/datasets\/index.html."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.cviu.2007.09.014","article-title":"Speeded-Up Robust Features (SURF)","volume":"110","author":"Bay","year":"2008","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1023\/B:VISI.0000020671.28016.e8","article-title":"Matching Widely Separated Views Based on Affine Invariant Regions","volume":"59","author":"Tuytelaars","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_53","unstructured":"Forstner, W., Dickscheid, T., and Schindler, F. (October, January 27). Detecting Interpretable and Accurate Scale-Invariant Keypoints. Proceedings of the 12th IEEE International Conference on Computer Vision, Kyoto, Japan."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Kadir, T., Zisserman, A., and Brady, M. (2004, January 11\u201314). An Affine Invariant Salient Region Detector. Proceedings of the 8th European Conference on Computer Vision, Prague, Czech Republic.","DOI":"10.1007\/978-3-540-24670-1_18"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1561\/0600000017","article-title":"Local Invariant Feature Detectors: A Survey","volume":"3","author":"Tuytelaars","year":"2007","journal-title":"Found. Trends Comput. Graph. Vis."},{"key":"ref_56","unstructured":"Remote Sensing Images. Available online: http:\/\/www.satimagingcorp.com\/."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/8\/11\/928\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:35:09Z","timestamp":1760211309000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/8\/11\/928"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,11,9]]},"references-count":56,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2016,11]]}},"alternative-id":["rs8110928"],"URL":"https:\/\/doi.org\/10.3390\/rs8110928","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2016,11,9]]}}}