{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T16:53:06Z","timestamp":1780764786810,"version":"3.54.1"},"reference-count":111,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,12,5]],"date-time":"2018-12-05T00:00:00Z","timestamp":1543968000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["13\/RC\/2094"],"award-info":[{"award-number":["13\/RC\/2094"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2019,1,31]]},"abstract":"<jats:p>Feature location (FL) is the task of finding the source code that implements a specific, user-observable functionality in a software system. It plays a key role in many software maintenance tasks and a wide variety of Feature Location Techniques (FLTs), which rely on source code structure or textual analysis, have been proposed by researchers. As FLTs evolve and more novel FLTs are introduced, it is important to perform comparison studies to investigate \u201cWhich are the best FLTs?\u201d However, an initial reading of the literature suggests that performing such comparisons would be an arduous process, based on the large number of techniques to be compared, the heterogeneous nature of the empirical designs, and the lack of transparency in the literature. This article presents a systematic review of 170 FLT articles, published between the years 2000 and 2015. Results of the systematic review indicate that 95% of the articles studied are directed towards novelty, in that they propose a novel FLT. Sixty-nine percent of these novel FLTs are evaluated through standard empirical methods but, of those, only 9% use baseline technique(s) in their evaluations to allow cross comparison with other techniques. The heterogeneity of empirical evaluation is also clearly apparent: altogether, over 60 different FLT evaluation metrics are used across the 170 articles, 272 subject systems have been used, and 235 different benchmarks employed. The review also identifies numerous user input formats as contributing to the heterogeneity. Analysis of the existing research also suggests that only 27% of the FLTs presented might be reproduced from the published material. These findings suggest that comparison across the existing body of FLT evaluations is very difficult. We conclude by providing guidelines for empirical evaluation of FLTs that may ultimately help to standardise empirical research in the field, cognisant of FLTs with different goals, leveraging common practices in existing empirical evaluations and allied with rationalisations. This is seen as a step towards standardising evaluation in the field, thus facilitating comparison across FLTs.<\/jats:p>","DOI":"10.1145\/3280988","type":"journal-article","created":{"date-parts":[[2018,12,5]],"date-time":"2018-12-05T08:04:59Z","timestamp":1543997099000},"page":"1-58","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["The State of Empirical Evaluation in Static Feature Location"],"prefix":"10.1145","volume":"28","author":[{"given":"Abdul","family":"Razzaq","sequence":"first","affiliation":[{"name":"The Irish Software Research Centre, LERO and CSIS, University of Limerick, LERO, Limerick, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Asanka","family":"Wasala","sequence":"additional","affiliation":[{"name":"The Irish Software Research Centre, LERO and CSIS, University of Limerick, Limerick, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris","family":"Exton","sequence":"additional","affiliation":[{"name":"The Irish Software Research Centre, LERO and CSIS, University of Limerick, Limerick, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jim","family":"Buckley","sequence":"additional","affiliation":[{"name":"The Irish Software Research Centre, LERO and CSIS, University of Limerick, LERO, Limerick, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,12,5]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2003.1183929"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.567"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.1016"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2017.10.048"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2648511.2648517"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1082983.1083129"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276933.1276934"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.36"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 2013 International Conference on Software Engineering. 522--531","author":"Panichella A.","unstructured":"A. Panichella , B. Dit , R. Oliveto , M. D. Penta , D. Poshyvanyk , and A. D. Lucia . 2013. How to effectively use topic models for software engineering tasks? An approach based on genetic algorithms . In Proceedings of the 2013 International Conference on Software Engineering. 522--531 . A. Panichella, B. Dit, R. Oliveto, M. D. Penta, D. Poshyvanyk, and A. D. Lucia. 2013. How to effectively use topic models for software engineering tasks? An approach based on genetic algorithms. In Proceedings of the 2013 International Conference on Software Engineering. 522--531."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1183005.1183006"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"J. Rubin and M. Chechik. 2013. A survey of feature location techniques. In Domain Engineering: Product Lines Languages and Conceptual Models. Springer Berlin 29--58. J. Rubin and M. Chechik. 2013. A survey of feature location techniques. In Domain Engineering: Product Lines Languages and Conceptual Models. Springer Berlin 29--58.","DOI":"10.1007\/978-3-642-36654-3_2"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.11.013"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.27"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(02)00052-3"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2824251"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2011.72"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/13487689.13487691"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581390"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2479232"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591064"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"N. Wilde M. Buckellew H. Page and V. Rajlich. A case study of feature location in unstructured legacy Fortran code. 68--76. N. Wilde M. Buckellew H. Page and V. Rajlich. A case study of feature location in unstructured legacy Fortran code. 68--76.","DOI":"10.1109\/CSMR.2001.914970"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2377656.2377660"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.45"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9233-9"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.32"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 17th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD'16)","author":"Sun X.","unstructured":"X. Sun , X. Liu , B. Li , Y. Duan , H. Yang , and J. Hu . 2016. Exploring topic models in software engineering data analysis: A survey . In Proceedings of the 17th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD'16) . IEEE, 357--362. X. Sun, X. Liu, B. Li, Y. Duan, H. Yang, and J. Hu. 2016. Exploring topic models in software engineering data analysis: A survey. In Proceedings of the 17th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD'16). IEEE, 357--362."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2993412.3003390"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2522920.2522930"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1134285.1134428"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09)","author":"Petrenko M.","unstructured":"M. Petrenko and V. Rajlich . 2009. Variable granularity for improving precision of impact analysis . In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09) . IEEE, 10--19. M. Petrenko and V. Rajlich. 2009. Variable granularity for improving precision of impact analysis. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09). IEEE, 10--19."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.28"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985441.1985451"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 34th International Conference on Software Engineering. 14--24","author":"Zhou J.","unstructured":"J. Zhou , H. Zhang , and D. Lo . 2012. Where should the bugs be fixed? -- More accurate information retrieval-based bug localization based on bug reports . In Proceedings of the 34th International Conference on Software Engineering. 14--24 . J. Zhou, H. Zhang, and D. Lo. 2012. Where should the bugs be fixed? -- More accurate information retrieval-based bug localization based on bug reports. In Proceedings of the 34th International Conference on Software Engineering. 14--24."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9095-3"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-013-0199-y"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-011-9194-4"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693093"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852761.1852777"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09)","author":"Revelle M.","unstructured":"M. Revelle and D. Poshyvanyk . 2009. An exploratory study on assessing feature location techniques . In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09) . IEEE, 218--222. M. Revelle and D. Poshyvanyk. 2009. An exploratory study on assessing feature location techniques. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09). IEEE, 218--222."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.04.002"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100062"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1041053"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 25th International Conference on Software Engineering. 125--135","author":"Marcus A.","unstructured":"A. Marcus and J. I. Maletic . 2003. Recovering documentation-to-source-code traceability links using latent semantic indexing . In Proceedings of the 25th International Conference on Software Engineering. 125--135 . A. Marcus and J. I. Maletic. 2003. Recovering documentation-to-source-code traceability links using latent semantic indexing. In Proceedings of the 25th International Conference on Software Engineering. 125--135."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, Italy. 68--77","author":"Petersen K.","unstructured":"K. Petersen , R. Feldt , S. Mujtaba , and M. Mattsson . 2008. Systematic mapping studies in software engineering . In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, Italy. 68--77 . K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, Italy. 68--77."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.52"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1475"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00127447"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.1678"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance (ICSM'09)","author":"Gay G.","unstructured":"G. Gay , S. Haiduc , A. Marcus , and T. Menzies . 2009. On the use of relevance feedback in IR-based concept location . In Proceedings of the IEEE International Conference on Software Maintenance (ICSM'09) . IEEE, 351--360. G. Gay, S. Haiduc, A. Marcus, and T. Menzies. 2009. On the use of relevance feedback in IR-based concept location. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM'09). IEEE, 351--360."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.04.004"},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","unstructured":"C. Wohlin P. Runeson M. H\u00f6st M. C. Ohlsson B. Regnell and A. Wessln. 2012. Experimentation in Software Engineering. Springer Publishing Company Inc. Berlin 2012. C. Wohlin P. Runeson M. H\u00f6st M. C. Ohlsson B. Regnell and A. Wessln. 2012. Experimentation in Software Engineering. Springer Publishing Company Inc. Berlin 2012.","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_2_1_54_1","volume-title":"Inc.","author":"Juristo N.","year":"2010","unstructured":"N. Juristo and A. M. Moreno . 2010. Basics of Software Engineering Experimentation. Springer Publishing Company , Inc. , 2010 . N. Juristo and A. M. Moreno. 2010. Basics of Software Engineering Experimentation. Springer Publishing Company, Inc., 2010."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.56"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316447303300309"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1093\/ptj\/85.3.257"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2001.972749"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.04.003"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13)","author":"Diaz D.","unstructured":"D. Diaz , G. Bavota , A. Marcus , R. Oliveto , S. Takahashi , and A. D. Lucia . 2013. Using code ownership to improve IR-based Traceability Link Recovery . In Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13) . IEEE, 123--132. D. Diaz, G. Bavota, A. Marcus, R. Oliveto, S. Takahashi, and A. D. Lucia. 2013. Using code ownership to improve IR-based Traceability Link Recovery. In Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13). IEEE, 123--132."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2013.48"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9255-y"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321631.1321702"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(98)00101-3"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/1189748.1189751"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1810295.1810345"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2011.6080780"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.38"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-011-9159-7"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13)","author":"Bassett B.","unstructured":"B. Bassett and N. A. Kraft . 2013. Structural information based term weighting in text retrieval for feature location . In Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13) . IEEE, 133--141. B. Bassett and N. A. Kraft. 2013. Structural information based term weighting in text retrieval for feature location. In Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13). IEEE, 133--141."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2597008.2597148"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2007.31"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/1314493.1314495"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321631.1321667"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9088-2"},{"key":"e_1_2_1_78_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories. 131--134","author":"Dit B.","unstructured":"B. Dit , A. Holtzhauer , D. Poshyvanyk , and H. Kagdi . 2013. A dataset from change history to support evaluation of software maintenance tasks . In Proceedings of the 10th Working Conference on Mining Software Repositories. 131--134 . B. Dit, A. Holtzhauer, D. Poshyvanyk, and H. Kagdi. 2013. A dataset from change history to support evaluation of software maintenance tasks. In Proceedings of the 10th Working Conference on Mining Software Repositories. 131--134."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.24"},{"key":"e_1_2_1_80_1","unstructured":"Y. Shin J. H. Hayes and J. Cleland-Huang. 2012. A framework for evaluating traceability benchmark metrics. Technical Reports. 21. https:\/\/via.library.depaul.edu\/tr\/21. Y. Shin J. H. Hayes and J. Cleland-Huang. 2012. A framework for evaluating traceability benchmark metrics. Technical Reports. 21. https:\/\/via.library.depaul.edu\/tr\/21."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2013.59"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.39"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, 309--318","author":"Sisman B.","unstructured":"B. Sisman and A. C. Kak . 2013. Assisting code search with automatic query reformulation for bug localization . In Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, 309--318 . B. Sisman and A. C. Kak. 2013. Assisting code search with automatic query reformulation for bug localization. In Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, 309--318."},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786859"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078841"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2008.33"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806827"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9290-8"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2006.10.017"},{"key":"e_1_2_1_90_1","volume-title":"Proceedings of the 37th International Conference on Software Engineering","volume":"2","author":"Maletic J. I.","unstructured":"J. I. Maletic and M. L. Collard . 2015. Exploration, analysis, and manipulation of source code using srcML . In Proceedings of the 37th International Conference on Software Engineering , Vol. 2 . IEEE Press, 951--952. J. I. Maletic and M. L. Collard. 2015. Exploration, analysis, and manipulation of source code using srcML. In Proceedings of the 37th International Conference on Software Engineering, Vol. 2. IEEE Press, 951--952."},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985939"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9327-7"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.71"},{"key":"e_1_2_1_94_1","volume-title":"Guide to Advanced Empirical Software Engineering: Springer-Verlag Inc","author":"Shull F.","year":"2007","unstructured":"F. Shull , J. Singer , and D. I. K. Sjberg . 2007. Guide to Advanced Empirical Software Engineering: Springer-Verlag Inc , New York , 2007 . F. Shull, J. Singer, and D. I. K. Sjberg. 2007. Guide to Advanced Empirical Software Engineering: Springer-Verlag Inc, New York, 2007."},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the 11th Working Conference on Reverse Engineering. IEEE, 214--223","author":"Marcus A.","unstructured":"A. Marcus , A. Sergeyev , V. Rajlich , and J. I. Maletic . 2004. An information retrieval approach to concept location in source code . In Proceedings of the 11th Working Conference on Reverse Engineering. IEEE, 214--223 . A. Marcus, A. Sergeyev, V. Rajlich, and J. I. Maletic. 2004. An information retrieval approach to concept location in source code. In Proceedings of the 11th Working Conference on Reverse Engineering. IEEE, 214--223."},{"key":"e_1_2_1_96_1","volume-title":"Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. 140--150","author":"Hill E.","unstructured":"E. Hill , D. Shepherd , and L. Pollock . 2015. Exploring the use of concern element role information in feature location evaluation . In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. 140--150 . E. Hill, D. Shepherd, and L. Pollock. 2015. Exploring the use of concern element role information in feature location evaluation. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. 140--150."},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.5555\/3044222.3051229"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.42"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.5555\/518049.856972"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9224-x"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2011.47"},{"key":"e_1_2_1_102_1","volume-title":"Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER'16)","author":"Panichella A.","unstructured":"A. Panichella , B. Dit , R. Oliveto , M. D. Penta , D. Poshyvanyk , and A. D. Lucia . 2016. Parameterizing and assembling IR-based solutions for SE tasks using genetic algorithms . In Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER'16) . IEEE, 314--325. A. Panichella, B. Dit, R. Oliveto, M. D. Penta, D. Poshyvanyk, and A. D. Lucia. 2016. Parameterizing and assembling IR-based solutions for SE tasks using genetic algorithms. In Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER'16). IEEE, 314--325."},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1213847"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9339-3"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.03.007"},{"key":"e_1_2_1_106_1","volume-title":"Reporting guidelines for controlled experiments in software engineering. 10","author":"Jedlitschka A.","unstructured":"A. Jedlitschka and D. Pfahl . Reporting guidelines for controlled experiments in software engineering. 10 pp. A. Jedlitschka and D. Pfahl. Reporting guidelines for controlled experiments in software engineering. 10 pp."},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2012.24"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.103"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-007-9053-5"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1145\/2812803"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/2951008"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3280988","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3280988","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T09:18:25Z","timestamp":1775294305000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3280988"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12,5]]},"references-count":111,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,31]]}},"alternative-id":["10.1145\/3280988"],"URL":"https:\/\/doi.org\/10.1145\/3280988","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,12,5]]},"assertion":[{"value":"2017-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-12-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}