{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:33:39Z","timestamp":1760150019737,"version":"build-2065373602"},"reference-count":67,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T00:00:00Z","timestamp":1697068800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002322","name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior-Brasil (CAPES)-Finance","doi-asserted-by":"publisher","award":["001"],"award-info":[{"award-number":["001"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>Software testing and debugging are standard practices of software quality assurance since they enable the identification and correction of failures. Benchmarks have been used in that context as a group of programs to support the comparison of different techniques according to pre-established parameters. However, the reasons that inspire researchers to propose novel benchmarks are not fully understood. This article reports the investigation, identification, classification, and externalization of the state of the art about the proposition of benchmarks on software testing and debugging domains. The study was carried out using systematic mapping procedures according to the guidelines widely followed by software engineering literature. The search identified 1674 studies, from which, 25 were selected for analysis. A list of benchmarks is provided and descriptively mapped according to their characteristics, motivations, and scope of use for their creation. The lack of data to support the comparison between available and novel software testing and debugging techniques is the main motivation for the proposition of benchmarks. Advancements in the standardization and prescription of benchmark structure and composition are still required. Establishing such a standard could foster benchmark reuse, thereby saving time and effort in the engineering of benchmarks for software testing and debugging.<\/jats:p>","DOI":"10.3390\/software2040021","type":"journal-article","created":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T14:22:17Z","timestamp":1697120537000},"page":"447-475","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A Systematic Mapping of the Proposition of Benchmarks in the Software Testing and Debugging Domain"],"prefix":"10.3390","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2921-3718","authenticated-orcid":false,"given":"Deuslirio da","family":"Silva-Junior","sequence":"first","affiliation":[{"name":"Instituto de Inform\u00e1tica, Universidade Federal de Goi\u00e1s, Goi\u00e2nia 74690-900, Goi\u00e1s, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2190-5477","authenticated-orcid":false,"given":"Valdemar V.","family":"Graciano-Neto","sequence":"additional","affiliation":[{"name":"Instituto de Inform\u00e1tica, Universidade Federal de Goi\u00e1s, Goi\u00e2nia 74690-900, Goi\u00e1s, Brazil"}]},{"given":"Diogo M.","family":"de-Freitas","sequence":"additional","affiliation":[{"name":"Instituto de Inform\u00e1tica, Universidade Federal de Goi\u00e1s, Goi\u00e2nia 74690-900, Goi\u00e1s, Brazil"}]},{"given":"Plinio de S\u00e1","family":"Leit\u00e3o-Junior","sequence":"additional","affiliation":[{"name":"Instituto de Inform\u00e1tica, Universidade Federal de Goi\u00e1s, Goi\u00e2nia 74690-900, Goi\u00e1s, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3647-8511","authenticated-orcid":false,"given":"Mohamad","family":"Kassab","sequence":"additional","affiliation":[{"name":"Engineering Division, The Pennsylvania State University, Malvern, PA 16801, USA"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,12]]},"reference":[{"key":"ref_1","unstructured":"Pelliccione, P., Kobetski, A., Larsson, T., Aramrattana, M., Aderum, T., Agren, S.M., Jonsson, G., Heldal, R., Bergenhem, C., and Thors\u00e9n, A. (December, January 28). Architecting cars as constituents of a system of systems. Proceedings of the International Colloquium on Software-Intensive Systems-of-Systems at 10th European Conference on Software Architecture (SiSoSECSA\u201916), Copenhagen, Denmark."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1109\/MCOM.2018.1701065","article-title":"A city-wide real-time traffic management system: Enabling crowdsensing in social internet of vehicles","volume":"56","author":"Wang","year":"2018","journal-title":"IEEE Commun. Mag."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Horita, F.E.A., Rhodes, D.H., Inoc\u00eancio, T.J., and Gonzales, G.R. (2019, January 20\u201324). Building a conceptual architecture and stakeholder map of a system-of-systems for disaster monitoring and early-warning: A case study in brazil. Proceedings of the XV Brazilian Symposium on Information Systems (SBSI\u201919), Aracaju, Brazil.","DOI":"10.1145\/3330204.3330215"},{"key":"ref_4","unstructured":"Cha, S., Taylor, R.N., and Kang, K. (2019). Software Testing, Springer International Publishing."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1109\/MSP.2009.56","article-title":"The real cost of software errors","volume":"7","author":"Zhivich","year":"2009","journal-title":"IEEE Secur. Priv."},{"key":"ref_6","unstructured":"(1994). IEEE Guide for Software Verification and Validation Plans. Standard No. IEEE Std 1059-1993."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lopes, V.C., Norberto, M., RS, D.W., Kassab, M., da Silva Soares, A., Oliveira, R., and Neto, V.V.G. (2020, January 20\u201321). A systematic mapping study on software testing for systems-of-systems. Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing, Natal, Brazil.","DOI":"10.1145\/3425174.3425216"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"68905","DOI":"10.1109\/ACCESS.2021.3077755","article-title":"Exploring the profiles of software testing jobs in the United States","volume":"9","author":"Kassab","year":"2021","journal-title":"IEEE Access"},{"key":"ref_9","unstructured":"Myers, G.J., and Sandler, C. (2004). The Art of Software Testing, John Wiley Sons, Inc."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1147\/sj.411.0004","article-title":"Software debugging, testing, and verification","volume":"41","author":"Hailpern","year":"2002","journal-title":"IBM Syst. J."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1109\/MS.2017.3571582","article-title":"Software testing: The state of the practice","volume":"34","author":"Kassab","year":"2017","journal-title":"IEEE Softw."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1109\/TSMC.1986.289308","article-title":"Expertise in debugging computer programs: An analysis of the content of verbal protocols","volume":"16","author":"Vessey","year":"1986","journal-title":"IEEE Trans. Syst. Man. Cybern."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/MS.2008.109","article-title":"Automating software testing using program analysis","volume":"25","author":"Godefroid","year":"2008","journal-title":"IEEE Softw."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"De-Freitas, D.M., Leitao-Junior, P.S., Camilo-Junior, C.G., and Harrison, R. (2018, January 8\u201313). Mutation-based evolutionary fault localisation. Proceedings of the 2018 IEEE Congress on Evolutionary Computation (CEC), Rio de Janeiro, Brazil.","DOI":"10.1109\/CEC.2018.8477719"},{"key":"ref_15","unstructured":"Goues, C.L., Dewey-Vogt, M., Forrest, S., and Weimer, W. (2012, January 2\u20139). A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. Proceedings of the 2012 34th International Conference on Software Engineering (ICSE), Zurich, Switzerland."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sim, S.E., Easterbrook, S., and Holt, R.C. (2003, January 3\u201310). Using benchmarking to advance research: A challenge to software engineering. Proceedings of the 25th International Conference on Software Engineering (ICSE\u201903), Portland, OR, USA. Available online: http:\/\/dl.acm.org\/citation.cfm?id=776816.776826.","DOI":"10.1109\/ICSE.2003.1201189"},{"key":"ref_17","unstructured":"McDaniel, G. (1994). IBM Dictionary of Computing, McGraw-Hill, Inc."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1109\/TSE.2016.2521368","article-title":"A survey on software fault localization","volume":"42","author":"Wong","year":"2016","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/TSE.2017.2755013","article-title":"Automatic software repair: A survey","volume":"45","author":"Gazzola","year":"2019","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M. (2008, January 26\u201327). Systematic mapping studies in software engineering. Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE\u201908), Bari, Italy.","DOI":"10.14236\/ewic\/EASE2008.8"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ammann, P., and Offutt, J. (2017). Introduction to Software Testing, Cambridge University Press. [2nd ed.].","DOI":"10.1017\/9781316771273"},{"key":"ref_22","unstructured":"Bertolino, A. (2000). Guide to the Software Engineering Body of Knowledge SWEBOK (v 07), IEEE."},{"key":"ref_23","unstructured":"Burnstein, I. (2003). Practical Software Testing: A Process-Oriented Approach, Springer. Springer Professional Computing."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Parnin, C., and Orso, A. (2011, January 17\u201321). Are automated debugging techniques actually helping programmers?. Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA\u201911), Toronto, ON, Canada.","DOI":"10.1145\/2001420.2001445"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"B\u00f6hme, M., Soremekun, E.O., Chattopadhyay, S., Ugherughe, E., and Zeller, A. (2017, January 4\u20138). Where is the bug and how is it fixed? An experiment with practitioners. Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC\/FSE 2017, Paderborn, Germany.","DOI":"10.1145\/3106237.3106255"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"B\u00f6hme, M., Soremekun, E.O., Chattopadhyay, S., Ugherughe, E.J., and Zeller, A. (2017, January 20\u201328). How Developers Debug Software\u2014The DBGBENCH Dataset. Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C), Buenos Aires, Argentina.","DOI":"10.1109\/ICSE-C.2017.94"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Qi, Y., Mao, X., Lei, Y., Dai, Z., and Wang, C. (7, January May). The strength of random search on automated program repair. Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), Hyderabad, India.","DOI":"10.1145\/2568225.2568254"},{"key":"ref_28","unstructured":"Hutchins, M., Foster, H., Goradia, T., and Ostrand, T. (1994, January 16\u201321). Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. Proceedings of the 16th International Conference on Software Engineering, Sorrento, Italy."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1049\/iet-sen.2018.5137","article-title":"Software fault localisation: A systematic mapping study","volume":"13","author":"Zakari","year":"2019","journal-title":"IET Softw."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Andrews, J.H., Briand, L.C., and Labiche, Y. (2005, January 15\u201321). Is mutation an appropriate tool for testing experiments? [software testing]. Proceedings of the 27th International Conference on Software Engineering (ICSE 2005), St. Louis, MO, USA.","DOI":"10.1145\/1062455.1062530"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1016\/j.infsof.2009.04.016","article-title":"Higher order mutation testing","volume":"51","author":"Jia","year":"2009","journal-title":"Inf. Softw. Technol."},{"key":"ref_32","unstructured":"(2019, January 15). Program-Repair.org-Community-Driven Website on Automated Program Repair (Automatic Bug Fixing). Available online: http:\/\/program-repair.org\/."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Just, R., Jalali, D., and Ernst, M.D. (2014, January 21\u201325). Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014), San Jose, CA, USA.","DOI":"10.1145\/2610384.2628055"},{"key":"ref_34","unstructured":"Tan, S.H., Yi, J., Mechtaev, S., and Roychoudhury, A. (2017, January 20\u201328). Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools. Proceedings of the 39th International Conference on Software Engineering Companion (ICSE-C\u201917), Buenos Aires, Argentina."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1109\/TSE.2015.2454513","article-title":"The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs","volume":"41","author":"Goues","year":"2015","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_36","unstructured":"Eytani, Y., and Ur, S. (2004, January 26\u201330). Compiling a benchmark of documented multi-threaded bugs. Proceedings of the International Parallel and Distributed Processing Symposium, IPDPS 2004 (Abstracts and CD-ROM), Santa Fe, NM, USA. Available online: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-12444295463%7B&%7DpartnerID=40%7B&%7Dmd5=f7c2175aeb242de8e8c043cce33de7db."},{"key":"ref_37","unstructured":"Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, Durham University. Technical Report EBSE, 2007-001."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Guessi, M., Graciano Neto, V.V., Bianchi, T., Felizardo, K., Oquendo, F., and Nakagawa, E.Y. (2015, January 13\u201317). A systematic literature review on the description of software architectures for systems of systems. Proceedings of the 30th Annual ACM Symposium on Applied Computing (SAC\u201915), New York, NY, USA. Association for Computing Machinery.","DOI":"10.1145\/2695664.2695795"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1109\/MS.2005.6","article-title":"Evidence-based software engineering for practitioners","volume":"22","author":"Dyba","year":"2005","journal-title":"IEEE Softw."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.infsof.2015.03.007","article-title":"Guidelines for conducting systematic mapping studies in software engineering: An update","volume":"64","author":"Petersen","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lin, D., Koppel, J., Chen, A., and Solar-Lezama, A. (2017, January 22\u201327). QuixBugs: A Multi-lingual Program Repair Benchmark Set Based on the Quixey Challenge. Proceedings of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH Companion 2017, Vancouver, BC, Canada.","DOI":"10.1145\/3135932.3135941"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Dallmeier, V., and Zimmermann, T. (2007, January 5\u20139). Extraction of Bug Localization Benchmarks from History. Proceedings of the Twenty-Second IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201907), Atlanta, GA, USA.","DOI":"10.1145\/1321631.1321702"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Rungta, N., and Mercer, E.G. (2009, January 19\u201320). Clash of the Titans: Tools and Techniques for Hunting Bugs in Concurrent Programs. Proceedings of the 7th Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging (PADTAD\u201909), Chicago, IL, USA.","DOI":"10.1145\/1639622.1639631"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Schuler, D., and Zeller, A. (2010, January 6\u201310). (Un-)Covering Equivalent Mutants. Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation (ICST\u201910), Paris, France.","DOI":"10.1109\/ICST.2010.30"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Bissyande, T.F., Thung, F., Wang, S., Lo, D., Jiang, L., and Reveillere, L. (2013, January 5\u20138). Empirical Evaluation of Bug Linking. Proceedings of the 2013 17th European Conference on Software Maintenance and Reengineering (CSMR\u201913), Genova, Italy.","DOI":"10.1109\/CSMR.2013.19"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Lamkanfi, A., P\u00e9rez, J., and Demeyer, S. (2013, January 18\u201319). The Eclipse and Mozilla Defect Tracking Dataset: A Genuine Dataset for Mining Bug Information. Proceedings of the 10th Working Conference on Mining Software Repositories (MSR\u201913), San Francisco, CA, USA. Available online: http:\/\/dl.acm.org\/citation.cfm?id=2487085.2487125.","DOI":"10.1109\/MSR.2013.6624028"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Abal, I., Brabrand, C., and Wasowski, A. (2014, January 15\u201319). 42 Variability Bugs in the Linux Kernel: A Qualitative Analysis. Proceedings of the 29th ACM\/IEEE International Conference on Automated Software Engineering (ASE\u201914), Vsters, Sweden.","DOI":"10.1145\/2642937.2642990"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Saha, R.K., Lawall, J., Khurshid, S., and Perry, D.E. (October, January 29). On the Effectiveness of Information Retrieval Based Bug Localization for C Programs. Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, Canada.","DOI":"10.1109\/ICSME.2014.38"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Varvaressos, S., Lavoie, K., Mass\u00e9, A.B., Gaboury, S., and Hall\u00e9, S. (4, January 31). Automated bug finding in video games: A case study for runtime monitoring. Proceedings of the 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, Cleveland, OH, USA.","DOI":"10.1109\/ICST.2014.27"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Ohira, M., Kashiwa, Y., Yamatani, Y., Yoshiyuki, H., Maeda, Y., Limsettho, N., Fujino, K., Hata, H., Ihara, A., and Matsumoto, K. (2015, January 16\u201317). A Dataset of High Impact Bugs: Manually-Classified Issue Reports. Proceedings of the 2015 IEEE\/ACM 12th Working Conference on Mining Software Repositories, Florence, Italy.","DOI":"10.1109\/MSR.2015.78"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Leesatapornwongsa, T., Lukman, J.F., Lu, S., and Gunawi, H.S. (2016, January 2\u20136). TaxDC: A taxonomy of non-deterministic concurrency bugs in datacenter distributed systems. Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems-ASPLOS, Atlanta, GA, USA.","DOI":"10.1145\/2872362.2872374"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Yi, J., Ahmed, U.Z., Karkare, A., Tan, S.H., and Roychoudhury, A. (2017, January 4\u20138). A Feasibility Study of Using Automated Program Repair for Introductory Programming Assignments. Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE 2017), Paderborn, Germany.","DOI":"10.1145\/3106237.3106262"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1016\/j.cose.2016.12.002","article-title":"Automatic detection and demonstrator generation for information flow leaks in object-oriented programs","volume":"67","author":"Do","year":"2017","journal-title":"Comput. Secur."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Saha, R.K., Lyu, Y., Yoshida, H., and Prasad, M.R. (November, January 30). ELIXIR: Effective Object Oriented Program Repair. Proceedings of the 32Nd IEEE\/ACM International Conference on Automated Software Engineering (ASE 2017), Urbana, IL, USA. Available online: http:\/\/dl.acm.org\/citation.cfm?id=3155562.3155643.","DOI":"10.1109\/ASE.2017.8115675"},{"key":"ref_55","unstructured":"Reis, S., and Abreu, R. CEUR Workshop Proceedings, Available online: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85035242201%7B&%7DpartnerID=40%7B&%7Dmd5=2677afd851481f16b0d8c44668e1d16b."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Timperley, C.S., Afzal, A., Katz, D.S., Hernandez, J.M., and Goues, C.L. (2018, January 9\u201313). Crashing Simulated Planes is Cheap: Can Simulation Detect Robotics Bugs Early?. Proceedings of the 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), V\u00e4ster\u00e5s, Sweden.","DOI":"10.1109\/ICST.2018.00040"},{"key":"ref_57","unstructured":"Fan, L., Su, T., Chen, S., Meng, G., Liu, Y., Xu, L., Pu, G., and Su, Z. (June, January 27). Large-scale Analysis of Framework-specific Exceptions in Android Apps. Proceedings of the 40th International Conference on Software Engineering (ICSE\u201918), Gothenburg, Sweden."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.jss.2018.03.046","article-title":"Mining repair model for exception-related bug","volume":"141","author":"Zhong","year":"2018","journal-title":"J. Syst. Softw."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Rahman, M.R., Golagha, M., and Pretschner, A. (2018\u20133, January 27). Pairika: A Failure Diagnosis Benchmark for C++ Programs. Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings (ICSE\u201918), Gothenburg, Sweden.","DOI":"10.1145\/3183440.3195097"},{"key":"ref_60","unstructured":"Tan, S.H., Dong, Z., Gao, X., and Roychoudhury, A. (June, January 27). Repairing Crashes in Android Apps. Proceedings of the 40th International Conference on Software Engineering (ICSE\u201918), Gothenburg, Sweden."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"833","DOI":"10.1016\/j.infsof.2008.01.006","article-title":"Empirical studies of agile software development: A systematic review","volume":"50","year":"2008","journal-title":"Inf. Softw. Technol."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1016\/j.infsof.2010.05.003","article-title":"A systematic review of comparative evidence of aspect-oriented programming","volume":"52","author":"Ali","year":"2010","journal-title":"Inf. Softw. Technol."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.infsof.2014.06.009","article-title":"Software test-code engineering: A systematic mapping","volume":"58","author":"Amannejad","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"1936","DOI":"10.1007\/s10664-016-9470-4","article-title":"Automatic repair of real bugs in java: A large-scale experiment on the defects4j dataset","volume":"22","author":"Martinez","year":"2017","journal-title":"Empir. Softw. Eng."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M.D., Pang, D., and Keller, B. (2017, January 20\u201328). Evaluating and improving fault localization. Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE), Buenos Aires, Argentina.","DOI":"10.1109\/ICSE.2017.62"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Snelting, G. (1998, January 16). Concept analysis\u2014A new framework for program understanding. Proceedings of the 1998 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE\u201998), Montreal, QC, Canada.","DOI":"10.1145\/277631.277634"},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Dyba, T., Dingsoyr, T., and Hanssen, G.K. (2007, January 20\u201321). Applying systematic reviews to diverse study types: An experience report. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), Madrid, Spain.","DOI":"10.1109\/ESEM.2007.59"}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/2\/4\/21\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:05:52Z","timestamp":1760130352000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/2\/4\/21"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,12]]},"references-count":67,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["software2040021"],"URL":"https:\/\/doi.org\/10.3390\/software2040021","relation":{},"ISSN":["2674-113X"],"issn-type":[{"type":"electronic","value":"2674-113X"}],"subject":[],"published":{"date-parts":[[2023,10,12]]}}}