{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T17:42:48Z","timestamp":1772041368882,"version":"3.50.1"},"reference-count":58,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,1,23]],"date-time":"2024-01-23T00:00:00Z","timestamp":1705968000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,23]],"date-time":"2024-01-23T00:00:00Z","timestamp":1705968000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Mocking frameworks provide convenient APIs, which create mock objects, manipulate their behavior, and verify their execution, for the purpose of isolating test dependencies in unit testing. This study contributes an in-depth empirical study of whether and how mocking frameworks are used in Apache projects. The key findings and insights of this study include: First, mocking frameworks are widely used in 66% of Apache Java projects, with Mockito, EasyMock, and PowerMock being the top three most popular frameworks. Larger-scale and more recent projects tend to observe a stronger need to use mocking frameworks. This underscores the importance of mocking in practice and related future research. Second, mocking is overall practiced quite selectively in software projects\u2014not all test files use mocking, nor all dependencies of a test target are mocked. It calls for more future research to gain a more systematic understanding of when and what to mock to provide formal guidance to practitioners. On top of this, the intensity of mocking in different projects shows different trends in the projects\u2019 evolution history\u2014implying the compound effects of various factors, such as the pace of a project\u2019s growth, the available resources, time pressure, and priority, etc. This points to an important future research direction in facilitating best mocking practices in software evolution. Furthermore, we revealed the most frequently used APIs in the three most popular frameworks, organized based on the function types. The top five APIs in each functional type of the three mocking frameworks usually take the majority (78% to 100%) of usage in Apache projects. This indicates that developers can focus on these APIs to quickly learn the common usage of these mocking frameworks. We further investigated informal methods of mocking, which do not rely on any mocking framework. These informal mocking methods point to potential sub-optimal mocking practices that could be improved, as well as limitations of existing mocking frameworks. Finally, we conducted a developer survey to collect additional insights regarding the above analysis based on their experience, which complements our analysis based on repository mining. Overall, this study offers practitioners profound empirical knowledge of how mocking frameworks are used in practice and sheds light on future research directions to enhancing mocking in practice.<\/jats:p>","DOI":"10.1007\/s10664-023-10410-y","type":"journal-article","created":{"date-parts":[[2024,1,23]],"date-time":"2024-01-23T14:02:23Z","timestamp":1706018543000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["An empirical study on the usage of mocking frameworks in Apache software foundation"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3202-3077","authenticated-orcid":false,"given":"Lu","family":"Xiao","sequence":"first","affiliation":[]},{"given":"Gengwu","family":"Zhao","sequence":"additional","affiliation":[]},{"given":"Xiao","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Keye","family":"Li","sequence":"additional","affiliation":[]},{"given":"Erick","family":"Lim","sequence":"additional","affiliation":[]},{"given":"Chenhao","family":"Wei","sequence":"additional","affiliation":[]},{"given":"Tingting","family":"Yu","sequence":"additional","affiliation":[]},{"given":"Xiaoyin","family":"Wang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,1,23]]},"reference":[{"key":"10410_CR1","unstructured":"Apache PDFBox | a java PDF library. https:\/\/pdfbox.apache.org\/"},{"key":"10410_CR2","unstructured":"Apache software foundation projects list. https:\/\/projects.apache.org\/projects.html. Accessed 8 May 2023"},{"key":"10410_CR3","doi-asserted-by":"crossref","unstructured":"Arcuri A, Fraser G, Just R (2017) Private api access and functional mocking in automated unit test generation. In: 2017 IEEE international conference on software testing, verification and validation (ICST), IEEE, pp 126\u2013137","DOI":"10.1109\/ICST.2017.19"},{"key":"10410_CR4","unstructured":"Barker D (2016) Web content management: systems, features, and best practices. \u201c O\u2019Reilly Media, Inc.\u201d"},{"key":"10410_CR5","doi-asserted-by":"publisher","unstructured":"Bertolino A (2007) Software testing research: achievements, challenges, dreams. In: Future of software engineering (FOSE\u201907), IEEE, pp 85\u2013103. https:\/\/doi.org\/10.1109\/FOSE.2007.25","DOI":"10.1109\/FOSE.2007.25"},{"key":"10410_CR6","unstructured":"Briney K (2015) Data Management for Researchers: Organize, maintain and share your data for research success. Pelagic Publishing Ltd"},{"key":"10410_CR7","unstructured":"Buyya R, Dastjerdi AV (2016) Internet of Things: Principles and paradigms. Elsevier"},{"issue":"5","key":"10410_CR8","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1109\/MC.2006.152","volume":"39","author":"K Crowston","year":"2006","unstructured":"Crowston K, Howison J (2006) Assessing the health of open source communities. Computer 39(5):89\u201391","journal-title":"Computer"},{"key":"10410_CR9","doi-asserted-by":"publisher","unstructured":"Daka E, Fraser G (2014) A survey on unit testing practices and problems. In: 2014 IEEE 25th International symposium on software reliability engineering, IEEE, pp 201\u2013211. https:\/\/doi.org\/10.1109\/ISSRE.2014.11","DOI":"10.1109\/ISSRE.2014.11"},{"key":"10410_CR10","unstructured":"dev: Apache wink \u2013 index. https:\/\/wink.apache.org\/"},{"issue":"6","key":"10410_CR11","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MS.2007.157","volume":"24","author":"JC Duenas","year":"2007","unstructured":"Duenas JC, Cuadrado F, Santill\u00e1n M, Ruiz JL et al (2007) Apache and eclipse: comparing open source project incubators. IEEE Soft 24(6):90\u201398","journal-title":"IEEE Soft"},{"key":"10410_CR12","unstructured":"EasyMock. https:\/\/easymock.org\/"},{"key":"10410_CR13","doi-asserted-by":"crossref","unstructured":"Fazzini M, Choi C, Copia JM, Lee G, Kakehi Y, Gorla A, Orso A (2022) Use of test doubles in android testing: an in-depth investigation. In: Proceedings of the 44th international conference on software engineering, pp 2266\u20132278","DOI":"10.1145\/3510003.3510175"},{"key":"10410_CR14","doi-asserted-by":"crossref","unstructured":"Fazzini M, Gorla A, Orso A (2020) A framework for automated test mocking of mobile apps. In: 2020 35th IEEE\/ACM International conference on automated software engineering (ASE), IEEE, pp 1204\u20131208","DOI":"10.1145\/3324884.3418927"},{"issue":"2","key":"10410_CR15","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1353\/rhe.2017.0003","volume":"40","author":"K Fosnacht","year":"2017","unstructured":"Fosnacht K, Sarraf S, Howe E, Peck LK (2017) How important are high response rates for college surveys? Rev Higher Educ 40(2):245\u2013265","journal-title":"Rev Higher Educ"},{"key":"10410_CR16","doi-asserted-by":"crossref","unstructured":"Freeman S, Mackinnon T, Pryce N, Walnes J (2004) jmock: supporting responsibility-based design with mock objects. In: Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, pp 4\u20135","DOI":"10.1145\/1028664.1028667"},{"key":"10410_CR17","doi-asserted-by":"publisher","unstructured":"Freeman S, Mackinnon T, Pryce N, Walnes J (2004) Mock roles, not objects. In: Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, pp 236\u2013246. https:\/\/doi.org\/10.1145\/1028664.1028765","DOI":"10.1145\/1028664.1028765"},{"key":"10410_CR18","doi-asserted-by":"publisher","unstructured":"Garousi V, Zhi J (2013) A survey of software testing practices in Canada. J Syst Softw 86(5):1354\u20131376. https:\/\/doi.org\/10.1016\/j.jss.2012.12.051. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121212003561","DOI":"10.1016\/j.jss.2012.12.051"},{"key":"10410_CR19","unstructured":"Henderson F (2017) Software engineering at google. arXiv:1702.01715"},{"key":"10410_CR20","unstructured":"Hunt A, Thomas D (2004) Pragmatic unit testing in c# with nunit. Pragmatic Programmers"},{"key":"10410_CR21","doi-asserted-by":"publisher","unstructured":"Ieee standard glossary of software engineering terminology (1990) IEEE Std 610(12\u20131990):1\u201384. https:\/\/doi.org\/10.1109\/IEEESTD.1990.101064","DOI":"10.1109\/IEEESTD.1990.101064"},{"key":"10410_CR22","unstructured":"JUnit 5. https:\/\/junit.org\/junit5\/"},{"key":"10410_CR23","unstructured":"Kaner C, Falk J, Nguyen HQ (1999) Testing computer software. John Wiley & Sons"},{"key":"10410_CR24","unstructured":"Karlesky M, Williams G, Bereza W, Fletcher M (2007) Mocking the embedded world: test-driven development, continuous integration, and design patterns. Proc. Emb. Systems Conf, CA, USA, pp 1518\u20131532"},{"key":"10410_CR25","unstructured":"Kim SS (2016) Mocking embedded hardware for software validation. Ph.D. thesis"},{"key":"10410_CR26","unstructured":"List of releases for the project. https:\/\/github.com\/gzhao9\/Mock-Apache-Empirical-Study\/blob\/main\/RQ1\/project%20tags_info.csv"},{"key":"10410_CR27","doi-asserted-by":"publisher","unstructured":"Marri MR, Xie T, Tillmann N, De\u00a0Halleux J, Schulte W (2009) An empirical study of testing file-system-dependent software with mock objects. In: 2009 ICSE Workshop on automation of software test, IEEE, pp 149\u2013153. https:\/\/doi.org\/10.1007\/s10664-018-9663-0","DOI":"10.1007\/s10664-018-9663-0"},{"key":"10410_CR28","unstructured":"Maven Repository: org.springframework \u00bb spring-mock. https:\/\/mvnrepository.com\/artifact\/org.springframework\/spring-mock"},{"key":"10410_CR29","unstructured":"Mockito release notes. https:\/\/code.google.com\/archive\/p\/mockito\/wikis\/ReleaseNotes.wiki. Accessed 8 May 2023"},{"key":"10410_CR30","unstructured":"mockito. https:\/\/site.mockito.org\/"},{"issue":"3","key":"10410_CR31","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1145\/567793.567795","volume":"11","author":"A Mockus","year":"2002","unstructured":"Mockus A, Fielding RT, Herbsleb JD (2002) Two case studies of open source software development: Apache and mozilla. ACM Trans Softw Eng Methodol (TOSEM) 11(3):309\u2013346","journal-title":"ACM Trans Softw Eng Methodol (TOSEM)"},{"key":"10410_CR32","doi-asserted-by":"crossref","unstructured":"Mockus A, Fielding RT, Herbsleb J (2000) A case study of open source software development: the apache server. In: Proceedings of the 22nd international conference on software engineering, pp 263\u2013272","DOI":"10.1145\/337180.337209"},{"key":"10410_CR33","unstructured":"moq. https:\/\/github.com\/moq\/moq4"},{"key":"10410_CR34","doi-asserted-by":"publisher","unstructured":"Mostafa S, Wang X (2014) An empirical study on the usage of mocking frameworks in software testing. In: 2014 14th international conference on quality software, IEEE, pp 127\u2013132. https:\/\/doi.org\/10.1109\/QSIC.2014.19","DOI":"10.1109\/QSIC.2014.19"},{"key":"10410_CR35","doi-asserted-by":"publisher","unstructured":"Myers GJ, Badgett T, Thomas TM, Sandler C (2004) The art of software testing, vol 2. Wiley Online Library. https:\/\/doi.org\/10.1002\/9781119202486","DOI":"10.1002\/9781119202486"},{"key":"10410_CR36","doi-asserted-by":"publisher","unstructured":"Nandigam J, Gudivada VN, Hamou-Lhadj A, Tao Y (2009) Interface-based object-oriented design with mock objects. In: 2009 Sixth international conference on information technology: new generations, IEEE, pp 713\u2013718. https:\/\/doi.org\/10.1109\/ITNG.2009.268","DOI":"10.1109\/ITNG.2009.268"},{"key":"10410_CR37","unstructured":"NMock: A Dynamic Mock Object Library for .NET. https:\/\/nmock.sourceforge.net\/"},{"key":"10410_CR38","doi-asserted-by":"publisher","unstructured":"Pereira G, Hora A (2020) Assessing mock classes: an empirical study. In: 2020 IEEE International conference on software maintenance and evolution (ICSME), IEEE, pp 453\u2013463. https:\/\/doi.org\/10.1109\/ICSME46990.2020.00050","DOI":"10.1109\/ICSME46990.2020.00050"},{"key":"10410_CR39","unstructured":"PowerMock framework site. https:\/\/powermock.github.io\/"},{"key":"10410_CR40","unstructured":"PowerMock framework site. https:\/\/powermock.github.io\/"},{"key":"10410_CR41","unstructured":"Projects by category in apache software foundation. https:\/\/projects.apache.org\/projects.html?category"},{"key":"10410_CR42","doi-asserted-by":"crossref","unstructured":"Rigby PC, German DM, Storey MA (2008) Open source software peer review practices: a case study of the apache server. In: Proceedings of the 30th international conference on Software engineering, pp 541\u2013550","DOI":"10.1145\/1368088.1368162"},{"issue":"4","key":"10410_CR43","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MS.2006.91","volume":"23","author":"P Runeson","year":"2006","unstructured":"Runeson P (2006) A survey of unit testing practices. IEEE Softw 23(4):22\u201329. https:\/\/doi.org\/10.1109\/MS.2006.91","journal-title":"IEEE Softw"},{"issue":"10","key":"10410_CR44","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/MC.2012.355","volume":"45","author":"C Severance","year":"2012","unstructured":"Severance C (2012) The apache software foundation: Brian behlendorf. Computer 45(10):8\u20139","journal-title":"Computer"},{"key":"10410_CR45","doi-asserted-by":"crossref","unstructured":"Solms F, Marshall L (2016) Contract-based mocking for services-oriented development. In: Proceedings of the annual conference of the south african institute of computer scientists and information technologists, pp 1\u20138","DOI":"10.1145\/2987491.2987534"},{"issue":"3","key":"10410_CR46","doi-asserted-by":"publisher","first-page":"1461","DOI":"10.1007\/s10664-018-9663-0","volume":"24","author":"D Spadini","year":"2019","unstructured":"Spadini D, Aniche M, Bruntink M, Bacchelli A (2019) Mock objects for testing java systems. Empirical Softw Eng 24(3):1461\u20131498. https:\/\/doi.org\/10.1007\/s10664-018-9663-0","journal-title":"Empirical Softw Eng"},{"key":"10410_CR47","doi-asserted-by":"publisher","unstructured":"Spadini D, Aniche M, Bruntink M, Bacchelli A (2017) To mock or not to mock? an empirical study on mocking practices. In: 2017 IEEE\/ACM 14th International conference on mining software repositories (MSR), IEEE, pp 402\u2013412. https:\/\/doi.org\/10.1109\/MSR.2017.61","DOI":"10.1109\/MSR.2017.61"},{"key":"10410_CR48","unstructured":"Svensg\u00e5rd S, Henriksson J (2017) Mocking saas cloud for testing"},{"key":"10410_CR49","doi-asserted-by":"crossref","unstructured":"Taneja K, Zhang Y, Xie T (2010) Moda: automated test generation for database applications via mock objects. In: Proceedings of the IEEE\/ACM international conference on Automated software engineering, pp 289\u2013292","DOI":"10.1145\/1858996.1859053"},{"key":"10410_CR50","unstructured":"Unittest.mock - mock object library. https:\/\/docs.python.org\/3\/library\/unittest.mock.html#module-unittest.mock"},{"key":"10410_CR51","unstructured":"Wang X (2021) Understanding and facilitating the usage of mocking frameworks for test dependency isolation. Ph.D. thesis, Stevens Institute of Technology"},{"key":"10410_CR52","doi-asserted-by":"publisher","unstructured":"Wang X, Xiao L, Yu T, Woepse A, Wong S (2022) From inheritance to mockito: An automatic refactoring approach. IEEE Trans Softw Eng 1\u201323. https:\/\/doi.org\/10.1109\/TSE.2022.3231850","DOI":"10.1109\/TSE.2022.3231850"},{"key":"10410_CR53","doi-asserted-by":"crossref","unstructured":"Wang X, Xiao L, Yu T, Woepse A, Wong S (2021) An automatic refactoring framework for replacing test-production inheritance by mocking mechanism. In: Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 540\u2013552","DOI":"10.1145\/3468264.3468590"},{"key":"10410_CR54","unstructured":"Web E (2013) Eclipse JDT\u2122(Java development tools). https:\/\/projects.eclipse.org\/projects\/eclipse.jdt"},{"key":"10410_CR55","doi-asserted-by":"crossref","unstructured":"Weiss M, Moroiu G, Zhao P (2006) Evolution of open source communities. In: Open source systems: IFIP working group 2.13 foundation on open source software, June 8\u201310, 2006, Como, Italy 2, Springer, pp 21\u201332","DOI":"10.1007\/0-387-34226-5_3"},{"key":"10410_CR56","unstructured":"Welcome to the Apache Struts project. https:\/\/struts.apache.org\/"},{"key":"10410_CR57","unstructured":"Why is it so bad to mock classes? \u2014 stackoverflow.com. https:\/\/stackoverflow.com\/questions\/1595166\/why-is-it-so-bad-to-mock-classes. Accessed 19 Jul 2023"},{"key":"10410_CR58","doi-asserted-by":"crossref","unstructured":"Zhu H, Wei L, Wen M, Liu Y, Cheung SC, Sheng Q, Zhou C (2020) Mocksniffer: characterizing and recommending mocking decisions for unit tests. In: Proceedings of the 35th IEEE\/ACM international conference on automated software engineering, pp 436\u2013447","DOI":"10.1145\/3324884.3416539"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10410-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10410-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10410-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,23]],"date-time":"2024-03-23T02:18:54Z","timestamp":1711160334000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10410-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,23]]},"references-count":58,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["10410"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10410-y","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,23]]},"assertion":[{"value":"10 October 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 January 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declared that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"39"}}