{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T12:46:30Z","timestamp":1765889190730,"version":"3.41.0"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,1,14]],"date-time":"2024-01-14T00:00:00Z","timestamp":1705190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Baker\/Koob endowments at California Polytechnic State University"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>\n            <jats:bold>Background and Context.<\/jats:bold>\n            Students\u2019 programming projects are often assessed on the basis of their tests as well as their implementations, most commonly using test adequacy criteria like branch coverage, or, in some cases, mutation analysis. As a result, students are implicitly encouraged to use these tools during their development process (i.e., so they have awareness of the strength of their own test suites).\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Objectives.<\/jats:bold>\n            Little is known about how students choose test cases for their software while being guided by these feedback mechanisms. We aim to explore the interaction between students and commonly used testing feedback mechanisms (in this case, branch coverage and mutation-based feedback).\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Method.<\/jats:bold>\n            We use grounded theory to explore this interaction. We conducted 12 think-aloud interviews with students as they were asked to complete a series of software testing tasks, each of which involved a different feedback mechanism. Interviews were recorded and transcripts were analyzed, and we present the overarching themes that emerged from our analysis.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Findings.<\/jats:bold>\n            Our findings are organized into a process model describing how students completed software testing tasks while being guided by a test adequacy criterion. Program comprehension strategies were commonly employed to reason about feedback and devise test cases. Mutation-based feedback tended to be cognitively overwhelming for students, and they resorted to weaker heuristics in order to address this feedback.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Implications.<\/jats:bold>\n            In the presence of testing feedback, students did not appear to consider\n            <jats:italic>problem coverage<\/jats:italic>\n            as a testing goal so much as\n            <jats:italic>program coverage<\/jats:italic>\n            . While test adequacy criteria can be useful for\n            <jats:italic>assessment<\/jats:italic>\n            of software tests, we must consider whether they represent good goals for testing, and if our current methods of practice and assessment are encouraging poor testing habits.\n          <\/jats:p>","DOI":"10.1145\/3628604","type":"journal-article","created":{"date-parts":[[2023,10,20]],"date-time":"2023-10-20T22:21:20Z","timestamp":1697840480000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["A Model of How Students Engineer Test Cases With Feedback"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-2899-4011","authenticated-orcid":false,"given":"Austin M.","family":"Shin","sequence":"first","affiliation":[{"name":"Ridgeline, Inc., USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6574-1278","authenticated-orcid":false,"given":"Ayaan M.","family":"Kazerouni","sequence":"additional","affiliation":[{"name":"California Polytechnic State University, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,1,14]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1869542.1869567"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.5555\/1355340"},{"key":"e_1_3_3_4_2","volume-title":"Effective Software Testing: A Developer\u2019s Guide","author":"Aniche Maur\u00edcio","year":"2022","unstructured":"Maur\u00edcio Aniche. 2022. Effective Software Testing: A Developer\u2019s Guide. Manning, Shelter Island, NY."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287461"},{"key":"e_1_3_3_6_2","article-title":"How developers engineer test cases: An observational study","author":"Aniche Maur\u00edcio","year":"2021","unstructured":"Maur\u00edcio Aniche, Christoph Treude, and Andy Zaidman. 2021. How developers engineer test cases: An observational study. IEEE Transactions on Software Engineering 48, 12 (2021), 4925\u20134946 DOI:10.1109\/ TSE.2021.3129889","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_7_2","volume-title":"16.9 \u2014 Rainfall Problem \u2014 AP CS Principles \u2014 Student Edition","author":"Michigan CS Learning 4 U. Group at The University of","year":"2015","unstructured":"CS Learning 4 U. Group at The University of Michigan. 2015. 16.9 \u2014 Rainfall Problem \u2014 AP CS Principles \u2014 Student Edition. Retrieved September 30, 2023 from https:\/\/runestone.academy\/ns\/books\/published\/StudentCSP\/CSPIntroData\/rainfall.html"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3430665.3456368"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366847"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1177\/2050312118822927"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/357766.351266"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/2648787"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3068324"},{"key":"e_1_3_3_14_2","volume-title":"Quasi-Experimentation: Design and Analysis Issues for Field Settings","author":"Cook Thomas D.","year":"1979","unstructured":"Thomas D. Cook, Donald Thomas Campbell, and Arles Day. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin Boston."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432417"},{"key":"e_1_3_3_16_2","first-page":"5","volume-title":"Proceedings of the PPIG","author":"Crosby Martha E.","year":"2002","unstructured":"Martha E. Crosby, Jean Scholtz, and Susan Wiedenbeck. 2002. The roles beacons play in comprehension for novice and expert programmers. In Proceedings of the PPIG. 5."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2014.12"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/C-M.1978.218136"},{"key":"e_1_3_3_19_2","first-page":"155","volume-title":"Proceedings of the Ninth International Conference on Dependability and Complex Systems DepCoS-RELCOMEX. Brun\u00f3w, Poland","author":"Derezi\u0144ska A.","year":"2014","unstructured":"A. Derezi\u0144ska and K. Ha\u0142as. 2014. Analysis of mutation operators for the python language. In Proceedings of the Ninth International Conference on Dependability and Complex Systems DepCoS-RELCOMEX. Brun\u00f3w, Poland, W. Zamojski, J. Mazurkiewicz, J. Sugier, T. Walkowiak, and J. Kacprzyk (Eds.), Springer International Publishing, Cham, 155\u2013164."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST57152.2023.00033"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1109\/QRS-C55045.2021.00161","volume-title":"Proceedings of the 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C)","author":"Doorn Niels","year":"2021","unstructured":"Niels Doorn, Tanja E. J. Vos, Beatriz Mar\u00edn, Harrie Passier, Lex Bijlsma, and Silvio Cacace. 2021. Exploring students\u2019 sensemaking of test case design. An initial study. In Proceedings of the 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C). 1069\u20131078. DOI:10.1109\/ QRS-C55045.2021.00161"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287474"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381707"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/1028174.971312"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591164"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591164"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157202"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE52884.2021.00026"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C51114.2020.00053"},{"key":"e_1_3_3_30_2","volume-title":"The Discovery of Grounded Theory: Strategies for Qualitative Research","author":"Glaser B. G.","year":"1967","unstructured":"B. G. Glaser and A. L. Strauss. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/563517.563446"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1975.6312836"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2014.40"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3563767.3568132"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2015.30"},{"key":"e_1_3_3_36_2","volume-title":"Exploring Code Coverage in Software Testing and its Correlation with Software Quality; A Systematic Literature Review","author":"Holl\u00e9n Joy W.","year":"2013","unstructured":"Joy W. Holl\u00e9n and Patrick S. Zacarias. 2013. Exploring Code Coverage in Software Testing and its Correlation with Software Quality; A Systematic Literature Review. Bachelor\u2019s Thesis. University of Gothenburg, 405 30 Gothenburg, Sweden."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/1352322.1352315"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/359369.359392"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.96.3.569"},{"key":"e_1_3_3_42_2","article-title":"Fast and accurate incremental feedback for students\u2019 software tests using selective mutation analysis","volume":"175","author":"Kazerouni Ayaan M.","year":"2021","unstructured":"Ayaan M. Kazerouni, James C. Davis, Arinjoy Basak, Clifford A. Shaffer, Francisco Servant, and Stephen H. Edwards. 2021. Fast and accurate incremental feedback for students\u2019 software tests using selective mutation analysis. Journal of Systems and Software 175, 110905 (2021). https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121221000029?via%3Dihub","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3105726.3106180"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287366"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1002\/spe.4380210704"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2016.28"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2015.7081877"},{"key":"e_1_3_3_48_2","unstructured":"Holger Krekel Bruno Oliveira Ronny Pfannschmidt Floris Bruynooghe Brianna Laugher and Florian Bruhin. 2004. pytest 6.2.2. Retrieved from https:\/\/github.com\/pytest-dev\/pytest. Accessed December 4 2023."},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.44"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/572133.572137"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/VL\/HCC53370.2022.9833117"},{"key":"e_1_3_3_52_2","doi-asserted-by":"crossref","DOI":"10.1002\/9781119202486","volume-title":"The Art of Software Testing (3rd. ed.)","author":"Myers Glenford J.","year":"2012","unstructured":"Glenford J. Myers, Corey Sandler, and Tom Badgett. 2012. The Art of Software Testing (3rd. ed.). John Wiley and Sons, Hoboken and N.J."},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/227607.227610"},{"key":"e_1_3_3_54_2","unstructured":"A. J. Offutt and J. M. Voas. 1996. Subsumption of Condition Coverage Techniques by Mutation Testing . Technical Report ISSE-TR-96-01. Information and Software Systems Engineering George Mason University."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0285(87)90007-7"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00087"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287374"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/HCC.2002.1046340"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993400500056563"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287384"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368136"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/1176617.1176743"},{"key":"e_1_3_3_63_2","volume-title":"Grounded Theory in Practice","author":"Strauss Anselm","year":"1997","unstructured":"Anselm Strauss and Juliet M. Corbin. 1997. Grounded Theory in Practice. Sage."},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2016.25"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3589004"},{"key":"e_1_3_3_66_2","volume-title":"Executable Examples: Empowering Students to Hone Their Problem Comprehension","author":"Wrenn John","year":"2022","unstructured":"John Wrenn. 2022. Executable Examples: Empowering Students to Hone Their Problem Comprehension. Ph. D. Dissertation. Brown University."},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339416"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230999"},{"issue":"2","key":"e_1_3_3_69_2","article-title":"Using relational problems to teach property-based testing","volume":"5","author":"Wrenn John","year":"2020","unstructured":"John Wrenn, Tim Nelson, and Shriram Krishnamurthi. 2020. Using relational problems to teach property-based testing. The Art, Science, and Engineering of Programming 5, 2(2020). https:\/\/programming-journal.org\/2021\/5\/9\/","journal-title":"The Art, Science, and Engineering of Programming"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3628604","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3628604","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:44Z","timestamp":1750291424000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3628604"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,14]]},"references-count":68,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3628604"],"URL":"https:\/\/doi.org\/10.1145\/3628604","relation":{},"ISSN":["1946-6226"],"issn-type":[{"type":"electronic","value":"1946-6226"}],"subject":[],"published":{"date-parts":[[2024,1,14]]},"assertion":[{"value":"2023-05-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-12","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}