{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T10:59:55Z","timestamp":1777633195669,"version":"3.51.4"},"reference-count":76,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T00:00:00Z","timestamp":1623196800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T00:00:00Z","timestamp":1623196800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008349","name":"Universit\u00e4t Duisburg-Essen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008349","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Students\u2019 experience is used in empirical software engineering research as well as in software engineering education to group students in either homogeneous or heterogeneous groups. To do so, students are commonly asked to self-rate their experience, as self-rated experience has been shown to be a good predictor for performance in programming tasks. Another experience-related measurement is participants\u2019 confidence (i.e., how confident is the person that their given answer is correct). Hence, self-rated experience and confidence are used as selector or control variables throughout empirical software engineering research and software engineering education. In this paper, we analyze data from several student experiments conducted in the past years to investigate whether self-rated experience and confidence are also good predictors for students\u2019 performance in model comprehension tasks. Our results show that while students can somewhat assess the correctness of a particular answer to one concrete question regarding a conceptual model (i.e., their confidence), their overall self-rated experience does not correlate with their actual performance. Hence, the use of the commonly used measurement of self-rated experience as a selector or control variable must be considered unreliable for model comprehension tasks.<\/jats:p>","DOI":"10.1007\/s10664-021-09972-6","type":"journal-article","created":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T14:21:44Z","timestamp":1623248504000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Reliability of self-rated experience and confidence as predictors for students\u2019 performance in software engineering"],"prefix":"10.1007","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9156-9731","authenticated-orcid":false,"given":"Marian","family":"Daun","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2918-5008","authenticated-orcid":false,"given":"Jennifer","family":"Brings","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4640-0598","authenticated-orcid":false,"given":"Patricia Aluko","family":"Obe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4936-1873","authenticated-orcid":false,"given":"Viktoria","family":"Stenkova","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,6,9]]},"reference":[{"issue":"2","key":"9972_CR1","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1109\/TSE.2007.17","volume":"33","author":"E Arisholm","year":"2007","unstructured":"Arisholm E, Gallis H, Dyba T, Sjoberg D (2007) Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Trans Softw Eng 33(2):65\u201386. https:\/\/doi.org\/10.1109\/TSE.2007.17","journal-title":"IEEE Trans Softw Eng"},{"key":"9972_CR2","doi-asserted-by":"publisher","unstructured":"Bastarrica MC, Simmonds J (2019) Gender differences in self and peer assessment in a software engineering capstone course. In: 2019 IEEE\/ACM 2nd international workshop on gender equality in software engineering (GE), pp 29\u201332. https:\/\/doi.org\/10.1109\/GE.2019.00014","DOI":"10.1109\/GE.2019.00014"},{"key":"9972_CR3","doi-asserted-by":"publisher","unstructured":"Berander P (2004) Using students as subjects in requirements prioritization. In: Proceedings. 2004 international symposium on empirical software engineering, 2004. ISESE \u201904, pp 167\u2013176. https:\/\/doi.org\/10.1109\/ISESE.2004.1334904","DOI":"10.1109\/ISESE.2004.1334904"},{"issue":"4","key":"9972_CR4","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1027\/1614-0001\/a000052","volume":"32","author":"GR Bergersen","year":"2011","unstructured":"Bergersen GR, Gustafsson JE (2011) Programming skill, knowledge, and working memory among professional software developers from an investment theory perspective. J Indiv Diff 32(4):201\u2013209. https:\/\/doi.org\/10.1027\/1614-0001\/a000052","journal-title":"J Indiv Diff"},{"key":"9972_CR5","doi-asserted-by":"publisher","unstructured":"Bergersen GR, Hannay JE, Sjoberg D, Dyba T, Karahasanovic A (2011) Inferring skill from tests of programming performance: combining time and quality. In: 2011 international symposium on empirical software engineering and measurement, iSSN: 1938-6451, pp 305\u2013314. https:\/\/doi.org\/10.1109\/ESEM.2011.39","DOI":"10.1109\/ESEM.2011.39"},{"issue":"12","key":"9972_CR6","doi-asserted-by":"publisher","first-page":"1163","DOI":"10.1109\/TSE.2014.2348997","volume":"40","author":"GR Bergersen","year":"2014","unstructured":"Bergersen GR, Sj\u00f8berg DIK, Dyb\u00e5 T (2014) Construction and validation of an instrument for measuring programming skill. IEEE Trans Softw Eng 40(12):1163\u20131184. https:\/\/doi.org\/10.1109\/TSE.2014.2348997","journal-title":"IEEE Trans Softw Eng"},{"issue":"1","key":"9972_CR7","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/S0164-1212(02)00025-0","volume":"65","author":"S Biffl","year":"2003","unstructured":"Biffl S (2003) Evaluating defect estimation models with major defects. J Syst Softw 65(1):13\u201329. https:\/\/doi.org\/10.1016\/S0164-1212(02)00025-0","journal-title":"J Syst Softw"},{"issue":"5","key":"9972_CR8","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1007\/BF00138746","volume":"18","author":"D Boud","year":"1989","unstructured":"Boud D, Falchikov N (1989) Quantitative studies of student self-assessment in higher education: a critical analysis of findings. High Educ 18(5):529\u2013549","journal-title":"High Educ"},{"issue":"2","key":"9972_CR9","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1007\/s10664-006-6403-7","volume":"11","author":"C Bunse","year":"2006","unstructured":"Bunse C (2006) Using patterns for the refinement and translationof UML models: a controlled experiment. Empir Softw Eng 11 (2):227\u2013267. https:\/\/doi.org\/10.1007\/s10664-006-6403-7","journal-title":"Empir Softw Eng"},{"issue":"3","key":"9972_CR10","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/507758.377467","volume":"33","author":"P Byrne","year":"2001","unstructured":"Byrne P, Lyons G (2001) The effect of student attributes on success in programming. SIGCSE Bull 33(3):49\u201352. https:\/\/doi.org\/10.1145\/507758.377467","journal-title":"SIGCSE Bull"},{"key":"9972_CR11","volume-title":"Experimental and quasi-experimental designs for research","author":"DT Campbell","year":"1963","unstructured":"Campbell DT, Stanley JC (1963) Experimental and quasi-experimental designs for research. Houghton Mifflin, Boston"},{"key":"9972_CR12","doi-asserted-by":"publisher","unstructured":"Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: Software metrics symposium, 2003. Proceedings. Ninth international, pp 239\u2013249. https:\/\/doi.org\/10.1109\/METRIC.2003.1232471","DOI":"10.1109\/METRIC.2003.1232471"},{"key":"9972_CR13","doi-asserted-by":"publisher","unstructured":"Cen L, Ruta D, Powell L, Ng J (2015) Interaction driven composition of student groups for optimal groupwork learning performance. In: 2015 IEEE frontiers in education conference (FIE), pp 1\u20136. https:\/\/doi.org\/10.1109\/FIE.2015.7344266","DOI":"10.1109\/FIE.2015.7344266"},{"key":"9972_CR14","volume-title":"Quasi-experimentation: design & analysis issues for field settings","author":"TD Cook","year":"1979","unstructured":"Cook TD, Campbell DT (1979) Quasi-experimentation: design & analysis issues for field settings. Houghton Mifflin, Boston"},{"issue":"3","key":"9972_CR15","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1080\/00336297.2003.10491800","volume":"55","author":"CJ Cushion","year":"2003","unstructured":"Cushion CJ, Armour KM, Jones RL (2003) Coach education and continuing professional development: Experience and learning to coach. Quest 55 (3):215\u2013230. https:\/\/doi.org\/10.1080\/00336297.2003.10491800","journal-title":"Quest"},{"key":"9972_CR16","doi-asserted-by":"publisher","unstructured":"Daun M, Brings J, Weyer T (2017) On the impact of the model-based representation of inconsistencies to manual reviews. In: Mayr H, Guizzardi G, Ma H, Pastor O (eds) Conceptual modeling. ER 2017. Lecture notes in computer science, vol 10650. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-319-69904-2_35","DOI":"10.1007\/978-3-319-69904-2_35"},{"key":"9972_CR17","doi-asserted-by":"publisher","unstructured":"Daun M, Brings J, Krajinski L, Weyer T (2019a) On the benefits of using dedicated models in validation processes for behavioral specifications. In: Sutton Jr SM, Armbrust O, Hebig R (eds) Proceedings of the international conference on software and system processes, ICSSP 2019, Montreal, QC, Canada, May 25-26, 2019. IEEE \/ ACM, pp 44\u201353. https:\/\/doi.org\/10.1109\/ICSSP.2019.00016","DOI":"10.1109\/ICSSP.2019.00016"},{"issue":"6","key":"9972_CR18","doi-asserted-by":"publisher","first-page":"3421","DOI":"10.1007\/s10270-019-00723-2","volume":"18","author":"M Daun","year":"2019","unstructured":"Daun M, Weyer T, Pohl K (2019b) Improving manual reviews in function-centered engineering of embedded systems using a dedicated review model. Softw Syst Model 18(6):3421\u20133459. https:\/\/doi.org\/10.1007\/s10270-019-00723-2","journal-title":"Softw Syst Model"},{"key":"9972_CR19","doi-asserted-by":"publisher","unstructured":"Daun M, Brings J, Weyer T (2020) Do instance-level review diagrams support validation processes of cyber-physical system specifications: Results from a controlled experiment. In: Proceedings of the international conference on software and system processes, ICSSP 2020, Seoul, Republic of Korea, October 10-11, 2020. ACM, p 10. https:\/\/doi.org\/10.1145\/3379177.3388893","DOI":"10.1145\/3379177.3388893"},{"key":"9972_CR20","unstructured":"DeVellis RF (2017) Scale development. 4th ed. Los Angeles: Sage"},{"key":"9972_CR21","doi-asserted-by":"publisher","unstructured":"Dick M, Postema M, Miller J (2001) Improving student performance in software engineering practice. In: Proceedings 14th conference on software engineering education and training. \u2019In search of a software engineering profession\u2019 (Cat. No.PR01059), pp 143\u2013152. https:\/\/doi.org\/10.1109\/CSEE.2001.913835","DOI":"10.1109\/CSEE.2001.913835"},{"issue":"4","key":"9972_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0121945","volume":"10","author":"B Diedenhofen","year":"2015","unstructured":"Diedenhofen B, Musch J (2015) cocor: A comprehensive solution for the statistical comparison of correlations. PLOS ONE 10(4):1\u201312. https:\/\/doi.org\/10.1371\/journal.pone.0121945","journal-title":"PLOS ONE"},{"issue":"3","key":"9972_CR23","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1007\/BF00127446","volume":"1","author":"K El Emam","year":"1996","unstructured":"El Emam K, Madhavji NH (1996) An instrument for measuring the success of the requirements engineering process in information systems development. Empir Softw Eng 1(3):201\u2013240. https:\/\/doi.org\/10.1007\/BF00127446","journal-title":"Empir Softw Eng"},{"issue":"1","key":"9972_CR24","first-page":"137","volume":"63","author":"RK Eskew","year":"1988","unstructured":"Eskew RK, Faley RH (1988) Some determinants of student performance in the first college-Level financial accounting course. Account Rev 63(1):137\u2013147","journal-title":"Account Rev"},{"issue":"4","key":"9972_CR25","doi-asserted-by":"publisher","first-page":"395","DOI":"10.3102\/00346543059004395","volume":"59","author":"N Falchikov","year":"1989","unstructured":"Falchikov N, Boud D (1989) Student self-Assessment in higher education: a meta-analysis. Rev Educ Res 59(4):395\u2013430. https:\/\/doi.org\/10.3102\/00346543059004395","journal-title":"Rev Educ Res"},{"key":"9972_CR26","doi-asserted-by":"publisher","unstructured":"Feigenspan J, K\u00e4stner C, Liebig J, Apel S, Hanenberg S (2012) Measuring programming experience. In: 2012 20th IEEE international conference on program comprehension (ICPC), iSSN: 1092-8138, pp 73\u201382. https:\/\/doi.org\/10.1109\/ICPC.2012.6240511","DOI":"10.1109\/ICPC.2012.6240511"},{"issue":"6","key":"9972_CR27","doi-asserted-by":"publisher","first-page":"3801","DOI":"10.1007\/s10664-018-9655-0","volume":"23","author":"R Feldt","year":"2018","unstructured":"Feldt R, Zimmermann T, Bergersen GR, Falessi D, Jedlitschka A, Juristo N, M\u00fcnch J, Oivo M, Runeson P, Shepperd M, Sj\u00f8berg DIK, Turhan B (2018) Four commentaries on the use of students and professionals in empirical software engineering experiments. Empir Softw Eng 23(6):3801\u20133820. https:\/\/doi.org\/10.1007\/s10664-018-9655-0","journal-title":"Empir Softw Eng"},{"key":"9972_CR28","unstructured":"Field A (2013) Discovering statistics using IBM SPSS statistics. 4th ed. London: Sage"},{"key":"9972_CR29","doi-asserted-by":"publisher","unstructured":"Fucci D, Turhan B, Oivo M (2015) On the effects of programming and testing skills on external quality and productivity in a test-driven development context. In: EASE \u201915: proceedings of the 19th international conference on evaluation and assessment in software engineering. ACM. https:\/\/doi.org\/10.1145\/2745802.2745826","DOI":"10.1145\/2745802.2745826"},{"key":"9972_CR30","unstructured":"Goodwin CJ, Goodwin KA (2016) Research in psychology methods and design. John Wiley & Sons, Hoboken, NJ, USA"},{"key":"9972_CR31","doi-asserted-by":"crossref","unstructured":"Hagan D, Markham S (2000) Does it help to have some programming experience before beginning a computing degree program?. In: Proceedings of the 5th annual SIGCSE\/SIGCUE conference on innovation and technology in computer science education (ITiCSE 2000). Association for Computing Machinery (ACM), pp 25\u201328","DOI":"10.1145\/353519.343063"},{"issue":"1","key":"9972_CR32","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1109\/TSE.2009.41","volume":"36","author":"JE Hannay","year":"2010","unstructured":"Hannay JE, Arisholm E, Engvik H, Sjoberg DI (2010) Effects of personality on pair programming. IEEE Trans Softw Eng 36(1):61\u201380. https:\/\/doi.org\/10.1109\/TSE.2009.41","journal-title":"IEEE Trans Softw Eng"},{"issue":"3","key":"9972_CR33","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1023\/A:1026586415054","volume":"5","author":"M H\u00f6st","year":"2000","unstructured":"H\u00f6st M, Regnell B, Wohlin C (2000) Using students as subjects-A comparative study of students and professionals in lead-Time impact assessment. Empir Softw Eng 5(3):201\u2013214","journal-title":"Empir Softw Eng"},{"key":"9972_CR34","doi-asserted-by":"publisher","unstructured":"ISO\/IEC\/IEEE (2010) International standard - Systems and software engineering \u2013 Vocabulary. In: ISO\/IEC\/IEEE 24765:2010(E), https:\/\/doi.org\/10.1109\/IEEESTD.2010.5733835","DOI":"10.1109\/IEEESTD.2010.5733835"},{"key":"9972_CR35","unstructured":"ITU (2016) International telecommunication union recommendation z.120: Message Sequence Chart (MSC). Tech. Rep. Z120, International Telecommunication Union"},{"key":"9972_CR36","doi-asserted-by":"publisher","unstructured":"James T, Galster M, Blincoe K, Miller G (2017) What is the perception of female and male software professionals on performance, team dynamics and job satisfaction? Insights from the trenches. In: 2017 IEEE\/ACM 39th international conference on software engineering: software engineering in practice track (ICSE-SEIP), pp 13\u201322. https:\/\/doi.org\/10.1109\/ICSE-SEIP.2017.31","DOI":"10.1109\/ICSE-SEIP.2017.31"},{"issue":"12","key":"9972_CR37","doi-asserted-by":"publisher","first-page":"1217","DOI":"10.1111\/j.1365-2929.2004.02012.x","volume":"38","author":"S Jamieson","year":"2004","unstructured":"Jamieson S (2004) Likert scales: How to (ab) use them? Med Educ 38(12):1217\u20131218","journal-title":"Med Educ"},{"key":"9972_CR38","doi-asserted-by":"crossref","unstructured":"Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Shull F, Singer J, Sj\u00f8berg DIK (eds). Springer, London, pp 201\u2013228","DOI":"10.1007\/978-1-84800-044-5_8"},{"key":"9972_CR39","unstructured":"Jensen LP (2015) Using consultation in student groups to improve development of team work skills amongst more reluctant students. In: Proceedings of the 43rd SEFI annual conference 2015 - diversity in engineering education: An opportunity to face the new trends of engineering, SEFI 2015"},{"issue":"1","key":"9972_CR40","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/S0164-1212(02)00160-7","volume":"70","author":"M J\u00f8rgensen","year":"2004","unstructured":"J\u00f8rgensen M, Teigen KH, Mol\u00f8kken K (2004) Better sure than safe? Over-confidence in judgement based software development effort prediction intervals. J Syst Softw 70(1):79\u201393. https:\/\/doi.org\/10.1016\/S0164-1212(02)00160-7","journal-title":"J Syst Softw"},{"issue":"1","key":"9972_CR41","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/1028174.971307","volume":"36","author":"N Katira","year":"2004","unstructured":"Katira N, Williams L, Wiebe E, Miller C, Balik S, Gehringer E (2004) On understanding compatibility of student pair programmers. SIGCSE Bull 36(1):7\u201311. https:\/\/doi.org\/10.1145\/1028174.971307","journal-title":"SIGCSE Bull"},{"issue":"3","key":"9972_CR42","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/BF00430277","volume":"1","author":"PA Kirschner","year":"1992","unstructured":"Kirschner PA (1992) Epistemology, practical work and Academic skills in science education. Sci Educ 1(3):273\u2013299. https:\/\/doi.org\/10.1007\/BF00430277","journal-title":"Sci Educ"},{"issue":"8","key":"9972_CR43","doi-asserted-by":"publisher","first-page":"721","DOI":"10.1109\/TSE.2002.1027796","volume":"28","author":"B Kitchenham","year":"2002","unstructured":"Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721\u2013734. https:\/\/doi.org\/10.1109\/TSE.2002.1027796","journal-title":"IEEE Trans Softw Eng"},{"key":"9972_CR44","doi-asserted-by":"publisher","unstructured":"Kumar AN (2008) The effect of using problem-solving software tutors on the self-confidence of female students. In: Proceedings of the 39th SIGCSE technical symposium on computer science education, SIGCSE \u201908. Association for Computing Machinery, New York, pp 523\u2013527. https:\/\/doi.org\/10.1145\/1352135.1352309","DOI":"10.1145\/1352135.1352309"},{"key":"9972_CR45","doi-asserted-by":"publisher","unstructured":"Layman L, Williams L, Osborne J, Berenson S, Slaten K, Vouk M (2005) How and why collaborative software development impacts the software engineering course. In: Proceedings frontiers in education 35th annual conference, pp T4C\u2013T4C. https:\/\/doi.org\/10.1109\/FIE.2005.1611964","DOI":"10.1109\/FIE.2005.1611964"},{"key":"9972_CR46","unstructured":"Likert R (1932) A technique for the measurement of attitudes. Archives of psychology"},{"issue":"1","key":"9972_CR47","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1146\/annurev.publhealth.23.100901.140546","volume":"23","author":"T Lumley","year":"2002","unstructured":"Lumley T, Diehr P, Emerson S, Chen L (2002) The importance of the normality assumption in large public health data sets. Annu Rev Public Health 23(1):151\u2013169. https:\/\/doi.org\/10.1146\/annurev.publhealth.23.100901.140546","journal-title":"Annu Rev Public Health"},{"issue":"2, Article 5","key":"9972_CR48","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/2791396","volume":"16","author":"L Marshall","year":"2016","unstructured":"Marshall L, Pieterse V, Thompson L, Venter MD (2016) Exploration of participation in student software engineering teams. ACM Trans Comput Educ 16(2, Article 5):38. https:\/\/doi.org\/10.1145\/2791396","journal-title":"ACM Trans Comput Educ"},{"key":"9972_CR49","doi-asserted-by":"publisher","unstructured":"Mcdowell C, Werner L, Bullock HE, Fernald J (2003) The impact of pair programming on student performance, perception and persistence. In: 25th international conference on software engineering, 2003. Proceedings, pp 602\u2013607. https:\/\/doi.org\/10.1109\/ICSE.2003.1201243","DOI":"10.1109\/ICSE.2003.1201243"},{"key":"9972_CR50","doi-asserted-by":"publisher","unstructured":"Mishra T, Kumar D, Gupta S (2014) Mining students\u2019 data for prediction performance. In: 2014 Fourth international conference on advanced computing communication technologies, iSSN: 2327-0659, pp 255\u2013262. https:\/\/doi.org\/10.1109\/ACCT.2014.105","DOI":"10.1109\/ACCT.2014.105"},{"issue":"1","key":"9972_CR51","doi-asserted-by":"publisher","first-page":"020092","DOI":"10.1063\/1.5005425","volume":"1891","author":"EOC Mkpojiogu","year":"2017","unstructured":"Mkpojiogu EOC, Hussain A (2017) Assessing students\u2019 performance in software requirements engineering education using scoring rubrics. AIP Conf Proc 1891(1):020092. https:\/\/doi.org\/10.1063\/1.5005425","journal-title":"AIP Conf Proc"},{"issue":"4","key":"9972_CR52","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1023\/B:EMSE.0000039883.47173.39","volume":"9","author":"MM M\u00fcller","year":"2004","unstructured":"M\u00fcller MM (2004) Are reviews an alternative to pair programming? Empir Softw Eng 9 (4):335\u2013351. https:\/\/doi.org\/10.1023\/B:EMSE.0000039883.47173.39","journal-title":"Empir Softw Eng"},{"issue":"6","key":"9972_CR53","doi-asserted-by":"publisher","first-page":"534","DOI":"10.1046\/j.1365-2923.2002.01228.x","volume":"36","author":"PJ Morgan","year":"2002","unstructured":"Morgan PJ, Cleave-Hogg D (2002) Comparison between medical students\u2019 experience, confidence and competence. Med Educ 36(6):534\u2013539. https:\/\/doi.org\/10.1046\/j.1365-2923.2002.01228.x","journal-title":"Med Educ"},{"key":"9972_CR54","doi-asserted-by":"publisher","unstructured":"Newhall T, Meeden L, Danner A, Soni A, Ruiz F, Wicentowski R (2014) A support program for introductory cs courses that improves student performance and retains students from underrepresented groups. In: Proceedings of the 45th ACM technical symposium on computer science education, SIGCSE \u201914. Association for Computing Machinery, New York, pp 433\u2013438. https:\/\/doi.org\/10.1145\/2538862.2538923","DOI":"10.1145\/2538862.2538923"},{"issue":"12","key":"9972_CR55","doi-asserted-by":"publisher","first-page":"1670","DOI":"10.1016\/j.infsof.2009.04.007","volume":"51","author":"A Nugroho","year":"2009","unstructured":"Nugroho A (2009) Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf Softw Technol 51 (12):1670\u20131685. https:\/\/doi.org\/10.1016\/j.infsof.2009.04.007","journal-title":"Inf Softw Technol"},{"key":"9972_CR56","unstructured":"Osgood CE, Suci GJ, Tannenbaum PH (1957) The measurement of meaning. Chicago: University of Illinois Press"},{"key":"9972_CR57","doi-asserted-by":"publisher","unstructured":"Pinto G, Ferreira C, Souza C, Steinmacher I, Meirelles P (2019) Training software engineers using open-source software: the students\u2019 perspective. In: Proceedings of the 41st international conference on software engineering: software engineering education and training, ICSE-SEET \u201919, event-place: Montreal, Quebec, Canada. IEEE Press, Piscataway, pp 147\u2013157. https:\/\/doi.org\/10.1109\/ICSE-SEET.2019.00024","DOI":"10.1109\/ICSE-SEET.2019.00024"},{"issue":"4","key":"9972_CR58","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1016\/j.jss.2009.10.036","volume":"83","author":"G Polan\u010di\u010d","year":"2010","unstructured":"Polan\u010di\u010d G, Heri\u010dko M, Rozman I (2010) An empirical examination of application frameworks success based on technology acceptance model. J Syst Softw 83(4):574\u2013584. https:\/\/doi.org\/10.1016\/j.jss.2009.10.036","journal-title":"J Syst Softw"},{"key":"9972_CR59","doi-asserted-by":"publisher","unstructured":"Raza M, Faria JP, Salazar R (2017) Helping software engineering students analyzing their performance data: tool support in an educational environment. In: Proceedings of the 39th international conference on software engineering companion, ICSE-C \u201917, event-place: Buenos Aires, Argentina. IEEE Press, Piscataway, pp 241\u2013243. https:\/\/doi.org\/10.1109\/ICSE-C.2017.61","DOI":"10.1109\/ICSE-C.2017.61"},{"key":"9972_CR60","volume-title":"Designing and conducting survey research: a comprehensive guide fourth edition","author":"LM Rea","year":"2014","unstructured":"Rea LM (2014) Designing and conducting survey research: a comprehensive guide fourth edition, 4th edn. Jossey-Bass, San Francisco","edition":"4"},{"issue":"1","key":"9972_CR61","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1080\/08886504.1998.10782238","volume":"31","author":"K Rex","year":"1998","unstructured":"Rex K, Roth RM (1998) The relationship of computer experience and computer self-Efficacy to performance in introductory computer literacy courses. J Res Comput Educ 31(1):14\u201324. https:\/\/doi.org\/10.1080\/08886504.1998.10782238","journal-title":"J Res Comput Educ"},{"key":"9972_CR62","doi-asserted-by":"publisher","unstructured":"Ricca F, Penta MD, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by UML stereotypes. In: 29Th international conference on software engineering, 2007. ICSE 2007, pp 375\u2013384. https:\/\/doi.org\/10.1109\/ICSE.2007.86","DOI":"10.1109\/ICSE.2007.86"},{"key":"9972_CR63","unstructured":"Robson C (2016) Real world research. 4th ed. Chichester: Wiley"},{"key":"9972_CR64","unstructured":"Runeson P (2003) Using students as experiment subjects \u2013 an analysis on graduate and freshmen student data. In: Proceedings 7th international conference on empirical assessment & evaluation in software engineering, pp 95\u2013102"},{"key":"9972_CR65","doi-asserted-by":"publisher","unstructured":"Salman I, Misirli AT, Juristo N (2015) Are students representatives of professionals in software engineering experiments?. In: 2015 IEEE\/ACM 37Th IEEE international conference on software engineering (ICSE), vol 1, pp 666\u2013676. https:\/\/doi.org\/10.1109\/ICSE.2015.82","DOI":"10.1109\/ICSE.2015.82"},{"issue":"4","key":"9972_CR66","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1109\/TSE.2008.26","volume":"34","author":"J Sillito","year":"2008","unstructured":"Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434\u2013451. https:\/\/doi.org\/10.1109\/TSE.2008.26","journal-title":"IEEE Trans Softw Eng"},{"issue":"2","key":"9972_CR67","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1007\/s10664-006-6401-9","volume":"11","author":"A Sinha","year":"2006","unstructured":"Sinha A, Smidts C (2006) An experimental evaluation of a higher-ordered-typed-functional specification-based test-generation technique. Empir Softw Eng 11 (2):173\u2013202. https:\/\/doi.org\/10.1007\/s10664-006-6401-9","journal-title":"Empir Softw Eng"},{"key":"9972_CR68","doi-asserted-by":"publisher","unstructured":"Sj\u00f8berg DIK, Anda B, Arisholm E, Dyb\u00e5 T, Jorgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. In: Empirical software engineering, 2002. Proceedings. 2002. International Symposium n, pp 17\u201326. https:\/\/doi.org\/10.1109\/ISESE.2002.1166921","DOI":"10.1109\/ISESE.2002.1166921"},{"key":"9972_CR69","doi-asserted-by":"crossref","unstructured":"Sj\u00f8berg DIK, Anda B, Arisholm E, Dyb\u00e5 T, J\u00f8rgensen M, Karahasanovi\u0107 A, Vok\u00e1\u010d M (2003) Challenges and recommendations when increasing the realism of controlled software engineering experiments. In: Conradi R, Wang AI (eds) Empirical methods and studies in software engineering: experiences from ESERNET, Lecture Notes in Computer Science. Springer, Berlin, pp 24\u201338","DOI":"10.1007\/978-3-540-45143-3_3"},{"issue":"4","key":"9972_CR70","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1023\/A:1009844119158","volume":"5","author":"WF Tichy","year":"2000","unstructured":"Tichy WF (2000) Hints for reviewing empirical work in software engineering. Empir Softw Eng 5(4):309\u2013312","journal-title":"Empir Softw Eng"},{"issue":"2","key":"9972_CR71","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1111\/j.1540-5915.2008.00192.x","volume":"39","author":"V Venkatesh","year":"2008","unstructured":"Venkatesh V, Bala H (2008) Technology acceptance model 3 and a research agenda on interventions. Decis Sci 39(2):273\u2013315. https:\/\/doi.org\/10.1111\/j.1540-5915.2008.00192.x","journal-title":"Decis Sci"},{"issue":"4","key":"9972_CR72","doi-asserted-by":"publisher","first-page":"607","DOI":"10.3102\/00028312035004607","volume":"35","author":"NM Webb","year":"1998","unstructured":"Webb NM, Nemer KM, Chizhik AW, Sugrue B (1998) Equity issues in collaborative group assessment: group composition and performance. Am Educ Res J 35(4):607\u2013651. https:\/\/doi.org\/10.3102\/00028312035004607","journal-title":"Am Educ Res J"},{"key":"9972_CR73","doi-asserted-by":"publisher","unstructured":"Wieringa R (2010) Design science methodology: Principles and practice. In: Proceedings of the 32nd ACM\/IEEE international conference on software engineering, ICSE \u201910, vol 2. ACM, New York, pp 493\u2013494. https:\/\/doi.org\/10.1145\/1810295.1810446","DOI":"10.1145\/1810295.1810446"},{"key":"9972_CR74","unstructured":"Witten IH, Frank E, Trigg LE, Hall MA, Holmes G, Cunningham SJ (1999) Weka: Practical machine learning tools and techniques with Java implementations. Working Paper, University of Waikato, Department of Computer Science, Hamilton, New Zealand, 99\/11"},{"key":"9972_CR75","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-4625-2","volume-title":"Experimentation in software engineering: An introduction, Kluwer international series in software engineering, vol 6","author":"C Wohlin","year":"2000","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson M, Regnell B, Wessl\u00e9n A (2000) Experimentation in software engineering: An introduction, Kluwer international series in software engineering, vol 6. Kluwer Academic, Boston"},{"key":"9972_CR76","doi-asserted-by":"publisher","unstructured":"Zhang D, Fonseca P, Cuthbert L, Ketteridge S (2014) An investigation of the team knowledge and team performance of the Chinese engineering students in a senior technical module. In: 2014 IEEE frontiers in education conference (FIE) Proceedings. https:\/\/doi.org\/10.1109\/FIE.2014.7044078, iSSN: 2377-634X, pp 1\u20138","DOI":"10.1109\/FIE.2014.7044078"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-021-09972-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-021-09972-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-021-09972-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,13]],"date-time":"2021-07-13T20:19:40Z","timestamp":1626207580000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-021-09972-6"}},"subtitle":["Results from multiple controlled experiments on model comprehension with graduate and undergraduate students"],"short-title":[],"issued":{"date-parts":[[2021,6,9]]},"references-count":76,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["9972"],"URL":"https:\/\/doi.org\/10.1007\/s10664-021-09972-6","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,9]]},"assertion":[{"value":"14 April 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 June 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"80"}}