{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T13:28:12Z","timestamp":1740144492778,"version":"3.37.3"},"reference-count":46,"publisher":"EDP Sciences","issue":"6","license":[{"start":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T00:00:00Z","timestamp":1732147200000},"content-version":"vor","delay-in-days":20,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["RAIRO-Oper. Res."],"accepted":{"date-parts":[[2024,10,23]]},"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:p>This work assesses the validity of the Brazilian Indicator of the Difference between Observed and Expected Performances (IDD), which applies a multilevel regression model to rank university courses. It proposes a methodology to test for inconsistencies and research for the causes. With this objective, we develop an experimental methodology, applied to a database with 4432 production engineering students distributed over 329 undergraduate courses, from the government\u2019s National Institute for Educational Studies and Research. We display a graphical analysis of the expected and achieved grades in final graduation exams, the gap in which determines the learning gains for every course. Then we apply the multilevel linear regression model of the IDD in a particular scenario, to test for consistency in value-added learning measurements using the technique <jats:italic>reductio ad absurdum<\/jats:italic>. The result reveals that the present methodology for the calculation of the IDD is inadequate and inconsistent, and that this can be explained by the ceiling effect. It needs to be radically revised to prevent unfair judgements. Finally, we propose and implement a production frontier technique to correct for the ensuing ceiling effect, namely a variable returns to scale data envelopment analysis model for adjusting the measurements using evidence based benchmarks.<\/jats:p>","DOI":"10.1051\/ro\/2024208","type":"journal-article","created":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T08:48:29Z","timestamp":1730105309000},"page":"4955-4969","source":"Crossref","is-referenced-by-count":0,"title":["Pitfalls in the assessment of higher education learning gains in Brazil through multilevel regression: handling the ceiling effect"],"prefix":"10.1051","volume":"58","author":[{"given":"Marcos Estellita","family":"Lins","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Musbah Koleilat","family":"Camara","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7824-7917","authenticated-orcid":false,"given":"Ana Luiza Lima","family":"de Souza","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"250","published-online":{"date-parts":[[2024,11,21]]},"reference":[{"key":"R1","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/BF00991998","volume":"27","author":"Banta","year":"1987","journal-title":"Res. Higher Educ."},{"key":"R2","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/s11577-019-00595-w","volume":"71","author":"Blossfeld","year":"2019","journal-title":"KZfSS K\u00f6lner Zeitschrift f\u00fcr Soziologie und Sozialpsychologie"},{"key":"R3","doi-asserted-by":"crossref","first-page":"76","DOI":"10.15446\/ing.investig.v33n2.39521","volume":"33","author":"Bogoya","year":"2013","journal-title":"Ingenier\u00eda e Investigaci\u00f3n"},{"key":"R4","first-page":"19","volume":"2008","author":"Borden","year":"2008","journal-title":"New Dir. Inst. Res."},{"key":"R5","unstructured":"Brasil L., Decree No 5,773. of May 9, 2006: que disp\u00f5e sobre o exerc\u00edcio das fun\u00e7\u00f5es de regula\u00e7\u00e3o, supervis\u00e3o e avalia\u00e7\u00e3o de institui\u00e7\u00f5es de educa\u00e7\u00e3o superior e cursos superiores de gradua\u00e7\u00e3o e seq\u00fcenciais no sistema federal de ensino. Presidencia da Rep\u00fablica (2006)."},{"key":"R6","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1051\/ro\/2022012","volume":"56","author":"Charles","year":"2022","journal-title":"RAIRO-Oper. Res."},{"key":"R7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1787\/hemp-v21-art5-en","volume":"21","author":"Coates","year":"2009","journal-title":"Higher Educ. Manage. Policy"},{"key":"R8","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.econedurev.2014.06.001","volume":"42","author":"Cunha","year":"2014","journal-title":"Econ. Educ. Rev."},{"key":"R9","doi-asserted-by":"crossref","first-page":"1819","DOI":"10.1111\/itor.12762","volume":"27","author":"De Witte","year":"2020","journal-title":"Int. Trans. Oper. Res."},{"key":"R10","doi-asserted-by":"crossref","first-page":"3116","DOI":"10.3390\/su13063116","volume":"13","author":"Dinc\u0103","year":"2021","journal-title":"Sustainability"},{"key":"R11","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1007\/s10734-011-9496-x","volume":"64","author":"Douglass","year":"2012","journal-title":"Higher Educ."},{"key":"R12","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1177\/003172171009100821","volume":"91","author":"Eckert","year":"2010","journal-title":"Phi Delta Kappan"},{"key":"R13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/23752696.2018.1508360","volume":"3","author":"Evans","year":"2018","journal-title":"Higher Educ. Pedagogies"},{"key":"R14","doi-asserted-by":"crossref","first-page":"149","DOI":"10.2307\/2983475","volume":"159","author":"Goldstein","year":"1996","journal-title":"J. R. Stat. Soc.: Ser. A (Stat. Soc.)"},{"key":"R15","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1177\/0013164414548576","volume":"75","author":"Ho","year":"2015","journal-title":"Educ. Psychol. Measur."},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Howson C.K., Learning gain in excellence frameworks and rankings, in Research Handbook on University Rankings. Edward Elgar Publishing (2021) 340\u2013352.","DOI":"10.4337\/9781788974981.00036"},{"key":"R17","unstructured":"Hujar J., Combatting ceiling effects: modeling high-ability student growth using multilevel tobit regression. Doctoral dissertation, The University of North Carolina at Charlotte (2022)."},{"key":"R18","unstructured":"INEP-Instituto Nacional de Estudos e Pesquisas Educacionais An\u00edsio Teixeira, Conceito Preliminar de Curso (CPC). INEP, Bras\u00edlia (2014). Retrieved September 5, 2017, from http:\/\/inep.gov.br\/conceito-preliminar-de-curso-cpc."},{"key":"R19","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1111\/j.0307-3378.2006.00238.x","volume":"58","author":"Johnes","year":"2006","journal-title":"Bull. Econ. Res."},{"key":"R20","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1177\/0013164412475300","volume":"73","author":"Keeley","year":"2013","journal-title":"Educ. Psychol. Measure."},{"key":"R21","unstructured":"Kim H. and Lalancette D., Literature Review on Value Added Measurement in Higher Education. OECD, Paris, France (2013). Retrieved May 2, 2015."},{"key":"R22","unstructured":"Koedel C. and Betts J.R., Re-examining the Role of Teacher Quality in the Educational Production Function. National Center on Performance Incentives, Vanderbilt, Peabody College (2007)."},{"key":"R23","unstructured":"Koedel C. and Betts J.R., Test-score ceiling effects and value added measurements of school quality. JSM Proc. Soc. Stat. Sect. (2008) 2370\u20132377."},{"key":"R24","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1162\/edfp.2009.5.1.5104","volume":"5","author":"Koedel","year":"2010","journal-title":"Educ. Finan. Policy"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Leckie G., Parker R., Goldstein H. and Tilling K., Mixed-effects location scale models for joint modeling school value-added effects on the mean and variance of student achievement. J. Educ. Behav. Stat. (2023). DOI: 10.3102\/1076998623121080.","DOI":"10.3102\/10769986231210808"},{"key":"R26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/09243453.2012.680892","volume":"24","author":"Lenkeit","year":"2013","journal-title":"School Effectiveness School Improv."},{"key":"R27","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1080\/02602930903197917","volume":"36","author":"Liu","year":"2011","journal-title":"Assess. Eval. Higher Educ."},{"key":"R28","doi-asserted-by":"crossref","first-page":"S1051","DOI":"10.1051\/ro\/2020027","volume":"55","author":"Marcenaro-Guti\u00e9rrez","year":"2021","journal-title":"RAIRO-Oper. Res."},{"key":"R29","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1177\/0016986210379095","volume":"54","author":"McBee","year":"2010","journal-title":"Gifted Child Q."},{"key":"R30","doi-asserted-by":"crossref","unstructured":"McGrath C.H., Guerin B., Harte E., Frearson M., and Manville C., Learning Gain in Higher Education. RAND Corporation, Santa Monica, CA (2015).","DOI":"10.7249\/RR996"},{"key":"R31","doi-asserted-by":"crossref","first-page":"1337","DOI":"10.1111\/itor.13012","volume":"29","author":"Mergoni","year":"2022","journal-title":"Int. Trans. Oper. Res."},{"key":"R32","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1111\/jedm.12114","volume":"53","author":"Milla","year":"2016","journal-title":"J. Educ. Measure"},{"key":"R33","unstructured":"Ng H.L., Investigating the Robustness of School-Performance Ratings to Three Factors Affecting the Underlying Student-Level Academic-Achievement Scores. Harvard University (2012)."},{"key":"R34","unstructured":"OECD, Measuring Improvements in Learning Outcomes: Best Practices to Assess the Value Added by Schools. OECD (2008)."},{"key":"R35","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1162\/edfp.2009.4.4.492","volume":"4","author":"Reardon","year":"2009","journal-title":"Educ. Finan. Policy"},{"key":"R36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/2330443X.2018.1460226","volume":"5","author":"Resch","year":"2018","journal-title":"Stat. Publ. Policy"},{"key":"R37","unstructured":"Resch A. and Isenberg E., How Do Test Scores at the Floor and Ceiling Affect Value Added Estimates? Mathematica Policy Research (2014)."},{"key":"R38","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1108\/09684880510594355","volume":"13","author":"Rodgers","year":"2005","journal-title":"Qual. Assur. Educ."},{"key":"R39","doi-asserted-by":"crossref","unstructured":"Rogaten J. and Rienties B., A critical review of learning gains methods and approaches, in Learning Gain in Higher Education (International Perspectives on Higher Education Research, Vol. 14), edited by Hughes C. and Tight M.. Emerald Publishing Limited, Leeds (2021) 17\u201331.","DOI":"10.1108\/S1479-362820210000014003"},{"key":"R40","doi-asserted-by":"crossref","first-page":"2284","DOI":"10.1080\/03075079.2016.1143925","volume":"42","author":"Roohr","year":"2017","journal-title":"Stud. Higher Educ."},{"key":"R41","doi-asserted-by":"crossref","first-page":"26","DOI":"10.3200\/CHNG.39.1.26-33","volume":"39","author":"Shavelson","year":"2007","journal-title":"Change: Mag. Higher Learn."},{"key":"R42","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1080\/02602938.2016.1168772","volume":"41","author":"Shavelson","year":"2016","journal-title":"Assess. Eval. Higher Educ."},{"key":"R43","doi-asserted-by":"crossref","first-page":"690431","DOI":"10.3389\/feduc.2021.690431","volume":"6","author":"Staus","year":"2021","journal-title":"Front. Educ."},{"key":"R44","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1080\/02602938.2011.560720","volume":"37","author":"Steedle","year":"2012","journal-title":"Assess. Eval. Higher Educ."},{"key":"R45","unstructured":"Tremblay K., Lalancette D. and Roseveare D., AHELO Feasibility Study Report. Volume 1 \u2013 Design and Implementation. Organization for Economic Co-operation and Development (OECD) (2012)."},{"key":"R46","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1007\/s10734-004-1704-5","volume":"51","author":"Yorke","year":"2006","journal-title":"Higher Educ."}],"container-title":["RAIRO - Operations Research"],"original-title":[],"link":[{"URL":"https:\/\/www.rairo-ro.org\/10.1051\/ro\/2024208\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T12:47:45Z","timestamp":1732193265000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.rairo-ro.org\/10.1051\/ro\/2024208"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11]]},"references-count":46,"journal-issue":{"issue":"6"},"alternative-id":["ro240285"],"URL":"https:\/\/doi.org\/10.1051\/ro\/2024208","relation":{},"ISSN":["0399-0559","2804-7303"],"issn-type":[{"type":"print","value":"0399-0559"},{"type":"electronic","value":"2804-7303"}],"subject":[],"published":{"date-parts":[[2024,11]]}}}