{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T17:14:11Z","timestamp":1655918051171},"reference-count":86,"publisher":"Springer Publishing Company","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Cogn Educ Psych"],"published-print":{"date-parts":[[2022,6,1]]},"abstract":"<jats:p>In order to assess the accuracy and validity of proficiency diagnostic tests in Second Language (L2), specifically regarding the linguistic (orthographic, semantic, syntactic, lexical) and cognitive (verbal reasoning, lexical decision) components for the immigrant population in Portugal, a study of cut-off points of 6 tests was conducted. This study was motivated by the unknown tests and the gap concerning the testing and rating of individuals in L2 settings, different from English as L2. The objective was to validate and inform about new tests (and how to establish rating scores and understand the proficiency level of students) in Second Language Testing (for European Portuguese) especially concerning different psychometrics for specific skills (e.g., phonemic blending). In this way, we contribute to identify new procedures for schools and professionals about testing of cognitive functions in immigrant populations. First, we examined 108 non-native students attending Portuguese schools, aged 7\u201317 years, speakers of mother tongues other than Portuguese and residing in Portugal for a period not exceeding eight years. Then, series of univariate analyses, non-parametric tests and the calculation of percentiles enabled the subsequent classification of the subjects\u2019 proficiency levels: With or without proficiency. Several levels were identified. The sensitivity and specificity indexes were calculated using the Receiver Operating Characteristic (ROC) analysis to define the cut-off points for each test. The score calculation took into account the chronological age, age of immigration and gender variables. The evidence from this study suggests that all tasks are suitable to assess immigrant students. However with the ROC analysis, this assessment differs regarding degrees of proficiency between groups given the characteristics of the tasks and of the subjects. The importance of doing tests with discriminatory power of the subjects\u2019 performance to be able to intervene in the linguistic and cognitive areas with the greatest deficit is examined here.<\/jats:p>","DOI":"10.1891\/jcep-2021-0021","type":"journal-article","created":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T16:55:23Z","timestamp":1655916923000},"page":"34-52","source":"Crossref","is-referenced-by-count":0,"title":["Test Difficulty in Second Language Setting: Measuring With Receiver Operating Characteristic"],"prefix":"10.1891","volume":"21","author":[{"given":"Sandra","family":"Figueiredo","sequence":"first","affiliation":[]},{"given":"Margarida Alves","family":"Martins","sequence":"additional","affiliation":[]}],"member":"793","published-online":{"date-parts":[[2022,6,22]]},"reference":[{"key":"2022062209550551000_21.1.34.1","doi-asserted-by":"publisher","DOI":"10.1177\/026553229501200206"},{"issue":"2","key":"2022062209550551000_21.1.34.2","first-page":"210","article-title":"The relationship between gender, motivation and achievement in learning English as a foreign language","year":"2017","journal-title":"European Journal of Contemporary Education"},{"key":"2022062209550551000_21.1.34.3","doi-asserted-by":"publisher","DOI":"10.1186\/cc3000"},{"issue":"6","key":"2022062209550551000_21.1.34.4","first-page":"666","article-title":"Bilingual education for young children: Review of the effects and consequences","year":"2018","journal-title":"International Journal of Bilingual Education and Bilingualism"},{"key":"2022062209550551000_21.1.34.5","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.34.4.859"},{"key":"2022062209550551000_21.1.34.6","unstructured":"Bialystok, E. , & Hakuta, K. (1999). Confounded age: Linguistic and cognitive factors in age differences for Second Language acquisition. Second Language Acquisition and the Critical Period Hypothesis . 161\u2013181."},{"key":"2022062209550551000_21.1.34.7","doi-asserted-by":"publisher","DOI":"10.1111\/j.1944-9720.2003.tb01930.x"},{"key":"2022062209550551000_21.1.34.8","unstructured":"Brennan, R. L. (1983). Elements of generalizability theory . American College Testing Program."},{"issue":"7","key":"2022062209550551000_21.1.34.9","first-page":"1601","article-title":"Monitoring speech production and comprehension: Where is the second-language delay?","year":"2019","journal-title":"Quarterly Journal of Experimental Psychology"},{"issue":"1","key":"2022062209550551000_21.1.34.10","first-page":"1","article-title":"The accessibility of Universal Grammar in adult language learning","year":"1988","journal-title":"Interlanguage studies bulletin (Utrecht)"},{"key":"2022062209550551000_21.1.34.11","first-page":"45","article-title":"Revisiting the bilingual lexical deficit: The impact of age of acquisition","year":"2019","journal-title":"Cognition"},{"issue":"1","key":"2022062209550551000_21.1.34.12","first-page":"47","article-title":"Poor written and oral text comprehension in third grade children: A multiple case study","year":"2018","journal-title":"Rivista di Psicolinguistica Applicata-Journal of Applied Psycholinguistics"},{"key":"2022062209550551000_21.1.34.13","doi-asserted-by":"crossref","unstructured":"Chen, Y. J. I. , Wilson, M. , Irey, R. C. , & Requa, M. K. (2020). An innovative measure of orthographic processing: Development and initial validation. Language Testing , 0265532220909310.","DOI":"10.1177\/0265532220909310"},{"key":"2022062209550551000_21.1.34.14","doi-asserted-by":"publisher","DOI":"10.1177\/0022219414538519"},{"key":"2022062209550551000_21.1.34.15","doi-asserted-by":"publisher","DOI":"10.1177\/0267658316637401"},{"key":"2022062209550551000_21.1.34.16","unstructured":"Council of Europe (2001). Common European framework of reference for languages: Learning, teaching, assessment . Cambridge University Press."},{"issue":"1","key":"2022062209550551000_21.1.34.17","first-page":"93","article-title":"The role of semantic transfer in clitic drop among simultaneous and sequential Chinese-Spanish bilinguals","year":"2013","journal-title":"Studies in Second Language Acquisition"},{"issue":"4","key":"2022062209550551000_21.1.34.18","first-page":"499","article-title":"The robustness of critical period effects in Second Language acquisition","year":"2000","journal-title":"Studies in Second Language Acquisition"},{"key":"2022062209550551000_21.1.34.19","doi-asserted-by":"crossref","unstructured":"D\u00f6rnyei, Z. (2014). The psychology of the language learner: Individual differences in Second Language acquisition . Routledge.","DOI":"10.4324\/9781410613349"},{"issue":"3","key":"2022062209550551000_21.1.34.20","first-page":"383","article-title":"Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis","year":"2017","journal-title":"Language Testing"},{"key":"2022062209550551000_21.1.34.21","doi-asserted-by":"publisher","DOI":"10.1037\/edu0000060"},{"issue":"1","key":"2022062209550551000_21.1.34.22","first-page":"79","article-title":"Evaluation of central auditory processing of Azeri-Persian bilinguals using dichotic","year":"2018","journal-title":"Iranian Journal of Child Neurology"},{"key":"2022062209550551000_21.1.34.23","doi-asserted-by":"crossref","unstructured":"Figueiredo S. (2017). Learning Portuguese as a second language . Springer International Publishing AG.","DOI":"10.1007\/978-3-319-55819-6"},{"issue":"2","key":"2022062209550551000_21.1.34.24","first-page":"184","article-title":"Second language education context and home language effect: Language dissimilarities and variation in immigrant students\u2019 outcomes","year":"2016","journal-title":"International Journal of Multilingualism"},{"issue":"2","key":"2022062209550551000_21.1.34.25","first-page":"192","article-title":"Psychological distress of immigrant population in host school systems and the critical period for second language development: A review","year":"2018","journal-title":"Clin. Exp Psychol"},{"key":"2022062209550551000_21.1.34.26","unstructured":"Feast, V. (2002). The impact of IELTS scores on performance at university (Doctoral dissertation). Flinders University."},{"key":"2022062209550551000_21.1.34.27","doi-asserted-by":"crossref","unstructured":"Frost, K. , Clothier, J. , Huisman, A. , & Wigglesworth, G. (2020). Responding to a TOEFL iBT integrated speaking task: Mapping task demands and test takers\u2019 use of stimulus content. Language Testing , 0265532219860750.","DOI":"10.1177\/0265532219860750"},{"key":"2022062209550551000_21.1.34.28","unstructured":"Fulcher, G. (2010). The reification of the Common European Framework of Reference (CEFR) and effect-driven testing. Advances in research on language acquisition and teaching: Selected papers (pp. 15\u201326)"},{"issue":"2","key":"2022062209550551000_21.1.34.29","first-page":"271","article-title":"Interpreting the relationships between TOEFL iBT scores and GPA: Language proficiency, policy, and profiles","year":"2018","journal-title":"Language Testing"},{"issue":"4","key":"2022062209550551000_21.1.34.30","first-page":"1006","article-title":"Development of first-and second-language vocabulary knowledge among language-minority children: Evidence from single language and conceptual scores","year":"2018","journal-title":"Journal of Child Language"},{"key":"2022062209550551000_21.1.34.31","unstructured":"Grubanov-Boskovic, S. , Natale, F. , & Scipioni, M. (2017). Patterns of immigrants\u2019 integration in European labour markets. European Commission: JRC Technical Reports . ISBN 978-92-79-74664-2. http:\/\/doi:10.2760\/408657"},{"key":"2022062209550551000_21.1.34.32","doi-asserted-by":"publisher","DOI":"10.1177\/0265532214564505"},{"issue":"1","key":"2022062209550551000_21.1.34.33","first-page":"1","article-title":"Designing and scaling level-specific writing tasks in alignment with the CEFR: A test-centered approach","year":"2011","journal-title":"Language Assessment Quarterly"},{"key":"2022062209550551000_21.1.34.34","unstructured":"Hartmann, N. , Fonseca, E. , Shulby, C. , Treviso, M. , Rodrigues, J. , & Aluisio, S. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. arXiv preprint arXiv , 1708.06025."},{"issue":"2","key":"2022062209550551000_21.1.34.35","first-page":"219","article-title":"The development of shared syntax in Second Language learning. Bilingualism","year":"2017","journal-title":"Language and Cognition"},{"issue":"1","key":"2022062209550551000_21.1.34.36","first-page":"2","article-title":"Language difficulty and prior learning influence foreign vocabulary acquisition","year":"2020","journal-title":"Languages"},{"key":"2022062209550551000_21.1.34.37","first-page":"371","article-title":"The role of grammatical development in oral assessment. Widening contexts for processability theory","year":"2019","journal-title":"Theories and Issues"},{"key":"2022062209550551000_21.1.34.38","doi-asserted-by":"crossref","unstructured":"Hille, K. , & Cho, Y. (2020). Placement testing: One test, two tests, three tests? How many tests are sufficient? Language Testing . https:\/\/doi.org\/10.1177\/0265532220912412","DOI":"10.1177\/0265532220912412"},{"issue":"2","key":"2022062209550551000_21.1.34.39","first-page":"305","article-title":"Book review: Assessing l2 listening: Moving towards authenticity","year":"2020","journal-title":"Language Testing"},{"key":"2022062209550551000_21.1.34.40","doi-asserted-by":"publisher","DOI":"10.1191\/0265532202lt229oa"},{"issue":"6","key":"2022062209550551000_21.1.34.41","first-page":"108","article-title":"Gender differences in speaking anxiety among English as a Second Language learners in a Malaysian tertiary context","year":"2017","journal-title":"International Journal for Studies on Children, Women, Elderly and Disabled"},{"issue":"2","key":"2022062209550551000_21.1.34.42","first-page":"193","article-title":"Developing a user-oriented Second Language comprehensibility scale for English-medium universities","year":"2018","journal-title":"Language Testing"},{"issue":"3","key":"2022062209550551000_21.1.34.43","first-page":"439","article-title":"Using the ACTFL OPIc to assess proficiency and monitor progress in a tertiary foreign languages program","year":"2019","journal-title":"Language Testing"},{"key":"2022062209550551000_21.1.34.44","first-page":"116","article-title":"Sequence learning in the human brain: A functional neuroanatomical meta-analysis of serial reaction time studies","year":"2020","journal-title":"NeuroImage"},{"key":"2022062209550551000_21.1.34.45","unstructured":"Kaftandjieva, F. (2010). Methods for setting cut scores in criterion-referenced achievement tests. A comparative analysis of six recent methods with an application to tests of reading in EFL . http:\/\/www.ealta.eu.org\/documents\/resources\/FK_second_doctorate.pdf"},{"key":"2022062209550551000_21.1.34.46","doi-asserted-by":"publisher","DOI":"10.1080\/13803395.2015.1065958"},{"key":"2022062209550551000_21.1.34.47","doi-asserted-by":"publisher","DOI":"10.1017\/S0142716411000099"},{"issue":"4","key":"2022062209550551000_21.1.34.48","first-page":"553","article-title":"Cloze testing for comprehension assessment: The HyTeC-cloze","year":"2019","journal-title":"Language Testing"},{"issue":"4","key":"2022062209550551000_21.1.34.49","first-page":"1399","article-title":"The ROC Toolbox: A toolbox for analyzing receiver-operating characteristics derived from confidence ratings","year":"2017","journal-title":"Behavior research methods"},{"issue":"3","key":"2022062209550551000_21.1.34.50","first-page":"321","article-title":"Functional adequacy in L2 writing: Towards a new rating scale","year":"2017","journal-title":"Language Testing"},{"issue":"8","key":"2022062209550551000_21.1.34.51","first-page":"968","article-title":"Another cup of TEE? The processing of Second Language near-cognates in first language reading","year":"2018","journal-title":"Language, Cognition and Neuroscience"},{"key":"2022062209550551000_21.1.34.52","unstructured":"Llosa, L. (2005). Building and supporting a validity argument for a standards-based classroom assessment of English proficiency . Unpublished Ph.D. dissertation, University of California, Los Angeles."},{"key":"2022062209550551000_21.1.34.53","doi-asserted-by":"publisher","DOI":"10.1177\/026553229801500202"},{"issue":"1","key":"2022062209550551000_21.1.34.54","first-page":"51","article-title":"Investigating the construct measured by banked gap-fill items: Evidence from eye-tracking","year":"2018","journal-title":"Language Testing"},{"key":"2022062209550551000_21.1.34.55","doi-asserted-by":"crossref","unstructured":"McLean, S. , Stewart, J. , & Batty, A. O. (2020). Predicting L2 reading proficiency with modalities of vocabulary knowledge: A bootstrapping approach. Language Testing , 0265532219898380.","DOI":"10.1177\/0265532219898380"},{"key":"2022062209550551000_21.1.34.56","doi-asserted-by":"crossref","unstructured":"Mouthon, M. , Khateb, A. , Lazeyras, F. , Pegna, A. J. , Lee-Jahnke, H. , Lehr, C. , & Annoni, J. M. (2019). Second-language proficiency modulates the brain language control network in bilingual translators: An event-related fMRI study. Bilingualism: Language and Cognition , 1\u201314.","DOI":"10.1017\/S1366728918001141"},{"issue":"1","key":"2022062209550551000_21.1.34.57","first-page":"105","article-title":"Acculturation gap distress among Latino youth: Prospective links to family processes and youth depressive symptoms, alcohol use, and academic performance","year":"2018","journal-title":"Journal of youth and adolescence"},{"issue":"3","key":"2022062209550551000_21.1.34.58","first-page":"110","article-title":"The role of gender in the accuracy and fluency of Iranian upper-intermediate EFL learners\u2019 l2 oral productions","year":"2019","journal-title":"Journal of Applied Linguistics and Language Research"},{"key":"2022062209550551000_21.1.34.59","doi-asserted-by":"publisher","DOI":"10.1016\/j.neures.2011.01.018"},{"key":"2022062209550551000_21.1.34.60","unstructured":"Pascoal, J. , & Oliveira, T. (2006). Portuguese as a Non-Mother Tongue in the National Syllabus. [Portugu\u00eas L\u00edngua n\u00e3o Materna no Curr\u00edculo Nacional, Orienta\u00e7\u00f5es Nacionais: Diagn\u00f3stico de compet\u00eancias em L\u00edngua Portuguesa da popula\u00e7\u00e3o escolar que frequenta as escolas portuguesas] . Ministry of Education."},{"issue":"1","key":"2022062209550551000_21.1.34.61","first-page":"23","article-title":"Cross-language transfer of morphological awareness in Chinese-English bilinguals","year":"2011","journal-title":"Journal of Research in Reading"},{"key":"2022062209550551000_21.1.34.62","first-page":"207","article-title":"Starting age overshadowed: The primacy of differential environmental and family support effects on Second Language attainment in an instructional context","year":"2019","journal-title":"Language Learning"},{"issue":"2","key":"2022062209550551000_21.1.34.63","first-page":"161","article-title":"Shaping a score: Complexity, accuracy, and fluency in integrated writing performances","year":"2019","journal-title":"Language Testing"},{"issue":"6","key":"2022062209550551000_21.1.34.64","first-page":"889912","article-title":"The price of uniformity: Language, family and personality adjustment in the immigrant second generation","year":"2002","journal-title":"Ethnic and Racial Studies"},{"key":"2022062209550551000_21.1.34.65","doi-asserted-by":"crossref","unstructured":"Saito, K. , Tierney, A. , & Sun, H. (2019, in press). Explicit and implicit aptitude effects on Second Language speech learning: Scrutinizing segmental, prosodic and temporal sensitivity and performance via behavioral and neurophysiological measures. Bilingualism: Language and Cognition . https:\/\/doi.org\/10.1017\/S1366728918000895","DOI":"10.1017\/S1366728918000895"},{"issue":"1","key":"2022062209550551000_21.1.34.66","first-page":"30","article-title":"Ethnic group differences in overweight and obese children and young people in England: Cross sectional survey","year":"2004","journal-title":"Archives of Disease in Childhood"},{"key":"2022062209550551000_21.1.34.67","doi-asserted-by":"publisher","DOI":"10.1177\/0265532207077205"},{"key":"2022062209550551000_21.1.34.68","unstructured":"Scaramucci, M. V. (2000). Proficiency for foreign language: Terminologies and conceptual considerations. Applied Linguistics , 36 ."},{"issue":"1","key":"2022062209550551000_21.1.34.69","first-page":"80","article-title":"Trauma, psychosocial factors, and help-seeking in three immigrant groups in Finland","year":"2019","journal-title":"The Journal of Behavioral Health Services & Research"},{"key":"2022062209550551000_21.1.34.70","first-page":"229","article-title":"A robust demonstration of the cognate facilitation effect in first-language and second-language naming","year":"2016","journal-title":"Journal of Experimental Child Psychology"},{"key":"2022062209550551000_21.1.34.71","doi-asserted-by":"publisher","DOI":"10.1191\/0265532205lt296oa"},{"key":"2022062209550551000_21.1.34.72","unstructured":"Sonnleitner, P. , Meier, P. , & Wrobel, G. (2018). Finding the right pace: How speech rate influences item difficulty of a listening comprehension test. Unpublished paper. http:\/\/hdl.handle.net\/10993\/37203"},{"issue":"2","key":"2022062209550551000_21.1.34.73","first-page":"217","article-title":"The teacher as examiner of L2 oral tests: A challenge to standardization","year":"2018","journal-title":"Language Testing"},{"key":"2022062209550551000_21.1.34.74","doi-asserted-by":"crossref","unstructured":"Suzuki, S. , & Kormos, J. (2019). Linguistic dimensions of comprehensibility and perceived fluency: An investigation of complexity, accuracy, and fluency in Second Language argumentative speech. Studies in Second Language Acquisition . 1\u201325.","DOI":"10.1017\/S0272263119000627"},{"key":"2022062209550551000_21.1.34.75","doi-asserted-by":"publisher","DOI":"10.1177\/002383098102400306"},{"key":"2022062209550551000_21.1.34.76","doi-asserted-by":"publisher","DOI":"10.1080\/15434303.2013.869815"},{"issue":"1","key":"2022062209550551000_21.1.34.77","first-page":"81","article-title":"Labor market integration, immigration experience, and psychological distress in a multi-ethnic sample of immigrants residing in Portugal","year":"2018","journal-title":"Ethnicity & Health"},{"key":"2022062209550551000_21.1.34.78","doi-asserted-by":"publisher","DOI":"10.1191\/0265532205lt301oa"},{"key":"2022062209550551000_21.1.34.79","doi-asserted-by":"crossref","unstructured":"Ulbrich, C. , & Wiese, R. (2018). Phonotactic principles and exposure in Second Language processing. Empirical Approaches to the Phonological Structure of Words (Linguistische Arbeiten). 153\u2013182.","DOI":"10.1515\/9783110542899-007"},{"issue":"3","key":"2022062209550551000_21.1.34.80","first-page":"401","article-title":"Acquisition of Dutch as a Second Language: The explanative power of cognate and genetic linguistic distance measures for 11 West European first languages","year":"2010","journal-title":"Studies in Second Language Acquisition"},{"key":"2022062209550551000_21.1.34.81","doi-asserted-by":"crossref","unstructured":"Van der Slik, F. W. , Hout, R. W. , & Schepens, J. J. (2015). The gender gap in Second Language acquisition: Gender differences in the acquisition of Dutch among immigrants from 88 countries with 49 mother tongues. PloS One , 10 (11).","DOI":"10.1371\/journal.pone.0142056"},{"issue":"1","key":"2022062209550551000_21.1.34.82","first-page":"47","article-title":"The role of morphological complexity in predicting the learnability of an additional language: The case of La (additional language) Dutch","year":"2019","journal-title":"Second Language Research"},{"key":"2022062209550551000_21.1.34.83","doi-asserted-by":"publisher","DOI":"10.1038\/sj.ejcn.1601415"},{"issue":"2","key":"2022062209550551000_21.1.34.84","first-page":"161","article-title":"A systematic review of methods for evaluating rating quality in language assessment","year":"2018","journal-title":"Language Testing"},{"issue":"1\u20132","key":"2022062209550551000_21.1.34.85","first-page":"25","article-title":"Examining the role of concentration, vocabulary and self-concept in listening and reading comprehension","year":"2016","journal-title":"International Journal of Listening"},{"key":"2022062209550551000_21.1.34.86","first-page":"104135","article-title":"Finding categories through words: More nameable features improve category learning","year":"2020","journal-title":"Cognition"}],"container-title":["Journal of Cognitive Education and Psychology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1891\/JCEP-2021-0021","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T16:57:08Z","timestamp":1655917028000},"score":1,"resource":{"primary":{"URL":"http:\/\/connect.springerpub.com\/lookup\/doi\/10.1891\/JCEP-2021-0021"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,1]]},"references-count":86,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,6,22]]},"published-print":{"date-parts":[[2022,6,1]]}},"alternative-id":["10.1891\/JCEP-2021-0021"],"URL":"https:\/\/doi.org\/10.1891\/jcep-2021-0021","relation":{},"ISSN":["1945-8959","1810-7621"],"issn-type":[{"value":"1945-8959","type":"print"},{"value":"1810-7621","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,1]]}}}