{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T08:49:18Z","timestamp":1768034958533,"version":"3.49.0"},"reference-count":66,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2023,9,16]],"date-time":"2023-09-16T00:00:00Z","timestamp":1694822400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001871","name":"National Funds through the FCT\u2014Foundation for Science and Technology, I.P.","doi-asserted-by":"publisher","award":["UI\/BD\/151034\/2021"],"award-info":[{"award-number":["UI\/BD\/151034\/2021"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"National Funds through the FCT\u2014Foundation for Science and Technology, I.P.","doi-asserted-by":"publisher","award":["UIDB\/04312\/2020"],"award-info":[{"award-number":["UIDB\/04312\/2020"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Education Sciences"],"abstract":"<jats:p>Teachers and researchers, in accordance with the main Portuguese curriculum documents in the area of Physical and Natural Sciences, agree that the development of scientific literacy is an integral part of basic education and report that the teaching of these skills is taking place in schools. However, few scientific literacy assessment instruments are available to assess students\u2019 proficiency in using these skills. In this article, we describe the design and processes for gathering validity evidence for the development of the Avalia\u00e7\u00e3o da Literacia Cient\u00edfica Essencial (ALCE) instrument. The ALCE assesses scientific literacy skills of students at the end of the 3rd cycle of Basic Education, in the cognitive domains of understanding; analysing and evaluating phenomena; problems and everyday situations involving content knowledge and skills developed in the subjects of Natural Sciences and Physical Chemistry. Our validity argument, which includes the gathering of evidence based on the content and internal structure of the instrument and is grounded in the current literature on the validation of assessment instruments, supports the use of the instrument for the assessment of students\u2019 scientific literacy level at the end of the 3rd cycle of Basic Education. The ALCE may be a useful tool to identify possible gaps between the teaching objectives and the students\u2019 scientific literacy proficiency and reflect on the methodologies, lesson plans and strategies used in the classroom in order to change them to better develop the students\u2019 scientific literacy.<\/jats:p>","DOI":"10.3390\/educsci13090941","type":"journal-article","created":{"date-parts":[[2023,9,17]],"date-time":"2023-09-17T23:57:46Z","timestamp":1694995066000},"page":"941","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Developing a Scientific Literacy Assessment Instrument for Portuguese 3rd Cycle Students"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6734-7592","authenticated-orcid":false,"given":"Marcelo","family":"Coppi","sequence":"first","affiliation":[{"name":"Center for Research in Education and Psychology, University of \u00c9vora, 7004-516 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1749-9077","authenticated-orcid":false,"given":"Isabel","family":"Fialho","sequence":"additional","affiliation":[{"name":"Center for Research in Education and Psychology, University of \u00c9vora, 7004-516 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6009-0242","authenticated-orcid":false,"given":"Mar\u00edlia","family":"Cid","sequence":"additional","affiliation":[{"name":"Center for Research in Education and Psychology, University of \u00c9vora, 7004-516 \u00c9vora, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1002\/1098-2736(200008)37:6<582::AID-TEA5>3.0.CO;2-L","article-title":"Scientific Literacy: Another Look at Its Historical and Contemporary Meanings and Its Relationship to Science Education Reform","volume":"37","author":"DeBoer","year":"2000","journal-title":"J. Res. Sci. Teach."},{"key":"ref_2","unstructured":"Abell, S.K., and Lederman, N.G. (2007). Handbook of Research on Science Education, Erlbaum."},{"key":"ref_3","unstructured":"OECD (2017). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy and Collaborative Problem Solving, revised edition, OECD Publishing."},{"key":"ref_4","first-page":"29","article-title":"Scientific Literacy: A Conceptual and Empirical Review","volume":"112","author":"Miller","year":"1983","journal-title":"Daedalus"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zeyer, A., and Kyburz-Graber, R. (2012). Science | Environment | Health: Towards a Renewed Pedagogy for Science Education, Springer.","DOI":"10.1007\/978-90-481-3949-1"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1002\/(SICI)1098-237X(200001)84:1<71::AID-SCE6>3.0.CO;2-C","article-title":"Scientific Literacy: A Conceptual Overview","volume":"84","author":"Laugksch","year":"2000","journal-title":"Sci. Educ."},{"key":"ref_7","unstructured":"OECD (2022, October 16). How Does PISA for Development Measure Scientific Literacy?. Available online: https:\/\/www.oecd.org\/pisa\/pisa-for-development\/10-How-PISA-D-measures-science-literacy.pdf."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Queiruga-Dios, M.\u00c1., L\u00f3pez-I\u00f1esta, E., Diez-Ojeda, M., S\u00e1iz-Manzanares, M.C., and Dorr\u00edo, J.B.V. (2020). Citizen Science for Scientific Literacy and the Attainment of Sustainable Development Goals in Formal Education. Sustainability, 12.","DOI":"10.3390\/su12104283"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e1912436117","DOI":"10.1073\/pnas.1912436117","article-title":"(Mis)Informed about What? What It Means to Be a Science-Literate Citizen in a Digital World","volume":"118","author":"Howell","year":"2021","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_10","unstructured":"European Union (2015). Science Education for Responsible Citizenship, European Union."},{"key":"ref_11","first-page":"1","article-title":"The Present Shock and Time Re-Appropriation in the Pandemic Era: Missed Opportunities for Science Education","volume":"159","author":"Levrini","year":"2020","journal-title":"Sci. Educ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1002\/sce.21612","article-title":"Recognition and Operationalization of Future-Scaffolding Skills: Results from an Empirical Study of a Teaching\u2013Learning Module on Climate Change and Futures Thinking","volume":"105","author":"Levrini","year":"2021","journal-title":"Sci. Educ."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"97","DOI":"10.33225\/jbse\/16.15.97","article-title":"Comparativeresearch on the Understandings of Nature of Science and Scientific Inquiry between Science Teachers from Shanghai and Chicago","volume":"15","author":"Wang","year":"2016","journal-title":"J. Balt. Sci. Educ."},{"key":"ref_14","unstructured":"World Economic Forum (2015). New Vision for Education\u2014Unlocking the Potential of Technology, World Economic Forum\u00ae."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1080\/09500693.2017.1420266","article-title":"Scientific Literacy for Democratic Decision-Making","volume":"40","author":"Yacoubian","year":"2018","journal-title":"Int. J. Sci. Educ."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1002\/sce.21115","article-title":"Developing a Measure of Scientific Literacy for Middle School Students","volume":"98","author":"Fives","year":"2014","journal-title":"Sci. Educ."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1187\/cbe.12-03-0026","article-title":"Developing a Test of Scientific Literacy Skills (TOSLS): Measuring Undergraduates\u2019 Evaluation of Scientific Information and Arguments","volume":"11","author":"Gormally","year":"2012","journal-title":"CBE Life Sci. Educ."},{"key":"ref_18","first-page":"e37523","article-title":"Scientific Literacy Assessment Instruments: A Systematic Literature Review","volume":"39","author":"Coppi","year":"2023","journal-title":"Educ. Rev."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1088\/0963-6625\/5\/4\/003","article-title":"Construction of a Paper-and-Pencil Test of Basic Scientific Literacy Based on Selected Literacy Goals Recommended by the American Association for the Advancement of Science","volume":"5","author":"Laugksch","year":"1996","journal-title":"Public Underst. Sci."},{"key":"ref_20","unstructured":"AAAS (1989). Project 2061: Science for All Americans, Oxford University Press."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1002\/sce.21575","article-title":"Reconceptualizing Nature-of-Science Education in the Age of Social Media","volume":"104","author":"Allchin","year":"2020","journal-title":"Sci. Educ."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"McComas, W.F. (2020). Nature of Science in Science Iinstruction: Rationales and Strategies, Springer Nature.","DOI":"10.1007\/978-3-030-57239-6"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1177\/0963662520977700","article-title":"Knowledge about the Nature of Science Increases Public Acceptance of Science Regardless of Identity Factors","volume":"30","author":"Weisberg","year":"2021","journal-title":"Public Underst. Sci."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Khishfe, R. (2022). Improving Students\u2019 Conceptions of Nature of Science: A Review of the Literature. Sci. Educ., 1\u201345.","DOI":"10.1007\/s11191-022-00390-8"},{"key":"ref_25","first-page":"826","article-title":"Estrat\u00e9gias Para a Coleta de Evid\u00eancias de Validade de Avalia\u00e7\u00f5es de Sala de Aula","volume":"14","author":"Coppi","year":"2022","journal-title":"Meta Avalia\u00e7\u00e3o"},{"key":"ref_26","unstructured":"Depresbiteris, L., and Tavares, M.R. (2009). Diversificar \u00e9 Preciso...:Instrumentos e T\u00e9cnicas de Avalia\u00e7\u00e3o de Aprendizagem, Senac S\u00e3o Paulo."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Haladyna, T.M., and Rodriguez, M.C. (2013). Developing and Validating Test Items, Taylor & Francis Group.","DOI":"10.4324\/9780203850381"},{"key":"ref_28","unstructured":"Popham, W.J. (2017). Classroom Assessment: What Teachers Need to Know, Pearson. [8th ed.]."},{"key":"ref_29","unstructured":"Russel, M.K., and Airasian, P.W. (2014). Avalia\u00e7\u00e3o Em Sala de Aula: Conceitos e Aplica\u00e7\u00f5es, AMGH. [7th ed.]."},{"key":"ref_30","unstructured":"Gipps, C.V. (2003). Beyond Testing: Towards a Theory of Educational Assessment, The Falmer Press."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"S85","DOI":"10.1016\/j.mjafi.2020.11.007","article-title":"Item Analysis of Multiple Choice Questions: A Quality Assurance Test for an Assessment Tool","volume":"77","author":"Kumar","year":"2021","journal-title":"Med. J. Armed Forces India"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1007\/s11165-020-09943-9","article-title":"Interdisciplinary Science Assessment of Carbon Cycling: Construct Validity Evidence Based on Internal Structure","volume":"52","author":"You","year":"2022","journal-title":"Res. Sci. Educ."},{"key":"ref_33","unstructured":"AERA, APA, and NCME (2014). Standards for Educational and Psychological Testing, American Educational Research Association."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Braun, H.I. (2016). Meeting the Challenges to Measurement in an Era of Accountability, Routledge.","DOI":"10.4324\/9780203781302"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"992","DOI":"10.1590\/S0080-62342009000500002","article-title":"Psicometria","volume":"43","author":"Pasquali","year":"2009","journal-title":"Rev. da Esc. Enferm. da USP"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"99","DOI":"10.18764\/2178-2229v29n2.2022.27","article-title":"Evid\u00eancias de Validade Baseadas No Conte\u00fado de Um Instrumento de Avalia\u00e7\u00e3o Da Literacia Cient\u00edfica","volume":"29","author":"Coppi","year":"2022","journal-title":"Cad. Pesqui."},{"key":"ref_37","unstructured":"Pasquali, L. (2009). Psicometria Teoria Dos Testes Na Psicologia e Na Educa\u00e7\u00e3o, Vozes. [4th ed.]."},{"key":"ref_38","unstructured":"Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., RATHS, J., and Wittrock, M.C. (2001). A Taxonomy for Learning, Teaching and Assessing: A Revison of Bloom\u2019s Taxonomy of Educational Objectives, Addison Wesley Longman."},{"key":"ref_39","unstructured":"Galv\u00e3o, C., Neves, A., Freire, A.M., Lopes, A.M.S., Santos, M.D.C., Vilela, M.D.C., Oliveira, M.T., and Pereira, M. (2001). Ci\u00eancias F\u00edsicas e Naturais\u2014Orienta\u00e7\u00f5es Curriculares Para o 3.o Ciclo Do Ensino B\u00e1sico, Minist\u00e9rio da Educa\u00e7\u00e3o."},{"key":"ref_40","unstructured":"DGE (2022, November 21). Aprendizagens Essenciais, articula\u00e7\u00e3o Com o Perfil dos Alunos\u2014Ci\u00eancias Naturais\u20149.o ano\u20143.o ciclo do Ensino B\u00e1sico. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Curriculo\/Aprendizagens_Essenciais\/3_ciclo\/ciencias_naturais_3c_9a_ff.pdf."},{"key":"ref_41","unstructured":"DGE (2022, November 21). Aprendizagens Essenciais, Articula\u00e7\u00e3o Com o Perfil dos Alunos\u2014Ci\u00eancias Naturais\u20148.o ano\u20143.o ciclo do Ensino B\u00e1sico. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Curriculo\/Aprendizagens_Essenciais\/3_ciclo\/ciencias_naturais_3c_8a_ff.pdf."},{"key":"ref_42","unstructured":"DGE (2022, November 21). Aprendizagens Essenciais, Articula\u00e7\u00e3o Com o Perfil dos Alunos\u2014Ci\u00eancias Naturais\u20147.o ano\u20143.o ciclo do Ensino B\u00e1sico. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Curriculo\/Aprendizagens_Essenciais\/3_ciclo\/ciencias_naturais_3c_7a_ff.pdf."},{"key":"ref_43","unstructured":"DGE (2022, November 21). Aprendizagens Essenciais, Articula\u00e7\u00e3o Com o Perfil dos Alunos\u2014F\u00edsico-qu\u00edmica\u20149.o ano\u20143.o ciclo do Ensino B\u00e1sico. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Curriculo\/Aprendizagens_Essenciais\/3_ciclo\/fisico-quimica_3c_9a.pdf."},{"key":"ref_44","unstructured":"DGE (2022, November 21). Aprendizagens Essenciais, Articula\u00e7\u00e3o Com o Perfil dos Alunos\u2014F\u00edsico-qu\u00edmica\u20147.o ano\u20143.o ciclo do Ensino B\u00e1sico. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Curriculo\/Aprendizagens_Essenciais\/3_ciclo\/fisico-quimica_3c_7a_ff.pdf."},{"key":"ref_45","unstructured":"DGE (2022, November 21). Aprendizagens Essenciais, Articula\u00e7\u00e3o Com o Perfil dos Alunos\u2014F\u00edsico-qu\u00edmica\u20148.o ano\u20143.o ciclo do Ensino B\u00e1sico. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Curriculo\/Aprendizagens_Essenciais\/3_ciclo\/fisico-quimica_3c_8a_ff.pdf."},{"key":"ref_46","unstructured":"Martins, G.O., Gomes, C.A.S., Brocardo, J.M.L., Pedroso, J.V., Carrillo, J.L.A., Silva, L.M.U., Encarna\u00e7\u00e3o, M.M.G.A.D., Horta, M.J.D.V.C., Cal\u00e7ada, M.T.C.S., and Nery, R.F.V. (2017). Perfil Dos Alunos \u00e0 Sa\u00edda Da Escolaridade Obrigat\u00f3ria, Minist\u00e9rio da Educa\u00e7\u00e3o e Ci\u00eancia\u2014DGE."},{"key":"ref_47","unstructured":"AAAS (1993). Project 2061: Benchmarks for Science Literacy, Oxford University Press."},{"key":"ref_48","unstructured":"Ebel, R.L., and Frisbie, D.A. (1991). Essentials of Educational Measurement, Prentice Hail International, Inc.. [5th ed.]."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Haladyna, T.M. (2004). Developing and Validating Multiple-Choice Test Items, Lawrence Erlbaum Associates. [3rd ed.].","DOI":"10.4324\/9780203825945"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"3061","DOI":"10.1590\/S1413-81232011000800006","article-title":"Validade de Conte\u00fado Nos Processos de Constru\u00e7\u00e3o e Adapta\u00e7\u00e3o de Instrumentos de Medidas","volume":"16","author":"Alexandre","year":"2011","journal-title":"Cien. Saude Colet."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1111\/j.1745-3992.2003.tb00141.x","article-title":"Reconsidering Reliability in Classroom Assessment and Grading","volume":"22","author":"Smith","year":"2003","journal-title":"Educ. Meas. Issues Pract."},{"key":"ref_52","first-page":"1000","article-title":"Teoria Da Resposta Ao Item","volume":"3","author":"Bortolotti","year":"2009","journal-title":"Rev. da Esc. Enferm. USP"},{"key":"ref_53","unstructured":"Baker, F.B. (2001). The Basics of Item Response Theory, ERIC."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3389\/fpsyg.2021.637547","article-title":"A Review of Key Likert Scale Development Advances: 1995\u20132019","volume":"12","author":"Jebb","year":"2021","journal-title":"Front. Psychol."},{"key":"ref_55","unstructured":"Ferreira, E.A. (2018). Teoria de Tesposta Ao Item\u2014TRI: An\u00e1lise de Algumas Quest\u00f5es Do ENEM: Habilidades 24 a 30. [Master\u2019s Thesis, Universidade Federal da grande Dourados]."},{"key":"ref_56","unstructured":"Mendon\u00e7a Filho, E.J.D. (2017). Evid\u00eancias de Validade Relacionadas \u00e0 Estrutura Interna Da Escala Cognitiva Do Invent\u00e1rio Dimensional de Avalia\u00e7\u00e3o Do Desenvolvimento Infantil. [Master\u2019s Thesis, Universidade Federal do Rio Grande do Sul]."},{"key":"ref_57","unstructured":"Pasquali, L. (2018). TRI\u2014Teoria de Resposta Ao Item: Teoria, Procedimentos e Aplica\u00e7\u00f5es, Appris."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Baker, F.B., and Kim, S.-H. (2017). The Basics of Item Response Theory. Statistics for Social and Behavioral Sciences, Springer.","DOI":"10.1007\/978-3-319-54205-8"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s11136-007-9198-0","article-title":"Applying Item Response Theory (IRT) Modeling to Questionnaire Development, Evaluation, and Refinement","volume":"16","author":"Edelen","year":"2007","journal-title":"Qual. Life Res."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Fayers, P.M., and Hays, R.D. (2005). Assessing Quality of Life in Clinical Trials: Methods and Practice, Oxford University Press.","DOI":"10.1093\/oso\/9780198527695.001.0001"},{"key":"ref_61","first-page":"35","article-title":"Alguns Aspectos Da Teoria de Resposta Ao Item Relativos \u00e0 Estima\u00e7\u00e3o Das Profici\u00eancias","volume":"21","author":"Klein","year":"2013","journal-title":"Ensaio"},{"key":"ref_62","unstructured":"Duarte, A., Nunes, A., Tavares, J., Mota, M., and Ven\u00e2ncio, T. (2020). TIMSS 2019\u2014Portugal. Resultados a Matem\u00e1tica e a Ci\u00eancias\u20144.\u00b0 Ano\u2014Volume 1, IAVE."},{"key":"ref_63","unstructured":"Louren\u00e7o, V., Duarte, A., Nunes, A., Amaral, A., Gon\u00e7alves, C., Mota, M., and Mendes, R. (2019). PISA 2018\u2014Portugal. Relat\u00f3rio Nacional, IAVE."},{"key":"ref_64","unstructured":"Portugal (2022, November 21). Despacho Normativo n.o 1-F\/2016 de 5 de abril de 2016. Available online: http:\/\/www.dge.mec.pt\/sites\/default\/files\/Noticias_Imagens\/2016_dn_1f.pdf."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Coppi, M., Fialho, I., and Cid, M. (2023). Assessing Portuguese Elementary School Students\u2019 Scientific Literacy: Application of the ALCE Instrument. Soc. Sci., 12.","DOI":"10.3390\/socsci12070374"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/jedm.12000","article-title":"Validating the Interpretations and Uses of Test Scores","volume":"50","author":"Kane","year":"2013","journal-title":"J. Educ. Meas."}],"container-title":["Education Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-7102\/13\/9\/941\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:52:05Z","timestamp":1760129525000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-7102\/13\/9\/941"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,16]]},"references-count":66,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["educsci13090941"],"URL":"https:\/\/doi.org\/10.3390\/educsci13090941","relation":{},"ISSN":["2227-7102"],"issn-type":[{"value":"2227-7102","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,16]]}}}