{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T15:24:07Z","timestamp":1776525847069,"version":"3.51.2"},"reference-count":16,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2021,11,20]],"date-time":"2021-11-20T00:00:00Z","timestamp":1637366400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study\u2019s findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.<\/jats:p>","DOI":"10.3390\/ijerph182212181","type":"journal-article","created":{"date-parts":[[2021,11,21]],"date-time":"2021-11-21T20:58:21Z","timestamp":1637528301000},"page":"12181","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030"],"prefix":"10.3390","volume":"18","author":[{"given":"Claudia","family":"Cappa","sequence":"first","affiliation":[{"name":"United Nations Children\u2019s Fund, New York, NY 10017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicole","family":"Petrowski","sequence":"additional","affiliation":[{"name":"United Nations Children\u2019s Fund, New York, NY 10017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elga Filipa","family":"De Castro","sequence":"additional","affiliation":[{"name":"United Nations Children\u2019s Fund, New York, NY 10017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emily","family":"Geisen","sequence":"additional","affiliation":[{"name":"Qualtrics, Raleigh, NC 27603, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patricia","family":"LeBaron","sequence":"additional","affiliation":[{"name":"RTI International, Research Triangle Park, NC 27709, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9097-2553","authenticated-orcid":false,"given":"Betania","family":"Allen-Leigh","sequence":"additional","affiliation":[{"name":"Instituto Nacional de Salud P\u00fablica, Cuernavaca, Morelos 62100, Mexico"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean Marie","family":"Place","sequence":"additional","affiliation":[{"name":"Department of Nutrition and Health Science, Ball State University, Muncie, IN 47303, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul J.","family":"Scanlon","sequence":"additional","affiliation":[{"name":"National Center for Health Statistics, Hyattsville, MD 20782, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1817","DOI":"10.1016\/j.jand.2018.09.005","article-title":"Establishing Validity and Cross-context Equivalence of Measures and Indicators","volume":"119","author":"Edward","year":"2019","journal-title":"J. Acad. Nutr. Diet."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Colbert, C.Y., French, J.C., Arroliga, A.C., and Bierer, S.B. (2019). Best Practice Versus Actual Practice: An Audit of Survey Pretesting Practices Reported in a Sample of Medical Education Journals. Med. Educ. Online, 24.","DOI":"10.1080\/10872981.2019.1673596"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bennett, C., Khangura, S., Brehaut, J.C., Graham, I.D., Moher, D., Potter, B.K., and Grimshaw, J.M. (2011). Reporting Guidelines for Survey Research: An Analysis of Published Guidance and Reporting Practices. PLoS Med., 8.","DOI":"10.1371\/journal.pmed.1001069"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"McCoy, D.C., Peet, E.D., Ezzati, M., Danaei, G., Black, M.M., Sudfeld, C.R., Fawzi, W., and Fink, G. (2016). Early Childhood Developmental Status in Low and Middle-Income Countries: National, Regional, and Global Prevalence Estimates Using Predictive Modeling. PLoS Med., 13.","DOI":"10.1371\/journal.pmed.1002034"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Fernald, L., Prado, E., Kariger, P., and Raikes, A. (2017). A Toolkit for Measuring Early Childhood Development in Low- and Middle-Income Countries, The World Bank.","DOI":"10.1596\/29000"},{"key":"ref_6","unstructured":"Loizillon, A., Petrowski, N., Britto, P., and Cappa, C. (2017). Development of the Early Childhood Development Index in MICS Surveys, UNICEF. MICS Methodological Papers, No. 6, Data and Analytics Section, Division of Data, Research and Policy."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Madans, J., Miller, K., Maitland, A., and Willis, G. (2011). Cognitive interviewing. Question Evaluation Methods: Contributing to the Science of Data Quality, John Wiley & Sons.","DOI":"10.1002\/9781118037003"},{"key":"ref_8","unstructured":"Jabine, T., Straf, M., Tanur, J., and Tourangeau, R. (1984). Cognitive science and survey methods: A cognitive perspective. Cognitive Aspects of Survey Design: Building a Bridge Between Disciplines, National Academy Press."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Miller, K., Chepp, V., Willson, S., and Padilla, J.L. (2014). Data collection. Cognitive Interviewing Methodology, John Wiley & Sons.","DOI":"10.1002\/9781118838860"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Miller, K., Chepp, V., Willson, S., and Padilla, J.L. (2014). Cognitive Interviewing Methodology, John Wiley & Sons.","DOI":"10.1002\/9781118838860"},{"key":"ref_11","unstructured":"Scanlon, P., and Miller, K. (2017). Cognitive Evaluation of the UNICEF Early Child Development Index (ECDI) and Inclusive Education Modules in the United States, India, and Jamaica, National Center for Health Statistics, Centers for Disease Control and Prevention."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1177\/1525822X11416092","article-title":"Cross-Cultural Cognitive Interviewing: Seeking Comparability and Enhancing Understanding","volume":"23","author":"Willis","year":"2011","journal-title":"Field Methods"},{"key":"ref_13","first-page":"69","article-title":"Summary of Washington Group Question Evaluation Studies","volume":"Volume 61","author":"Altman","year":"2016","journal-title":"International Measurement of Disability"},{"key":"ref_14","unstructured":"(2021, September 17). National Center for Health Statistics, Q-Notes, Available online: https:\/\/www.cdc.gov\/nchs\/ccqder\/products\/qnotes.htm."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Harkness, J.A., Braun, M., Edwards, B., Johnson, T.P., Lyberg, L.E., Mohler, P.P., Pennell, B.-E., and Smith, T.W. (2010). Survey Methods in Multinational, Multiregional, and Multicultural Contexts, John Wiley & Sons.","DOI":"10.1002\/9780470609927"},{"key":"ref_16","unstructured":"Lyberg, L.E., Pennell, B.E., Cibelli Hibben, K.L., and de Jong, J.A. (2021). Task Force Report on Quality in Comparative Surveys, American Association for Public Opinion Research\/World Association for Public Opinion Research."}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/18\/22\/12181\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:33:08Z","timestamp":1760167988000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/18\/22\/12181"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,20]]},"references-count":16,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["ijerph182212181"],"URL":"https:\/\/doi.org\/10.3390\/ijerph182212181","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,20]]}}}