{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T05:19:42Z","timestamp":1794028782449,"version":"4.8.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T00:00:00Z","timestamp":1696291200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T00:00:00Z","timestamp":1696291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Behav Res"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Tools like ChatGPT, which allow people to unlock the potential of large language models (LLMs), have taken the world by storm. ChatGPT\u2019s ability to produce written output of remarkable quality has inspired, or forced, academics to consider its consequences for both research and education. In particular, the question of what constitutes authorship, and how to evaluate (scientific) contributions has received a lot of attention. However, its impact on (online) human data collection has mostly flown under the radar. The current paper examines how ChatGPT can be (mis)used in the context of generating norming data. We found that ChatGPT is able to produce sensible output, resembling that of human participants, for a typicality rating task. Moreover, the test\u2013retest reliability of ChatGPT\u2019s ratings was similar to that of human participants tested 1\u00a0day apart. We discuss the relevance of these findings in the context of (online) human data collection, focusing both on opportunities (e.g., (risk-)free pilot data) and challenges (e.g., data fabrication).<\/jats:p>","DOI":"10.3758\/s13428-023-02235-w","type":"journal-article","created":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T16:03:53Z","timestamp":1696349033000},"page":"4974-4981","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["The impact of ChatGPT on human data collection: A case study involving typicality norming data"],"prefix":"10.3758","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0565-441X","authenticated-orcid":false,"given":"Tom","family":"Heyman","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geert","family":"Heyman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,10,3]]},"reference":[{"key":"2235_CR1","unstructured":"Allaire, J., Cheng, J., Xie, Y., McPherson, J., Chang, W., Allen, J., ... Hyndman, R. (2016). rmarkdown: Dynamic Documents for R. https:\/\/CRAN.R-project.org\/package=rmarkdown"},{"key":"2235_CR2","unstructured":"Aust, F., & Barth, M. (2017). papaja: Create APA Manuscripts with R Markdown. https:\/\/github.com\/crsh\/papaja"},{"key":"2235_CR3","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-021-01787-z","author":"B Banks","year":"2022","unstructured":"Banks, B., & Connell, L. (2022). Category production norms for 117 concrete and abstract categories. Behavior Research Methods. https:\/\/doi.org\/10.3758\/s13428-021-01787-z","journal-title":"Behavior Research Methods"},{"key":"2235_CR4","doi-asserted-by":"publisher","unstructured":"Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don\u2019t count, predict! A systematic comparison of context-counting vs. Context-predicting semantic vectors. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 238\u2013247. https:\/\/doi.org\/10.3115\/v1\/P14-1023","DOI":"10.3115\/v1\/P14-1023"},{"key":"2235_CR5","unstructured":"Barsalou, L. W. (1987). Concepts and conceptual development: Ecological and intellectual factors in categorization, Neisser, U. (Ed.); pp. 101\u2013140. Cambridge: Cambridge University Press."},{"key":"2235_CR6","unstructured":"Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., ... Zhang, Y. (2023). Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303.12712"},{"key":"2235_CR7","unstructured":"Connell, L., & Ramscar, M. (2001). Using distributional measures to model typicality in categorization. Proceedings of the 23rd Annual Conference of the Cognitive Science Society, 226\u2013231. https:\/\/escholarship.org\/uc\/item\/0kq3h2sm"},{"key":"2235_CR8","doi-asserted-by":"publisher","unstructured":"Cuskley, C., & Sulik, J. (2022). The burden for high-quality online data collection lies with researchers, not recruitment platforms. https:\/\/doi.org\/10.31234\/osf.io\/w7qy9","DOI":"10.31234\/osf.io\/w7qy9"},{"key":"2235_CR9","doi-asserted-by":"publisher","unstructured":"De Deyne, S., Verheyen, S., Ameel, E., Vanpaemel, W., Dry, M. J., Voorspoels, W., & Storms, G. (2008). Exemplar by feature applicability matrices and other Dutch normative data for semantic concepts. Behavior Research Methods, 40(4), 1030\u20131048. https:\/\/doi.org\/10.3758\/brm.40.4.1030","DOI":"10.3758\/brm.40.4.1030"},{"key":"2235_CR10","unstructured":"Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805"},{"issue":"7","key":"2235_CR11","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1016\/j.tics.2023.04.008","volume":"27","author":"D Dillion","year":"2023","unstructured":"Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023). Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7), 597\u2013600. https:\/\/doi.org\/10.1016\/j.tics.2023.04.008","journal-title":"Trends in Cognitive Sciences"},{"key":"2235_CR12","doi-asserted-by":"publisher","unstructured":"Heyman, T., & Heyman, G. (2019). Can prediction-based distributional semantic models predict typicality? Quarterly Journal of Experimental Psychology, 72(8), 2084\u20132109. https:\/\/doi.org\/10.1177\/1747021819830949","DOI":"10.1177\/1747021819830949"},{"key":"2235_CR13","doi-asserted-by":"publisher","unstructured":"Kasneci, E., Sessler, K., K\u00fcchemann, S., Bannert, M., Dementieva, D., Fischer, F., ... Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https:\/\/doi.org\/10.1016\/j.lindif.2023.102274","DOI":"10.1016\/j.lindif.2023.102274"},{"key":"2235_CR14","unstructured":"Kirchner, J. H., Ahmad, L., Aaronson, S., & Leike, J. (2023). New AI classifier for Indicating AI-Written Text. https:\/\/openai.com\/blog\/new-ai-classifier-for-indicating-ai-written-text"},{"key":"2235_CR15","unstructured":"Kosinski, M. (2023). Theory of mind may have spontaneously emerged in large language models. http:\/\/arxiv.org\/abs\/2302.02083"},{"key":"2235_CR16","doi-asserted-by":"crossref","unstructured":"Loconte, R., Orr\u00fa, G., Tribastone, M., Pietrini, P., & Sartori, G. (2023). Challenging ChatGPT \u2019intelligence\u2019 with human tools: A neuropsychological investigation on prefrontal functioning of a large language model. https:\/\/ssrn.com\/abstract=4377371","DOI":"10.2139\/ssrn.4471829"},{"issue":"1","key":"2235_CR17","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/S0749-596X(03)00021-4","volume":"49","author":"BC Malt","year":"2003","unstructured":"Malt, B. C., Sloman, S. A., & Gennari, S. P. (2003). Universality and language specificity in object naming. Journal of Memory and Language, 49(1), 20\u201342. https:\/\/doi.org\/10.1016\/S0749-596X(03)00021-4","journal-title":"Journal of Memory and Language"},{"key":"2235_CR18","unstructured":"Marcus, G., & Davis, E. (2023). How not to test GPT-3. https:\/\/garymarcus.substack.com\/p\/how-not-to-test-gpt-3"},{"key":"2235_CR19","unstructured":"Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic regularities in continuous space word representations. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language tTchnologies, 746\u2013751. https:\/\/aclanthology.org\/N13-1090"},{"issue":"4","key":"2235_CR20","doi-asserted-by":"publisher","first-page":"608","DOI":"10.3758\/bf03192731","volume":"37","author":"LI Morrow","year":"2005","unstructured":"Morrow, L. I., & Duffy, M. F. (2005). The representation of ontological category concepts as affected by healthy aging: Normative data and theoretical implications. Behavior Research Methods, 37(4), 608\u2013625. https:\/\/doi.org\/10.3758\/bf03192731","journal-title":"Behavior Research Methods"},{"key":"2235_CR21","unstructured":"OpenAI. (2023). GPT-4 technical report. arXiv:2303.08774"},{"key":"2235_CR22","unstructured":"Park, P. S., Schoenegger, P., & Zhu, C. (2023). \u201cCorrect answers\u201d from the psychology of artificial intelligence. arXiv:2302.07267"},{"key":"2235_CR23","doi-asserted-by":"publisher","DOI":"10.1101\/2023.01.21.525030","author":"M Pividori","year":"2023","unstructured":"Pividori, M., & Greene, C. S. (2023). A publishing infrastructure for AI-assisted academic authoring.https:\/\/doi.org\/10.1101\/2023.01.21.525030","journal-title":"A publishing infrastructure for AI-assisted academic authoring."},{"key":"2235_CR24","unstructured":"R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https:\/\/www.R-project.org\/"},{"key":"2235_CR25","unstructured":"Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training."},{"key":"2235_CR26","doi-asserted-by":"crossref","unstructured":"Reiss, M. V. (2023). Testing the reliability of ChatGPT for text annotation and classification: A cautionary remark. arXiv:2304.11085","DOI":"10.31219\/osf.io\/rvy5p"},{"key":"2235_CR27","doi-asserted-by":"crossref","unstructured":"Renner, J., Denis, P., Gilleron, R., & Brunelli\u00e9re, A. (2023). Exploring category structure with contextual language models and lexical semantic networks. arXiv:2302.06942","DOI":"10.18653\/v1\/2023.eacl-main.167"},{"issue":"4","key":"2235_CR28","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1016\/0010-0285(75)90024-9","volume":"7","author":"E Rosch","year":"1975","unstructured":"Rosch, E., & Mervis, C. B. (1975). Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7(4), 573\u2013605. https:\/\/doi.org\/10.1016\/0010-0285(75)90024-9","journal-title":"Cognitive Psychology"},{"issue":"2","key":"2235_CR29","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1177\/2515245919838781","volume":"2","author":"K Sassenberg","year":"2019","unstructured":"Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 2(2), 107\u2013114. https:\/\/doi.org\/10.1177\/2515245919838781","journal-title":"Advances in Methods and Practices in Psychological Science"},{"issue":"7951","key":"2235_CR30","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1038\/d41586-023-00680-3","volume":"615","author":"ML Seghier","year":"2023","unstructured":"Seghier, M. L. (2023). ChatGPT: Not all languages are equal. Nature, 615(7951), 216. https:\/\/doi.org\/10.1038\/d41586-023-00680-3","journal-title":"Nature"},{"issue":"1","key":"2235_CR31","doi-asserted-by":"publisher","first-page":"72","DOI":"10.2307\/1412159","volume":"15","author":"C Spearman","year":"1904","unstructured":"Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72\u2013101. https:\/\/doi.org\/10.2307\/1412159","journal-title":"The American Journal of Psychology"},{"issue":"5","key":"2235_CR32","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0063507","volume":"8","author":"S Verheyen","year":"2013","unstructured":"Verheyen, S., & Storms, G. (2013). A mixture approach to vagueness and ambiguity. PloS One, 8(5), e63507. https:\/\/doi.org\/10.1371\/journal.pone.0063507","journal-title":"PloS One"},{"key":"2235_CR33","unstructured":"Veselovsky, V., Ribeiro, M. H., & West, R. (2023). Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv:2306.07899"},{"key":"2235_CR34","doi-asserted-by":"publisher","unstructured":"Webb, M. A., & Tangney, J. P. (2022). Too good to be true: Bots and bad data from mechanical turk. Perspectives on Psychological Science, 17456916221120028,. https:\/\/doi.org\/10.1177\/17456916221120027","DOI":"10.1177\/17456916221120027"},{"key":"2235_CR35","unstructured":"Zero, G. P. T. (2023). In AI Text Detector. https:\/\/www.zerogpt.com"}],"container-title":["Behavior Research Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-023-02235-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-023-02235-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-023-02235-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T18:16:53Z","timestamp":1722363413000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.3758\/s13428-023-02235-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,3]]},"references-count":35,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["2235"],"URL":"https:\/\/doi.org\/10.3758\/s13428-023-02235-w","relation":{},"ISSN":["1554-3528"],"issn-type":[{"value":"1554-3528","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,3]]},"assertion":[{"value":"1 September 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 October 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that there were no conflicts of interest with respect to the authorship or the publication of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}