{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T17:05:51Z","timestamp":1790787951578,"version":"4.1.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,9,10]],"date-time":"2021-09-10T00:00:00Z","timestamp":1631232000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,10]],"date-time":"2021-09-10T00:00:00Z","timestamp":1631232000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009828","name":"Universit\u00e4t Erfurt","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100009828","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Behav Res"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Because of the increasing popularity of voice-controlled virtual assistants, such as Amazon\u2019s Alexa and Google Assistant, they should be considered a new medium for psychological and behavioral research. We developed Survey Mate, an extension of Google Assistant, and conducted two studies to analyze the reliability and validity of data collected through this medium. In the first study, we assessed validated procrastination and shyness scales as well as social desirability indicators for both the virtual assistant and an online questionnaire. The results revealed comparable internal consistency and construct and criterion validity. In the second study, five social psychological experiments, which have been successfully replicated by the Many Labs projects, were successfully reproduced using a virtual assistant for data collection. Comparable effects were observed for users of both smartphones and smart speakers. Our findings point to the applicability of virtual assistants in data collection independent of the device used. While we identify some limitations, including data privacy concerns and a tendency toward more socially desirable responses, we found that virtual assistants could allow the recruitment of participants who are hard to reach with established data collection techniques, such as people with visual impairment, dyslexia, or lower education. This new medium could also be suitable for recruiting samples from non-Western countries because of its wide availability and easily adaptable language settings. It could also support an increase in the generalizability of theories in the future.<\/jats:p>","DOI":"10.3758\/s13428-021-01629-y","type":"journal-article","created":{"date-parts":[[2021,9,10]],"date-time":"2021-09-10T21:02:47Z","timestamp":1631307767000},"page":"1227-1239","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Ok Google: Using virtual assistants for data collection in psychological and behavioral research"],"prefix":"10.3758","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9925-1920","authenticated-orcid":false,"given":"Philipp","family":"Sprengholz","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cornelia","family":"Betsch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,9,10]]},"reference":[{"issue":"3","key":"1629_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3311956","volume":"26","author":"T Ammari","year":"2019","unstructured":"Ammari, T., Kaye, J., Tsai, J. Y., & Bentley, F. (2019). Music, Search, and IoT. ACM Transactions on Computer-Human Interaction, 26(3), 1\u201328. https:\/\/doi.org\/10.1145\/3311956","journal-title":"ACM Transactions on Computer-Human Interaction"},{"issue":"1","key":"1629_CR2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1745691610393980","volume":"6","author":"M Buhrmester","year":"2011","unstructured":"Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon\u2019s Mechanical Turk. Perspectives on Psychological Science, 6(1), 3\u20135. https:\/\/doi.org\/10.1177\/1745691610393980","journal-title":"Perspectives on Psychological Science"},{"key":"1629_CR3","doi-asserted-by":"crossref","unstructured":"Cheek, J. M. (1983). The Revised Cheek and Buss Shyness Scale (RCBS). Unpublished Manuscript. http:\/\/academics.wellesley.edu\/Psychology\/Cheek\/research.html","DOI":"10.1037\/t05422-000"},{"issue":"1","key":"1629_CR4","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1177\/014662168500900103","volume":"9","author":"DV Cicchetti","year":"1985","unstructured":"Cicchetti, D. V., Shoinralter, D., & Tyrer, P. J. (1985). The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability\u00a0: A Monte Carlo Investigation. Applied Psychological Measurement, 9(1), 31\u201336. https:\/\/doi.org\/10.1177\/014662168500900103","journal-title":"Applied Psychological Measurement"},{"key":"1629_CR5","unstructured":"Davie, N., & Hilber, T. (2018). Opportunities and challenges of using Amazon Echo in education. Proceedings of 2018 IADIS International Conference Mobile Learning, 205\u2013208."},{"issue":"1","key":"1629_CR6","first-page":"51","volume":"11","author":"B Diedenhofen","year":"2016","unstructured":"Diedenhofen, B., & Musch, J. (2016). cocron: A Web Interface and R Package for the Statistical Comparison of Cronbach\u2019s Alpha Coefficients. International Journal of Internet Science, 11(1), 51\u201360.","journal-title":"International Journal of Internet Science"},{"issue":"1","key":"1629_CR7","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1207\/s15327752jpa4901_13","volume":"49","author":"E Diener","year":"1985","unstructured":"Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The Satisfaction With Life Scale. Journal of Personality Assessment, 49(1), 71\u201375. https:\/\/doi.org\/10.1207\/s15327752jpa4901_13","journal-title":"Journal of Personality Assessment"},{"issue":"4","key":"1629_CR8","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1002\/tesj.353","volume":"8","author":"G Dizon","year":"2017","unstructured":"Dizon, G. (2017). Using Intelligent Personal Assistants for Second Language Learning: A Case Study of Alexa. TESOL Journal, 8(4), 811\u2013830. https:\/\/doi.org\/10.1002\/tesj.353","journal-title":"TESOL Journal"},{"key":"1629_CR9","unstructured":"Dubiel, M., Halvey, M., & Azzopardi, L. (2018). A Survey Investigating Usage of Virtual Personal Assistants. http:\/\/arxiv.org\/abs\/1807.04606"},{"issue":"1","key":"1629_CR10","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1093\/ijpor\/edl022","volume":"19","author":"KM Hansen","year":"2006","unstructured":"Hansen, K. M. (2006). The Effects of Incentives, Interview Length, and Interviewer Characteristics on Response Rates in a CATI-Study. International Journal of Public Opinion Research, 19(1), 112\u2013121. https:\/\/doi.org\/10.1093\/ijpor\/edl022","journal-title":"International Journal of Public Opinion Research"},{"issue":"1","key":"1629_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1111\/j.1468-0017.2006.00297.x","volume":"22","author":"M Hauser","year":"2007","unstructured":"Hauser, M., Cushman, F., Young, L., Kang-Xing Jin, R., & Mikhail, J. (2007). A Dissociation Between Moral Judgments and Justifications. Mind & Language, 22(1), 1\u201321. https:\/\/doi.org\/10.1111\/j.1468-0017.2006.00297.x","journal-title":"Mind & Language"},{"issue":"2\u20133","key":"1629_CR12","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1017\/S0140525X0999152X","volume":"33","author":"J Henrich","year":"2010","unstructured":"Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2\u20133), 61\u201383. https:\/\/doi.org\/10.1017\/S0140525X0999152X","journal-title":"Behavioral and Brain Sciences"},{"issue":"6245","key":"1629_CR13","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1126\/science.aaa8685","volume":"349","author":"J Hirschberg","year":"2015","unstructured":"Hirschberg, J., & Manning, C. D. (2015). Advances in natural language processing. Science, 349(6245), 261\u2013266. https:\/\/doi.org\/10.1126\/science.aaa8685","journal-title":"Science"},{"key":"1629_CR14","doi-asserted-by":"crossref","unstructured":"Hopko, D. R., Stowell, J., Jones, W. H., Armento, M. E., & Cheek, J. M. (2005). Psychometric properties of the revised Cheek and Buss shyness scale. Journal of Personality Assessment, 84(2), 185\u2013192.","DOI":"10.1207\/s15327752jpa8402_08"},{"issue":"1","key":"1629_CR15","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1080\/02763869.2018.1404391","volume":"37","author":"MB Hoy","year":"2018","unstructured":"Hoy, M. B. (2018). Alexa, Siri, Cortana, and More: An Introduction to Voice Assistants. Medical Reference Services Quarterly, 37(1), 81\u201388. https:\/\/doi.org\/10.1080\/02763869.2018.1404391","journal-title":"Medical Reference Services Quarterly"},{"issue":"2","key":"1629_CR16","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1002\/(SICI)1099-0771(199806)11:2<107::AID-BDM292>3.0.CO;2-Y","volume":"11","author":"CK Hsee","year":"1998","unstructured":"Hsee, C. K. (1998). Less is better: when low-value options are valued more highly than high-value options. Journal of Behavioral Decision Making, 11(2), 107\u2013121. https:\/\/doi.org\/10.1002\/(SICI)1099-0771(199806)11:2<107::AID-BDM292>3.0.CO;2-Y","journal-title":"Journal of Behavioral Decision Making"},{"issue":"11","key":"1629_CR17","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1177\/01461672952111004","volume":"21","author":"KE Jacowitz","year":"1995","unstructured":"Jacowitz, K. E., & Kahneman, D. (1995). Measures of Anchoring in Estimation Tasks. Personality and Social Psychology Bulletin, 21(11), 1161\u20131166. https:\/\/doi.org\/10.1177\/01461672952111004","journal-title":"Personality and Social Psychology Bulletin"},{"key":"1629_CR18","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1016\/j.ijmedinf.2019.07.001","volume":"129","author":"T Jadczyk","year":"2019","unstructured":"Jadczyk, T., Kiwic, O., Khandwalla, R. M., Grabowski, K., Rudawski, S., Magaczewski, P., Benyahia, H., Wojakowski, W., & Henry, T. D. (2019). Feasibility of a voice-enabled automated platform for medical data collection: CardioCube. International Journal of Medical Informatics, 129, 388\u2013393. https:\/\/doi.org\/10.1016\/j.ijmedinf.2019.07.001","journal-title":"International Journal of Medical Informatics"},{"key":"1629_CR19","unstructured":"Juniper Research. (2019). Digital Voice Assistants. http:\/\/www.juniperresearch.com\/researchstore\/innovation-disruption\/digital-voice-assistants"},{"issue":"22","key":"1629_CR20","doi-asserted-by":"publisher","first-page":"10031","DOI":"10.1073\/pnas.92.22.10031","volume":"92","author":"C Kamm","year":"1995","unstructured":"Kamm, C. (1995). User interfaces for voice applications. Proceedings of the National Academy of Sciences, 92(22), 10031\u201310037. https:\/\/doi.org\/10.1073\/pnas.92.22.10031","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"3","key":"1629_CR21","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1027\/1864-9335\/a000373","volume":"50","author":"RA Klein","year":"2019","unstructured":"Klein, R. A. (2019). Correction to Klein et al. (2014). Social Psychology, 50(3), 211\u2013213. https:\/\/doi.org\/10.1027\/1864-9335\/a000373","journal-title":"Social Psychology"},{"issue":"3","key":"1629_CR22","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1027\/1864-9335\/a000178","volume":"45","author":"RA Klein","year":"2014","unstructured":"Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahn\u00edk, \u0160., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., \u2026 Nosek, B. A. (2014). Investigating Variation in Replicability. Social Psychology, 45(3), 142\u2013152. https:\/\/doi.org\/10.1027\/1864-9335\/a000178","journal-title":"Social Psychology"},{"issue":"4","key":"1629_CR23","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1177\/2515245918810225","volume":"1","author":"RA Klein","year":"2018","unstructured":"Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahn\u00edk, \u0160., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., \u2026 Nosek, B. A. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443\u2013490. https:\/\/doi.org\/10.1177\/2515245918810225","journal-title":"Advances in Methods and Practices in Psychological Science"},{"issue":"5","key":"1629_CR24","doi-asserted-by":"publisher","first-page":"847","DOI":"10.1093\/poq\/nfn063","volume":"72","author":"F Kreuter","year":"2008","unstructured":"Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity. Public Opinion Quarterly, 72(5), 847\u2013865. https:\/\/doi.org\/10.1093\/poq\/nfn063","journal-title":"Public Opinion Quarterly"},{"issue":"4","key":"1629_CR25","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1177\/1948550617697177","volume":"8","author":"D Lakens","year":"2017","unstructured":"Lakens, D. (2017). Equivalence Tests. Social Psychological and Personality Science, 8(4), 355\u2013362. https:\/\/doi.org\/10.1177\/1948550617697177","journal-title":"Social Psychological and Personality Science"},{"issue":"4","key":"1629_CR26","doi-asserted-by":"publisher","first-page":"e0194559","DOI":"10.1371\/journal.pone.0194559","volume":"13","author":"C Liu","year":"2018","unstructured":"Liu, C., Cheng, Y., Hsu, A. S. C., Chen, C., Liu, J., & Yu, G. (2018). Optimism and self-efficacy mediate the association between shyness and subjective well-being among Chinese working adults. PLOS ONE, 13(4), e0194559. https:\/\/doi.org\/10.1371\/journal.pone.0194559","journal-title":"PLOS ONE"},{"key":"1629_CR27","doi-asserted-by":"publisher","unstructured":"Moore, R. J., & Arar, R. (2019). Conversational UX Design: A Practitioner\u2019s Guide to the Natural Conversation Framework. Association for Computing Machinery. https:\/\/doi.org\/10.1145\/3304087","DOI":"10.1145\/3304087"},{"issue":"3","key":"1629_CR28","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1002\/ejsp.2420150303","volume":"15","author":"AJ Nederhof","year":"1985","unstructured":"Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15(3), 263\u2013280. https:\/\/doi.org\/10.1002\/ejsp.2420150303","journal-title":"European Journal of Social Psychology"},{"issue":"1","key":"1629_CR29","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/s42409-018-0005-1","volume":"1","author":"D Nie\u00dfen","year":"2019","unstructured":"Nie\u00dfen, D., Partsch, M. V., Kemper, C. J., & Rammstedt, B. (2019). An English-Language Adaptation of the Social Desirability\u2013Gamma Short Scale (KSE-G). Measurement Instruments for the Social Sciences, 1(1), 2. https:\/\/doi.org\/10.1186\/s42409-018-0005-1","journal-title":"Measurement Instruments for the Social Sciences"},{"key":"1629_CR30","unstructured":"NPR, & Edison Research. (2020). The Smart Audio Report. http:\/\/www.nationalpublicmedia.com\/insights\/reports\/smart-audio-report\/"},{"key":"1629_CR31","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.jbef.2017.12.004","volume":"17","author":"S Palan","year":"2018","unstructured":"Palan, S., & Schitter, C. (2018). Prolific.ac\u2014A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22\u201327. https:\/\/doi.org\/10.1016\/j.jbef.2017.12.004","journal-title":"Journal of Behavioral and Experimental Finance"},{"key":"1629_CR32","doi-asserted-by":"publisher","unstructured":"P\u00e9rez Garc\u00eda, M., Saffon L\u00f3pez, S., & Donis, H. (2018, July 1). Everybody is talking about Virtual Assistants, but how are people really using them? Proceedings of the 32nd International BCS Human Computer Interaction Conference. https:\/\/doi.org\/10.14236\/ewic\/HCI2018.96","DOI":"10.14236\/ewic\/HCI2018.96"},{"key":"1629_CR33","unstructured":"Pew Research Center. (2019). Mobile Connectivity in Emerging Economies. http:\/\/www.internet\/internet\/2019\/03\/07\/mobile-connectivity-in-emerging-economies\/"},{"issue":"1","key":"1629_CR34","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1086\/265467","volume":"5","author":"D Rugg","year":"1941","unstructured":"Rugg, D. (1941). Experiments in Wording Questions: II. Public Opinion Quarterly, 5(1), 91. https:\/\/doi.org\/10.1086\/265467","journal-title":"Public Opinion Quarterly"},{"issue":"2","key":"1629_CR35","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1177\/2515245919838781","volume":"2","author":"K Sassenberg","year":"2019","unstructured":"Sassenberg, K., & Ditrich, L. (2019). Research in Social Psychology Changed Between 2011 and 2016: Larger Sample Sizes, More Self-Report Measures, and More Online Studies. Advances in Methods and Practices in Psychological Science, 2(2), 107\u2013114. https:\/\/doi.org\/10.1177\/2515245919838781","journal-title":"Advances in Methods and Practices in Psychological Science"},{"issue":"8","key":"1629_CR36","doi-asserted-by":"publisher","first-page":"926","DOI":"10.1016\/j.paid.2010.02.025","volume":"48","author":"P Steel","year":"2010","unstructured":"Steel, P. (2010). Arousal, avoidant and decisional procrastinators: Do they exist? Personality and Individual Differences, 48(8), 926\u2013934. https:\/\/doi.org\/10.1016\/j.paid.2010.02.025","journal-title":"Personality and Individual Differences"},{"key":"1629_CR37","doi-asserted-by":"publisher","unstructured":"Svartdal, F., & Steel, P. (2017). Irrational Delay Revisited: Examining Five Procrastination Scales in a Global Sample. Frontiers in Psychology, 8. https:\/\/doi.org\/10.3389\/fpsyg.2017.01927","DOI":"10.3389\/fpsyg.2017.01927"},{"issue":"5","key":"1629_CR38","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1037\/0033-2909.133.5.859","volume":"133","author":"R Tourangeau","year":"2007","unstructured":"Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859\u2013883. https:\/\/doi.org\/10.1037\/0033-2909.133.5.859","journal-title":"Psychological Bulletin"},{"issue":"4481","key":"1629_CR39","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1126\/science.7455683","volume":"211","author":"A Tversky","year":"1981","unstructured":"Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453\u2013458. https:\/\/doi.org\/10.1126\/science.7455683","journal-title":"Science"},{"issue":"9","key":"1629_CR40","doi-asserted-by":"publisher","first-page":"1666","DOI":"10.2105\/AJPH.2011.300194","volume":"101","author":"S Yeatman","year":"2011","unstructured":"Yeatman, S., & Trinitapoli, J. (2011). Best-Friend Reports: A Tool for Measuring the Prevalence of Sensitive Behaviors. American Journal of Public Health, 101(9), 1666\u20131667. https:\/\/doi.org\/10.2105\/AJPH.2011.300194","journal-title":"American Journal of Public Health"}],"container-title":["Behavior Research Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-021-01629-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-021-01629-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-021-01629-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,8]],"date-time":"2022-06-08T10:06:32Z","timestamp":1654682792000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.3758\/s13428-021-01629-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,10]]},"references-count":40,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["1629"],"URL":"https:\/\/doi.org\/10.3758\/s13428-021-01629-y","relation":{},"ISSN":["1554-3528"],"issn-type":[{"value":"1554-3528","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,10]]},"assertion":[{"value":"18 May 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 September 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Both studies were conducted in accordance with German Psychological Association guidelines. All participants provided informed consent to use and share their data for scientific purposes without disclosure of their identities. The research was conducted at a German university, where institutional review boards or committees are not mandatory. It is negligible-risk research, with no foreseeable risk of harm or discomfort other than potential inconvenience expected from participation. All participants knew that they were free to quit the study at any time.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interest"}}]}}