{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T15:15:23Z","timestamp":1773933323936,"version":"3.50.1"},"reference-count":88,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T00:00:00Z","timestamp":1603152000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"name":"The University of Chicago, Center for Decision Research","award":["Postdoctoral Fellowship"],"award-info":[{"award-number":["Postdoctoral Fellowship"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2021,3,31]]},"abstract":"<jats:p>We provide guidelines for planning, executing, analyzing, and reporting hypothesis-driven experiments in Human\u2013Robot Interaction (HRI). The intended audience are researchers in the field of HRI who are not trained in empirical research but who are interested in conducting rigorous human-participant studies to support their research. Following the chronological order of research activities and grounded in updated research practices in psychological and behavioral sciences, this primer covers recommended methods and common pitfalls for defining research questions, identifying constructs and hypotheses, choosing appropriate study designs, operationalizing constructs as variables, planning and executing studies, sampling, choosing statistical tools for data analysis, and reporting results.<\/jats:p>","DOI":"10.1145\/3412374","type":"journal-article","created":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T12:20:19Z","timestamp":1603196419000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":95,"title":["A Primer for Conducting Experiments in Human\u2013Robot Interaction"],"prefix":"10.1145","volume":"10","author":[{"given":"Guy","family":"Hoffman","sequence":"first","affiliation":[{"name":"Cornell University, Ithaca, New York"}]},{"given":"Xuan","family":"Zhao","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, Illinois"}]}],"member":"320","published-online":{"date-parts":[[2020,10,20]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Adler and Peter Adler","author":"Patricia","year":"1994","unstructured":"Patricia A. Adler and Peter Adler. 1994. Observational Techniques. Sage, Thousand Oaks, CA."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797617723724"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797616647519"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612459060"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-017-0189-z"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-010-0064-9"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/job.1960"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020211"},{"key":"e_1_2_1_10_1","volume-title":"Abbott","author":"Bordens Kenneth S.","year":"2018","unstructured":"Kenneth S. Bordens and Bruce B. Abbott. 2018. Research Design and Methods: A Process Approach (10th ed.). McGraw--Hill, New York, NY."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2015.01.006"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020208"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0365-7"},{"key":"e_1_2_1_14_1","first-page":"205316801562207","article-title":"Are samples drawn from mechanical turk valid for research on political ideology","volume":"2","author":"Clifford Scott","year":"2015","unstructured":"Scott Clifford, Ryan M. Jewell, and Philip D. Waggoner. 2015. Are samples drawn from mechanical turk valid for research on political ideology? Res. Pol. 2, 4 (2015), 2053168015622072.","journal-title":"Res. Pol."},{"key":"e_1_2_1_15_1","volume-title":"Statistical Power Analysis for the Social Sciences","author":"Cohen Jacob","unstructured":"Jacob Cohen. 1988. Statistical Power Analysis for the Social Sciences. Erlbaum, Hillsdale, NJ."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.112.1.155"},{"key":"e_1_2_1_17_1","volume-title":"Designing Experiments for the Social Sciences: How to Plan, Create, and Execute Research Using Experiments","author":"Coleman Renita","unstructured":"Renita Coleman. 2018. Designing Experiments for the Social Sciences: How to Plan, Create, and Execute Research Using Experiments. Sage, Thousand Oaks, CA."},{"key":"e_1_2_1_18_1","volume-title":"Research Methods and Statistics in Psychology","author":"Coolican Hugh","unstructured":"Hugh Coolican. 2017. Research Methods and Statistics in Psychology. Psychology Press, Road Hove, UK."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2019.12.009"},{"key":"e_1_2_1_20_1","volume-title":"Confidence Intervals, and Meta-analysis","author":"Cumming Geoff","unstructured":"Geoff Cumming. 2013. Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis. Routledge, New York, NY."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2020.01.007"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2006.2004"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020236"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/xap0000092"},{"key":"e_1_2_1_25_1","unstructured":"Edward Diener and Robert Biswas-Diener. 2019. The Replication Crisis in Psychology."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-017-1266-z"},{"key":"e_1_2_1_27_1","unstructured":"Victor DiFate. 2007. Evidence. Retrieved from https:\/\/www.iep.utm.edu\/evidence\/."},{"key":"e_1_2_1_28_1","first-page":"1","article-title":"Machine translation: Mining text for social theory. Annu","volume":"42","author":"Evans James A.","year":"2016","unstructured":"James A. Evans and Pedro Aceves. 2016. Machine translation: Mining text for social theory. Annu. Rev. Sociol. 42, 1 (July 2016), 21--50.","journal-title":"Rev. Sociol."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03193146"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612459059"},{"key":"e_1_2_1_31_1","volume-title":"Discovering Statistics Using R","author":"Field Andy","unstructured":"Andy Field, Jeremy Miles, and Zo\u00eb Field. 2012. Discovering Statistics Using R. Sage, Thousand Oaks, CA."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550617693063"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1323051111"},{"key":"e_1_2_1_34_1","volume-title":"Strauss","author":"Glaser Barney G.","year":"2017","unstructured":"Barney G. Glaser and Anselm L. Strauss. 2017. Discovery of Grounded Theory: Strategies for Qualitative Research. Routledge, Abingdon-on-Thames, UK."},{"key":"e_1_2_1_35_1","volume-title":"Wallnau","author":"Gravetter Frederick J.","year":"2020","unstructured":"Frederick J. Gravetter and Larry B. Wallnau. 2020. Essentials of Statistics for Behavioral Sciences. Cengage Learning, Boston, MA."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10654-016-0149-3"},{"key":"e_1_2_1_37_1","volume-title":"Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-based Approach","author":"Hayes Andrew F.","unstructured":"Andrew F. Hayes. 2017. Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-based Approach. Guilford Publications, New York, NY."},{"key":"e_1_2_1_38_1","volume-title":"Philosophy of Natural Science","author":"Hempel Carl G.","unstructured":"Carl G. Hempel. 1966. Philosophy of Natural Science. Prentice-Hall, Englewood Cliffs, N.J."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1038\/466029a"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2018.1543616"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173386.3173389"},{"key":"e_1_2_1_43_1","volume-title":"An Introduction to Statistical Learning","author":"James Gareth","unstructured":"Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An Introduction to Statistical Learning. Vol. 112. Springer, Heidelberg, Germany."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1313476110"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-122414-033702"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr0203_4"},{"key":"e_1_2_1_47_1","volume-title":"Beyond Significance Testing: Statistics Reform in the Behavioral Sciences","author":"Kline Rex B.","unstructured":"Rex B. Kline. 2013. Beyond Significance Testing: Statistics Reform in the Behavioral Sciences. American Psychological Association, Washington, DC."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2017.00908"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2010.05.001"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2013.00863"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0122565"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720819872672"},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","unstructured":"D. Stephen Lindsay. 2015. Replication in psychological science.","DOI":"10.1177\/0956797615616374"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.58.110405.085542"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2015.09.008"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1751-9004.2009.00239.x"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-016-0021"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10869-017-9524-7"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-122216-011836"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.84.3.231"},{"key":"e_1_2_1_61_1","volume-title":"et\u00a0al","author":"Nosek Brian A.","year":"2015","unstructured":"Brian A. Nosek, George Alter, George C. Banks, Denny Borsboom, Sara D. Bowman, Steven J. Breckler, Stuart Buck, Christopher D. Chambers, Gilbert Chin, Garret Christensen, et\u00a0al. 2015. Promoting an open research culture. Science 348, 6242 (2015), 1422--1425."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708274114"},{"key":"e_1_2_1_63_1","volume-title":"Estimating the reproducibility of psychological science. Science 349, 6251","author":"Collaboration Open Science","year":"2015","unstructured":"Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1177\/0963721414531598"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171268"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1038\/nrn.2016.167"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.1.1.Riek"},{"key":"e_1_2_1_68_1","volume-title":"Proceedings of the 11th ACM\/IEEE International Conference on Human Robot Interaction (HRI\u201916)","author":"Robinette Paul","unstructured":"Paul Robinette, Wenchen Li, Robert Allen, Ayanna M. Howard, and Alan R. Wagner. 2016. Overtrust of robots in emergency evacuation scenarios. In Proceedings of the 11th ACM\/IEEE International Conference on Human Robot Interaction (HRI\u201916). IEEE Press, New York, NY, 101--108."},{"key":"e_1_2_1_69_1","volume-title":"Self-reports: How the questions shape the answers.American Psychologist 54, 2","author":"Schwarz Norbert","year":"1999","unstructured":"Norbert Schwarz. 1999. Self-reports: How the questions shape the answers.American Psychologist 54, 2 (1999), 93."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-122216-011845"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611417632"},{"key":"e_1_2_1_72_1","volume-title":"New Methods in Cognitive Psychology","author":"Singmann Henrik","unstructured":"Henrik Singmann and David Kellen. 2020. An introduction to linear mixed modeling in experimental psychology. In New Methods in Cognitive Psychology. Psychology Press, Road Hove, UK."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-019-03350-5"},{"key":"e_1_2_1_74_1","volume-title":"Qualitative Psychology: A Practical Guide to Research Methods","author":"Smith Jonathan A.","year":"2015","unstructured":"Jonathan A. Smith. 2015. Qualitative Psychology: A Practical Guide to Research Methods. Sage, Thousand Oaks, CA."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.44.5.1075"},{"key":"e_1_2_1_76_1","volume-title":"How to Think Straight About Psychology","author":"Stanovich Keith E.","unstructured":"Keith E. Stanovich. 2013. How to Think Straight About Psychology (10th ed.). Pearson, Boston, MA.","edition":"10"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2017.06.007"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1177\/0261927X09351676"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1080\/01973533.2015.1012991"},{"key":"e_1_2_1_80_1","volume-title":"Proceedings of the 14th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201919)","author":"Ullman Daniel","unstructured":"Daniel Ullman and Bertram F. Malle. 2019. Measuring gains and losses in human-robot trust: Evidence for differentiable components of trust. In Proceedings of the 14th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201919). IEEE Press, New York, NY, 618--619."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03193889"},{"key":"e_1_2_1_82_1","first-page":"40","article-title":"Faking it: Social desirability response bias in self-report research","volume":"25","author":"\u00a0al Thea F.","year":"2008","unstructured":"Thea F. Van de Mortel et\u00a0al. 2008. Faking it: Social desirability response bias in self-report research. Austr. J. Adv. Nurs. 25, 4 (2008), 40.","journal-title":"Austr. J. Adv. Nurs."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2016.03.004"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-017-1343-3"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0022790"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2014.01.005"},{"key":"e_1_2_1_87_1","doi-asserted-by":"crossref","unstructured":"Timothy D. Wilson Elliot Aronson and Kevin Carlsmith. 2010. The art of laboratory experimentation. In Handbook of Social Psychology. 51--81.","DOI":"10.1002\/9780470561119.socpsy001002"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1037\/pspa0000056"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412374","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3412374","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:25:01Z","timestamp":1750195501000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412374"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,20]]},"references-count":88,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,3,31]]}},"alternative-id":["10.1145\/3412374"],"URL":"https:\/\/doi.org\/10.1145\/3412374","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,20]]},"assertion":[{"value":"2019-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}