{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T04:57:35Z","timestamp":1769749055944,"version":"3.49.0"},"reference-count":113,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2023,7,24]],"date-time":"2023-07-24T00:00:00Z","timestamp":1690156800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NSF","award":["2016600, 2016586, 2016604"],"award-info":[{"award-number":["2016600, 2016586, 2016604"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,9,30]]},"abstract":"<jats:p>A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users\u2014software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.<\/jats:p>","DOI":"10.1145\/3587157","type":"journal-article","created":{"date-parts":[[2023,3,14]],"date-time":"2023-03-14T12:15:44Z","timestamp":1678796144000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["What\u2019s (Not) Working in Programmer User Studies?"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2366-8436","authenticated-orcid":false,"given":"Matthew C.","family":"Davis","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0607-4257","authenticated-orcid":false,"given":"Emad","family":"Aghayi","sequence":"additional","affiliation":[{"name":"George Mason University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9564-3337","authenticated-orcid":false,"given":"Thomas D.","family":"Latoza","sequence":"additional","affiliation":[{"name":"George Mason University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9079-5534","authenticated-orcid":false,"given":"Xiaoyin","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Texas at San Antonio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4769-0219","authenticated-orcid":false,"given":"Brad A.","family":"Myers","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9672-5297","authenticated-orcid":false,"given":"Joshua","family":"Sunshine","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,7,24]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1017\/psrm.2021.57"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183440.3195001"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-019-01237-x"},{"key":"e_1_3_2_5_2","unstructured":"2022. An author of \u201cArrestt: A framework to create reproducible experiments to evaluate software testing techniques.\u201d Personal communication."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962628"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1882362.1882368"},{"key":"e_1_3_2_8_2","first-page":"1","volume-title":"Experimental Software Engineering Issues: Critical Assessment and Future Directions","author":"Basili Victor R.","year":"1993","unstructured":"Victor R. Basili. 1993. The experimental paradigm in software engineering. In Experimental Software Engineering Issues: Critical Assessment and Future Directions. Springer, 1\u201312."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.1996.493439"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1986.6312975"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799939"},{"key":"e_1_3_2_12_2","volume-title":"Understanding Your Users: A Practical Guide to User Research Methods","author":"Baxter Kathy","year":"2015","unstructured":"Kathy Baxter, Catherine Courage, and Kelly Caine. 2015. Understanding Your Users: A Practical Guide to User Research Methods. Morgan Kaufmann."},{"key":"e_1_3_2_13_2","unstructured":"Moritz Beller. 2020. Why I Will Never Join an Artifacts Evaluation Committee Again. Retrieved September 25 2022 from https:\/\/inventitech.com\/blog\/why-i-will-never-review-artifacts-again\/."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISESE.2004.1334904"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2348997"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2007.50"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/2994031"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00054"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_14"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518948"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2076021.2048117"},{"issue":"4","key":"e_1_3_2_23_2","first-page":"231","article-title":"A web-based support environment for software engineering experiments","volume":"9","author":"Arisholm E.","year":"2002","unstructured":"E. Arisholm, D. I. K. Sj\u00f8berg, G. J. Carelius, and Y. Lindsj\u00f8rn. 2002. A web-based support environment for software engineering experiments. Nord. J. Comput. 9, 4 (2002), 231\u2013247.","journal-title":"Nord. J. Comput."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3452379"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2993288.2993303"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-014-0458-y"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2652524.2652566"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40411-015-0023-0"},{"key":"e_1_3_2_29_2","first-page":"575","volume-title":"Proceedings of the IEEE\/ACM 37th IEEE International Conference on Software Engineering","volume":"2","author":"Neto Francisco Gomes de Oliveira","year":"2015","unstructured":"Francisco Gomes de Oliveira Neto, Richard Torkar, and Patricia D. L. Machado. 2015. An initiative to improve reproducibility and empirical evaluation of software testing techniques. In Proceedings of the IEEE\/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 575\u2013578."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/226295.226310"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE.2015.32"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2017.68"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9523-3"},{"key":"e_1_3_2_34_2","volume-title":"Personal communication","author":"Felderer Michael","year":"2022","unstructured":"Michael Felderer. 2022. Personal communication."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9655-0"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.5555\/1809745"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39259-7_8"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1142\/S0218194014400178"},{"key":"e_1_3_2_39_2","first-page":"504","volume-title":"Proceedings of the International Conference on Software Engineering and Knowledge Engineering (SEKE\u201913)","author":"Freire Mar\u00edlia Aranha","year":"2013","unstructured":"Mar\u00edlia Aranha Freire, Daniel Alencar da Costa, Edmilson Campos Neto, Tain\u00e1 Medeiros, Uir\u00e1 Kulesza, Eduardo Aranha, and S\u00e9rgio Soares. 2013. Automated support for controlled experiments in software engineering: A systematic review (S). In Proceedings of the International Conference on Software Engineering and Knowledge Engineering (SEKE\u201913). 504\u2013509."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1037\/1524-9220.5.1.40"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/THFE.1967.233303"},{"key":"e_1_3_2_42_2","unstructured":"HPCS Development Time Working Group. 2007. Exp. Manager. Retrieved October 5 2022 from https:\/\/www.cs.umd.edu\/basili\/hpcs\/index.php_id=17."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/2915970.2916010"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-71440-0_9"},{"key":"e_1_3_2_45_2","first-page":"319","volume-title":"Handbook of Research Methods in Consumer Psychology","author":"Hauser David","year":"2019","unstructured":"David Hauser, Gabriele Paolacci, and Jesse Chandler. 2019. Common concerns with MTurk as a participant pool: Evidence and solutions. In Handbook of Research Methods in Consumer Psychology. Routledge, 319\u2013337."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09851-6"},{"key":"e_1_3_2_47_2","volume-title":"Personal communication","author":"Hochstein Lorin","year":"2022","unstructured":"Lorin Hochstein. 2022. Personal communication."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2458(08)00605-0"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026586415054"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-92898-2_14"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581397"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-010-9141-9"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9061-0"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9279-3"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2089155.2089166"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445265"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3475716.3484273"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2016.30"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.229"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368115"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS.2008.38"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(00)00024-8"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2004.08.005"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635870"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376791"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbef.2017.12.004"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF02249052"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.06.009"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462176"},{"key":"e_1_3_2_71_2","volume-title":"Essentials of Behavioral Research: Methods and Data Analysis","author":"Rosenthal Robert","year":"2008","unstructured":"Robert Rosenthal and Ralph L. Rosnow. 2008. Essentials of Behavioral Research: Methods and Data Analysis."},{"key":"e_1_3_2_72_2","volume-title":"Qualitative Interviewing: The Art of Hearing Data","author":"Rubin Herbert J.","year":"2011","unstructured":"Herbert J. Rubin and Irene S. Rubin. 2011. Qualitative Interviewing: The Art of Hearing Data. SAGE."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9262-z"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.82"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070508"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2599030"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2017.2720171"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1162\/OPMI_a_00002"},{"key":"e_1_3_2_79_2","unstructured":"William R. Shadish Thomas D. Cook and Donald T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston MA: Houghton Mifflin."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.5555\/857197.857885"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:EMSE.0000013516.80487.33"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.5555\/1324786"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9060-1"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.24"},{"key":"e_1_3_2_85_2","volume-title":"Personal communication","author":"Silveira F\u00e1bio Fagundes","year":"2022","unstructured":"F\u00e1bio Fagundes Silveira. 2022. Personal communication."},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-87013-3_18"},{"key":"e_1_3_2_87_2","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1007\/978-3-540-71301-2_23","volume-title":"Empirical Software Engineering Issues: Critical Assessment and Future Directions","author":"Sj\u00f8berg Dag I. K.","year":"2007","unstructured":"Dag I. K. Sj\u00f8berg. 2007. Knowledge acquisition in software engineering requires sharing of data and artifacts. In Empirical Software Engineering Issues: Critical Assessment and Future Directions. Springer, 77\u201382."},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.5555\/857197.857886"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-45143-3_3"},{"key":"e_1_3_2_90_2","volume-title":"Personal communication","author":"Sj\u00f8berg Dag I. K.","year":"2022","unstructured":"Dag I. K. Sj\u00f8berg and Gunnar R. Bergersen. 2022. Personal communication."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/FOSE.2007.30"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.97"},{"key":"e_1_3_2_93_2","unstructured":"Davide Spadini. 2020. CRExperiment. Retrieved March 11 2022 from https:\/\/github.com\/ishepard\/CRExperiment."},{"key":"e_1_3_2_94_2","first-page":"1171","volume-title":"Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering (ICSE\u201920)","author":"Spadini Davide","year":"2020","unstructured":"Davide Spadini, G\u00fcl \u00c7alikli, and Alberto Bacchelli. 2020. Primers or reminders? The effects of existing review comments on code review. In Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering (ICSE\u201920). IEEE, 1171\u20131182."},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00110"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/1852786.1852832"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501957"},{"key":"e_1_3_2_98_2","unstructured":"JetBrains Team. 2015. MPS: Meta Programming System. https:\/\/www.jetbrains.com\/opensource\/mps\/."},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09973-5"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799942"},{"key":"e_1_3_2_101_2","volume-title":"Personal communication","author":"Travassos Guilherme H.","year":"2022","unstructured":"Guilherme H. Travassos. 2022. Personal communication."},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS.2008.30"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/1159733.1159741"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771797"},{"key":"e_1_3_2_105_2","volume-title":"The Psychology of Computer Programming","author":"Weinberg Gerald M.","year":"1971","unstructured":"Gerald M. Weinberg. 1971. The Psychology of Computer Programming. Vol. 29. Van Nostrand Reinhold New York."},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392375"},{"key":"e_1_3_2_107_2","volume-title":"Backtracking Support in Code Editing","author":"Yoon YoungSeok","year":"2015","unstructured":"YoungSeok Yoon. 2015. Backtracking Support in Code Editing. Ph.D. Dissertation."},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/2089155.2089163"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.5555\/2663638.2663669"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2014.6883030"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.43"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2013.6645254"},{"key":"e_1_3_2_113_2","volume-title":"Personal communication","author":"Zazworka Nico","year":"2022","unstructured":"Nico Zazworka. 2022. Personal communication."},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1011489321999"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587157","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3587157","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:15Z","timestamp":1750178835000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587157"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,24]]},"references-count":113,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,9,30]]}},"alternative-id":["10.1145\/3587157"],"URL":"https:\/\/doi.org\/10.1145\/3587157","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,24]]},"assertion":[{"value":"2022-06-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}