{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T17:03:29Z","timestamp":1791911009977,"version":"4.3.4"},"reference-count":113,"publisher":"Emerald","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,5,8]]},"abstract":"<jats:p>Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like \u201cwhich image is better, A or B?\u201d, leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper\u2019s contributions. On the one hand we argue that, when conducted hastily as an afterthought, such studies lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception. We call for increased attention to both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. Together with this call, we offer an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community.<\/jats:p>","DOI":"10.1561\/0600000106","type":"journal-article","created":{"date-parts":[[2023,5,8]],"date-time":"2023-05-08T08:25:18Z","timestamp":1683534318000},"page":"201-252","source":"Crossref","is-referenced-by-count":6,"title":["Towards Better User Studies in Computer Graphics and Vision"],"prefix":"10.1108","volume":"15","author":[{"given":"Zoya","family":"Bylinskii","sequence":"first","affiliation":[{"name":"Adobe Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron","family":"Hertzmann","sequence":"additional","affiliation":[{"name":"Adobe Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yile","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Herman","sequence":"additional","affiliation":[{"name":"Adobe Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefanie","family":"Hutka","sequence":"additional","affiliation":[{"name":"Adobe Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2023,5,8]]},"reference":[{"issue":"4","key":"2026032615065963000_ref001","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1145\/1924421.1924439","article-title":"Design principles for visual communication","volume":"54","author":"Agrawala","year":"2011","journal-title":"Commun. ACM"},{"key":"2026032615065963000_ref002","volume-title":"Measuring the user experience: collecting, analyzing, and presenting usability metrics","author":"Albert","year":"2013"},{"key":"2026032615065963000_ref003","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1016\/S0065-2601(08)60386-X","article-title":"Situated optimism: Specific outcome expectancies and self-regulation","volume":"30","author":"Armor","year":"1998","journal-title":"Advances in experimental social psychology"},{"issue":"4","key":"2026032615065963000_ref004","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1093\/jcr\/ucx096","article-title":"Asking questions with reflexive focus: A tutorial on designing and conducting interviews","volume":"44","author":"Arsel","year":"2017","journal-title":"Journal of Consumer Research"},{"key":"2026032615065963000_ref005","article-title":"Large scale qualitative evaluation of generative image model outputs","volume-title":"arXiv preprint arXiv:2301.04518","author":"Assogba","year":"2023"},{"key":"2026032615065963000_ref006","volume-title":"Terminologies for reproducible research","author":"Barba","year":"2018"},{"key":"2026032615065963000_ref007","first-page":"1","volume-title":"2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops","author":"Berg","year":"2009"},{"issue":"7611","key":"2026032615065963000_ref008","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1038\/535201a","article-title":"Watch out for cheats in citation game","volume":"535","author":"Biagioli","year":"2016","journal-title":"Nature News"},{"key":"2026032615065963000_ref009","unstructured":"M.\n              Bloom\n            , E.Ellis, N.Era, L.Herman, F.Kazerooni, L.Meckler, and E. C.Siegel, A practical guide to inclusive research, 2021. url: https:\/\/medium.com\/inclusive-research."},{"issue":"4","key":"2026032615065963000_ref010","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1145\/3386569.3392413","article-title":"Code replicability in computer graphics","volume":"39","author":"Bonneel","year":"2020","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"2026032615065963000_ref011","unstructured":"B.\n              Brown\n             and R.Hart, Forrester infographic: The roi of design thinking, 2019. url: https:\/\/www.forrester.com\/report\/Forrester-Infographic-The-ROI-Of-Design-Thinking\/RES157995."},{"key":"2026032615065963000_ref012","unstructured":"R.\n              Budiu\n            \n          , You are not the user: The false-consensus effect, 2017. url: https:\/\/www.nngroup.com\/articles\/false-consensus\/."},{"key":"2026032615065963000_ref013","volume-title":"Amazon\u2019s mechanical turk: A new source of inexpensive, yet high-quality data?","author":"Buhrmester","year":"2016"},{"key":"2026032615065963000_ref014","first-page":"40","article-title":"Pairedcyclegan: Asymmetric style transfer for applying and removing makeup","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Chang","year":"2018"},{"key":"2026032615065963000_ref015","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1145\/2501988.2502008","article-title":"Attribit: Content creation with semantic attributes","volume-title":"Proceedings of the 26th annual ACM symposium on User interface software and technology","author":"Chaudhuri","year":"2013"},{"issue":"4","key":"2026032615065963000_ref016","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964930","article-title":"Probabilistic reasoning for assembly-based 3d modeling","volume":"30","author":"Chaudhuri","year":"2011","journal-title":"ACM Trans. Graph"},{"issue":"4","key":"2026032615065963000_ref017","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392409","article-title":"Human-in-the-loop differential subspace search in high-dimensional latent space","volume":"39","author":"Chiu","year":"2020","journal-title":"ACM Trans. Graph"},{"issue":"4","key":"2026032615065963000_ref018","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1177\/1948550619875149","article-title":"An mturk crisis? shifts in data quality and the impact on study results","volume":"11","author":"Chmielewski","year":"2020","journal-title":"Social Psychological and Personality Science"},{"key":"2026032615065963000_ref019","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1109\/CVPR.2010.5540214","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Cho","year":"2010"},{"key":"2026032615065963000_ref020","volume-title":"Influence: The psychology of persuasion","author":"Cialdini","year":"55"},{"issue":"8","key":"2026032615065963000_ref021","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1145\/3360311","article-title":"Threats of a replication crisis in empirical computer science","volume":"63","author":"Cockburn","year":"2020","journal-title":"Communications of the ACM"},{"key":"2026032615065963000_ref022","doi-asserted-by":"crossref","DOI":"10.4324\/9780203224342","volume-title":"Research methods in education","author":"Cohen","year":"2002"},{"key":"2026032615065963000_ref023","doi-asserted-by":"crossref","DOI":"10.1145\/1531326.1531334","article-title":"How well do line drawings depict shape?","volume":"28","author":"Cole","year":"2009","journal-title":"ACM Transactions on Graphics (Proc. SIGGRAPH)"},{"key":"2026032615065963000_ref024","volume-title":"Research design: Qualitative, quantitative, and mixed methods approaches","author":"Creswell","year":"2017"},{"key":"2026032615065963000_ref025","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.destud.2019.10.006","article-title":"Methodological diversity and theoretical integration: Research in design fixation as an example of fixation in research design?","volume":"65","author":"Crilly","year":"2019","journal-title":"Design Studies"},{"key":"2026032615065963000_ref026","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/w7qy9","volume-title":"The burden for high-quality online data collection lies with researchers, not recruitment platforms","author":"Cuskley","year":"2022"},{"key":"2026032615065963000_ref027","first-page":"429","volume-title":"International Conference on Ubiquitous Computing","author":"Davidoff","year":"2007"},{"key":"2026032615065963000_ref028","volume-title":"The benchmark lottery","author":"Dehghani","year":"2021"},{"key":"2026032615065963000_ref029","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","volume-title":"2009 IEEE conference on computer vision and pattern recognition","author":"Deng","year":"2009"},{"key":"2026032615065963000_ref030","article-title":"Deep generative image models using a laplacian pyramid of adversarial networks","volume":"28","author":"Denton","year":"2015","journal-title":"Advances in neural information processing systems"},{"issue":"4","key":"2026032615065963000_ref031","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/S1057-7408(07)70033-5","article-title":"Self-image motives and consumer behavior: How sacrosanct self-beliefs sway preferences in the marketplace","volume":"17","author":"Dunning","year":"2007","journal-title":"Journal of consumer psychology"},{"key":"2026032615065963000_ref032","volume-title":"Research Methods in Psychology, 6th Edition","author":"Elmes","year":"1999"},{"issue":"9","key":"2026032615065963000_ref033","doi-asserted-by":"crossref","first-page":"101515","DOI":"10.1016\/j.isci.2020.101515","article-title":"Who gets credit for ai-generated art?","volume":"23","author":"Epstein","year":"2020","journal-title":"Iscience"},{"issue":"10","key":"2026032615065963000_ref034","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1177\/030802269806101005","article-title":"Reflexivity: An essential component for all research?","volume":"61","author":"Finlay","year":"1998","journal-title":"British Journal of Occupational Therapy"},{"key":"2026032615065963000_ref035","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1145\/3379337.3415825","article-title":"Predicting visual importance across graphic design types","volume-title":"Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology","author":"Fosco","year":"2020"},{"key":"2026032615065963000_ref036","doi-asserted-by":"publisher","DOI":"10.1177\/0276237421994697","article-title":"The role of ai attribution knowledge in the evaluation of artwork","volume-title":"Empirical Studies of the Arts","author":"Gangadharbatla","year":"2021"},{"key":"2026032615065963000_ref037","volume-title":"Observing the user experience: A practitioner\u2019s guide to user research","author":"Goodman","year":"2012"},{"key":"2026032615065963000_ref038","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1145\/1357054.1357074","article-title":"Usability evaluation considered harmful (some of the time)","volume-title":"Proceedings of the SIGCHI conference on Human factors in computing systems","author":"Greenberg","year":"2008"},{"key":"2026032615065963000_ref039","volume-title":"The weak science of humancomputer interaction","author":"Greenberg","year":"1991"},{"key":"2026032615065963000_ref040","volume-title":"Is replication important for hci?","author":"Greiffenhagen","year":"2014"},{"issue":"7829","key":"2026032615065963000_ref041","doi-asserted-by":"crossref","first-page":"E14","DOI":"10.1038\/s41586-020-2766-y","article-title":"Transparency and reproducibility in artificial intelligence","volume":"586","author":"Haibe-Kains","year":"2020","journal-title":"Nature"},{"key":"2026032615065963000_ref042","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1145\/3379337.3415846","volume-title":"Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, ser. UIST \u201920","author":"Hartmann","year":"2020"},{"key":"2026032615065963000_ref043","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1145\/1753326.1753357","article-title":"Crowdsourcing graphical perception: Using mechanical turk to assess visualization design","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems","author":"Heer","year":"2010"},{"key":"2026032615065963000_ref044","doi-asserted-by":"publisher","DOI":"10.1177\/14614448221089604","article-title":"In the eye of the beholder: A viewer-defined conception of online visual creativity","volume-title":"New Media & Society","author":"Herman","year":"2022"},{"key":"2026032615065963000_ref045","doi-asserted-by":"crossref","DOI":"10.1145\/1809939.1809957","article-title":"Non-Photorealistic Rendering and the Science of Art","volume-title":"Proc. NPAR","author":"Hertzmann","year":"2010"},{"key":"2026032615065963000_ref046","doi-asserted-by":"crossref","first-page":"3523","DOI":"10.1145\/2556288.2557004","article-title":"Is once enough? on the extent and content of replications in human-computer interaction","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Hornb\u00e6k","year":"2014"},{"key":"2026032615065963000_ref047","volume-title":"einops: Clear and reliable tensor manipulations with einstein-like notation","author":"ICLR, Reviewer discussion for iclr 2022 submission","year":"2022"},{"key":"2026032615065963000_ref048","article-title":"From moral principles to ethics codes","volume-title":"Research Methods in Psychology","author":"Jhangiani","year":"2020"},{"key":"2026032615065963000_ref049","doi-asserted-by":"crossref","DOI":"10.1126\/science.328.5986.1627","volume-title":"A weird view of human nature skews psychologists\u2019 studies","author":"Jones","year":"2010"},{"issue":"4","key":"2026032615065963000_ref050","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3450626.3459821","article-title":"Shapemod: Macro operation discovery for 3d shape programs","volume":"40","author":"Jones","year":"2021","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"2026032615065963000_ref051","volume-title":"Countering racial bias in computer graphics research","author":"Kim","year":"2021"},{"key":"2026032615065963000_ref052","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1145\/1357054.1357127","article-title":"Crowdsourcing user studies with mechanical turk","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems","author":"Kittur","year":"2008"},{"issue":"1","key":"2026032615065963000_ref053","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1007\/s10618-008-0114-1","article-title":"Controlled experiments on the web: Survey and practical guide","volume":"18","author":"Kohavi","year":"2009","journal-title":"Data mining and knowledge discovery"},{"issue":"3","key":"2026032615065963000_ref054","doi-asserted-by":"crossref","DOI":"10.1561\/0600000071","article-title":"Crowdsourcing in computer vision","volume":"10","author":"Kovashka","year":"2016","journal-title":"Foundations and Trends in Computer Graphics and Vision"},{"key":"2026032615065963000_ref055","volume-title":"Observing the user experience: a practitioner\u2019s guide to user research","author":"Kuniavsky","year":"2003"},{"key":"2026032615065963000_ref056","doi-asserted-by":"crossref","first-page":"3336","DOI":"10.1109\/CVPR.2010.5540027","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Li","year":"2010"},{"key":"2026032615065963000_ref057","first-page":"453","article-title":"A closedform solution to photorealistic image stylization","volume-title":"Proceedings of the European conference on computer vision (ECCV)","author":"Li","year":"2018"},{"issue":"4","key":"2026032615065963000_ref058","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2461912.2461998","article-title":"Realbrush: Painting with examples of physical media","volume":"32","author":"Lu","year":"2013","journal-title":"ACM Transactions on Graphics (TOG)"},{"issue":"4","key":"2026032615065963000_ref059","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2185520.2185542","article-title":"Helpinghand: Example-based stroke stylization","volume":"31","author":"Lu","year":"2012","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"2026032615065963000_ref060","volume-title":"The metric society: On the quantification of the social","author":"Mau","year":"2019"},{"issue":"4","key":"2026032615065963000_ref061","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185587","article-title":"Render me real? investigating the effect of render style on the perception of animated virtual humans","volume":"31","author":"McDonnell","year":"2012","journal-title":"ACM Trans. Graph"},{"issue":"3","key":"2026032615065963000_ref062","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1360612.1360625","article-title":"Clone attack! perception of crowd variety","volume":"27","author":"McDonnell","year":"2008"},{"issue":"4","key":"2026032615065963000_ref063","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2010324.1964982","article-title":"Interactive furniture layout using interior design guidelines","volume":"30","author":"Merrell","year":"2011","journal-title":"ACM transactions on graphics (TOG)"},{"key":"2026032615065963000_ref064","volume-title":"Reproducibility and replicability in science","author":"National Academies of Sciences Engineering and Medicine","year":"2019"},{"issue":"2","key":"2026032615065963000_ref065","doi-asserted-by":"crossref","first-page":"151","DOI":"10.3200\/GENP.135.2.151-166","article-title":"The good-subject effect: Investigating participant demand characteristics","volume":"135","author":"Nichols","year":"2008","journal-title":"The Journal of general psychology"},{"key":"2026032615065963000_ref066","unstructured":"J.\n              Nielsen\n            \n          , Field studies done right: Fast and observational, 2002. url: https:\/\/www.nngroup.com\/articles\/field-studies-done-right-fast-and-observational\/."},{"key":"2026032615065963000_ref067","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1145\/259963.260531","article-title":"Usability inspection methods","volume-title":"Conference companion on Human factors in computing systems","author":"Nielsen","year":"1994"},{"key":"2026032615065963000_ref068","first-page":"249","article-title":"Heuristic evaluation of user interfaces","volume-title":"Proceedings of the SIGCHI conference on Human factors in computing systems","author":"Nielsen","year":"1990"},{"key":"2026032615065963000_ref069","doi-asserted-by":"crossref","DOI":"10.18574\/nyu\/9781479833641.001.0001","volume-title":"Algorithms of Oppression: How Search Engines Reinforce Racism","author":"Noble","year":"2018"},{"key":"2026032615065963000_ref070","doi-asserted-by":"crossref","DOI":"10.3726\/978-1-4539-1717-6","volume-title":"The intersectional internet: Race, sex, class, and culture online","author":"Noble","year":"2016"},{"key":"2026032615065963000_ref071","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1964921.1964958","article-title":"Color compatibility from large datasets","volume-title":"ACM SIGGRAPH 2011 papers","author":"O\u2019Donovan","year":"2011"},{"issue":"3","key":"2026032615065963000_ref072","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1145\/501786.501788","article-title":"Collisions and perception","volume":"20","author":"O\u2019Sullivan","year":"2001","journal-title":"ACM Trans. Graph"},{"key":"2026032615065963000_ref073","doi-asserted-by":"publisher","DOI":"10.1145\/882262.882303","article-title":"Evaluating the visual fidelity of physically based animations","volume":"22","author":"O\u2019Sullivan","year":"2003","journal-title":"ACM Transactions on Graphics"},{"issue":"6251","key":"2026032615065963000_ref074","doi-asserted-by":"crossref","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"key":"2026032615065963000_ref075","doi-asserted-by":"crossref","first-page":"2328","DOI":"10.1109\/CVPR.2010.5539920","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Parikh","year":"2010"},{"issue":"2","key":"2026032615065963000_ref076","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1111\/j.1948-7169.1999.tb00250.x","article-title":"Needfinding: The why and how of uncovering people\u2019s needs","volume":"10","author":"Patnaik","year":"1999","journal-title":"Design Management Journal (Former Series)"},{"issue":"6060","key":"2026032615065963000_ref077","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1126\/science.1213847","article-title":"Reproducible research in computational science","volume":"334","author":"Peng","year":"2011","journal-title":"Science"},{"issue":"2","key":"2026032615065963000_ref078","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3503460","article-title":"A primitive for manual hatching","volume":"41","author":"Philbrick","year":"2022","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"2026032615065963000_ref079","article-title":"Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program)","volume":"22","author":"Pineau","year":"2021","journal-title":"Journal of Machine Learning Research"},{"key":"2026032615065963000_ref080","doi-asserted-by":"publisher","first-page":"76","DOI":"10.3389\/fninf.2017.00076","article-title":"Reproducibility vs. replicability: A brief history of a confused terminology","volume":"11","author":"Plesser","year":"2018","journal-title":"Frontiers in Neuroinformatics"},{"key":"2026032615065963000_ref081","volume-title":"Interviewing users: how to uncover compelling insights","author":"Portigal","year":"2013"},{"key":"2026032615065963000_ref082","volume-title":"Designing and conducting survey research: A comprehensive guide","author":"Rea","year":"2014"},{"key":"2026032615065963000_ref083","first-page":"1364","article-title":"Labinthewild: Conducting largescale online experiments with uncompensated samples","volume-title":"Proceedings of the 18th ACM conference on computer supported cooperative work & social computing","author":"Reinecke","year":"2015"},{"key":"2026032615065963000_ref084","first-page":"387","article-title":"Usability evaluation with the cognitive walkthrough","volume-title":"Conference companion on Human factors in computing systems","author":"Rieman","year":"1995"},{"key":"2026032615065963000_ref085","first-page":"6182","article-title":"Fast and flexible indoor scene synthesis via deep convolutional generative models","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Ritchie","year":"2019"},{"key":"2026032615065963000_ref086","unstructured":"C.\n              Rohrer\n            \n          , When to use which: User-experience research methods, 2014. url: https:\/\/www.nngroup.com\/articles\/which-ux-research-methods\/."},{"key":"2026032615065963000_ref087","unstructured":"M.\n              Rosala\n            \n          , Rating scales in ux research: Likert or semantic differential?2020. url: https:\/\/www.nngroup.com\/articles\/rating-scales\/."},{"issue":"3","key":"2026032615065963000_ref088","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1111\/j.1540-5885.2006.00195.x","article-title":"Ethnographies in the front end: Designing for enhanced customer experiences","volume":"23","author":"Rosenthal","year":"2006","journal-title":"Journal of Product Innovation Management"},{"issue":"3","key":"2026032615065963000_ref089","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/0022-1031(77)90049-X","article-title":"The \u201cfalse consensus effect\u201d: An egocentric bias in social perception and attribution processes","volume":"13","author":"Ross","year":"1977","journal-title":"Journal of experimental social psychology"},{"issue":"2","key":"2026032615065963000_ref090","first-page":"348","article-title":"How to plan, design, and conduct effective tests","volume":"17","author":"Rubin","year":"2008","journal-title":"Handbook of usability testing"},{"issue":"1-3","key":"2026032615065963000_ref091","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1007\/s11263-007-0090-8","article-title":"Labelme: A database and web-based tool for image annotation","volume":"77","author":"Russell","year":"2008","journal-title":"International journal of computer vision"},{"key":"2026032615065963000_ref092","unstructured":"J.\n              Sauro\n            \n          , How many people cheat in online surveys?2010. url: https:\/\/measuringu.com\/cheat-survey\/."},{"key":"2026032615065963000_ref093","volume-title":"Quantifying the user experience: Practical statistics for user research","author":"Sauro","year":"2016"},{"key":"2026032615065963000_ref094","volume-title":"Large-scale needfinding methods, quality metrics, and need prioritization in user-centered design","author":"Schaffhausen","year":"2015"},{"key":"2026032615065963000_ref095","first-page":"10","article-title":"\u201cthis is a problem, don\u2019t you agree?\u201d framing and bias in human evaluation for natural language generation","volume-title":"Proceedings of the 1st Workshop on Evaluating NLG Evaluation","author":"Schoch","year":"2020"},{"issue":"4","key":"2026032615065963000_ref096","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459813","article-title":"The effect of shape and illumination on material perception: Model and applications","volume":"40","author":"Serrano","year":"2021","journal-title":"ACM Trans. Graph"},{"key":"2026032615065963000_ref097","volume-title":"Validating product ideas: Through lean user research","author":"Sharon","year":"2016"},{"issue":"2","key":"2026032615065963000_ref098","doi-asserted-by":"publisher","DOI":"10.1145\/2699649","article-title":"Augmented airbrush for computer aided painting (cap)","volume":"34","author":"Shilkrot","year":"2015","journal-title":"ACM Trans. Graph"},{"issue":"4","key":"2026032615065963000_ref099","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3072959.3073690","article-title":"Playful palette: An interactive parametric color mixer for artists","volume":"36","author":"Shugrina","year":"2017","journal-title":"ACM Transactions on Graphics (TOG)"},{"issue":"11","key":"2026032615065963000_ref100","doi-asserted-by":"crossref","first-page":"1359","DOI":"10.1177\/0956797611417632","article-title":"False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant","volume":"22","author":"Simmons","year":"2011","journal-title":"Psychological science"},{"key":"2026032615065963000_ref101","first-page":"1","volume-title":"2008 IEEE computer society conference on computer vision and pattern recognition workshops","author":"Sorokin","year":"2008"},{"key":"2026032615065963000_ref102","first-page":"523","volume-title":"European conference on computer vision","author":"Spain","year":"2008"},{"key":"2026032615065963000_ref103","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-15524-1","volume-title":"Why greatness cannot be planned: The myth of the objective","author":"Stanley","year":"2015"},{"issue":"5","key":"2026032615065963000_ref104","doi-asserted-by":"crossref","DOI":"10.1145\/1618452.1618513","article-title":"Exploratory modeling with collaborative design spaces","volume":"28","author":"Talton","year":"2009","journal-title":"ACM Trans. on Graphics"},{"issue":"5","key":"2026032615065963000_ref105","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1618452.1618513","article-title":"Exploratory modeling with collaborative design spaces","volume":"28","author":"Talton","year":"2009","journal-title":"ACM Trans. Graph"},{"key":"2026032615065963000_ref106","doi-asserted-by":"publisher","DOI":"10.1145\/1275808.1276473","volume-title":"The influence of shape on the perception of material reflectance","author":"Vangorp","year":"2007"},{"key":"2026032615065963000_ref107","volume-title":"Multi-level active prediction of useful image annotations for recognition","author":"Vijayanarasimhan","year":"2008"},{"issue":"7","key":"2026032615065963000_ref108","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1287\/mnsc.32.7.791","article-title":"Lead users: A source of novel product concepts","volume":"32","author":"Von Hippel","year":"1986","journal-title":"Management science"},{"issue":"4","key":"2026032615065963000_ref109","first-page":"1","article-title":"Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks","volume":"38","author":"Wang","year":"2019","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"2026032615065963000_ref110","doi-asserted-by":"crossref","DOI":"10.1037\/aca0000579","article-title":"A computational approach to studying aesthetic judgments of ambiguous artworks","volume-title":"Psychology of Aesthetics, Creativity, and the Arts, To appear","author":"Wang"},{"key":"2026032615065963000_ref111","first-page":"33","article-title":"Replichi: The workshop ii","volume-title":"CHI\u201914 Extended Abstracts on Human Factors in Computing Systems","author":"Wilson","year":"2014"},{"key":"2026032615065963000_ref112","first-page":"1","article-title":"Iconate: Automatic compound icon generation and ideation","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems","author":"Zhao","year":"2020"},{"key":"2026032615065963000_ref113","article-title":"Hype: A benchmark for human eye perceptual evaluation of generative models","volume":"32","author":"Zhou","year":"2019","journal-title":"Advances in neural information processing systems"}],"container-title":["Foundations and Trends\u00ae in Computer Graphics and Vision"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/ftcgv\/article-pdf\/15\/3\/201\/10864582\/0600000106en.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/ftcgv\/article-pdf\/15\/3\/201\/10864582\/0600000106en.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T14:09:10Z","timestamp":1777471750000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/ftcgv\/article\/15\/3\/201\/1319328\/Towards-Better-User-Studies-in-Computer-Graphics"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,8]]},"references-count":113,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,5,8]]}},"URL":"https:\/\/doi.org\/10.1561\/0600000106","relation":{},"ISSN":["1572-2740","1572-2759"],"issn-type":[{"value":"1572-2740","type":"print"},{"value":"1572-2759","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,8]]}}}