{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T07:41:30Z","timestamp":1767339690107,"version":"build-2065373602"},"reference-count":34,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2020,12,4]],"date-time":"2020-12-04T00:00:00Z","timestamp":1607040000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1948254","BCS-1928502"],"award-info":[{"award-number":["IIS-1948254","BCS-1928502"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Gesture elicitation studies are a popular means of gaining valuable insights into how users interact with novel input devices. One of the problems elicitation faces is that of legacy bias, when elicited interactions are biased by prior technologies use. In response, methodologies have been introduced to reduce legacy bias. This is the first study that formally examines the production method of reducing legacy bias (i.e., repeated proposals for a single referent). This is done through a between-subject study that had 27 participants per group (control and production) with 17 referents placed in a virtual environment using a head-mounted display. This study found that over a range of referents, legacy bias was not significantly reduced over production trials. Instead, production reduced participant consensus on proposals. However, in the set of referents that elicited the most legacy biased proposals, production was an effective means of reducing legacy bias, with an overall reduction of 11.93% for the chance of eliciting a legacy bias proposal.<\/jats:p>","DOI":"10.3390\/mti4040088","type":"journal-article","created":{"date-parts":[[2020,12,4]],"date-time":"2020-12-04T11:59:00Z","timestamp":1607083140000},"page":"88","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["The Cost of Production in Elicitation Studies and the Legacy Bias-Consensus Trade off"],"prefix":"10.3390","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7358-8333","authenticated-orcid":false,"given":"Adam S.","family":"Williams","sequence":"first","affiliation":[{"name":"Department of Computer Science, Colorado State University, Fort Collins, CO 80523, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jason","family":"Garcia","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Colorado State University, Fort Collins, CO 80523, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fernando","family":"De Zayas","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Florida International University, Miami, FL 33199, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fidel","family":"Hernandez","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Florida International University, Miami, FL 33199, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9428-6518","authenticated-orcid":false,"given":"Julia","family":"Sharp","sequence":"additional","affiliation":[{"name":"Department of Statistics, Colorado State University, Fort Collins, CO 80523, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2449-3802","authenticated-orcid":false,"given":"Francisco R.","family":"Ortega","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Colorado State University, Fort Collins, CO 80523, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,12,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Villarreal-Narvaez, S., Vanderdonckt, J., Vatavu, R.D., and Wobbrock, J.A. (2020, January 6\u201310). A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies. Proceedings of the ACM International Conference on Designing Interactive Systems (DIS\u201920), Eindhoven, The Netherlands.","DOI":"10.1145\/3357236.3395511"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Wobbrock, J.O., Aung, H.H., Rothrock, B., and Myers, B.A. (2005, January 2). Maximizing the guessability of symbolic input. Proceedings of the Extended Abstracts on Human Factors in Computing Systems, Portland, OR, USA.","DOI":"10.1145\/1056808.1057043"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wobbrock, J.O., Morris, M.R., and Wilson, A.D. (2009, January 4). User-defined Gestures for Surface Computing. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/1518701.1518866"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Vatavu, R.D., and Wobbrock, J.O. (2015, January 18). Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.","DOI":"10.1145\/2702123.2702223"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Vatavu, R.D., and Wobbrock, J.O. (2016, January 7). Between-Subjects Elicitation Studies: Formalization and Tool Support. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2858036.2858228"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Vatavu, R.D. (2019, January 2). The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow Scotland, UK.","DOI":"10.1145\/3290605.3300454"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1145\/3182168","article-title":"Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation","volume":"25","author":"Tsandilas","year":"2018","journal-title":"ACM Trans. Comput. Hum. Interact."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ortega, F.R., Galvan, A., Tarre, K., Barreto, A., Rishe, N., Bernal, J., Balcazar, R., and Thomas, J. (2017, January 18\u201319). Gesture elicitation for 3D travel via multi-touch and mid-Air systems for procedurally generated pseudo-universe. Proceedings of the 2017 IEEE Symposium on 3D User Interfaces (3DUI), Los Angeles, CA, USA.","DOI":"10.1109\/3DUI.2017.7893331"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ortega, F.R., Tarre, K., Kress, M., Williams, A.S., Barreto, A.B., and Rishe, N.D. (2019, January 23\u201327). Selection and Manipulation Whole-Body Gesture Elicitation Study In Virtual Reality. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.","DOI":"10.1109\/VR.2019.8798105"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3479","DOI":"10.1109\/TVCG.2020.3023566","article-title":"Understanding Multimodal User Gesture and Speech Behavior for Object Manipulation in Augmented Reality Using Elicitation","volume":"26","author":"Williams","year":"2020","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3427330","article-title":"Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation","volume":"4","author":"Williams","year":"2020","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Morris, M.R. (2012, January 11). Web on the Wall: Insights from a Multimodal Interaction Elicitation Study. Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, Cambridge, MA, USA.","DOI":"10.1145\/2396636.2396651"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"102847","DOI":"10.1016\/j.autcon.2019.102847","article-title":"Gesture and speech elicitation for 3D CAD modeling in conceptual design","volume":"106","author":"Khan","year":"2019","journal-title":"Autom. Constr."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1145\/2591689","article-title":"Reducing Legacy Bias in Gesture Elicitation Studies","volume":"21","author":"Morris","year":"2014","journal-title":"Interactions"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1145\/2803169","article-title":"Benefiting from Legacy Bias","volume":"22","author":"Bubalo","year":"2015","journal-title":"Interactions"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ruiz, J., and Vogel, D. (2015, January 18). Soft-Constraints to Reduce Legacy and Performance Bias to Elicit Whole-Body Gestures with Low Arm Fatigue. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.","DOI":"10.1145\/2702123.2702583"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Hoff, L., Hornecker, E., and Bertel, S. (2016, January 14). Modifying Gesture Elicitation: Do Kinaesthetic Priming and Increased Production Reduce Legacy Bias?. Proceedings of the TEI \u201916: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, Eindhoven, The Netherlands.","DOI":"10.1145\/2839462.2839472"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Camurri, A., and Volpe, G. (2004). A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. Gesture-Based Communication in Human-Computer Interaction, Springer.","DOI":"10.1007\/b95740"},{"key":"ref_19","unstructured":"Piumsomboon, T., Clark, A., Billinghurst, M., and Cockburn, A. (May, January 27). User-Defined Gestures for Augmented Reality. Proceedings of the CHI\u201913 Extended Abstracts on Human Factors in Computing Systems, Paris, France."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nacenta, M.A., Kamber, Y., Qiang, Y., and Kristensson, P.O. (2013, January 27). Memorability of pre-designed and user-defined gesture sets. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.","DOI":"10.1145\/2470654.2466142"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1007\/s00779-015-0863-y","article-title":"On free-hand TV control: Experimental results on user-elicited gestures with Leap Motion","volume":"19","author":"Pentiuc","year":"2015","journal-title":"Pers. Ubiquit. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Gheran, B.F., Vanderdonckt, J., and Vatavu, R.D. (2018, January 8). Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. Proceedings of the 2018 Designing Interactive Systems Conference, Hong Kong, China.","DOI":"10.1145\/3196709.3196741"},{"key":"ref_23","unstructured":"Huang, Y.J., Fujiwara, T., Lin, Y.X., Lin, W.C., and Ma, K.L. (2017, January 18\u201321). A gesture system for graph visualization in virtual reality environments. Proceedings of the 2017 IEEE Pacific Visualization Symposium (PacificVis), Seoul, Korea."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., and Billinghurst, M. (2014, January 10\u201312). Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality. Proceedings of the 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany.","DOI":"10.1109\/ISMAR.2014.6948411"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"992","DOI":"10.1177\/0018720818780544","article-title":"Developing a 3d gestural interface for anesthesia-related human-computer interaction tasks using both experts and novices","volume":"60","author":"Jurewicz","year":"2018","journal-title":"Hum. Factors"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wittorf, M.L., and Jakobsen, M.R. (2016, January 23). Eliciting Mid-Air Gestures for Wall-Display Interaction. Proceedings of the 9th Nordic Conference on Human-Computer Interaction, Gothenburg, Sweden.","DOI":"10.1145\/2971485.2971503"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Nebeling, M., Huber, A., Ott, D., and Norrie, M.C. (2014, January 6). Web on the Wall Reloaded: Implementation, Replication and Refinement of User-Defined Interaction Sets. Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces, Dresden, Germany.","DOI":"10.1145\/2669485.2669497"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chan, E., Seyed, T., Stuerzlinger, W., Yang, X.D., and Maurer, F. (2016, January 7). User Elicitation on Single-Hand Microgestures. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2858036.2858589"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cafaro, F., Lyons, L., Tarre, K., and Antle, A. (2018, January 21\u201326). Framed Guessability: Improving the Discoverability of Gestures and Body Movements for Full-Body Interaction. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174167"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Vogiatzidakis, P., and Koutsabasis, P. (2018). Gesture Elicitation Studies for Mid-Air Interaction: A Review. Multimodal Technol. Interact., 2.","DOI":"10.3390\/mti2040065"},{"key":"ref_31","unstructured":"Lee, L., Javed, Y., Danilowicz, S., and Maher, M.L. (2014, January 10). Information at the Wave of Your Hand. Proceedings of the HCI Korea, Seoul, Korea."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Koutsabasis, P., and Domouzis, C.K. (2016, January 7). Mid-Air Browsing and Selection in Image Collections. Proceedings of the International Working Conference on Advanced Visual Interfaces, Bari, Italy.","DOI":"10.1145\/2909132.2909248"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1145\/3412499","article-title":"Evolutionary Gestures: When a Gesture is Not Quite Legacy Biased","volume":"27","author":"Williams","year":"2020","journal-title":"Interactions"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Marriott, K., Chen, J., Hlawatsch, M., Itoh, T., Nacenta, M.A., Reina, G., and Stuerzlinger, W. (2018). Immersive analytics: Time to reconsider the value of 3d for information visualisation. Immersive Analytics, Springer.","DOI":"10.1007\/978-3-030-01388-2_2"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/4\/4\/88\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:41:26Z","timestamp":1760179286000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/4\/4\/88"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,4]]},"references-count":34,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["mti4040088"],"URL":"https:\/\/doi.org\/10.3390\/mti4040088","relation":{},"ISSN":["2414-4088"],"issn-type":[{"type":"electronic","value":"2414-4088"}],"subject":[],"published":{"date-parts":[[2020,12,4]]}}}