{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T20:49:35Z","timestamp":1776286175868,"version":"3.50.1"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T00:00:00Z","timestamp":1512518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1530837"],"award-info":[{"award-number":["1530837"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2017,12,6]]},"abstract":"<jats:p>Online feedback exchange (OFE) systems are an increasingly popular way to test concepts with millions of target users before going to market. Yet, we know little about how designers make sense of this abundant feedback. This empirical study investigates how expert and novice designers make sense of feedback in OFE systems. We observed that when feedback conflicted with frames originating from the participant's design knowledge, experts were more likely than novices to question the inconsistency, seeking critical information to expand their understanding of the design goals. Our results suggest that in order for OFE systems to be truly effective, they must be able to support nuances in sensemaking activities of novice and expert users.<\/jats:p>","DOI":"10.1145\/3134680","type":"journal-article","created":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T21:23:15Z","timestamp":1512595395000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Novice and Expert Sensemaking of Crowdsourced Design Feedback"],"prefix":"10.1145","volume":"1","author":[{"given":"Eureka","family":"Foong","sequence":"first","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Darren","family":"Gergle","sequence":"additional","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elizabeth M.","family":"Gerber","sequence":"additional","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,12,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"d.}. BetaFamily. ({n. d.}). Retrieved","year":"2017","unstructured":"{n. d.}. BetaFamily. ({n. d.}). Retrieved July 7, 2017 from www.betafamily.com {n. d.}. BetaFamily. ({n. d.}). Retrieved July 7, 2017 from www.betafamily.com"},{"key":"e_1_2_1_2_1","volume-title":"d.}. Dribbble. ({n. d.}). Retrieved","year":"2017","unstructured":"{n. d.}. Dribbble. ({n. d.}). Retrieved July 7, 2017 from www.dribbble.com {n. d.}. Dribbble. ({n. d.}). Retrieved July 7, 2017 from www.dribbble.com"},{"key":"e_1_2_1_3_1","volume-title":"d.}. MURAL. ({n. d.}). Retrieved","year":"2017","unstructured":"{n. d.}. MURAL. ({n. d.}). Retrieved July 7, 2017 from www.mural.ly {n. d.}. MURAL. ({n. d.}). Retrieved July 7, 2017 from www.mural.ly"},{"key":"e_1_2_1_4_1","volume-title":"d.}. UserTesting. ({n. d.}). Retrieved","year":"2017","unstructured":"{n. d.}. UserTesting. ({n. d.}). Retrieved July 7, 2017 from www.usertesting.com {n. d.}. UserTesting. ({n. d.}). Retrieved July 7, 2017 from www.usertesting.com"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0142-694X(98)00031-3"},{"key":"e_1_2_1_6_1","volume-title":"Innovation as a learning process: Embedding design thinking. California management review 50, 1 (Oct","author":"Beckman Sara L","year":"2007","unstructured":"Sara L Beckman and Michael Barry . 2007. Innovation as a learning process: Embedding design thinking. California management review 50, 1 (Oct . 2007 ), 25--56. Sara L Beckman and Michael Barry. 2007. Innovation as a learning process: Embedding design thinking. California management review 50, 1 (Oct. 2007), 25--56."},{"key":"e_1_2_1_7_1","volume-title":"Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4","author":"Cohen Jacob","year":"1968","unstructured":"Jacob Cohen . 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 ( 1968 ), 213--220. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213--220."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.destud.2004.06.002"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1177\/1050651907311923"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025791"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2757226.2757249"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/62266.62273"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858195"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13229"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2006.100"},{"key":"e_1_2_1_18_1","unstructured":"Gary Klein Jennifer K Phillips Erica L Rall and Deborah Peluso. 2007. A data-frame theory of sensemaking. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.  Gary Klein Jennifer K Phillips Erica L Rall and Deborah Peluso. 2007. A data-frame theory of sensemaking. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making."},{"key":"e_1_2_1_19_1","volume-title":"FOCUS: A model of sensemaking. Technical Report.","author":"Klein Gary","year":"2007","unstructured":"Gary Klein , W Seick , D Peluso , and J Smith . 2007 . FOCUS: A model of sensemaking. Technical Report. Gary Klein, W Seick, D Peluso, and J Smith. 2007. FOCUS: A model of sensemaking. Technical Report."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025883"},{"key":"e_1_2_1_21_1","volume-title":"How Designers Think: The Design Process Demysti ed","author":"Lawson Bryan","unstructured":"Bryan Lawson . 2005. How Designers Think: The Design Process Demysti ed . Elsevier , Oxford, England . Bryan Lawson. 2005. How Designers Think: The Design Process Demysti ed. Elsevier, Oxford, England."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675283"},{"key":"e_1_2_1_23_1","volume-title":"Eames Design: The Work of the Office of Charles and Ray Eames. Harry N. Abrams.","author":"Neuhart John","year":"1989","unstructured":"John Neuhart , Charles Eames , Ray Eames , and Marilyn Neuhart . 1989 . Eames Design: The Work of the Office of Charles and Ray Eames. Harry N. Abrams. John Neuhart, Charles Eames, Ray Eames, and Marilyn Neuhart. 1989. Eames Design: The Work of the Office of Charles and Ray Eames. Harry N. Abrams."},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Duyen T Nguyen Thomas R Garncarz Felicia Ng Laura Dabbish and Steven Dow. 2016. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.  Duyen T Nguyen Thomas R Garncarz Felicia Ng Laura Dabbish and Steven Dow. 2016. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.","DOI":"10.1145\/2998181.2998319"},{"key":"e_1_2_1_25_1","volume-title":"The Design of Everyday Things: Revised and Expanded Edition","author":"Norman Don","unstructured":"Don Norman . 2013. The Design of Everyday Things: Revised and Expanded Edition . Basic Books . Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books."},{"key":"e_1_2_1_26_1","volume-title":"Design for the Real World. Thames and Hudson","author":"Papanek Victor","unstructured":"Victor Papanek and R Buckminster Fuller . 1972. Design for the Real World. Thames and Hudson , London . Victor Papanek and R Buckminster Fuller. 1972. Design for the Real World. Thames and Hudson, London."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of international conference on.","author":"Pirolli Peter","year":"2005","unstructured":"Peter Pirolli and Stuart Card . 2005 . The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis . In Proceedings of international conference on. Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169209"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.2307\/1511391"},{"key":"e_1_2_1_30_1","unstructured":"James P Spradley. 1980. Participant Observation. Wadsworth Publishing Company.  James P Spradley. 1980. Participant Observation. Wadsworth Publishing Company."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753578"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1287\/orsc.1050.0133"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145252"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531604"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675140"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901790.2901820"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819953"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134680","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134680","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134680","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:24Z","timestamp":1750212684000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134680"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,12,6]]},"references-count":37,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2017,12,6]]}},"alternative-id":["10.1145\/3134680"],"URL":"https:\/\/doi.org\/10.1145\/3134680","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,12,6]]},"assertion":[{"value":"2017-12-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}