{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,27]],"date-time":"2025-09-27T08:17:16Z","timestamp":1758961036667},"reference-count":53,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2013,9,23]],"date-time":"2013-09-23T00:00:00Z","timestamp":1379894400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/2.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Hum. Cent. Comput. Inf. Sci."],"published-print":{"date-parts":[[2013,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Social design feedback is a novel approach to usability evaluation where user participants are asked to comment on designs asynchronously in online ad-hoc groups. Two key features of this approach are that (1) it supports interaction between user participants and development team representatives and (2) user participants can see and respond to other participants\u2019 comments. Two design cases, involving 250 user participants, were studied to explore the output of social design feedback and investigate the effect of the two key features of this approach. Of all the design feedback, 17% was rated highly useful, and 21% contained change suggestions. The presence of an active moderator, representing the development team and interacting with the user participants, increased the usefulness of the design feedback. The opportunity to see and respond to others\u2019 design feedback had a minor effect on the kind of design feedback provided, but no effect on usefulness. Based on the findings, we offer advice on how to implement social design feedback and suggest future research.<\/jats:p>","DOI":"10.1186\/2192-1962-3-18","type":"journal-article","created":{"date-parts":[[2013,9,28]],"date-time":"2013-09-28T09:31:48Z","timestamp":1380360708000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Social design feedback: evaluations with users in online ad-hoc groups"],"prefix":"10.1186","volume":"3","author":[{"given":"Asbj\u00f8rn","family":"F\u00f8lstad","sequence":"first","affiliation":[]},{"given":"Kasper","family":"Hornb\u00e6k","sequence":"additional","affiliation":[]},{"given":"P\u00e5l","family":"Ulleberg","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2013,9,23]]},"reference":[{"key":"40_CR1","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1006\/ijhc.2001.0503","volume":"55","author":"M Maguire","year":"2001","unstructured":"Maguire M: Methods to support human-centred design. Int J Hum Comput Stud 2001, 55: 587\u2013634. 10.1006\/ijhc.2001.0503","journal-title":"Int J Hum Comput Stud"},{"key":"40_CR2","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1145\/182966.182969","volume":"1","author":"M Hammontree","year":"1994","unstructured":"Hammontree M, Weiler P, Nayak N: Remote usability testing. Interactions 1994, 1: 21\u201325. 10.1145\/182966.182969","journal-title":"Interactions"},{"key":"40_CR3","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1300\/J111v45n03_14","volume":"45","author":"BC Thomset-Scott","year":"2006","unstructured":"Thomset-Scott BC: Web site usability with remote users. J Libr Adm 2006, 45: 517\u2013547. 10.1300\/J111v45n03_14","journal-title":"J Libr Adm"},{"key":"40_CR4","first-page":"1151","volume-title":"The human-computer interaction handbook","author":"AA Ozok","year":"2008","unstructured":"Ozok AA: Survey design and implementation in HCI. In The human-computer interaction handbook. 2nd edition. Edited by: Sears A, Jacko JA. New York: Lawrence Erlbaum Associates; 2008:1151\u20131170.","edition":"2"},{"key":"40_CR5","doi-asserted-by":"crossref","first-page":"2117","DOI":"10.1145\/2207676.2208364","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203212","author":"A Bruun","year":"2012","unstructured":"Bruun A, Stage J: The effect of task assignments and instruction types on remote asynchronous usability testing. In Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203212. Edited by: Konstan JA, Chi EH, H\u00f6\u00f6k K. New York: ACM Press; 2012:2117\u20132126."},{"key":"40_CR6","first-page":"3","volume":"18","author":"H Holmstr\u00f6m","year":"2006","unstructured":"Holmstr\u00f6m H, Henfridsson O: Improving package software through online community knowledge. Scand J Inf Syst 2006, 18: 3\u201336.","journal-title":"Scand J Inf Syst"},{"key":"40_CR7","first-page":"31","volume-title":"Proceedings of the Tenth Anniversary Conference on Participatory Design 2008","author":"J Hess","year":"2008","unstructured":"Hess J, Offenberg S, Pipek V: Community driven development as participation? Involving user communities in a software design process. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008. Edited by: Simonsen J, Robertson T. New York: ACM Press; 2008:31\u201340."},{"key":"40_CR8","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.bushor.2009.09.003","volume":"53","author":"AM Kaplan","year":"2010","unstructured":"Kaplan AM, Haenlein M: Users of the world, unite! The challenges and opportunities of social media. Bus Horiz 2010, 53: 59\u201368. 10.1016\/j.bushor.2009.09.003","journal-title":"Bus Horiz"},{"key":"40_CR9","unstructured":"Notable http:\/\/notableapp.com"},{"key":"40_CR10","unstructured":"ConseptShare http:\/\/conceptshare.com"},{"key":"40_CR11","first-page":"719","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203209","author":"ELC Law","year":"2009","unstructured":"Law ELC, Roto V, Hassenzahl M, Vermeeren APOS, Kort J: Understanding, scoping and defining user experience: a survey approach. In Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203209. Edited by: Olsen DRJr, Arthur RB, Hinckley K, Morris MR, Hudson S, Greenberg S. New York: ACM Press; 2009:719\u2013728."},{"key":"40_CR12","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1080\/10447310709336963","volume":"22","author":"A F\u00f8lstad","year":"2007","unstructured":"F\u00f8lstad A: Work-domain experts as evaluators: usability inspection of domain-specific work-support systems. Int J Hum Comput Interact 2007, 22: 217\u2013245. 10.1080\/10447310709336963","journal-title":"Int J Hum Comput Interact"},{"key":"40_CR13","doi-asserted-by":"publisher","first-page":"2019","DOI":"10.1016\/j.jss.2010.02.026","volume":"83","author":"A F\u00f8lstad","year":"2010","unstructured":"F\u00f8lstad A, Hornb\u00e6k K: Work-domain knowledge in usability evaluation: experiences with cooperative usability testing. J Syst Softw 2010, 83: 2019\u20132030. 10.1016\/j.jss.2010.02.026","journal-title":"J Syst Softw"},{"key":"40_CR14","first-page":"1356","volume-title":"Proceedings of the Human Factors and Ergonomics Society 55th annual meeting, HFES 2011","author":"JA Cowley","year":"2011","unstructured":"Cowley JA, Radford-Davenport J: Qualitative data differences between a focus group and online forum hosting a usability design review: a case study. In Proceedings of the Human Factors and Ergonomics Society 55th annual meeting, HFES 2011. Santa Monica: Human Factors and Ergonomics Society; 2011:1356\u20131360."},{"key":"40_CR15","first-page":"471","volume-title":"Proceedings of CHI \u203202","author":"K Vredenburg","year":"2002","unstructured":"Vredenburg K, Mao J, Smith PW, Carey T: A survey of user-centered design practice. In Proceedings of CHI \u203202. Edited by: Wixon D. New York: ACM Press; 2002:471\u2013478."},{"key":"40_CR16","first-page":"200","volume-title":"People and computers XIX - the bigger picture: Proceedings of HCI 2005","author":"I Bark","year":"2005","unstructured":"Bark I, F\u00f8lstad A, Gulliksen J: Use and usefulness of HCI methods: results from an exploratory survey among Nordic HCI practitioners. In People and computers XIX - the bigger picture: Proceedings of HCI 2005. Edited by: McEwan T, Gulliksen J, Benyon D. London: Springer-Verlag; 2005:200\u2013217."},{"key":"40_CR17","volume-title":"Handbook of usability testing: how to plan, design and conduct effective tests","author":"J Rubin","year":"2008","unstructured":"Rubin J, Chisnell D: Handbook of usability testing: how to plan, design and conduct effective tests. 2nd edition. Indianapolis: Wiley Publishing; 2008.","edition":"2"},{"key":"40_CR18","volume-title":"Proceedings of the 5th ERCIM workshop on user interfaces for all","author":"M Hertzum","year":"1999","unstructured":"Hertzum M: User testing in industry: a case study of laboratory, workshop, and field tests. In Proceedings of the 5th ERCIM workshop on user interfaces for all. Edited by: Kobsa A, Stephanidis C. Germany: Dagstuhl; 1999."},{"key":"40_CR19","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1016\/S0953-5438(02)00023-1","volume":"14","author":"M Hertzum","year":"2002","unstructured":"Hertzum M, Andersen HHK, Andersen V, Hansen CB: Trust in information sources: Seeking information from people, documents, and virtual agents. Interact Comput 2002, 14: 575\u2013599. 10.1016\/S0953-5438(02)00023-1","journal-title":"Interact Comput"},{"key":"40_CR20","volume-title":"Measuring the user experience","author":"T Tullis","year":"2008","unstructured":"Tullis T, Albert B: Measuring the user experience. Boston: Morgan Kaufmann; 2008."},{"key":"40_CR21","first-page":"1383","volume-title":"Proceedings of CHI \u203205","author":"E Fr\u00f8kj\u00e6r","year":"2005","unstructured":"Fr\u00f8kj\u00e6r E, Hornb\u00e6k K: Cooperative usability testing: complementing usability tests with user-supported interpretation sessions. In Proceedings of CHI \u203205. Edited by: van der Veer G, Gale C. New York: Extended Abstracts on Human Factors in Computing Systems. ACM Press; 2005:1383\u20131386."},{"key":"40_CR22","first-page":"63","volume-title":"Usability inspection methods","author":"RG Bias","year":"1994","unstructured":"Bias RG: The pluralistic usability walkthrough: coordinated empathies. In Usability inspection methods. Edited by: Nielsen J, Mack R. New York: Wiley; 1994:63\u201376."},{"key":"40_CR23","volume-title":"R3UEMs: Review, Report and Refine Usability Evaluation Methods. Proceedings of the 3. COST294-MAUSE international workshop","author":"A F\u00f8lstad","year":"2007","unstructured":"F\u00f8lstad A: Group-based expert walkthrough. In R3UEMs: Review, Report and Refine Usability Evaluation Methods. Proceedings of the 3. COST294-MAUSE international workshop. Edited by: Scapin D, Law E. Athens: R3UEMs; 2007."},{"key":"40_CR24","doi-asserted-by":"crossref","first-page":"2127","DOI":"10.1145\/2207676.2208365","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203212","author":"A F\u00f8lstad","year":"2012","unstructured":"F\u00f8lstad A, Law ELC, Hornb\u00e6k K: Analysis in practical usability evaluation: a survey study. In Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203212. Edited by: Konstan JA, Chi EH, H\u00f6\u00f6k K. New York: ACM Press; 2012:2127\u20132136."},{"key":"40_CR25","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/1900441.1900447","volume-title":"Proceedings of the biennial participatory design conference, PDC \u203210","author":"P Hagen","year":"2010","unstructured":"Hagen P, Robertson T: Social technologies: Challenges and opportunities for participation. In Proceedings of the biennial participatory design conference, PDC \u203210. Edited by: Robertson T, B\u00f8dker K, Bratteteig T, Loi D. New York: ACM Press; 2010:31\u201340."},{"key":"40_CR26","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1080\/01449299408914597","volume":"13","author":"ED Smilowitz","year":"1994","unstructured":"Smilowitz ED, Darnell MJ, Benson AE: Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests. Behav Inf Technol 1994, 13: 183\u2013190. 10.1080\/01449299408914597","journal-title":"Behav Inf Technol"},{"key":"40_CR27","doi-asserted-by":"crossref","first-page":"1619","DOI":"10.1145\/1518701.1518948","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203209","author":"A Bruun","year":"2009","unstructured":"Bruun A, Gull P, Hofmeister L, Stage J: Let your users do the testing: a comparison of three remote asynchronous usability testing methods. In Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203209. Edited by: Hickley K, Morris MR, Hudson S, Greenberg S. New York: ACM Press; 2009:1619\u20131628."},{"key":"40_CR28","volume-title":"Experimental and quasi-experimental designs for generalized causal inference","author":"WR Shadish","year":"2002","unstructured":"Shadish WR, Cook TD, Campbell DT: Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin; 2002."},{"key":"40_CR29","unstructured":"Notebox http:\/\/noteboxapp.com"},{"key":"40_CR30","unstructured":"Cage http:\/\/cageapp.com"},{"key":"40_CR31","unstructured":"Owela http:\/\/owela.fi"},{"key":"40_CR32","unstructured":"RECORD online Living Lab http:\/\/livinglab.origo.no"},{"key":"40_CR33","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1207\/S15327590IJHC1304_03","volume":"13","author":"HR Hartson","year":"2001","unstructured":"Hartson HR, Andre TS, Williges RC: Criteria for evaluating usability evaluation methods. Int J Hum Comput Interact 2001, 13: 373\u2013410. 10.1207\/S15327590IJHC1304_03","journal-title":"Int J Hum Comput Interact"},{"key":"40_CR34","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1080\/014492997119789","volume":"16","author":"BE John","year":"1997","unstructured":"John BE, Marks SJ: Tracking the effectiveness of usability evaluation methods. Behav Inf Technol 1997, 16: 188\u2013202. 10.1080\/014492997119789","journal-title":"Behav Inf Technol"},{"key":"40_CR35","volume-title":"Advances in Human-Computer Interaction","author":"A F\u00f8lstad","year":"2010","unstructured":"F\u00f8lstad A, Knutsen J: Online user feedback in early phases of the design process: lessons learnt from four design cases. Advances in Human-Computer Interaction 2010. doi:10.1155\/2010\/956918 doi:10.1155\/2010\/956918"},{"key":"40_CR36","first-page":"183","volume-title":"Proceedings of the 2008 international conference on web search and data mining","author":"E Agichtein","year":"2008","unstructured":"Agichtein E, Castillo C, Donato D, Gionis A, Mishne G: Finding high-quality content in social media. In Proceedings of the 2008 international conference on web search and data mining. Edited by: Najorc M, Broder A, Chacrabarti S. New York: ACM Press; 2008:183\u2013194."},{"key":"40_CR37","first-page":"11","volume-title":"Proceedings of GROUP \u203205","author":"C Lampe","year":"2005","unstructured":"Lampe C, Johnston E: Follow the (Slash) dot: effects of feedback on new members in an online community. In Proceedings of GROUP \u203205. Edited by: Pandergast M, Schmidt K, Mark G, Ackerman M. New York: ACM Press; 2005:11\u201320."},{"key":"40_CR38","first-page":"61","volume-title":"Proceedings of GROUP \u203209","author":"MJ Brzozowski","year":"2009","unstructured":"Brzozowski MJ, Sandholm T, Hogg T: Effects of feedback and peer pressure on contributions to enterprise social media. In Proceedings of GROUP \u203209. Edited by: Teasley S, Havn E, Prinz W, Lutters W. New York: ACM Press; 2009:61\u201370."},{"key":"40_CR39","volume-title":"J Comput Mediat Commun","author":"M L\u00fcders","year":"2013","unstructured":"L\u00fcders M, F\u00f8lstad A, Waldal E: Expectations and experiences with MyLabourParty: From right to know to right to participate? J Comput Mediat Commun 2013. (in press) (in press)"},{"key":"40_CR40","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1145\/1518701.1518847","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203209","author":"M Burke","year":"2009","unstructured":"Burke M, Marlow C, Lento T: Feed me: motivating newcomer contribution in social network sites. In Proceedings of the SIGCHI conference on human factors in computing systems, CHI \u203209. Edited by: Olsen DRJr, Arthur RB, Hinckley K, Morris MR, Hudson S, Greenberg S. New York: ACM Press; 2009:945\u2013954."},{"key":"40_CR41","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1080\/01449290600811594","volume":"26","author":"A Kienle","year":"2007","unstructured":"Kienle A, Ritterskamp C: Facilitating asynchronous discussions in learning communities: the impact of moderation strategies. Behav Inf Technol 2007, 26: 73\u201380. 10.1080\/01449290600811594","journal-title":"Behav Inf Technol"},{"key":"40_CR42","volume-title":"The handbook of internet studies","author":"J Stomer-Galley","year":"2011","unstructured":"Stomer-Galley J, Wichowski A: Political discussion online. In The handbook of internet studies. Edited by: Consalvo M, Ess C. Chichester: Wiley-Blackwell; 2011."},{"key":"40_CR43","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/0030-5073(83)90156-3","volume":"32","author":"SJ Ashford","year":"1983","unstructured":"Ashford SJ, Cummings LL: Feedback as an individual resource: personal strategies of creating information. Organ Behav Hum Perform 1983, 32: 370\u2013398. 10.1016\/0030-5073(83)90156-3","journal-title":"Organ Behav Hum Perform"},{"key":"40_CR44","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1037\/0021-9010.78.4.531","volume":"78","author":"AR Dennis","year":"1993","unstructured":"Dennis AR, Valacich JS: Computer brainstorms: more heads are better than one. J Appl Psychol 1993, 78: 531\u2013537.","journal-title":"J Appl Psychol"},{"key":"40_CR45","doi-asserted-by":"publisher","first-page":"1549","DOI":"10.1016\/j.chb.2005.07.003","volume":"23","author":"DM DeRosa","year":"2007","unstructured":"DeRosa DM, Smith CL, Hantula DA: The medium matters: mining the long-promised merit of group interaction in creative idea generation tasks in a meta-analysis of the electronic group brainstorming literature. Comput Human Behav 2007, 23: 1549\u20131581. 10.1016\/j.chb.2005.07.003","journal-title":"Comput Human Behav"},{"key":"40_CR46","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1037\/0022-3514.65.4.681","volume":"65","author":"SJ Karau","year":"1993","unstructured":"Karau SJ, Williams KD: Social loafing: a meta-analytic review and theoretical integration. J Pers Soc Psychol 1993, 65: 681\u2013706.","journal-title":"J Pers Soc Psychol"},{"key":"40_CR47","volume-title":"Proceedings of the Hawaii international conference on system sciences, HICSS \u203205","author":"JJH Jung","year":"2005","unstructured":"Jung JJH, Schneider C, Valacich JS: The Influence of real-time identifiability and evaluability performance feedback on group electronic brainstorming performance. In Proceedings of the Hawaii international conference on system sciences, HICSS \u203205. Los Alamitos: IEEE; 2005. doi:10.1109\/HICSS.2005.566 doi:10.1109\/HICSS.2005.566"},{"key":"40_CR48","doi-asserted-by":"publisher","first-page":"38","DOI":"10.4018\/jec.2005100103","volume":"1","author":"AR Dennis","year":"2005","unstructured":"Dennis AR, Pinsonneault A, Hilmer KM, Barki H, Galupe B, Huber MW, Bellavance F: Patterns in electronic brainstorming. International Journal of eCollaboration 2005, 1: 38\u201357.","journal-title":"International Journal of eCollaboration"},{"key":"40_CR49","volume-title":"Creativity in context","author":"TM Amabile","year":"1996","unstructured":"Amabile TM: Creativity in context. Boulder: Westview Press; 1996."},{"key":"40_CR50","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 1977, 33: 159\u2013174. 10.2307\/2529310","journal-title":"Biometrics"},{"key":"40_CR51","volume-title":"Discovering statistics using SPSS","author":"A Field","year":"2009","unstructured":"Field A: Discovering statistics using SPSS. London: Sage; 2009."},{"key":"40_CR52","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1037\/0033-2909.112.1.155","volume":"112","author":"J Cohen","year":"1992","unstructured":"Cohen J: A power primer. Psychol Bull 1992, 112: 155\u2013159.","journal-title":"Psychol Bull"},{"key":"40_CR53","unstructured":"G*Power 3 http:\/\/www.psycho.uni-duesseldorf.de\/abteilungen\/aap\/gpower3"}],"container-title":["Human-centric Computing and Information Sciences"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/2192-1962-3-18.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/2192-1962-3-18\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/2192-1962-3-18.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,30]],"date-time":"2021-07-30T07:00:15Z","timestamp":1627628415000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/2192-1962-3-18"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,9,23]]},"references-count":53,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2013,12]]}},"alternative-id":["40"],"URL":"https:\/\/doi.org\/10.1186\/2192-1962-3-18","relation":{},"ISSN":["2192-1962"],"issn-type":[{"value":"2192-1962","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,9,23]]},"assertion":[{"value":"13 September 2012","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2013","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 September 2013","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"18"}}