{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T11:28:02Z","timestamp":1740137282421,"version":"3.37.3"},"reference-count":50,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2020,8,9]],"date-time":"2020-08-09T00:00:00Z","timestamp":1596931200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,9]],"date-time":"2020-08-09T00:00:00Z","timestamp":1596931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cogn Tech Work"],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The evaluation of groupware has a long history; several researchers have investigated this research area and made attempts to develop evaluation methods. This paper aims to make a contribution to this research topic by introducing a groupware evaluation method called Team Usability Testing. The goal of this method is to evaluate the usability of real-time distributed groupware. The Team Usability Test consists of a combination of questionnaires, on-screen behaviour recordings and interviews. The data analysis is based on the mechanics of collaboration theoretical framework and involves communication analysis, behaviour analysis, and analysis of post-experiment interviews. A laboratory experiment and a field study constitute the two main phases of the creation of the usability testing method. In this paper, the results of the laboratory experiment and its implications for the field study will be discussed. According to the results, the Team Usability Test is able to explore team-level usability problems and contextual problems. The paper ends with the discussion of future field research considerations related to the possible application of the Team Usability Test in real-life work settings.<\/jats:p>","DOI":"10.1007\/s10111-020-00647-8","type":"journal-article","created":{"date-parts":[[2020,8,9]],"date-time":"2020-08-09T17:02:38Z","timestamp":1596992558000},"page":"755-769","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Empirical study of Team Usability Testing: a laboratory experiment"],"prefix":"10.1007","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4336-0674","authenticated-orcid":false,"given":"Dalma","family":"Geszten","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8085-1259","authenticated-orcid":false,"given":"Bal\u00e1zs P\u00e9ter","family":"H\u00e1mornik","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3015-7100","authenticated-orcid":false,"given":"K\u00e1roly","family":"Hercegfi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,8,9]]},"reference":[{"key":"647_CR1","doi-asserted-by":"crossref","unstructured":"Baker K, Greenberg S, Gutwin C (2001) Heuristic evaluation of groupware based on the mechanics of collaboration. In: Proceedings of the 8th IFIP international conference on engineering for human-computer interaction\u00a0(EHCI '01). Springer, London, pp 123\u2013140","DOI":"10.1007\/3-540-45348-2_14"},{"key":"647_CR2","doi-asserted-by":"crossref","unstructured":"Baker K, Greenberg S, Gutwin C (2002) Empirical development of a heuristic evaluation methodology for shared workspace groupware. In: Proceedings of the 2002 ACM conference on computer supported cooperative work\u00a0(CSCW '02). ACM, New York, pp 96\u2013105","DOI":"10.1145\/587078.587093"},{"key":"647_CR3","doi-asserted-by":"crossref","unstructured":"Bly SA (1988): A use of drawing surfaces in different collaborative settings. In: Proceedings of the 1988 ACM conference on computer-supported cooperative work. Portland, Oregon, pp 250\u2013256","DOI":"10.1145\/62266.62286"},{"key":"647_CR4","doi-asserted-by":"crossref","unstructured":"Buur J, B\u00f8dker S (2000) From usability lab to \u201cdesign collaboratorium\u201d: reframing usability practice. In: Proceedings of the 3rd conference on designing interactive systems: processes, practices, methods, and techniques, pp 297\u2013307","DOI":"10.1145\/347642.347768"},{"key":"647_CR5","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1016\/j.sbspro.2013.10.289","volume":"97","author":"CJ Chen","year":"2013","unstructured":"Chen CJ, Lau SY, Chuah KM, Teh CS (2013) Group usability testing of virtual reality-based learning environments: a modified approach. Proc Soc Behav Sci 97:691\u2013699","journal-title":"Proc Soc Behav Sci"},{"issue":"2","key":"647_CR6","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1007\/s10111-005-0175-5","volume":"7","author":"C Chisalita","year":"2005","unstructured":"Chisalita C, Melguizo MCP, Hoorn JF, Van der Veer GC, Kok E (2005) Cultural differences in user groups: a multi-angle understanding of IT use in large organizations. Cogn Technol Work 7(2):101\u2013110","journal-title":"Cogn Technol Work"},{"key":"647_CR7","doi-asserted-by":"crossref","unstructured":"Cho JY, Cho MH (2019) Students\u2019 use of social media in collaborative design: a case study of an advanced interior design studio. Cogn Technol Work 1\u201316","DOI":"10.1007\/s10111-019-00597-w"},{"key":"647_CR8","unstructured":"Christensen B, Ellingsen G (2014) User-controlled standardisation of health care practices. In Proceedings of the 22nd European conference of on information systems (ECIS), Tel Aviv"},{"key":"647_CR9","doi-asserted-by":"crossref","unstructured":"Dew K, Turner AM, Desai L, Martin N, Kirchhoff K (2015) Evaluating groupware prototypes with discount methods. In: Proceedings of the 18th ACM conference companion on computer supported cooperative work & social computing, pp 183\u2013186","DOI":"10.1145\/2685553.2699002"},{"issue":"2","key":"647_CR10","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s10111-005-0177-3","volume":"7","author":"B Fields","year":"2005","unstructured":"Fields B, Amaldi P, Tassi A (2005) Representing collaborative work: the airport as common information space. Cogn Technol Work 7(2):119\u2013133","journal-title":"Cogn Technol Work"},{"issue":"3","key":"647_CR11","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1016\/j.pec.2011.01.004","volume":"82","author":"I Fletcher","year":"2011","unstructured":"Fletcher I, Mazzi M, Nuebling M (2011) When coders are reliable: the application of three measures to assess inter-rater reliability\/agreement with doctor\u2013patient communication data coded with the VR-CoDES. Patient Educ Couns 82(3):341\u2013345","journal-title":"Patient Educ Couns"},{"key":"647_CR12","doi-asserted-by":"crossref","unstructured":"Fr\u00edas SGD, Mezura-Godoy C, Ben\u00edtez-Guerrero E (2019) FrUtEG: a conceptual framework for utility evaluation in groupware. In: Proceedings of the IX Latin American conference on human computer interaction, pp 1\u20138","DOI":"10.1145\/3358961.3358965"},{"issue":"3","key":"647_CR13","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1016\/0953-5438(92)90023-9","volume":"4","author":"S Greenberg","year":"1992","unstructured":"Greenberg S, Roseman M, Webster D, Bohnet R (1992) Human and technical factors of distributed group drawing tools. Interact Comput 4(3):364\u2013392","journal-title":"Interact Comput"},{"key":"647_CR14","doi-asserted-by":"crossref","unstructured":"Grudin J (1988) Why CSCW applications fail: problems in the design and evaluation of organizational interfaces. In: Proceedings of the 1988 ACM conference on computer-supported cooperative work\u00a0(CSCW '88). ACM, New York, pp 85\u201393","DOI":"10.1145\/62266.62273"},{"key":"647_CR15","doi-asserted-by":"crossref","unstructured":"Gumienny R, Gericke L, Wenzel M, Meinel C (2013) Supporting creative collaboration in globally distributed companies. In: Proceedings of the 2013 conference on computer supported cooperative work, pp 995\u20131007","DOI":"10.1145\/2441776.2441890"},{"key":"647_CR16","doi-asserted-by":"crossref","unstructured":"Gutwin C, Greenberg S (1998) Design for individuals, design for groups: tradeoffs between power and workspace awareness. In: Proceedings of the ACM 1998 conference on computer supported cooperative work, Seattle","DOI":"10.1145\/289444.289495"},{"issue":"3","key":"647_CR17","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1145\/329693.329696","volume":"6","author":"C Gutwin","year":"1999","unstructured":"Gutwin C, Greenberg S (1999) The effects of workspace awareness support on the usability of real-time distributed groupware. ACM Trans Comput Hum Interact (TOCHI) 6(3):243\u2013281","journal-title":"ACM Trans Comput Hum Interact (TOCHI)"},{"key":"647_CR18","doi-asserted-by":"crossref","unstructured":"Gutwin C, Greenberg S (2000) The mechanics of collaboration: developing low cost usability evaluation methods for shared workspaces. In: Proceedings of the 9th IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises (WETICE '00). IEEE Computer Society, Washington, DC, pp 98\u2013103","DOI":"10.1109\/ENABL.2000.883711"},{"issue":"3\u20134","key":"647_CR19","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1023\/A:1021271517844","volume":"11","author":"C Gutwin","year":"2002","unstructured":"Gutwin C, Greenberg S (2002) A descriptive framework of workspace awareness for real-time groupware. Comput Support Coop Work 11(3\u20134):411\u2013446","journal-title":"Comput Support Coop Work"},{"key":"647_CR20","doi-asserted-by":"crossref","unstructured":"Gutwin C, Roseman M, Greenberg S (1996) A usability study of awareness widgets in a shared workspace groupware system. In: Proceedings of the ACM conference on computer supported cooperative work (CSCW), pp 258\u2013267","DOI":"10.1145\/240080.240298"},{"key":"647_CR21","doi-asserted-by":"crossref","unstructured":"Hackman GS, Biers DW (1992) Team usability testing: are two heads better than one? In: Proceedings of the human factors and ergonomics society annual meeting, vol 36, no 16, pp 1205\u20131209","DOI":"10.1177\/154193129203601605"},{"key":"647_CR22","doi-asserted-by":"crossref","unstructured":"Haynes SR, Purao S, Skattebo AL (2004) Situating evaluation in scenarios of use. In: Proceedings of the 2004 ACM conference on computer supported cooperative work, pp 92\u2013101","DOI":"10.1145\/1031607.1031624"},{"issue":"8","key":"647_CR23","first-page":"988","volume":"25","author":"V Herskovic","year":"2019","unstructured":"Herskovic V, Ochoa SF, Pino JA (2019) Identifying groupware requirements in people-driven mobile collaborative processes. J UCS 25(8):988\u20131017","journal-title":"J UCS"},{"key":"647_CR24","doi-asserted-by":"crossref","unstructured":"Ignat CL, Oster G, Fox O, Shalin VL, Charoy F (2015) How do user groups cope with delay in real-time collaborative note taking. In: Proceedings of the 14th European conference on computer supported cooperative work, Oslo, pp 223\u2013242","DOI":"10.1007\/978-3-319-20499-4_12"},{"key":"647_CR25","unstructured":"Khoshafian S, Buckiewicz M (1995) Introduction to groupware, workflow, and workgroup computing. Wiley"},{"key":"647_CR26","doi-asserted-by":"crossref","unstructured":"Kjeldskov J, Skov MB (2014) Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations. In: Proceedings of the 16th international conference on human-computer interaction with mobile devices & services\u00a0(MobileHCI '14). ACM, New York, pp 43\u201352","DOI":"10.1145\/2628363.2628398"},{"key":"647_CR27","doi-asserted-by":"crossref","unstructured":"Laufer L, Halacsy P, Somlai-Fischer A (2011) Prezi meeting: collaboration in a zoomable canvas based environment. In: CHI'11 extended abstracts on human factors in computing systems, pp 749\u2013752","DOI":"10.1145\/1979742.1979673"},{"key":"647_CR28","doi-asserted-by":"crossref","unstructured":"Lopez G, Guerrero LA (2017) Awareness supporting technologies used in collaborative systems: a systematic literature review. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pp 808\u2013820","DOI":"10.1145\/2998181.2998281"},{"key":"647_CR29","doi-asserted-by":"crossref","unstructured":"Marlow J, Carter S, Good N, Chen JW (2016) Beyond talking heads: multimedia artifact creation, use, and sharing in distributed meetings. In: Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing, pp 1703\u20131715","DOI":"10.1145\/2818048.2819958"},{"key":"647_CR30","doi-asserted-by":"crossref","unstructured":"Nielsen J (1994) Usability inspection methods. In: Conference companion on human factors in computing systems, pp 413\u2013414","DOI":"10.1145\/259963.260531"},{"issue":"3","key":"647_CR31","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1080\/07421222.2015.1094961","volume":"32","author":"JF Nunamaker Jr","year":"2015","unstructured":"Nunamaker JF Jr, Briggs RO, Derrick DC, Schwabe G (2015) The last research mile: achieving both rigor and relevance in information systems research. J Manag Inf Syst 32(3):10\u201347","journal-title":"J Manag Inf Syst"},{"issue":"6","key":"647_CR32","doi-asserted-by":"publisher","first-page":"737","DOI":"10.1016\/j.intcom.2003.09.003","volume":"15","author":"TC Pargman","year":"2003","unstructured":"Pargman TC (2003) Collaborating with writing tools: an instrumental perspective on the problem of computer-supported collaborative activities. Interact Comput 15(6):737\u2013757","journal-title":"Interact Comput"},{"issue":"2\u20133","key":"647_CR33","doi-asserted-by":"publisher","first-page":"315","DOI":"10.1007\/s10111-017-0407-5","volume":"19","author":"C Passos","year":"2017","unstructured":"Passos C, Da Silva MH, Mol AC, Carvalho PV (2017) Design of a collaborative virtual environment for training security agents in big events. Cogn Technol Work 19(2\u20133):315\u2013328","journal-title":"Cogn Technol Work"},{"key":"647_CR34","unstructured":"Pinelle D (2000) A survey of groupware evaluations in CSCW Proceedings. In: CSCW proceedings,\u00a0pp 22\u201326"},{"issue":"3","key":"647_CR35","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1007\/s00779-007-0145-4","volume":"12","author":"D Pinelle","year":"2008","unstructured":"Pinelle D, Gutwin C (2008) Evaluating teamwork support in tabletop groupware applications using collaboration usability analysis. Pers Ubiquit Comput 12(3):237\u2013254","journal-title":"Pers Ubiquit Comput"},{"issue":"4","key":"647_CR36","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1145\/966930.966932","volume":"10","author":"D Pinelle","year":"2003","unstructured":"Pinelle D, Gutwin C, Greenberg S (2003) Task analysis for groupware usability evaluation. ACM Trans Comput Hum Interact 10(4):281\u2013311","journal-title":"ACM Trans Comput Hum Interact"},{"issue":"4","key":"647_CR37","doi-asserted-by":"publisher","first-page":"657","DOI":"10.1016\/j.intcom.2004.07.004","volume":"16","author":"M Romero-Salcedo","year":"2004","unstructured":"Romero-Salcedo M, Osuna-G\u00f3mez CA, Sheremetov L, Villa L, Morales C, Rocha L, Chi M (2004) Study and analysis of workspace awareness in CDebate: a groupware application for collaborative debates. Interact Comput 16(4):657\u2013681","journal-title":"Interact Comput"},{"key":"647_CR38","doi-asserted-by":"crossref","unstructured":"Salom\u00f3n S, Duque R, Monta\u00f1a JL, Ten\u00e9s L (2019) Modeling users behavior in groupware applications. In: International conference on cooperative design, visualization and engineering. Springer, Cham, pp 11\u201321","DOI":"10.1007\/978-3-030-30949-7_2"},{"issue":"4\u20136","key":"647_CR41","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1007\/s10606-013-9193-7","volume":"22","author":"K Schmidt","year":"2013","unstructured":"Schmidt K, Bannon L (2013) Constructing CSCW: the first quarter century. Comput Support Coop Work (CSCW) 22(4\u20136):345\u2013372","journal-title":"Comput Support Coop Work (CSCW)"},{"key":"647_CR42","doi-asserted-by":"crossref","unstructured":"Silsand L, Christensen B, Ellingsen G (2012) Towards an empowered user role in the design of large-scale electronic patient records. In: Proceedings of the 12th participatory design conference: exploratory papers, workshop descriptions, industry cases, vol 2, pp 65\u201368","DOI":"10.1145\/2348144.2348165"},{"issue":"2","key":"647_CR43","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1007\/s10111-005-0180-8","volume":"7","author":"G Spinelli","year":"2005","unstructured":"Spinelli G, Perry M, O\u2019Hara K (2005) Understanding complex cognitive systems: the role of space in the organisation of collaborative work. Cogn Technol Work 7(2):111\u2013118","journal-title":"Cogn Technol Work"},{"key":"647_CR44","doi-asserted-by":"crossref","unstructured":"Steves MP, Morse E, Gutwin C, Greenberg S (2001) A comparison of usage evaluation and inspection methods for assessing groupware usability. In: Proceedings of the 2001 international ACM SIGGROUP conference on supporting group work, pp 125\u2013134","DOI":"10.1145\/500286.500306"},{"key":"647_CR45","first-page":"273","volume":"17","author":"A Strauss","year":"1994","unstructured":"Strauss A, Corbin J (1994) Grounded theory methodology. Handb Qual Res 17:273\u2013285","journal-title":"Handb Qual Res"},{"key":"647_CR46","doi-asserted-by":"crossref","unstructured":"Svensson \u00c5, Ohlander U, Lundberg J (2019) Design implications for teamwork in ATC. In: Cognition, technology & work, pp 1\u201318","DOI":"10.1007\/s10111-019-00579-y"},{"issue":"2","key":"647_CR47","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1016\/0020-7373(91)90039-A","volume":"34","author":"JC Tang","year":"1991","unstructured":"Tang JC (1991) Findings from observational studies of collaborative work. Int J Man Mach Stud 34(2):143\u2013160","journal-title":"Int J Man Mach Stud"},{"key":"647_CR48","doi-asserted-by":"crossref","unstructured":"Tang JC, Isaacs EA, Rua M (1994) Supporting distributed groups with a montage of lightweight interactions. In: Proceedings of the 1994 ACM conference on computer supported cooperative work, pp 23\u201334","DOI":"10.1145\/192844.192861"},{"key":"647_CR49","doi-asserted-by":"crossref","unstructured":"Van der Veer G, van Welie M (2000) Task based groupware design: putting theory into practice. In: Proceedings of the 3rd conference on designing interactive systems: processes, practices, methods, and techniques, pp 326\u2013337","DOI":"10.1145\/347642.347781"},{"issue":"4","key":"647_CR51","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1007\/s10111-012-0232-9","volume":"15","author":"D Vyas","year":"2013","unstructured":"Vyas D, van der Veer G, Nijholt A (2013) Creative practices in the design studio culture: collaboration and communication. Cogn Technol Work 15(4):415\u2013443","journal-title":"Cogn Technol Work"},{"key":"647_CR52","doi-asserted-by":"crossref","unstructured":"Yamauchi T, Ohno T, Nakatani M, Kato Y, Markman A (2012) Psychology of user experience in a collaborative video-conference system. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, pp 187\u2013196","DOI":"10.1145\/2145204.2145234"},{"key":"647_CR53","doi-asserted-by":"crossref","unstructured":"Yen DC, Wen HJ, Lin B, Chou DC (1999) Groupware: a strategic analysis and implementation. Industrial management & data systems","DOI":"10.1108\/02635579910243879"}],"container-title":["Cognition, Technology &amp; Work"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10111-020-00647-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10111-020-00647-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10111-020-00647-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,17]],"date-time":"2021-10-17T09:07:44Z","timestamp":1634461664000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10111-020-00647-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,9]]},"references-count":50,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["647"],"URL":"https:\/\/doi.org\/10.1007\/s10111-020-00647-8","relation":{},"ISSN":["1435-5558","1435-5566"],"issn-type":[{"type":"print","value":"1435-5558"},{"type":"electronic","value":"1435-5566"}],"subject":[],"published":{"date-parts":[[2020,8,9]]},"assertion":[{"value":"5 March 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 August 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This study was approved by the (Hungarian) Joint Psychological Research Ethics Committee (Reference no.: 2018\/03).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}}]}}