{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T02:25:21Z","timestamp":1778034321303,"version":"3.51.4"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2013,11,1]],"date-time":"2013-11-01T00:00:00Z","timestamp":1383264000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/F063822\/1, EP\/G012393\/1"],"award-info":[{"award-number":["EP\/F063822\/1, EP\/G012393\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"name":"MATCH programme"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2013,11]]},"abstract":"<jats:p>The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach -- which we call the<jats:italic>Grounded Procedure<\/jats:italic>-- is introduced and its value argued.<\/jats:p>","DOI":"10.1145\/2506210","type":"journal-article","created":{"date-parts":[[2013,11,12]],"date-time":"2013-11-12T15:29:09Z","timestamp":1384270149000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Reviewing and Extending the Five-User Assumption"],"prefix":"10.1145","volume":"20","author":[{"given":"Simone","family":"Borsci","sequence":"first","affiliation":[{"name":"Brunel University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert D.","family":"Macredie","sequence":"additional","affiliation":[{"name":"Brunel University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julie","family":"Barnett","sequence":"additional","affiliation":[{"name":"Brunel University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer","family":"Martin","sequence":"additional","affiliation":[{"name":"University of Nottingham"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jasna","family":"Kuljis","sequence":"additional","affiliation":[{"name":"Brunel University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Terry","family":"Young","sequence":"additional","affiliation":[{"name":"Brunel University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2013,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.4103\/0256-4602.57825"},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Bias R. G. and Mayhew D. J. 2005. Cost-Justifying Usability: An Update for the Internet Age. Morgan Kaufmann Publishers San Francisco CA. Bias R. G. and Mayhew D. J. 2005. Cost-Justifying Usability: An Update for the Internet Age . Morgan Kaufmann Publishers San Francisco CA.","DOI":"10.1016\/B978-012095811-5\/50022-5"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Borsci S. Federici S. Mele M. L. Polimeno D. and Londei A. 2012. The bootstrap discovery behaviour model: Why five users are not enough to test user experience. In Cognitively Informed Intelligent Interfaces: Systems Design and Development E. M. Alkhalifa and K. Gaid Eds. IGI Global Press Hershey PA. Borsci S. Federici S. Mele M. L. Polimeno D. and Londei A. 2012. The bootstrap discovery behaviour model: Why five users are not enough to test user experience. In Cognitively Informed Intelligent Interfaces: Systems Design and Development E. M. Alkhalifa and K. Gaid Eds. IGI Global Press Hershey PA.","DOI":"10.4018\/978-1-4666-1628-8.ch015"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10339-010-0376-6"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/01449290010020648"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.lisr.2005.01.012"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176344552"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03195514"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10339-010-0358-8"},{"key":"e_1_2_1_10_1","volume-title":"Monte Carlo: Concepts, Algorithms, and Applications","author":"Fishman G. S.","year":"1995","unstructured":"Fishman , G. S. 1995 . Monte Carlo: Concepts, Algorithms, and Applications . Springer , New York . Fishman, G. S. 1995. Monte Carlo: Concepts, Algorithms, and Applications. Springer, New York."},{"key":"e_1_2_1_11_1","volume-title":"Draft guidance for industry and Food and Drug Administration staff - Applying human factors and usability engineering to optimize medical device design. U.S. Food and Drug Administration","author":"Food and Drug Administration (FDA) 2011.","unstructured":"Food and Drug Administration (FDA) 2011. Draft guidance for industry and Food and Drug Administration staff - Applying human factors and usability engineering to optimize medical device design. U.S. Food and Drug Administration , Silver Spring, MD . Food and Drug Administration (FDA) 2011. Draft guidance for industry and Food and Drug Administration staff - Applying human factors and usability engineering to optimize medical device design. U.S. Food and Drug Administration, Silver Spring, MD."},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Fox J. 2002. An R and S-Plus Companion to Applied Regression. SAGE. Fox J. 2002. An R and S-Plus Companion to Applied Regression . SAGE.","DOI":"10.32614\/CRAN.package.car"},{"key":"e_1_2_1_13_1","unstructured":"Glaser B. G. and Strauss A. L. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company Chicago IL. Glaser B. G. and Strauss A. L. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research . Aldine Publishing Company Chicago IL."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/40.3-4.237"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327590IJHC1501_14"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/502115.502118"},{"key":"e_1_2_1_17_1","volume-title":"Ergonomic requirements for office work with visual display terminals","unstructured":"ISO 1998. ISO 9241-11:1998 , Ergonomic requirements for office work with visual display terminals , CEN , Brussels, BE . ISO 1998. ISO 9241-11:1998, Ergonomic requirements for office work with visual display terminals, CEN, Brussels, BE."},{"key":"e_1_2_1_18_1","volume-title":"Statistical Methods for Speech Recognition","author":"Jelinek F.","unstructured":"Jelinek , F. 1997. Statistical Methods for Speech Recognition . MIT Press , Cambridge, MA . Jelinek, F. 1997. Statistical Methods for Speech Recognition. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Kirakowski J. 2005. Chapter 18 - Summative usability testing: Measurement and sample size. In Cost-Justifying Usability 2nd Ed. R. G. Bias and D. J. Mayhew Eds. Morgan Kaufmann San Francisco CA 519--553. Kirakowski J. 2005. Chapter 18 - Summative usability testing: Measurement and sample size. In Cost-Justifying Usability 2nd Ed. R. G. Bias and D. J. Mayhew Eds. Morgan Kaufmann San Francisco CA 519--553.","DOI":"10.1016\/B978-012095811-5\/50018-3"},{"key":"e_1_2_1_20_1","volume-title":"Human-Computer Interaction: Interaction Design and Usability","author":"Kurosu M.","unstructured":"Kurosu , M. 2007. Concept of usability revisited . In Human-Computer Interaction: Interaction Design and Usability , J. Jacko Ed., Springer , Berlin , 579--586. Kurosu, M. 2007. Concept of usability revisited. In Human-Computer Interaction: Interaction Design and Usability, J. Jacko Ed., Springer, Berlin, 579--586."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872089403600215"},{"key":"e_1_2_1_22_1","volume-title":"Validation of Monte Carlo estimation of problem discovery likelihood. Tech. rep. 29.3357","author":"Lewis J. R.","unstructured":"Lewis , J. R. 2000. Validation of Monte Carlo estimation of problem discovery likelihood. Tech. rep. 29.3357 , IBM , Raleigh, NC . Lewis, J. R. 2000. Validation of Monte Carlo estimation of problem discovery likelihood. Tech. rep. 29.3357, IBM, Raleigh, NC."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327590IJHC1304_06"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1167948.1167973"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240839"},{"key":"e_1_2_1_26_1","unstructured":"Manning C. D. and Schutze H. 1999. Foundations of Statistical Natural Language Processing. MIT Press Cambridge MA. Manning C. D. and Schutze H. 1999. Foundations of Statistical Natural Language Processing . MIT Press Cambridge MA."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/77481.77486"},{"key":"e_1_2_1_28_1","unstructured":"Nielsen J. 1995. Severity ratings for usability problems. http:\/\/useit.com\/papers\/heuristic\/severityrating.html. Nielsen J. 1995. Severity ratings for usability problems. http:\/\/useit.com\/papers\/heuristic\/severityrating.html."},{"key":"e_1_2_1_29_1","unstructured":"Nielsen J. 2000. Why you only need to test with 5 users. http:\/\/www.useit.com\/alertbox\/20000319.html. Nielsen J. 2000. Why you only need to test with 5 users. http:\/\/www.useit.com\/alertbox\/20000319.html."},{"key":"e_1_2_1_30_1","unstructured":"Nielsen J. 2012. How many test users in a usability study? http:\/\/www.useit.com\/alertbox\/number-of-test-users.html. Nielsen J. 2012. How many test users in a usability study? http:\/\/www.useit.com\/alertbox\/number-of-test-users.html."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169166"},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Nielsen J. and Mack R. L. 1994. Usability Inspection Methods. Wiley New York. Nielsen J. and Mack R. L. 1994. Usability Inspection Methods . Wiley New York.","DOI":"10.1145\/259963.260531"},{"key":"e_1_2_1_33_1","volume-title":"Some observations on mental models","author":"Norman D. A.","unstructured":"Norman , D. A. 1983. Some observations on mental models . In Mental Models, D. Gentner and A. Steven Eds., Lawrence Earlbaum Associates, Hillsdale, NJ , 7--14. Norman, D. A. 1983. Some observations on mental models. In Mental Models, D. Gentner and A. Steven Eds., Lawrence Earlbaum Associates, Hillsdale, NJ, 7--14."},{"key":"e_1_2_1_34_1","volume-title":"The Psychology of Everyday Things","author":"Norman D. A.","unstructured":"Norman , D. A. 1988. The Psychology of Everyday Things . Basic Books , New York . Norman, D. A. 1988. The Psychology of Everyday Things. Basic Books, New York."},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Norman D. A. and Draper S. W. 1986. User Centered System Design: New Perspectives on Human-Computer Interaction. Lawrence Erlbaum Associates Inc Hillsdale NJ. Norman D. A. and Draper S. W. 1986. User Centered System Design: New Perspectives on Human-Computer Interaction . Lawrence Erlbaum Associates Inc Hillsdale NJ.","DOI":"10.1201\/b15703"},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Petrie H. and Bevan N. 2009. The evaluation of accessibility usability and user experience. In The Universal Access Handbook C. Stephanidis Ed. CRC Press London UK. Petrie H. and Bevan N. 2009. The evaluation of accessibility usability and user experience. In The Universal Access Handbook C. Stephanidis Ed. CRC Press London UK.","DOI":"10.1201\/9781420064995-c20"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Sauro J. and Lewis J. R. 2012. Quantifying the User Experience. Morgan Kaufmann Waltham MA. Sauro J. and Lewis J. R. 2012. Quantifying the User Experience . Morgan Kaufmann Waltham MA.","DOI":"10.1016\/B978-0-12-384968-7.00002-3"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1531514.1531527"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/1671011.1671034"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2133806.2133824"},{"key":"e_1_2_1_41_1","volume-title":"Designing the User Interface: Strategies for Effective Human-Computer Interaction","author":"Shneiderman B.","unstructured":"Shneiderman , B. 1986. Designing the User Interface: Strategies for Effective Human-Computer Interaction . Addison-Wesley Longman Publishing Co., Inc , Boston, MA . Shneiderman, B. 1986. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Longman Publishing Co., Inc, Boston, MA."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/634067.634236"},{"key":"e_1_2_1_43_1","volume-title":"User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan, Kaufmann.","author":"Tullis T.","year":"2008","unstructured":"Tullis , T. and Albert , W . 2008 . Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan, Kaufmann. Tullis, T. and Albert, W. 2008. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan, Kaufmann."},{"key":"e_1_2_1_44_1","unstructured":"Turner C. W. Lewis J. R. and Nielsen J. 2006. Determining usability test sample size. In International Encyclopedia of Ergonomics and Human Factors W. Karwowski Ed. CRC Press Boca Raton FL 3084--3088. Turner C. W. Lewis J. R. and Nielsen J. 2006. Determining usability test sample size. In International Encyclopedia of Ergonomics and Human Factors W. Karwowski Ed. CRC Press Boca Raton FL 3084--3088."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1177\/154193129003400411"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872089203400407"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the Proceedings of IHM-HCI 2001 Conference, J. Vanderdonckt, A. Blandford, and A. Derycke Eds., C\u00e9padu\u00e8s Editions","author":"Woolrych A.","unstructured":"Woolrych , A. and Cockton , G . 2001. Why and when five test users aren\u2019t enough . In Proceedings of the Proceedings of IHM-HCI 2001 Conference, J. Vanderdonckt, A. Blandford, and A. Derycke Eds., C\u00e9padu\u00e8s Editions , London, UK, 105--108. Woolrych, A. and Cockton, G. 2001. Why and when five test users aren\u2019t enough. In Proceedings of the Proceedings of IHM-HCI 2001 Conference, J. Vanderdonckt, A. Blandford, and A. Derycke Eds., C\u00e9padu\u00e8s Editions, London, UK, 105--108."}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2506210","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2506210","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:28:43Z","timestamp":1750231723000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2506210"}},"subtitle":["A Grounded Procedure for Interaction Evaluation"],"short-title":[],"issued":{"date-parts":[[2013,11]]},"references-count":47,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2013,11]]}},"alternative-id":["10.1145\/2506210"],"URL":"https:\/\/doi.org\/10.1145\/2506210","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,11]]},"assertion":[{"value":"2012-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}