{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T12:30:07Z","timestamp":1765888207242,"version":"3.41.0"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,9,9]],"date-time":"2019-09-09T00:00:00Z","timestamp":1567987200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1750987"],"award-info":[{"award-number":["1750987"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2019,9,9]]},"abstract":"<jats:p>Distinguishing identities is useful for several applications such as automated grocery or personalized recommendations. Unfortunately, several recent proposals for identification systems are evaluated using poor recruitment practices. We discovered that 23 out of 30 surveyed systems used datasets with 20 participants or less. Those studies achieved an average classification accuracy of 93%. We show that the classifier performance is misleading when the participant count is small. This is because the finite precision of measurements creates upper limits on the number of users that can be distinguished.<\/jats:p>\n          <jats:p>To demonstrate why classifier performance is misleading, we used publicly available datasets. The data was collected from human subjects. We created five systems with at least 20 participants each. In three cases we achieved accuracies greater than 90% by merely applying readily available machine learning software packages, often with default parameters. For datasets where we had sufficient participants, we evaluated how the performance degrades as the number of participants increases. One of the systems built suffered a drop in accuracy that was over 35% as the participant count increased from 20 to 250. We argue that data from small participant count datasets do not adequately explore variations. Systems trained on such limited data are likely to incorrectly identify users when the user base increases beyond what was tested. We conclude by explaining generalizable reasons for this issue and provide insights on how to conduct more robust system analysis and design.<\/jats:p>","DOI":"10.1145\/3351262","type":"journal-article","created":{"date-parts":[[2019,9,10]],"date-time":"2019-09-10T15:58:26Z","timestamp":1568131106000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Recruit Until It Fails"],"prefix":"10.1145","volume":"3","author":[{"given":"Shridatt","family":"Sugrim","sequence":"first","affiliation":[{"name":"Rutgers University, Piscataway, New Jersey"}]},{"given":"Can","family":"Liu","sequence":"additional","affiliation":[{"name":"Rutgers University, Piscataway, New Jersey"}]},{"given":"Janne","family":"Lindqvist","sequence":"additional","affiliation":[{"name":"Rutgers University, Piscataway, New Jersey"}]}],"member":"320","published-online":{"date-parts":[[2019,9,9]]},"reference":[{"edition":"1","volume-title":"Learning From Data","author":"Abu-Mostafa Yaser Said","key":"e_1_2_2_1_1"},{"key":"e_1_2_2_2_1","unstructured":"Amazon. 2019. http:\/\/amazon.com\/go.  Amazon. 2019. http:\/\/amazon.com\/go."},{"key":"e_1_2_2_3_1","unstructured":"Amazon. 2019. https:\/\/www.amazon.com\/.  Amazon. 2019. https:\/\/www.amazon.com\/."},{"key":"e_1_2_2_4_1","first-page":"1411","article-title":"Small Samples: Does Size Matter","volume":"42","author":"Anderson Andrew John","year":"2001","journal-title":"Investigative Ophthalmology and Visual Science"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126649"},{"key":"e_1_2_2_6_1","unstructured":"Davide Anguita Alessandro Ghio Luca Oneto Xavier Parra and Jorge Luis Reyes-Ortiz. 2013. A public domain dataset for human activity recognition using smartphones.. In ESANN.  Davide Anguita Alessandro Ghio Luca Oneto Xavier Parra and Jorge Luis Reyes-Ortiz. 2013. A public domain dataset for human activity recognition using smartphones.. In ESANN."},{"volume-title":"Practical Statistics for Data Scientists: 50 Essential Concepts. \" O'Reilly Media","author":"Bruce Peter","key":"e_1_2_2_7_1"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/nrn3475"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858498"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-011-0415-z"},{"volume-title":"Nonparametric estimation of Shannon\u00e2\u0102&Zacute;s index of diversity when there are unseen species in sample. Environmental and ecological statistics 10, 4","year":"2003","author":"Chao Anne","key":"e_1_2_2_11_1"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984541"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130906"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025991"},{"key":"e_1_2_2_15_1","unstructured":"Dua Dheeru and Efi Karra Taniskidou. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml  Dua Dheeru and Efi Karra Taniskidou. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"e_1_2_2_16_1","unstructured":"FaceBook. 2019. https:\/\/www.facebook.com\/business\/ads.  FaceBook. 2019. https:\/\/www.facebook.com\/business\/ads."},{"volume-title":"Did I do that? Abnormal predictive processes in schizophrenia when button pressing to deliver a tone. Schizophrenia bulletin 40, 4","year":"2013","author":"Ford Judith M","key":"e_1_2_2_17_1"},{"volume-title":"The elements of statistical learning","author":"Friedman Jerome","key":"e_1_2_2_18_1"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2017.09.005"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523813"},{"volume-title":"NBA Players stats since","year":"1950","author":"Goldstein Omri","key":"e_1_2_2_21_1"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126615"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/2044749.2044829"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2906388.2906402"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1183\/09031936.00136408"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328919"},{"key":"e_1_2_2_27_1","first-page":"61","article-title":"Fundamentals of statistical signal processing: Detection theory, vol. 2. Prentice Hall Upper Saddle River, NJ, USA","volume":"3","author":"Kay Steven M","year":"1998","journal-title":"Chapter"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3098279.3098553"},{"volume-title":"Determining sample size for research activities. Educational and psychological measurement 30, 3","year":"1970","author":"Krejcie Robert V","key":"e_1_2_2_29_1"},{"volume-title":"Introduction to sample size determination and power analysis for clinical trials. Controlled clinical trials 2, 2","year":"1981","author":"Lachin John M","key":"e_1_2_2_30_1"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3191755"},{"volume-title":"Power analysis and determination of sample size for covariance structure modeling. Psychological methods 1, 2","year":"1996","author":"MacCallum Robert C","key":"e_1_2_2_32_1"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126604"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025807"},{"key":"e_1_2_2_35_1","unstructured":"N\/A. 2018. Data.gov. https:\/\/www.data.gov  N\/A. 2018. Data.gov. https:\/\/www.data.gov"},{"key":"e_1_2_2_36_1","unstructured":"Netflix. 2019. https:\/\/help.netflix.com\/en\/node\/100639.  Netflix. 2019. https:\/\/help.netflix.com\/en\/node\/100639."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300340"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025678"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.75512"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/245108.245121"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00283-013-9389-1"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971736"},{"volume-title":"Sample size in qualitative research. Research in nursing & health 18, 2","year":"1995","author":"Sandelowski Margarete","key":"e_1_2_2_44_1"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025536"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858152"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173705"},{"volume-title":"Computational Methods in Biometric Authentication","author":"Schuckers Michael E.","key":"e_1_2_2_48_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-84996-202-5"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264944"},{"volume-title":"Measurement of diversity. Nature 163, 4148","year":"1949","author":"Simpson Edward H","key":"e_1_2_2_50_1"},{"key":"e_1_2_2_51_1","unstructured":"Spotify. 2019. https:\/\/www.spotify.com\/.  Spotify. 2019. https:\/\/www.spotify.com\/."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287067"},{"volume-title":"Robust Performance Metrics for Authentication Systems. In Network and Distributed Systems Security (NDSS) Symposium","year":"2019","author":"Sugrim Shridatt","key":"e_1_2_2_53_1"},{"key":"e_1_2_2_54_1","unstructured":"Twitter. 2019. https:\/\/ads.twitter.com\/.  Twitter. 2019. https:\/\/ads.twitter.com\/."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3214287"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3210240.3210335"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3191772"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3081333.3081340"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2018.8486346"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984565"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971670"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130975"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287071"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858466"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130985"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174011"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264959"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2018.8486006"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287080"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3351262","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3351262","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3351262","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:25:51Z","timestamp":1750206351000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3351262"}},"subtitle":["Exploring Performance Limits for Identification Systems"],"short-title":[],"issued":{"date-parts":[[2019,9,9]]},"references-count":69,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9,9]]}},"alternative-id":["10.1145\/3351262"],"URL":"https:\/\/doi.org\/10.1145\/3351262","relation":{},"ISSN":["2474-9567"],"issn-type":[{"type":"electronic","value":"2474-9567"}],"subject":[],"published":{"date-parts":[[2019,9,9]]},"assertion":[{"value":"2019-09-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}