{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T21:40:21Z","timestamp":1771623621834,"version":"3.50.1"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,10,15]],"date-time":"2021-10-15T00:00:00Z","timestamp":1634256000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["1763569"],"award-info":[{"award-number":["1763569"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Access. Comput."],"published-print":{"date-parts":[[2021,12,31]]},"abstract":"<jats:p>Advances in sign-language recognition technology have enabled researchers to investigate various methods that can assist users in searching for an unfamiliar sign in ASL using sign-recognition technology. Users can generate a query by submitting a video of themselves performing the sign they believe they encountered somewhere and obtain a list of possible matches. However, there is disagreement among developers of such technology on how to report the performance of their systems, and prior research has not examined the relationship between the performance of search technology and users\u2019 subjective judgements for this task. We conducted three studies using a Wizard-of-Oz prototype of a webcam-based ASL dictionary search system to investigate the relationship between the performance of such a system and user judgements. We found that, in addition to the position of the desired word in a list of results, the placement of the desired word above or below the fold and the similarity of the other words in the results list affected users\u2019 judgements of the system. We also found that metrics that incorporate the precision of the overall list correlated better with users\u2019 judgements than did metrics currently reported in prior ASL dictionary research.<\/jats:p>","DOI":"10.1145\/3470650","type":"journal-article","created":{"date-parts":[[2021,10,16]],"date-time":"2021-10-16T01:21:13Z","timestamp":1634347273000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary Search"],"prefix":"10.1145","volume":"14","author":[{"given":"Saad","family":"Hassan","sequence":"first","affiliation":[{"name":"School of Information, Rochester, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Oliver","family":"Alonzo","sequence":"additional","affiliation":[{"name":"School of Information, Rochester, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abraham","family":"Glasser","sequence":"additional","affiliation":[{"name":"School of Information, Rochester, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matt","family":"Huenerfauth","sequence":"additional","affiliation":[{"name":"School of Information, Rochester, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,15]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277902"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308561.3353791"},{"key":"e_1_3_3_4_2","first-page":"11","volume-title":"Proceedings of the Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies","volume":"2","author":"Athitsos Vassilis","year":"2010","unstructured":"Vassilis Athitsos, Carol Neidle, Stan Sclaroff, Joan Nash, Alexandra Stefan, Ashwin Thangali, Haijing Wang, and Quan Yuan. 2010. Large lexicon project: American sign language video corpus and sign language indexing\/retrieval algorithms. In Proceedings of the Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies (CSLT\u201910), Vol. 2. European Language Resources Association (ELRA), Valletta, Malta, 11\u201314."},{"issue":"260","key":"e_1_3_3_5_2","article-title":"Summary health statistics for U.S. adults: National health interview survey, 2012","volume":"1","author":"Blackwell Debra L.","year":"2014","unstructured":"Debra L. Blackwell, Jacqueline W. Lucas, and Tainya C. Clarke. 2014. Summary health statistics for U.S. adults: National health interview survey, 2012. Vital and Health Statistics. Series 10, Data from the National Health Survey 1, 260 (Feb. 2014), 171. Retrieved from https:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24819891.","journal-title":"Vital and Health Statistics. Series 10, Data from the National Health Survey"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675226"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/792550.792552"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-016-0742-0"},{"key":"e_1_3_3_9_2","unstructured":"Dorothy Casterline Carl Croneberg et\u00a0al. 1965. A dictionary of American Sign Language on linguistic principles. Gallaudet College."},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","unstructured":"Matias D. Cattaneo Nicolas Idrobo and Rocio Titiunik. 2019. A Practical Introduction to Regression Discontinuity Designs: Foundations . Retrieved from https:\/\/ideas.repec.org\/p\/arx\/papers\/1911.09511.html.","DOI":"10.1017\/9781108684606"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2769493.2769534"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2011.6130349"},{"key":"e_1_3_3_13_2","first-page":"92","volume-title":"Proceedings of the Modern Language Association of America","author":"Looney Natalia Lusin and Dennis","year":"2018","unstructured":"Natalia Lusin and Dennis Looney. 2018. Enrollments in languages other than english in united states institutions of higher education, summer 2016 and fall 2016: Preliminary report. In Proceedings of the Modern Language Association of America. 92. Retrieved from https:\/\/www.mla.org\/content\/download\/110154\/2406932\/2016-Enrollments-Final-Report.pdf."},{"key":"e_1_3_3_14_2","volume-title":"Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology","author":"Elliott Ralph","year":"2011","unstructured":"Ralph Elliott, Helen Cooper, John Glauert, Richard Bowden, and Fran\u00e7ois Lefebvre-Albaret. 2011. Search-by-example in multilingual sign language databases. In Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology (SLTAT\u201911). SLTAT, Dundee, Scotland, 8. Retrieved from http:\/\/personal.ee.surrey.ac.uk\/Personal\/H.Cooper\/research\/papers\/SBE_SLTAT.pdf."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277839"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1093\/restud\/rdr043"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2854946.2855005"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076063"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00013-4"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767732"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23187"},{"key":"e_1_3_3_23_2","unstructured":"J. Lapiak. 2021. Handspeak. Retrieved from https:\/\/www.handspeak.com\/."},{"key":"e_1_3_3_24_2","first-page":"404","article-title":"Grammar, gesture, and meaning in american sign language","volume":"1","author":"Liddell Scott","year":"2003","unstructured":"Scott Liddell. 2003. Grammar, gesture, and meaning in american sign language. Grammar, Gesture, and Meaning in American Sign Language 1 (Mar. 2003), 404. https:\/\/doi.org\/10.1017\/CBO9780511615054","journal-title":"Grammar, Gesture, and Meaning in American Sign Language"},{"key":"e_1_3_3_25_2","volume-title":"Proceedings of the 11th International Conference on Language Resources and Evaluation","author":"Metaxas Dimitris","year":"2018","unstructured":"Dimitris Metaxas, Mark Dilsizian, and Carol Neidle. 2018. Linguistically-driven framework for computationally efficient and scalable sign recognition. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC\u201918). European Language Resources Association (ELRA), Miyazaki, Japan, 8. Retrieved from https:\/\/www.aclweb.org\/anthology\/L18-1271."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1353\/sls.2004.0005"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1353\/sls.2006.0019"},{"issue":"1","key":"e_1_3_3_28_2","first-page":"31","article-title":"American sign language","volume":"1","author":"Neidle Carol","year":"2015","unstructured":"Carol Neidle and Joan Cottle Poole Nash. 2015. American sign language. Jullie Bakken Jepsen, Goedele De Clerck, Sam Lutalo Kiingi, and William B. McGregor (eds.), Sign Languages of the World 1, 1 (2015), 31\u201370.","journal-title":"Jullie Bakken Jepsen, Goedele De Clerck, Sam Lutalo Kiingi, and William B. McGregor (eds.), Sign Languages of the World"},{"key":"e_1_3_3_29_2","volume-title":"Proceedings of the 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon","author":"Neidle Carol","year":"2012","unstructured":"Carol Neidle and Christian Vogler. 2012. A new web interface to facilitate access to corpora: Development of the ASLLRP data access interface (DAI). In Proceedings of the 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon (LREC\u201912). Citeseer, OpenBU, Istanbul, Turkey, 8. https:\/\/open.bu.edu\/handle\/2144\/31886."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572030"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2838739.2838825"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00138-012-0450-4"},{"key":"e_1_3_3_33_2","first-page":"9","volume-title":"Proceedings of the Joint Statistical Meeting","author":"Robbins Naomi","year":"2011","unstructured":"Naomi Robbins and Richard Heiberger. 2011. Plotting likert and other rating scales. Proceedings of the Joint Statistical Meeting. 9."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/988672.988675"},{"key":"e_1_3_3_35_2","unstructured":"ShuR. 2021. SLintoDictionary. Retrieved from http:\/\/slinto.com\/us."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogpsych.2004.05.001"},{"key":"e_1_3_3_37_2","unstructured":"Richard A. Tennant Marianne Gluszak and Marianne Gluszak Brown. 1998. The American Sign Language Handshape Dictionary . Gallaudet University Press Washington D.C."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2008.4761363"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35749-7_27"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2049536.2049554"},{"key":"e_1_3_3_41_2","first-page":"3067","volume-title":"Proceedings of the 10th International Conference on Language Resources and Evaluation","author":"Yanovich Polina","year":"2016","unstructured":"Polina Yanovich, Carol Neidle, and Dimitris Metaxas. 2016. Detection of major ASL sign types in continuous signing for ASL recognition. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC\u201916). European Language Resources Association (ELRA), Portoro\u017e, Slovenia, 3067\u20133073. Retrieved from https:\/\/www.aclweb.org\/anthology\/L16-1490."}],"container-title":["ACM Transactions on Accessible Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470650","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3470650","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3470650","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:55Z","timestamp":1750191535000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470650"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,15]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12,31]]}},"alternative-id":["10.1145\/3470650"],"URL":"https:\/\/doi.org\/10.1145\/3470650","relation":{},"ISSN":["1936-7228","1936-7236"],"issn-type":[{"value":"1936-7228","type":"print"},{"value":"1936-7236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,15]]},"assertion":[{"value":"2020-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}