{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,8]],"date-time":"2025-09-08T06:12:46Z","timestamp":1757311966585,"version":"3.41.0"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2013,10,1]],"date-time":"2013-10-01T00:00:00Z","timestamp":1380585600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100006462","name":"City University of New York","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006462","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["0746556, 1065009"],"award-info":[{"award-number":["0746556, 1065009"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Visage Technologies AB"},{"DOI":"10.13039\/501100004830","name":"Siemens","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004830","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Access. Comput."],"published-print":{"date-parts":[[2013,10]]},"abstract":"<jats:p>Many researchers internationally are studying how to synthesize computer animations of sign language; such animations have accessibility benefits for people who are deaf and have lower literacy in written languages. The field has not yet formed a consensus as to how to best conduct evaluations of the quality of sign language animations, and this article explores an important methodological issue for researchers conducting experimental studies with participants who are deaf. Traditionally, when evaluating an animation, some lower and upper baselines are shown for comparison during the study. For the upper baseline, some researchers use carefully produced animations, and others use videos of human signers. Specifically, this article investigates, in studies where signers view animations of sign language and are asked subjective and comprehension questions, whether participants differ in their subjective and comprehension responses when actual videos of human signers are shown during the study. Through three sets of experiments, we characterize how the Likert-scale subjective judgments of participants about sign language animations are negatively affected when they are also shown videos of human signers for comparison -- especially when displayed side-by-side. We also identify a small positive effect on the comprehension of sign language animations when studies also contain videos of human signers. Our results enable direct comparison of previously published evaluations of sign language animations that used different types of upper baselines -- video or animation. Our results also provide methodological guidance for researchers who are designing evaluation studies of sign language animation or designing experimental stimuli or questions for participants who are deaf.<\/jats:p>","DOI":"10.1145\/2517038","type":"journal-article","created":{"date-parts":[[2013,10,17]],"date-time":"2013-10-17T12:23:34Z","timestamp":1382012614000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Effect of Displaying Human Videos During an Evaluation Study of American Sign Language Animation"],"prefix":"10.1145","volume":"5","author":[{"given":"Hernisa","family":"Kacorri","sequence":"first","affiliation":[{"name":"The City University of New York, Graduate Center"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengfei","family":"Lu","sequence":"additional","affiliation":[{"name":"The City University of New York, Graduate Center"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matt","family":"Huenerfauth","sequence":"additional","affiliation":[{"name":"The City University of New York, Queens College"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2013,10]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Facial Animation: The Standard, Implementations and Applications","author":"Ahlberg J.","year":"2002","unstructured":"Ahlberg , J. , Pandzic , I. S. , and You , L . 2002 . Evaluating face models animated by MPEG-4. In MPEG-4 Facial Animation: The Standard, Implementations and Applications , I. S. Pandzic and R. Forchheimer Eds., Wiley , 291--296. Ahlberg, J., Pandzic, I. S., and You, L. 2002. Evaluating face models animated by MPEG-4. In MPEG-4 Facial Animation: The Standard, Implementations and Applications, I. S. Pandzic and R. Forchheimer Eds., Wiley, 291--296."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03655-2_23"},{"key":"e_1_2_1_3_1","volume-title":"Maxine: Embodied conversational agents for multimodal affective communication","author":"Baldassarri S.","year":"2012","unstructured":"Baldassarri , S. and Cerezo , E . 2012 . Maxine: Embodied conversational agents for multimodal affective communication . In Computer Graphics, N. Mukai Ed. Baldassarri, S. and Cerezo, E. 2012. Maxine: Embodied conversational agents for multimodal affective communication. In Computer Graphics, N. Mukai Ed."},{"volume-title":"The production of co-speech iconic gestures: Empirical study and computational simulation with virtual agents. Dissertation","author":"Bergmann K.","key":"e_1_2_1_4_1","unstructured":"Bergmann , K. 2012. The production of co-speech iconic gestures: Empirical study and computational simulation with virtual agents. Dissertation , Bielefeld University , Germany. Bergmann, K. 2012. The production of co-speech iconic gestures: Empirical study and computational simulation with virtual agents. Dissertation, Bielefeld University, Germany."},{"volume-title":"Proceedings of the DePaul CTI Research Conference.","author":"Davidson M. J.","key":"e_1_2_1_5_1","unstructured":"Davidson , M. J. , Alkoby , K. , Sedgwick , E. , Berthiaume , A. , Carter , R. , Christopher , J. , Craft , B. , Furst , J. , Hinkle , D. , Konie , B. , Lancaster , G. , Luecking , S. , Morris , A. , McDonald , J. , Tomuro , N. , Toro , J. , and Wolfe , R . 2000. Usability testing of computer animation of fingerspelling for American Sign Language . In Proceedings of the DePaul CTI Research Conference. Davidson, M. J., Alkoby, K., Sedgwick, E., Berthiaume, A., Carter, R., Christopher, J., Craft, B., Furst, J., Hinkle, D., Konie, B., Lancaster, G., Luecking, S., Morris, A., McDonald, J., Tomuro, N., Toro, J., and Wolfe, R. 2000. Usability testing of computer animation of fingerspelling for American Sign Language. In Proceedings of the DePaul CTI Research Conference."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-007-0102-z"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1093\/deafed\/enn037"},{"volume-title":"Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Language Resources and Evaluation Conference (LREC).","author":"Filhol M.","key":"e_1_2_1_8_1","unstructured":"Filhol , M. , Delorme , M. , and Braffort , A . 2010. Combining constraint-based models for sign language synthesis . In Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Language Resources and Evaluation Conference (LREC). Filhol, M., Delorme, M., and Braffort, A. 2010. Combining constraint-based models for sign language synthesis. In Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Language Resources and Evaluation Conference (LREC)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-007-0094-8"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/365024.365121"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2030365.2030371"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/11550617_32"},{"key":"e_1_2_1_14_1","first-page":"1","article-title":"Sign language in the interface: Access for deaf signers. In Universal Access Handbook, C. Stephanidis Ed","volume":"38","author":"Huenerfauth M.","year":"2009","unstructured":"Huenerfauth , M. and Hanson , V. 2009 . Sign language in the interface: Access for deaf signers. In Universal Access Handbook, C. Stephanidis Ed ., Erlbaum , 38 . 1 -- 38 .18. Huenerfauth, M. and Hanson, V. 2009. Sign language in the interface: Access for deaf signers. In Universal Access Handbook, C. Stephanidis Ed., Erlbaum, 38.1--38.18.","journal-title":"Erlbaum"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1878803.1878823"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-011-0247-7"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1361203.1361206"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2049536.2049556"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39188-0_55"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1279700.1279705"},{"key":"e_1_2_1_21_1","volume-title":"Intelligent Virtual Agents","volume":"6895","author":"Kipp M.","unstructured":"Kipp , M. , Heloir , A. , and Nguyen , Q . 2011a. Sign language avatars: Animation and comprehensibility . In Intelligent Virtual Agents , vol. 6895 . H. Vilhj\u00e1lmsson et al. Eds., Springer, Berlin, 113--126. Kipp, M., Heloir, A., and Nguyen, Q. 2011a. Sign language avatars: Animation and comprehensibility. In Intelligent Virtual Agents, vol. 6895. H. Vilhj\u00e1lmsson et al. Eds., Springer, Berlin, 113--126."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2049536.2049557"},{"key":"e_1_2_1_23_1","first-page":"399","article-title":"The synthesis of LSE classifiers: From representation to evaluation","volume":"17","author":"L\u00f3pez-Colino F.","year":"2011","unstructured":"L\u00f3pez-Colino , F. and Col\u00e1s , J. 2011 . The synthesis of LSE classifiers: From representation to evaluation . J. Univ. Comput. Sci. 17 , 3, 399 -- 425 . L\u00f3pez-Colino, F. and Col\u00e1s, J. 2011. The synthesis of LSE classifiers: From representation to evaluation. J. Univ. Comput. Sci. 17, 3, 399--425.","journal-title":"J. Univ. Comput. Sci."},{"volume-title":"Proceedings of the 1st Workshop on Speech and Language Processing for Assistive Technologies (SLPAT).","author":"Lu P.","key":"e_1_2_1_24_1","unstructured":"Lu , P. and Huenerfauth , M . 2010. Collecting a motion-capture corpus of American sign language for data-driven generation research . In Proceedings of the 1st Workshop on Speech and Language Processing for Assistive Technologies (SLPAT). Lu, P. and Huenerfauth, M. 2010. Collecting a motion-capture corpus of American sign language for data-driven generation research. In Proceedings of the 1st Workshop on Speech and Language Processing for Assistive Technologies (SLPAT)."},{"volume-title":"Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology (SLTAT), in conjunction with ASSETS\u201911","author":"Lu P.","key":"e_1_2_1_25_1","unstructured":"Lu , P. and Huenerfauth , M . 2011. Synthesizing American sign language spatially inflected verbs from motion-capture data . In Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology (SLTAT), in conjunction with ASSETS\u201911 . Lu, P. and Huenerfauth, M. 2011. Synthesizing American sign language spatially inflected verbs from motion-capture data. In Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology (SLTAT), in conjunction with ASSETS\u201911."},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Lu P. and Huenerfauth M. 2013. Collecting and evaluating the CUNY ASL corpus for research on American sign language animation. Submitted for publication.  Lu P. and Huenerfauth M. 2013. Collecting and evaluating the CUNY ASL corpus for research on American sign language animation. Submitted for publication.","DOI":"10.1016\/j.csl.2013.10.004"},{"volume-title":"Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies (SLPAT).","author":"Lu P.","key":"e_1_2_1_27_1","unstructured":"Lu , P. and Huenerfauth , M . 2012. Learning a vector-based model of American sgn language inflecting verbs from motion-capture data . In Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies (SLPAT). Lu, P. and Huenerfauth, M. 2012. Learning a vector-based model of American sgn language inflecting verbs from motion-capture data. In Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies (SLPAT)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2384916.2384949"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1394281.1394294"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1353\/sls.2006.0019"},{"volume-title":"Proceedings of the Asia Design Conference.","author":"Morimoto K.","key":"e_1_2_1_31_1","unstructured":"Morimoto , K. , Kurokawa , T. , Kentarou , U. , Teruyo , K. , Kazushi , T. , Katsuo , N. , and Tamotsu , F . 2003. Design of an agent to represent Japanese sign language for hearing-impaired people in stomach x-ray inspection . In Proceedings of the Asia Design Conference. Morimoto, K., Kurokawa, T., Kentarou, U., Teruyo, K., Kazushi, T., Katsuo, N., and Tamotsu, F. 2003. Design of an agent to represent Japanese sign language for hearing-impaired people in stomach x-ray inspection. In Proceedings of the Asia Design Conference."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/11788713_90"},{"key":"e_1_2_1_33_1","unstructured":"Neidle C. Kegl J. MacLaughlin D. Bahan B. and Lee R. G. 2000. The Syntax of ASL: Functional Categories And Hierarchical Structure. MIT Press Cambridge MA.  Neidle C. Kegl J. MacLaughlin D. Bahan B. and Lee R. G. 2000. The Syntax of ASL: Functional Categories And Hierarchical Structure . MIT Press Cambridge MA."},{"key":"e_1_2_1_34_1","first-page":"34","article-title":"User evaluation of an electronic Malaysian sign language dictionary: E-sign dictionary","volume":"2","author":"Ow S. H.","year":"2009","unstructured":"Ow , S. H. 2009 . User evaluation of an electronic Malaysian sign language dictionary: E-sign dictionary . Comput. Inf. Sci. 2 , 2, 34 -- 52 . Ow, S. H. 2009. User evaluation of an electronic Malaysian sign language dictionary: E-sign dictionary. Comput. Inf. Sci. 2, 2, 34--52.","journal-title":"Comput. Inf. Sci."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1899950.1899980"},{"key":"e_1_2_1_36_1","first-page":"3","article-title":"Computer-based signing accommodations: Comparing a recorded human with an avatar","volume":"10","author":"Russell M.","year":"2009","unstructured":"Russell , M. , Kavanaugh , M. , Masters , J. , Higgins , J. , and Hoffmann , T. 2009 . Computer-based signing accommodations: Comparing a recorded human with an avatar . J. Appl. Testing Technol. 10 , 3 , 21. Russell, M., Kavanaugh, M., Masters, J., Higgins, J., and Hoffmann, T. 2009. Computer-based signing accommodations: Comparing a recorded human with an avatar. J. Appl. Testing Technol. 10, 3, 21.","journal-title":"J. Appl. Testing Technol."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2049536.2049594"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages (LREC\u201910)","author":"Schnepp J.","year":"2010","unstructured":"Schnepp , J. , Wolfe , R. , and McDonald , J. 2010 . Synthetic corpora: A synergy of linguistics and computer animation . In Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages (LREC\u201910) . Schnepp, J., Wolfe, R., and McDonald, J. 2010. Synthetic corpora: A synergy of linguistics and computer animation. In Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages (LREC\u201910)."},{"volume-title":"Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology.","author":"Schnepp J.","key":"e_1_2_1_39_1","unstructured":"Schnepp , J. , Wolfe , R. , Shiver , B. , McDonald , J. , and Toro , J . 2011. SignQUOTE: A remote testing facility for eliciting signed qualitative feedback . In Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology. Schnepp, J., Wolfe, R., Shiver, B., McDonald, J., and Toro, J. 2011. SignQUOTE: A remote testing facility for eliciting signed qualitative feedback. In Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01068419"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1093\/deafed\/5.4.337"},{"key":"e_1_2_1_42_1","unstructured":"VCOM3D. 2012. Homepage. http:\/\/www.vcom3d.com\/.  VCOM3D. 2012. Homepage. http:\/\/www.vcom3d.com\/."}],"container-title":["ACM Transactions on Accessible Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2517038","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2517038","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:28:40Z","timestamp":1750231720000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2517038"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,10]]},"references-count":41,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2013,10]]}},"alternative-id":["10.1145\/2517038"],"URL":"https:\/\/doi.org\/10.1145\/2517038","relation":{},"ISSN":["1936-7228","1936-7236"],"issn-type":[{"type":"print","value":"1936-7228"},{"type":"electronic","value":"1936-7236"}],"subject":[],"published":{"date-parts":[[2013,10]]},"assertion":[{"value":"2013-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}