{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:48:52Z","timestamp":1776084532653,"version":"3.50.1"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,6,11]],"date-time":"2019-06-11T00:00:00Z","timestamp":1560211200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Google Faculty Research Award"},{"name":"National Technical Institute for the Deaf"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Access. Comput."],"published-print":{"date-parts":[[2019,6,30]]},"abstract":"<jats:p>Automatic Speech Recognition (ASR) technology has seen major advancements in its accuracy and speed in recent years, making it a possible mechanism for supporting communication between people who are Deaf or Hard-of-Hearing (DHH) and their hearing peers. However, state-of-the-art ASR technology is still imperfect in many realistic settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but it has been found to have little correlation with human-subject performance for many applications. This article describes and evaluates several new captioning-focused evaluation metrics for predicting the impact of ASR errors on the understandability of automatically generated captions for people who are DHH. Through experimental studies with DHH users, we have found that our new metric (based on word-importance and semantic-difference scoring) is more closely correlated with DHH user's judgements of caption quality\u2014as compared to pre-existing metrics for ASR evaluation.<\/jats:p>","DOI":"10.1145\/3325862","type":"journal-article","created":{"date-parts":[[2019,6,11]],"date-time":"2019-06-11T13:28:16Z","timestamp":1560259696000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Predicting the Understandability of Imperfect English Captions for People Who Are Deaf or Hard of Hearing"],"prefix":"10.1145","volume":"12","author":[{"given":"Sushant","family":"Kafle","sequence":"first","affiliation":[{"name":"Rochester Institute of Technology, Rochester, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matt","family":"Huenerfauth","sequence":"additional","affiliation":[{"name":"Rochester Institute of Technology, Rochester, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,6,11]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Caption accuracy metrics project. Caption viewer survey: Error ranking of real-time captions in live television news programs","author":"Apone Tom","unstructured":"Tom Apone , Marcia Brooks , and Trisha O'Connell . 2010. Caption accuracy metrics project. Caption viewer survey: Error ranking of real-time captions in live television news programs . WGBH National Center for Accessible Media Boston . Retrieved from https:\/\/dcmp.org\/learn\/static-assets\/nadh300.pdf. Tom Apone, Marcia Brooks, and Trisha O'Connell. 2010. Caption accuracy metrics project. Caption viewer survey: Error ranking of real-time captions in live television news programs. WGBH National Center for Accessible Media Boston. Retrieved from https:\/\/dcmp.org\/learn\/static-assets\/nadh300.pdf."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/638249.638284"},{"key":"e_1_2_2_3_1","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. 65--72","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie . 2005 . METEOR: An automatic metric for MT evaluation with improved correlation with human judgements . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. 65--72 . Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgements. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. 65--72."},{"key":"e_1_2_2_4_1","volume-title":"New Era for Robust Speech Recognition, Exploiting Deep Learning","author":"Barker Jon P.","unstructured":"Jon P. Barker , Ricard Marxer , Emmanuel Vincent , and Shinji Watanabe . 2017. The CHiME challenges: Robust speech recognition in everyday environments . In New Era for Robust Speech Recognition, Exploiting Deep Learning . Springer , 327--344. Jon P. Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe. 2017. The CHiME challenges: Robust speech recognition in everyday environments. In New Era for Robust Speech Recognition, Exploiting Deep Learning. Springer, 327--344."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/13506285.2013.804016"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132525.3132541"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173665"},{"key":"e_1_2_2_8_1","volume-title":"Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL\u201907)","author":"Brants Thorsten","year":"2007","unstructured":"Thorsten Brants , Ashok C. Popat , Peng Xu , Franz J. Och , and Jeffrey Dean . 2007 . Large language models in machine translation . In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL\u201907) . Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL\u201907)."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0127577"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1093\/deafed\/enp033"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ridd.2014.03.039"},{"key":"e_1_2_2_12_1","doi-asserted-by":"crossref","unstructured":"J. R. Duffy and T. G. Giolas. 1971. The Effect of Word Predictability on Sentence Intelligibility. Technical report Submarine Medical Research Laboratory.  J. R. Duffy and T. G. Giolas. 1971. The Effect of Word Predictability on Sentence Intelligibility. Technical report Submarine Medical Research Laboratory.","DOI":"10.21236\/AD0746118"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0005640"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2982142.2982198"},{"key":"e_1_2_2_15_1","volume-title":"Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13)","author":"Favre Beno\u00eet","year":"2013","unstructured":"Beno\u00eet Favre , Kyla Cheung , Siavash Kazemian , Adam Lee , Yang Liu , Cosmin Munteanu , Ani Nenkova , Dennis Ochei , Gerald Penn , Stephen Tratz , Clare R. Voss , and Frauke Zeller . 2013 . Automatic human utility evaluation of ASR systems: Does WER really predict performance? In Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13) . 3463--3467. Retrieved from http:\/\/www.isca-speech.org\/archive\/interspeech_ 2013\/i13_3463.html. Beno\u00eet Favre, Kyla Cheung, Siavash Kazemian, Adam Lee, Yang Liu, Cosmin Munteanu, Ani Nenkova, Dennis Ochei, Gerald Penn, Stephen Tratz, Clare R. Voss, and Frauke Zeller. 2013. Automatic human utility evaluation of ASR systems: Does WER really predict performance? In Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13). 3463--3467. Retrieved from http:\/\/www.isca-speech.org\/archive\/interspeech_2013\/i13_3463.html."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207016.2207053"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2384916.2384966"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/23273798.2017.1323109"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-005-0005-9"},{"key":"e_1_2_2_20_1","volume-title":"Proceedings of the 6th International Conference Computer-Assisted Information Retrieval (Recherche d'Information et ses Applications) (RIAO'00)","author":"Garofolo John S.","unstructured":"John S. Garofolo , Cedric G. P. Auzanne , and Ellen M. Voorhees . 2000. The TREC Spoken Document Retrieval Track: A Success Story . In Proceedings of the 6th International Conference Computer-Assisted Information Retrieval (Recherche d'Information et ses Applications) (RIAO'00) . 1--20. John S. Garofolo, Cedric G. P. Auzanne, and Ellen M. Voorhees. 2000. The TREC Spoken Document Retrieval Track: A Success Story. In Proceedings of the 6th International Conference Computer-Assisted Information Retrieval (Recherche d'Information et ses Applications) (RIAO'00). 1--20."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2899475.2899478"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1992.225858"},{"key":"e_1_2_2_24_1","volume-title":"Entropy and Information Theory","author":"Gray Robert M.","unstructured":"Robert M. Gray . 2011. Entropy and Information Theory . Springer Science 8 Business Media. Robert M. Gray. 2011. Entropy and Information Theory. Springer Science 8 Business Media."},{"key":"e_1_2_2_25_1","volume-title":"Proceedings of the 4th Workshop on Child, Computer and Interaction (WOCCI'14)","author":"Gray Sharmistha S.","year":"2014","unstructured":"Sharmistha S. Gray , Daniel Willett , Jianhua Lu , Joel Pinto , Paul Maergner , and Nathan Bodenstab . 2014 . Child automatic speech recognition for US English: Child interaction with living-room-electronic-devices . In Proceedings of the 4th Workshop on Child, Computer and Interaction (WOCCI'14) . 21--26. Retrieved from http:\/\/www.isca-speech.org\/archive\/wocci_ 2014\/wc14_021.html. Sharmistha S. Gray, Daniel Willett, Jianhua Lu, Joel Pinto, Paul Maergner, and Nathan Bodenstab. 2014. Child automatic speech recognition for US English: Child interaction with living-room-electronic-devices. In Proceedings of the 4th Workshop on Child, Computer and Interaction (WOCCI'14). 21--26. Retrieved from http:\/\/www.isca-speech.org\/archive\/wocci_2014\/wc14_021.html."},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS\u201913)","author":"Harrington Rebecca Perkins","unstructured":"Rebecca Perkins Harrington and Gregg C. Vanderheiden . 2013. Crowd caption correction (CCC) . In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS\u201913) . 45:1--45:2. Rebecca Perkins Harrington and Gregg C. Vanderheiden. 2013. Crowd caption correction (CCC). In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS\u201913). 45:1--45:2."},{"key":"e_1_2_2_27_1","doi-asserted-by":"crossref","unstructured":"Nobuyasu Itoh Gakuto Kurata Ryuki Tachibana and Masafumi Nishimura. 2014. A metric for evaluating speech recognition accuracy based on human perception model. In Proceeding of INTERSPEECH.  Nobuyasu Itoh Gakuto Kurata Ryuki Tachibana and Masafumi Nishimura. 2014. A metric for evaluating speech recognition accuracy based on human perception model. In Proceeding of INTERSPEECH.","DOI":"10.21437\/Interspeech.2015-321"},{"key":"e_1_2_2_28_1","volume-title":"Smith","author":"Jackson Dorothy W.","year":"1997","unstructured":"Dorothy W. Jackson , Peter V. Paul , and Jonathan C . Smith . 1997 . Prior knowledge and reading comprehension ability of deaf adolescents. J. Deaf Studies Deaf Educat . (1997), 172--184. Dorothy W. Jackson, Peter V. Paul, and Jonathan C. Smith. 1997. Prior knowledge and reading comprehension ability of deaf adolescents. J. Deaf Studies Deaf Educat. (1997), 172--184."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.21437\/SLPAT.2016-4"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132525.3132542"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2982142.2982164"},{"key":"e_1_2_2_32_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba . 2014 . Adam : A method for stochastic optimization. Retrieved from http:\/\/arxiv.org\/abs\/1412.6980. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Retrieved from http:\/\/arxiv.org\/abs\/1412.6980."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/09541440340000213"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2543578"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380122"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594459"},{"key":"e_1_2_2_37_1","volume-title":"Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13)","author":"Lei Xin","year":"2013","unstructured":"Xin Lei , Andrew W. Senior , Alexander Gruenstein , and Jeffrey Sorensen . 2013 . Accurate and compact large vocabulary speech recognition on mobile devices . In Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13) . 662--665. http:\/\/www.isca-speech.org\/archive\/interspeech_2013\/i13_0662.html. Xin Lei, Andrew W. Senior, Alexander Gruenstein, and Jeffrey Sorensen. 2013. Accurate and compact large vocabulary speech recognition on mobile devices. In Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13). 662--665. http:\/\/www.isca-speech.org\/archive\/interspeech_2013\/i13_0662.html."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2014.2304637"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1353\/aad.0.0006"},{"key":"e_1_2_2_40_1","unstructured":"Iain A. McCowan Darren Moore John Dines Daniel Gatica-Perez Mike Flynn Pierre Wellner and Herv\u00e9 Bourlard. 2004. On the use of Information Retrieval Measures for Speech Recognition Evaluation. Technical report IDIAP.  Iain A. McCowan Darren Moore John Dines Daniel Gatica-Perez Mike Flynn Pierre Wellner and Herv\u00e9 Bourlard. 2004. On the use of Information Retrieval Measures for Speech Recognition Evaluation. Technical report IDIAP."},{"key":"e_1_2_2_41_1","volume-title":"Proceedings of the 12th Annual Conference of the International Speech Communication Association (INTERSPEECH'11)","author":"Mishra Taniya","year":"2011","unstructured":"Taniya Mishra , Andrej Ljolje , and Mazin Gilbert . 2011 . Predicting human perceived accuracy of ASR systems . In Proceedings of the 12th Annual Conference of the International Speech Communication Association (INTERSPEECH'11) . 1945--1948. Retrieved from http:\/\/www.isca-speech.org\/archive\/interspeech_ 2011\/i11_1945.html. Taniya Mishra, Andrej Ljolje, and Mazin Gilbert. 2011. Predicting human perceived accuracy of ASR systems. In Proceedings of the 12th Annual Conference of the International Speech Communication Association (INTERSPEECH'11). 1945--1948. Retrieved from http:\/\/www.isca-speech.org\/archive\/interspeech_2011\/i11_1945.html."},{"key":"e_1_2_2_42_1","volume-title":"Proceedings of the 8th International Conference on Spoken Language Processing (ICSLP\u201904)","author":"Morris Andrew Cameron","year":"2004","unstructured":"Andrew Cameron Morris , Viktoria Maier , and Phil D. Green . 2004. From WER and RIL to MER and WIL:Improved evaluation measures for connected speech recognition . In Proceedings of the 8th International Conference on Spoken Language Processing (ICSLP\u201904) . Retrieved from http:\/\/www.isca-speech.org\/archive\/interspeech_ 2004 \/i04_2765.html. Andrew Cameron Morris, Viktoria Maier, and Phil D. Green. 2004. From WER and RIL to MER and WIL:Improved evaluation measures for connected speech recognition. In Proceedings of the 8th International Conference on Spoken Language Processing (ICSLP\u201904). Retrieved from http:\/\/www.isca-speech.org\/archive\/interspeech_2004\/i04_2765.html."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2005.1415298"},{"key":"e_1_2_2_44_1","volume-title":"Sabine Schulte im Walde, and Ngoc Thang Vu","author":"Nguyen Kim Anh","year":"2016","unstructured":"Kim Anh Nguyen , Sabine Schulte im Walde, and Ngoc Thang Vu . 2016 . Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. arXiv preprint arXiv:1605.07766. Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. arXiv preprint arXiv:1605.07766."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178964"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1002\/1520-6807(199007)27:3<210::AID-PITS2310270306>3.0.CO;2-S"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.124.3.372"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03206448"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1037\/0882-7974.21.3.448"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0020990"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1194"},{"key":"e_1_2_2_53_1","volume-title":"Proceedings of the Intern LREC. 125--129","author":"Rousseau Anthony","year":"2012","unstructured":"Anthony Rousseau , Paul Del\u00e9glise , and Yannick Esteve . 2012 . TED-LIUM: An automatic speech recognition dedicated corpus . In Proceedings of the Intern LREC. 125--129 . Anthony Rousseau, Paul Del\u00e9glise, and Yannick Esteve. 2012. TED-LIUM: An automatic speech recognition dedicated corpus. In Proceedings of the Intern LREC. 125--129."},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1074"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661334.2661337"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1353\/aad.2011.0039"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2745555.2746648"},{"key":"e_1_2_2_58_1","unstructured":"Yuan Tang. 2016. TF.Learn: TensorFlow's High-level Module for Distributed Machine Learning. Retrieved from http:\/\/arxiv.org\/abs\/1612.04251.  Yuan Tang. 2016. TF.Learn: TensorFlow's High-level Module for Distributed Machine Learning. Retrieved from http:\/\/arxiv.org\/abs\/1612.04251."},{"key":"e_1_2_2_59_1","volume-title":"Heeman","author":"Traum David R.","year":"1996","unstructured":"David R. Traum and Peter A . Heeman . 1996 . Utterance units in spoken dialogue. In Proceedings of the Workshop on Dialogue Processing in Spoken Language Systems. Springer , 125--140. David R. Traum and Peter A. Heeman. 1996. Utterance units in spoken dialogue. In Proceedings of the Workshop on Dialogue Processing in Spoken Language Systems. Springer, 125--140."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1969289.1969318"},{"key":"e_1_2_2_61_1","volume-title":"Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU\u201903)","author":"Wang Ye-Yi","year":"2003","unstructured":"Ye-Yi Wang , Alex Acero , and Ciprian Chelba . 2003 . Is word error rate a good indicator for spoken language understanding accuracy . In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU\u201903) . IEEE, 577--582. Ye-Yi Wang, Alex Acero, and Ciprian Chelba. 2003. Is word error rate a good indicator for spoken language understanding accuracy. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU\u201903). IEEE, 577--582."},{"key":"e_1_2_2_62_1","volume-title":"Proceedings of the Workshop on Standardization for Speech I\/O technology.","author":"Woodard J. P","unstructured":"J. P Woodard and J. T. Nelson . 1982. An information theoretic measure of speech recognition performance . In Proceedings of the Workshop on Standardization for Speech I\/O technology. J. P Woodard and J. T. Nelson. 1982. An information theoretic measure of speech recognition performance. In Proceedings of the Workshop on Standardization for Speech I\/O technology."},{"key":"e_1_2_2_63_1","doi-asserted-by":"crossref","unstructured":"Wayne Xiong Jasha Droppo Xuedong Huang Frank Seide Mike Seltzer Andreas Stolcke Dong Yu and Geoffrey Zweig. 2016. Achieving Human Parity in Conversational Speech Recognition. Retrieved from http:\/\/arxiv.org\/abs\/1610.05256.  Wayne Xiong Jasha Droppo Xuedong Huang Frank Seide Mike Seltzer Andreas Stolcke Dong Yu and Geoffrey Zweig. 2016. Achieving Human Parity in Conversational Speech Recognition. Retrieved from http:\/\/arxiv.org\/abs\/1610.05256.","DOI":"10.1109\/ICASSP.2017.7953159"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1353\/aad.2011.0026"}],"container-title":["ACM Transactions on Accessible Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3325862","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3325862","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:08Z","timestamp":1750204388000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3325862"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,11]]},"references-count":63,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,6,30]]}},"alternative-id":["10.1145\/3325862"],"URL":"https:\/\/doi.org\/10.1145\/3325862","relation":{},"ISSN":["1936-7228","1936-7236"],"issn-type":[{"value":"1936-7228","type":"print"},{"value":"1936-7236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,11]]},"assertion":[{"value":"2018-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-06-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}