{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:38:41Z","timestamp":1750307921942,"version":"3.41.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2006,10,1]],"date-time":"2006-10-01T00:00:00Z","timestamp":1159660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2006,10]]},"abstract":"<jats:p>Confidence measures are a practical solution for improving the usefulness of Natural Language Processing applications. Confidence estimation is a generic machine learning approach for deriving confidence measures. We give an overview of the application of confidence estimation in various fields of Natural Language Processing, and present experimental results for speech recognition, spoken language understanding, and statistical machine translation.<\/jats:p>","DOI":"10.1145\/1177055.1177057","type":"journal-article","created":{"date-parts":[[2007,1,16]],"date-time":"2007-01-16T19:38:29Z","timestamp":1168976309000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Confidence estimation for NLP applications"],"prefix":"10.1145","volume":"3","author":[{"given":"Simona","family":"Gandrabur","sequence":"first","affiliation":[{"name":"RALI, Universit\u00e9 de Montr\u00e9al, Montr\u00e9al, Qu\u00e9bec, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"George","family":"Foster","sequence":"additional","affiliation":[{"name":"National Research Council, Gatineau, Quebec, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guy","family":"Lapalme","sequence":"additional","affiliation":[{"name":"RALI, Universit\u00e9 de Montr\u00e9al, Montr\u00e9al, Qu\u00e9bec, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2006,10]]},"reference":[{"volume-title":"Neural Networks for Pattern Recognition","author":"Bishop C. M.","key":"e_1_2_1_1_1","unstructured":"Bishop , C. M. 1995. Neural Networks for Pattern Recognition . Oxford University Press , Oxford, England . Bishop, C. M. 1995. Neural Networks for Pattern Recognition. Oxford University Press, Oxford, England."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of COLING 2004","author":"Blatz J.","year":"2035","unstructured":"Blatz , J. , Fitzgerald , E. , Foster , G. , Gandrabur , S. , Goutte , C. , Kulesza , A. , Sanchis , A. , and Ueffing , N . 2004a. Confidence estimation for machine translation . In Proceedings of COLING 2004 ( Geneva). 315--321. 10.3115\/12 2035 5.1220401 Blatz, J., Fitzgerald, E., Foster, G., Gandrabur, S., Goutte, C., Kulesza, A., Sanchis, A., and Ueffing, N. 2004a. Confidence estimation for machine translation. In Proceedings of COLING 2004 (Geneva). 315--321. 10.3115\/1220355.1220401"},{"key":"e_1_2_1_3_1","unstructured":"Blatz J. Fitzgerald E. Foster G. Gandrabur S. Goutte C. Kulesza A. Sanchis A. and Ueffing N. 2004b. Confidence estimation for machine translation. Tech. Rep. Final report JHU \/CLSP 2003 Summer Workshop Baltimore MD. 10.3115\/1220355.1220401   Blatz J. Fitzgerald E. Foster G. Gandrabur S. Goutte C. Kulesza A. Sanchis A. and Ueffing N. 2004b. Confidence estimation for machine translation. Tech. Rep. Final report JHU \/CLSP 2003 Summer Workshop Baltimore MD. 10.3115\/1220355.1220401"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 4th ICSLP","volume":"4","author":"Caminero J.","year":"1996","unstructured":"Caminero , J. , De La Torre , D. , Villarubia , L. , Martin , C. , and Hernandez , L . 1996. On-line garbage modeling with discriminant analysis for utterance verification . In Proceedings of the 4th ICSLP 1996 . Vol. 4 . Philadelphia, PA, 2115--2118. Caminero, J., De La Torre, D., Villarubia, L., Martin, C., and Hernandez, L. 1996. On-line garbage modeling with discriminant analysis for utterance verification. In Proceedings of the 4th ICSLP 1996. Vol. 4. Philadelphia, PA, 2115--2118."},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Carpenter P. Jin C. Wilson D. Zhang R. Bohus D. and Rudnicky A. 2001. Is this conversation on track? In Eurospeech 2001 (Aalborg Denmark). 2121--2124.  Carpenter P. Jin C. Wilson D. Zhang R. Bohus D. and Rudnicky A. 2001. Is this conversation on track? In Eurospeech 2001 (Aalborg Denmark). 2121--2124.","DOI":"10.21437\/Eurospeech.2001-500"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 40th Annual Meeting of the ACL","author":"Collins M.","year":"2002","unstructured":"Collins , M. 2002 . Ranked algorithms for named entity extraction: Boosting and the voted perceptron . In Proceedings of the 40th Annual Meeting of the ACL . Philadelphia, PA, 489--496. 10.3115\/1073083.1073165 Collins, M. 2002. Ranked algorithms for named entity extraction: Boosting and the voted perceptron. In Proceedings of the 40th Annual Meeting of the ACL. Philadelphia, PA, 489--496. 10.3115\/1073083.1073165"},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Collins M. and Koo T. 2005. Discriminative reranking for natural language parsing. Computational Linguistics 1 (Mar.) 25--70. 10.1162\/0891201053630273   Collins M. and Koo T. 2005. Discriminative reranking for natural language parsing. Computational Linguistics 1 (Mar.) 25--70. 10.1162\/0891201053630273","DOI":"10.1162\/0891201053630273"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)","author":"Collins M.","year":"1984","unstructured":"Collins , M. , Roark , B. , and Saraclar , M . 2005. Discriminative syntactic language modeling for speech recognition . In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05) . Association for Computational Linguistics, Ann Arbor, MI. 507--514. 10.3115\/12 1984 0.1219903 Collins, M., Roark, B., and Saraclar, M. 2005. Discriminative syntactic language modeling for speech recognition. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05). Association for Computational Linguistics, Ann Arbor, MI. 507--514. 10.3115\/1219840.1219903"},{"key":"e_1_2_1_9_1","volume-title":"Torch: A modular machine learning software library. Tech. Rep. IDIAP-RR 02-46, IDIAP.","author":"Collobert R.","year":"2002","unstructured":"Collobert , R. , Bengio , S. , and Mari\u00e9thoz , J . 2002 . Torch: A modular machine learning software library. Tech. Rep. IDIAP-RR 02-46, IDIAP. Collobert, R., Bengio, S., and Mari\u00e9thoz, J. 2002. Torch: A modular machine learning software library. Tech. Rep. IDIAP-RR 02-46, IDIAP."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of HLT-NAACL.","author":"Culotta A.","year":"2004","unstructured":"Culotta , A. and McCallum , A. 2004 . Confidence estimation for information extraction . In Proceedings of HLT-NAACL. Culotta, A. and McCallum, A. 2004. Confidence estimation for information extraction. In Proceedings of HLT-NAACL."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of AAAI.","author":"Culotta A.","year":"2005","unstructured":"Culotta , A. and McCallum , A. 2005 . Reducing labeling effort for structured prediction tasks . In Proceedings of AAAI. Culotta, A. and McCallum, A. 2005. Reducing labeling effort for structured prediction tasks. In Proceedings of AAAI."},{"key":"e_1_2_1_12_1","volume-title":"Tech. Rep. TCD-CS-2005-20","author":"Delany S. J.","year":"2005","unstructured":"Delany , S. J. , Cunningham , P. , and Doyle , D . 2005 . Generating estimates of classification confidence for a case-based spam filter. Tech. Rep. TCD-CS-2005-20 , Trinity College Dublin Computer Science Department . Delany, S. J., Cunningham, P., and Doyle, D. 2005. Generating estimates of classification confidence for a case-based spam filter. Tech. Rep. TCD-CS-2005-20, Trinity College Dublin Computer Science Department."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 38th Annual Meeting of the ACL","author":"Foster G.","year":"2000","unstructured":"Foster , G. 2000 . A Maximum Entropy \/Minimum Divergence translation model . In Proceedings of the 38th Annual Meeting of the ACL ( Hong Kong). 37--42. 10.3115\/1075218.1075224 Foster, G. 2000. A Maximum Entropy \/Minimum Divergence translation model. In Proceedings of the 38th Annual Meeting of the ACL (Hong Kong). 37--42. 10.3115\/1075218.1075224"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007999327580"},{"key":"e_1_2_1_15_1","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP-02)","author":"Foster G.","year":"1869","unstructured":"Foster , G. , Langlais , P. , and Lapalme , G . 2002. User-friendly text prediction for translators . In Conference on Empirical Methods in Natural Language Processing (EMNLP-02) , (Philadelphia, PA). 148--155. 10.3115\/11 1869 3.1118713 Foster, G., Langlais, P., and Lapalme, G. 2002. User-friendly text prediction for translators. In Conference on Empirical Methods in Natural Language Processing (EMNLP-02), (Philadelphia, PA). 148--155. 10.3115\/1118693.1118713"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of CoNLL-2003","author":"Gandrabur S.","year":"1917","unstructured":"Gandrabur , S. and Foster , G . 2003. Confidence estimation for translation prediction . In Proceedings of CoNLL-2003 , W. Daelemans and M. Osborne, Eds. Edmonton, B.C., Canada, 95--102. 10.3115\/11 1917 6.1119189 Gandrabur, S. and Foster, G. 2003. Confidence estimation for translation prediction. In Proceedings of CoNLL-2003, W. Daelemans and M. Osborne, Eds. Edmonton, B.C., Canada, 95--102. 10.3115\/1119176.1119189"},{"key":"e_1_2_1_17_1","volume-title":"ICASSP","author":"Gillick L.","year":"1997","unstructured":"Gillick , L. , Ito , Y. , and Young , J . 1997. A probabilistic approach to confidence measure estimation and evaluation . In ICASSP 1997 . 879--882. Gillick, L., Ito, Y., and Young, J. 1997. A probabilistic approach to confidence measure estimation and evaluation. In ICASSP 1997. 879--882."},{"volume-title":"Proceedings of the 25th International Symposium on Computer Architecture. 122--131","author":"Grunwald D.","key":"e_1_2_1_18_1","unstructured":"Grunwald , D. , Klauser , A. , Manne , S. , and Pleszkun , A. R . 1998. Confidence estimation for speculation control . In Proceedings of the 25th International Symposium on Computer Architecture. 122--131 . 10.1145\/279358.279376 Grunwald, D., Klauser, A., Manne, S., and Pleszkun, A. R. 1998. Confidence estimation for speculation control. In Proceedings of the 25th International Symposium on Computer Architecture. 122--131. 10.1145\/279358.279376"},{"volume-title":"Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP) 2002","author":"Guillevic D.","key":"e_1_2_1_19_1","unstructured":"Guillevic , D. , Gandrabur , S. , and Normandin , Y . 2002. Robust semantic confidence scoring . In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP) 2002 ( Denver, CO). Guillevic, D., Gandrabur, S., and Normandin, Y. 2002. Robust semantic confidence scoring. In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP) 2002 (Denver, CO)."},{"key":"e_1_2_1_20_1","unstructured":"Hasegawa-Johnson M. Baker J. Borys S. Chen K. Coogan E. Greenberg S. Juneja A. Kirchhoff K. Livescu K. Mohan S. Muller J. Sonmez K. and Wang T. 2005. Landmark-based speech recognition: Report of the 2004 Johns Hopkins Summer Workshop. In ICASSP.  Hasegawa-Johnson M. Baker J. Borys S. Chen K. Coogan E. Greenberg S. Juneja A. Kirchhoff K. Livescu K. Mohan S. Muller J. Sonmez K. and Wang T. 2005. Landmark-based speech recognition: Report of the 2004 Johns Hopkins Summer Workshop. In ICASSP."},{"volume-title":"Proc. Automatic Speech Recognition Workshop","author":"Hazen T.","key":"e_1_2_1_21_1","unstructured":"Hazen , T. , Burianek , T. , Polifroni , J. , and Seneff , S . 2000. Recognition confidence scoring for use in speech understanding systems . In Proc. Automatic Speech Recognition Workshop ( Paris, France). 213--220. Hazen, T., Burianek, T., Polifroni, J., and Seneff, S. 2000. Recognition confidence scoring for use in speech understanding systems. In Proc. Automatic Speech Recognition Workshop (Paris, France). 213--220."},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Hazen T. Seneff S. and Polifroni J. 2002. Recognition confidence scoring and its use in speech understanding systems. Compu. Speech Lang. 49--67.  Hazen T. Seneff S. and Polifroni J. 2002. Recognition confidence scoring and its use in speech understanding systems. Compu. Speech Lang. 49--67.","DOI":"10.1006\/csla.2001.0183"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of ECML-98","volume":"1398","author":"Joachims T.","year":"1998","unstructured":"Joachims , T. 1998 . Text categorization with support vector machines: Learning with many relevant features . In Proceedings of ECML-98 , 10th European Conference on Machine Learning, C. N\u00e9dellec and C. Rouveirol, Eds. Lecture Notes in Computer Sciences , vol. 1398 . Springer Verlag, Heidelberg, DE, Chemnitz, DE, 137--142. Joachims, T. 1998. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of ECML-98, 10th European Conference on Machine Learning, C. N\u00e9dellec and C. Rouveirol, Eds. Lecture Notes in Computer Sciences, vol. 1398. Springer Verlag, Heidelberg, DE, Chemnitz, DE, 137--142."},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Kemp T. and Schaaf T. 1997. Estimating confidence using word lattices. In Eurospeech. 827--830.  Kemp T. and Schaaf T. 1997. Estimating confidence using word lattices. In Eurospeech. 827--830.","DOI":"10.21437\/Eurospeech.1997-281"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:COAT.0000010117.98933.a0"},{"key":"e_1_2_1_26_1","volume-title":"ICASSP","author":"Ma C.","year":"2001","unstructured":"Ma , C. , Randolph , M. , and Drish , J . 2001. A support vector machines-based rejection technique for speech recognition . In ICASSP 2001 . Ma, C., Randolph, M., and Drish, J. 2001. A support vector machines-based rejection technique for speech recognition. In ICASSP 2001."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of LREC","author":"Macklovitch E.","year":"2006","unstructured":"Macklovitch , E. 2006 . TransType2: The last word . In Proceedings of LREC 2006. Genoa, Italy. Macklovitch, E. 2006. TransType2: The last word. In Proceedings of LREC 2006. Genoa, Italy."},{"key":"e_1_2_1_28_1","volume-title":"ICASSP","author":"Maison B.","year":"2001","unstructured":"Maison , B. and Gopinath , R . 2001. Robust confidence annotation and rejection for continuous speech recognition . In ICASSP 2001 . Maison, B. and Gopinath, R. 2001. Robust confidence annotation and rejection for continuous speech recognition. In ICASSP 2001."},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1006\/csla.2000.0152","article-title":"Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Compu","volume":"14","author":"Mangu L.","year":"2000","unstructured":"Mangu , L. , Brill , E. , and Stolcke , A. 2000 . Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Compu . Speech Lang. 14 , 4, 373 -- 400 . Mangu, L., Brill, E., and Stolcke, A. 2000. Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Compu. Speech Lang. 14, 4, 373--400.","journal-title":"Speech Lang."},{"volume-title":"Proceedings of HLT 2002, 2nd International Conference on Human Language Technology Research, M. Marcus, Ed. Morgan Kaufmann","author":"Manmatha R.","key":"e_1_2_1_30_1","unstructured":"Manmatha , R. and Sever , H . 2002. A formal approach to score normalization for meta-search . In Proceedings of HLT 2002, 2nd International Conference on Human Language Technology Research, M. Marcus, Ed. Morgan Kaufmann , San Francisco, CA. 98--103. Manmatha, R. and Sever, H. 2002. A formal approach to score normalization for meta-search. In Proceedings of HLT 2002, 2nd International Conference on Human Language Technology Research, M. Marcus, Ed. Morgan Kaufmann, San Francisco, CA. 98--103."},{"volume-title":"Proceedings of Eurospeech 2001","author":"Moreno P.","key":"e_1_2_1_31_1","unstructured":"Moreno , P. , Logan , B. , and Raj , B . 2001. A boosting approach for confidence scoring . In Proceedings of Eurospeech 2001 ( Aalborg, Denmark). 2109--2112. Moreno, P., Logan, B., and Raj, B. 2001. A boosting approach for confidence scoring. In Proceedings of Eurospeech 2001 (Aalborg, Denmark). 2109--2112."},{"volume-title":"EMLNP'04","author":"Nepveu L.","key":"e_1_2_1_32_1","unstructured":"Nepveu , L. , Lapalme , G. , Langlais , P. , and Foster , G . 2004. Adaptive language and translation models for interactive machine translation . In EMLNP'04 ( Barcelona, Spain). 190--197. Nepveu, L., Lapalme, G., Langlais, P., and Foster, G. 2004. Adaptive language and translation models for interactive machine translation. In EMLNP'04 (Barcelona, Spain). 190--197."},{"key":"e_1_2_1_33_1","volume-title":"ICASSP","author":"Neti C.","year":"1997","unstructured":"Neti , C. , Roukos , S. , and Eide , E . 1997. Word-based confidence measures as a guide for stack search in speech recognition . In ICASSP 1997 . 883--886. Neti, C., Roukos, S., and Eide, E. 1997. Word-based confidence measures as a guide for stack search in speech recognition. In ICASSP 1997. 883--886."},{"key":"e_1_2_1_34_1","unstructured":"NIST. 2001. Automatic evaluation of machine translation quality using n-gram co-occurence statistics. www.nist.gov\/speech\/tests\/mt\/doc\/ngram-study.pdf.  NIST. 2001. Automatic evaluation of machine translation quality using n-gram co-occurence statistics. www.nist.gov\/speech\/tests\/mt\/doc\/ngram-study.pdf."},{"volume-title":"Proceedings of the 40th Annual Meeting of the ACL","author":"Och F. J.","key":"e_1_2_1_35_1","unstructured":"Och , F. J. and Ney , H . 2002. Discriminative training and maximum entropy models for statistical machine translation . In Proceedings of the 40th Annual Meeting of the ACL ( Philadelphia, PA). 295--302. 10.3115\/1073083.1073133 Och, F. J. and Ney, H. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the 40th Annual Meeting of the ACL (Philadelphia, PA). 295--302. 10.3115\/1073083.1073133"},{"key":"e_1_2_1_36_1","volume-title":"Bleu: A method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division","author":"Papineni K.","year":"2001","unstructured":"Papineni , K. , Roukos , S. , Ward , T. , and Zhu , W . 2001 . Bleu: A method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division , Thomas J. Watson Research Center ( 2001). Papineni, K., Roukos, S., Ward, T., and Zhu, W. 2001. Bleu: A method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division, Thomas J. Watson Research Center (2001)."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of LREC (Lisbon, Portual). 825--828","author":"Quirk C. B.","year":"2004","unstructured":"Quirk , C. B. 2004 . Training a sentence-level machine translation confidence measure . In Proceedings of LREC (Lisbon, Portual). 825--828 . Quirk, C. B. 2004. Training a sentence-level machine translation confidence measure. In Proceedings of LREC (Lisbon, Portual). 825--828."},{"volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP-2001)","author":"San-Secundo R.","key":"e_1_2_1_38_1","unstructured":"San-Secundo , R. , Pellom , B. , Hacioglu , K. , Ward , W. , and Pardo , J . 2001. Confidence measures for spoken dialogue systems . In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP-2001) (Salt Lake City, UT). San-Secundo, R., Pellom, B., Hacioglu, K., Ward, W., and Pardo, J. 2001. Confidence measures for spoken dialogue systems. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP-2001) (Salt Lake City, UT)."},{"key":"e_1_2_1_39_1","volume-title":"ICASSP","author":"Sanchis A.","year":"2003","unstructured":"Sanchis , A. , Juan , A. , and Vidal , E . 2003. Improving utterance verification using a smoothed naive Bayes model . In ICASSP 2003 , (Hong Kong). 29--35. Sanchis, A., Juan, A., and Vidal, E. 2003. Improving utterance verification using a smoothed naive Bayes model. In ICASSP 2003, (Hong Kong). 29--35."},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the MSRI Workshop on Nonlinear Estimation and Classification","author":"Schapire R.","year":"2001","unstructured":"Schapire , R. 2001 . The boosting approach to machine learning: An overview . In Proceedings of the MSRI Workshop on Nonlinear Estimation and Classification ( Berkeley, CA, Mar.). Schapire, R. 2001. The boosting approach to machine learning: An overview. In Proceedings of the MSRI Workshop on Nonlinear Estimation and Classification (Berkeley, CA, Mar.)."},{"key":"e_1_2_1_41_1","unstructured":"Shen L. Sarkar A. and Och F. J. 2004. Discriminative reranking for machine translation. In HLT-NAACL. 177--184.  Shen L. Sarkar A. and Och F. J. 2004. Discriminative reranking for machine translation. In HLT-NAACL. 177--184."},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1006\/csla.1999.0126","article-title":"Evaluation of word confidence for speech recognition systems","volume":"13","author":"Siu M.","year":"1999","unstructured":"Siu , M. and Gish , H. 1999 . Evaluation of word confidence for speech recognition systems . Comput. Speech Lang. 13 , 4, 299 -- 318 . Siu, M. and Gish, H. 1999. Evaluation of word confidence for speech recognition systems. Comput. Speech Lang. 13, 4, 299--318.","journal-title":"Comput. Speech Lang."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 5th European Conference on Speech Communication and Technology.","volume":"1","author":"Stolcke A.","unstructured":"Stolcke , A. , Koenig , Y. , and Weintraub , M . 1997. Explicit word error minimization in N-best list rescoring . In Proceedings of the 5th European Conference on Speech Communication and Technology. Vol. 1 . (Rhodes, Greece). 163--166. Stolcke, A., Koenig, Y., and Weintraub, M. 1997. Explicit word error minimization in N-best list rescoring. In Proceedings of the 5th European Conference on Speech Communication and Technology. Vol. 1. (Rhodes, Greece). 163--166."},{"volume-title":"Proceedings of the 14th International Conference on Machine Learning","author":"Ting K. M.","key":"e_1_2_1_44_1","unstructured":"Ting , K. M. and Witten , I. H . 1997. Stacking bagged and dagged models . In Proceedings of the 14th International Conference on Machine Learning ( Nashville, TN). Morgan Kaufmann, San Matco, CA, 367--375. Ting, K. M. and Witten, I. H. 1997. Stacking bagged and dagged models. In Proceedings of the 14th International Conference on Machine Learning (Nashville, TN). Morgan Kaufmann, San Matco, CA, 367--375."},{"volume-title":"Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (New-Orleans, LA). Springer-Verlag","author":"Ueffing N.","key":"e_1_2_1_45_1","unstructured":"Ueffing , N. , Macherey , K. , and Ney , H . 2003. Confidence measures for statistical machine translation . In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (New-Orleans, LA). Springer-Verlag , New York, 394--401. Ueffing, N., Macherey, K., and Ney, H. 2003. Confidence measures for statistical machine translation. In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (New-Orleans, LA). Springer-Verlag, New York, 394--401."},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of HLT\/EMNLP","author":"Ueffing N.","year":"2057","unstructured":"Ueffing , N. and Ney , H . 2005. Word-level confidence estimation for machine translation using phrase-based translation models . In Proceedings of HLT\/EMNLP . Vancouver, BC, Canada. 10.3115\/12 2057 5.1220671 Ueffing, N. and Ney, H. 2005. Word-level confidence estimation for machine translation using phrase-based translation models. In Proceedings of HLT\/EMNLP. Vancouver, BC, Canada. 10.3115\/1220575.1220671"},{"key":"e_1_2_1_47_1","volume-title":"ICASSP","author":"Weintraub M.","year":"1997","unstructured":"Weintraub , M. , Beaufays , F. , Rivlin , Z. , Konig , Y. , and Stolcke , A . 1997. Neural-network based measures of confidence for word recognition . In ICASSP 1997 . 887--890. Weintraub, M., Beaufays, F., Rivlin, Z., Konig, Y., and Stolcke, A. 1997. Neural-network based measures of confidence for word recognition. In ICASSP 1997. 887--890."},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1109\/89.906002","article-title":"Confidence measures for large vocabulary continuous speech recognition","volume":"9","author":"Wessel F.","year":"2001","unstructured":"Wessel , F. , Schluter , R. , Macherey , K. , and Ney , H. 2001 . Confidence measures for large vocabulary continuous speech recognition . IEEE Trans. Speech Audio Process. 9 , 3, 288 -- 298 . Wessel, F., Schluter, R., Macherey, K., and Ney, H. 2001. Confidence measures for large vocabulary continuous speech recognition. IEEE Trans. Speech Audio Process. 9, 3, 288--298.","journal-title":"IEEE Trans. Speech Audio Process."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 10th Text REtrieval Conference (TREC","author":"Xu J.","year":"2002","unstructured":"Xu , J. , Licuanan , A. , May , J. , Miller , S. , and Weischedel , R . 2002. Trec2002qa at bbn: Answer selection and confidence estimation . In Proceedings of the 10th Text REtrieval Conference (TREC 2002 ). Xu, J., Licuanan, A., May, J., Miller, S., and Weischedel, R. 2002. Trec2002qa at bbn: Answer selection and confidence estimation. In Proceedings of the 10th Text REtrieval Conference (TREC 2002)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Zhang R. and Rudnicky A. 2001. Word level confidence annotation using combinations of features. In Eurospeech. Aalborg Denmark 2105--2108.  Zhang R. and Rudnicky A. 2001. Word level confidence annotation using combinations of features. In Eurospeech. Aalborg Denmark 2105--2108.","DOI":"10.21437\/Eurospeech.2001-496"},{"key":"e_1_2_1_51_1","unstructured":"Zou K. H. 2004. Receiver operating characteristic (ROC) literature research. http:\/\/splweb.bwh.harvard.edu:8000\/pages\/ppl\/zou\/roc.html.  Zou K. H. 2004. Receiver operating characteristic (ROC) literature research. http:\/\/splweb.bwh.harvard.edu:8000\/pages\/ppl\/zou\/roc.html."}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1177055.1177057","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1177055.1177057","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T14:51:38Z","timestamp":1750258298000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1177055.1177057"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2006,10]]},"references-count":51,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2006,10]]}},"alternative-id":["10.1145\/1177055.1177057"],"URL":"https:\/\/doi.org\/10.1145\/1177055.1177057","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"type":"print","value":"1550-4875"},{"type":"electronic","value":"1550-4883"}],"subject":[],"published":{"date-parts":[[2006,10]]},"assertion":[{"value":"2006-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}