{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T16:20:26Z","timestamp":1761582026633,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":70,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3409696","type":"proceedings-article","created":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T06:03:52Z","timestamp":1604815432000},"page":"901-913","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["Operational calibration: debugging confidence errors for DNNs in the field"],"prefix":"10.1145","author":[{"given":"Zenan","family":"Li","sequence":"first","affiliation":[{"name":"Nanjing University, China"}]},{"given":"Xiaoxing","family":"Ma","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}]},{"given":"Chang","family":"Xu","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}]},{"given":"Jingwei","family":"Xu","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}]},{"given":"Chun","family":"Cao","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}]},{"given":"Jian","family":"L\u00fc","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"volume-title":"Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs\/1206.5538 1 ( 2012 )","year":"2012","author":"Bengio Yoshua","key":"e_1_3_2_2_1_1"},{"volume-title":"Pattern recognition and machine learning","year":"2016","author":"Bishop Christopher M","key":"e_1_3_2_2_2_1"},{"volume-title":"Davide Del Testa","year":"2016","author":"Bojarski Mariusz","key":"e_1_3_2_2_3_1"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"crossref","unstructured":"Glenn W Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review 78 1 ( 1950 ) 1-3.  Glenn W Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review 78 1 ( 1950 ) 1-3.","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2"},{"key":"e_1_3_2_2_5_1","unstructured":"John Burkardt. 2014. The truncated normal distribution. 32 pages. http:\/\/people.sc.fsu.edu\/~jburkardt\/presentations\/truncatednormal.pdf  John Burkardt. 2014. The truncated normal distribution. 32 pages. http:\/\/people.sc.fsu.edu\/~jburkardt\/presentations\/truncatednormal.pdf"},{"key":"e_1_3_2_2_6_1","unstructured":"Aleksandar Chakarov Aditya Nori Sriram Rajamani Shayak Sen and Deepak Vijaykeerthy. 2016. Debugging machine learning tasks. arXiv preprint arXiv:1603.07292 ( 2016 ) 23.  Aleksandar Chakarov Aditya Nori Sriram Rajamani Shayak Sen and Deepak Vijaykeerthy. 2016. Debugging machine learning tasks. arXiv preprint arXiv:1603.07292 ( 2016 ) 23."},{"volume-title":"Proceedings of the fourteenth international conference on artificial intelligence and statistics. aistats, 215-223","year":"2011","author":"Coates Adam","key":"e_1_3_2_2_7_1"},{"key":"e_1_3_2_2_8_1","unstructured":"Council of European Union. 2014. Council regulation (EU) no 269\/2014. http:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?qid=1416170084502&uri= CELEX: 32014R0269.  Council of European Union. 2014. Council regulation (EU) no 269\/2014. http:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?qid=1416170084502&uri= CELEX: 32014R0269."},{"volume-title":"Kai Li, and Li Fei-fei.","year":"2009","author":"Deng Jia","key":"e_1_3_2_2_9_1"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7502-7_900-1"},{"volume-title":"The elements of statistical learning","author":"Friedman Jerome","key":"e_1_3_2_2_11_1"},{"key":"e_1_3_2_2_12_1","unstructured":"Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. arXiv preprint arXiv:1506.02142 ( 2015 ) 12.  Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. arXiv preprint arXiv:1506.02142 ( 2015 ) 12."},{"volume-title":"Deep Learning","author":"Goodfellow Ian","key":"e_1_3_2_2_13_1"},{"first-page":"1321","volume-title":"Proceedings of the 34th International Conference on Machine Learning-Volume 70","author":"Guo Chuan","key":"e_1_3_2_2_14_1"},{"key":"e_1_3_2_2_15_1","unstructured":"T. Hastie R. Tibshirani and J.H. Friedman. 2009. The Elements of Statistical Learning: Data Mining Inference and Prediction. Springer. https:\/\/books.google. com\/books?id=eBSgoAEACAAJ  T. Hastie R. Tibshirani and J.H. Friedman. 2009. The Elements of Statistical Learning: Data Mining Inference and Prediction. Springer. https:\/\/books.google. com\/books?id=eBSgoAEACAAJ"},{"key":"e_1_3_2_2_16_1","unstructured":"Geofrey Hinton Oriol Vinyals and Jef Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 ( 2015 ) 9.  Geofrey Hinton Oriol Vinyals and Jef Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 ( 2015 ) 9."},{"volume-title":"abs\/","year":"1905","author":"Ilyas Andrew","key":"e_1_3_2_2_17_1"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2007.4408844"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"volume-title":"Felix X Yu, Peter Richt\u00e1rik, Ananda Theertha Suresh, and Dave Bacon.","year":"2016","author":"Kone\u010dny Jakub","key":"e_1_3_2_2_20_1"},{"key":"e_1_3_2_2_21_1","unstructured":"Alex Krizhevsky Geofrey Hinton etal 2009. Learning multiple layers of features from tiny images. Technical Report. Citeseer.  Alex Krizhevsky Geofrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Technical Report. Citeseer."},{"volume-title":"Accurate uncertainties for deep learning using calibrated regression. arXiv preprint arXiv","year":"1807","author":"Kuleshov Volodymyr","key":"e_1_3_2_2_22_1"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_2_2_24_1","first-page":"2278","volume-title":"Proc. IEEE 86","author":"LeCun Yann","year":"1998"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"volume-title":"Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE '19)","year":"2019","author":"Li Zenan","key":"e_1_3_2_2_26_1"},{"volume-title":"Stumpe","year":"2017","author":"Liu Yun","key":"e_1_3_2_2_27_1"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"edition":"1","volume-title":"ImageCLEF: Experimental Evaluation in Visual Information Retrieval","author":"Mller Henning","key":"e_1_3_2_2_30_1"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13244-019-0832-5"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1175\/1520-0450(1973)012%26#60;0595:ANVPOT%26#62;2.0.CO;2"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/2888116.2888120"},{"key":"e_1_3_2_2_34_1","unstructured":"Andrew Ng. 2016. Nuts and bolts of building AI applications using Deep Learning. Neurips-Keynote ( 2016 ) 5.  Andrew Ng. 2016. Nuts and bolts of building AI applications using Deep Learning. Neurips-Keynote ( 2016 ) 5."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102430"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"crossref","unstructured":"Ziad Obermeyer and Ezekiel J Emanuel. 2016. Predicting the future-big data machine learning and clinical medicine. The New England journal of medicine 375 13 ( 2016 ) 1216.  Ziad Obermeyer and Ezekiel J Emanuel. 2016. Predicting the future-big data machine learning and clinical medicine. The New England journal of medicine 375 13 ( 2016 ) 1216.","DOI":"10.1056\/NEJMp1606181"},{"key":"e_1_3_2_2_37_1","first-page":"4901","volume-title":"Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"97","author":"Odena Augustus","year":"2019"},{"key":"e_1_3_2_2_38_1","article-title":"A survey on transfer learning","volume":"22","author":"Pan Sinno Jialin","year":"2009","journal-title":"IEEE Transactions on knowledge and data engineering"},{"key":"e_1_3_2_2_39_1","first-page":"79","article-title":"Thumbs Up? Sentiment Classification Using Machine Learning Techniques","author":"Pang Bo","year":"2002","journal-title":"Proceedings of EMNLP. EMNLP"},{"key":"e_1_3_2_2_40_1","unstructured":"Nicolas Papernot and Patrick McDaniel. 2018. Deep k-nearest neighbors: Towards confident interpretable and robust deep learning. arXiv preprint arXiv:1803. 04765 ( 2018 ) 18.  Nicolas Papernot and Patrick McDaniel. 2018. Deep k-nearest neighbors: Towards confident interpretable and robust deep learning. arXiv preprint arXiv:1803. 04765 ( 2018 ) 18."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/IGARSS.2011.6049994"},{"volume-title":"Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 ( 2011 ), 2825-2830.","year":"2011","author":"Pedregosa F.","key":"e_1_3_2_2_42_1"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"volume-title":"Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods","author":"Platt John C.","key":"e_1_3_2_2_45_1"},{"volume-title":"Williams","year":"2005","author":"Rasmussen Carl Edward","key":"e_1_3_2_2_46_1"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-019-02307-y"},{"key":"e_1_3_2_2_48_1","unstructured":"Peter Schulam and Suchi Saria. 2019. Can you trust this prediction? Auditing pointwise reliability after learning. arXiv preprint arXiv:1901. 00403 ( 2019 ) 10.  Peter Schulam and Suchi Saria. 2019. Can you trust this prediction? Auditing pointwise reliability after learning. arXiv preprint arXiv:1901. 00403 ( 2019 ) 10."},{"key":"e_1_3_2_2_49_1","first-page":"3179","article-title":"Evidential deep learning to quantify classification uncertainty","author":"Sensoy Murat","year":"2018","journal-title":"Advances in Neural Information Processing Systems. NeurIps"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-59802-9_4"},{"first-page":"371","volume-title":"Mar ( 2008 )","author":"Shafer Glenn","key":"e_1_3_2_2_52_1"},{"key":"e_1_3_2_2_53_1","first-page":"7375","article-title":"Out-of-distribution detection using multiple semantic label representations","author":"Shalev Gabi","year":"2018","journal-title":"Advances in Neural Information Processing Systems."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2810103.2813687"},{"volume-title":"International Conference on Learning Representations. ICLR, 19","year":"2018","author":"Shu Rui","key":"e_1_3_2_2_55_1"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_3_2_2_58_1","article-title":"On the consistency of multiclass classification methods","author":"Tewari Ambuj","year":"2007","journal-title":"Journal of Machine Learning Research 8"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.4233"},{"volume-title":"Proceedings of the 42st International Conference on Software Engineering. IEEE Press, ICSE' 20","year":"2020","author":"Wang Huiyan","key":"e_1_3_2_2_60_1"},{"key":"e_1_3_2_2_61_1","unstructured":"Jindong Wang et al. 2019. Everything about Transfer Learning and Domain Adapation. http:\/\/transferlearning.xyz.  Jindong Wang et al. 2019. Everything about Transfer Learning and Domain Adapation. http:\/\/transferlearning.xyz."},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_2_2_63_1","first-page":"1305","volume-title":"Proceedings of the 31st International Conference on Machine Learning (Proceedings of Machine Learning Research, 2), Eric P. Xing and Tony Jebara (Eds.). PMLR","author":"Wang Xuezhi","year":"2014"},{"key":"e_1_3_2_2_64_1","unstructured":"Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. Citeseer Citeseer 7.  Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. Citeseer Citeseer 7."},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775151"},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002683"},{"volume-title":"Machine Learning Testing: Survey, Landscapes and Horizons. CoRR abs\/","year":"1906","author":"Zhang Jie M.","key":"e_1_3_2_2_67_1"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-016-6906-3"},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"crossref","unstructured":"Zhi-Hua Zhou. 2019. Abductive learning: Towards bridging machine learning and logical reasoning. Science China Information Sciences 62 7 ( 2019 ) 76101.  Zhi-Hua Zhou. 2019. Abductive learning: Towards bridging machine learning and logical reasoning. Science China Information Sciences 62 7 ( 2019 ) 76101.","DOI":"10.1007\/s11432-018-9801-4"},{"key":"e_1_3_2_2_71_1","article-title":"Active learning with sampling by uncertainty and density for data annotations","volume":"18","author":"Zhu Jingbo","year":"2009","journal-title":"IEEE Transactions on audio, speech, and language processing"}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"],"location":"Virtual Event USA","acronym":"ESEC\/FSE '20"},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409696","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3409696","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:40Z","timestamp":1750203880000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409696"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":70,"alternative-id":["10.1145\/3368089.3409696","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3409696","relation":{},"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}