{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:10:26Z","timestamp":1750219826515,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":45,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,8,4]],"date-time":"2023-08-04T00:00:00Z","timestamp":1691107200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,4]]},"DOI":"10.1145\/3609437.3609457","type":"proceedings-article","created":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T22:11:25Z","timestamp":1696543885000},"page":"205-215","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Practical Accuracy Evaluation for Deep Learning Systems via Latent Representation Discrepancy"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-1574-2758","authenticated-orcid":false,"given":"Yining","family":"Yin","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7477-3642","authenticated-orcid":false,"given":"Yang","family":"Feng","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3271-7255","authenticated-orcid":false,"given":"Zixi","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4629-6372","authenticated-orcid":false,"given":"Zhihong","family":"Zhao","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]}],"member":"320","published-online":{"date-parts":[[2023,10,5]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n. d.]. scipy.stats.wasserstein_distance \u2014 SciPy v1.11.1 Manual. https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.stats.wasserstein_distance.html. (Accessed on 07\/16\/2023).  [n. d.]. scipy.stats.wasserstein_distance \u2014 SciPy v1.11.1 Manual. https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.stats.wasserstein_distance.html. (Accessed on 07\/16\/2023)."},{"key":"e_1_3_2_1_2_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a028)","author":"Bengio Yoshua","year":"2013","unstructured":"Yoshua Bengio , Gregoire Mesnil , Yann Dauphin , and Salah Rifai . 2013 . Better Mixing via Deep Representations . In Proceedings of the 30th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a028) , Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, Georgia, USA, 552\u2013560. https:\/\/proceedings.mlr.press\/v28\/bengio13.html Yoshua Bengio, Gregoire Mesnil, Yann Dauphin, and Salah Rifai. 2013. Better Mixing via Deep Representations. In Proceedings of the 30th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a028), Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, Georgia, USA, 552\u2013560. https:\/\/proceedings.mlr.press\/v28\/bengio13.html"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416609"},{"key":"e_1_3_2_1_4_1","volume-title":"Pattern recognition. Machine learning 128, 9","author":"Bishop M","year":"2006","unstructured":"Christopher\u00a0 M Bishop . 2006. Pattern recognition. Machine learning 128, 9 ( 2006 ). Christopher\u00a0M Bishop. 2006. Pattern recognition. Machine learning 128, 9 (2006)."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394112"},{"key":"e_1_3_2_1_6_1","volume-title":"ARTICLE","author":"Collobert Ronan","year":"2011","unstructured":"Ronan Collobert , Jason Weston , L\u00e9on Bottou , Michael Karlen , Koray Kavukcuoglu , and Pavel Kuksa . 2011. Natural language processing (almost) from scratch. Journal of machine learning research 12 , ARTICLE ( 2011 ), 2493\u20132537. Ronan Collobert, Jason Weston, L\u00e9on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research 12, ARTICLE (2011), 2493\u20132537."},{"volume-title":"Applied nonparametric statistics. Vol.\u00a02. PWS-Kent Pub","author":"W Daniel","key":"e_1_3_2_1_7_1","unstructured":"Wayne\u00a0 W Daniel 1990. Applied nonparametric statistics. Vol.\u00a02. PWS-Kent Pub . Boston . Wayne\u00a0W Daniel 1990. Applied nonparametric statistics. Vol.\u00a02. PWS-Kent Pub. Boston."},{"key":"e_1_3_2_1_8_1","unstructured":"Alex Davies. 2019. Tesla\u2019s Latest Autopilot Death Looks Just Like a Prior Crash. https:\/\/www.wired.com\/story\/teslas-latest-autopilot-death-looks-like-prior-crash\/  Alex Davies. 2019. Tesla\u2019s Latest Autopilot Death Looks Just Like a Prior Crash. https:\/\/www.wired.com\/story\/teslas-latest-autopilot-death-looks-like-prior-crash\/"},{"key":"e_1_3_2_1_9_1","volume-title":"The mahalanobis distance. Chemometrics and intelligent laboratory systems 50, 1","author":"De\u00a0Maesschalck Roy","year":"2000","unstructured":"Roy De\u00a0Maesschalck , Delphine Jouan-Rimbaud , and D\u00e9sir\u00e9\u00a0 L Massart . 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems 50, 1 ( 2000 ), 1\u201318. Roy De\u00a0Maesschalck, Delphine Jouan-Rimbaud, and D\u00e9sir\u00e9\u00a0L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems 50, 1 (2000), 1\u201318."},{"key":"e_1_3_2_1_10_1","volume-title":"Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410","author":"Feinman Reuben","year":"2017","unstructured":"Reuben Feinman , Ryan\u00a0 R Curtin , Saurabh Shintre , and Andrew\u00a0 B Gardner . 2017. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410 ( 2017 ). Reuben Feinman, Ryan\u00a0R Curtin, Saurabh Shintre, and Andrew\u00a0B Gardner. 2017. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410 (2017)."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380415"},{"volume-title":"Deep learning","author":"Goodfellow Ian","key":"e_1_3_2_1_13_1","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep learning . MIT press . Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press."},{"key":"e_1_3_2_1_14_1","volume-title":"Properties of Euclidean and non-Euclidean distance matrices. Linear algebra and its applications 67","author":"Gower John\u00a0Clifford","year":"1985","unstructured":"John\u00a0Clifford Gower . 1985. Properties of Euclidean and non-Euclidean distance matrices. Linear algebra and its applications 67 ( 1985 ), 81\u201397. John\u00a0Clifford Gower. 1985. Properties of Euclidean and non-Euclidean distance matrices. Linear algebra and its applications 67 (1985), 81\u201397."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_2_1_16_1","volume-title":"A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136","author":"Hendrycks Dan","year":"2016","unstructured":"Dan Hendrycks and Kevin Gimpel . 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 ( 2016 ). Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 (2016)."},{"key":"e_1_3_2_1_17_1","volume-title":"Deep Anomaly Detection with Outlier Exposure. In 7th International Conference on Learning Representations, ICLR 2019","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks , Mantas Mazeika , and Thomas\u00a0 G. Dietterich . 2019 . Deep Anomaly Detection with Outlier Exposure. In 7th International Conference on Learning Representations, ICLR 2019 , New Orleans, LA, USA , May 6-9, 2019. OpenReview.net. https:\/\/openreview.net\/forum?id=HyxCxhRcY7 Dan Hendrycks, Mantas Mazeika, and Thomas\u00a0G. Dietterich. 2019. Deep Anomaly Detection with Outlier Exposure. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https:\/\/openreview.net\/forum?id=HyxCxhRcY7"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00126"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206627"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10958-006-0049-2"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_1_22_1","unstructured":"John Lafferty Andrew McCallum and Fernando\u00a0CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. (2001).  John Lafferty Andrew McCallum and Fernando\u00a0CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. (2001)."},{"key":"e_1_3_2_1_23_1","volume-title":"Deep learning. nature 521, 7553","author":"LeCun Yann","year":"2015","unstructured":"Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. nature 521, 7553 ( 2015 ), 436\u2013444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436\u2013444."},{"key":"e_1_3_2_1_24_1","volume-title":"A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems 31","author":"Lee Kimin","year":"2018","unstructured":"Kimin Lee , Kibok Lee , Honglak Lee , and Jinwoo Shin . 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems 31 ( 2018 ). Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems 31 (2018)."},{"volume-title":"Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV)","author":"Levinson Jesse","key":"e_1_3_2_1_25_1","unstructured":"Jesse Levinson , Jake Askeland , Jan Becker , Jennifer Dolson , David Held , Soeren Kammel , J\u00a0Zico Kolter , Dirk Langer , Oliver Pink , Vaughan Pratt , 2011. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV) . IEEE , 163\u2013168. Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J\u00a0Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, 2011. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV). IEEE, 163\u2013168."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_3_2_1_27_1","volume-title":"Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In International Conference on Learning Representations.","author":"Liang Shiyu","year":"2018","unstructured":"Shiyu Liang , Yixuan Li , and R Srikant . 2018 . Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In International Conference on Learning Representations. Shiyu Liang, Yixuan Li, and R Srikant. 2018. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_28_1","volume-title":"Energy-based Out-of-distribution Detection. Advances in Neural Information Processing Systems 33","author":"Liu Weitang","year":"2020","unstructured":"Weitang Liu , Xiaoyun Wang , John Owens , and Yixuan Li. 2020. Energy-based Out-of-distribution Detection. Advances in Neural Information Processing Systems 33 ( 2020 ). Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based Out-of-distribution Detection. Advances in Neural Information Processing Systems 33 (2020)."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_1_31_1","unstructured":"Matt McFarland. 2016. Tesla\u2019s autopilot probed by government after crash kills driver. https:\/\/money.cnn.com\/2016\/06\/30\/technology\/tesla-autopilot-death\/index.html.  Matt McFarland. 2016. Tesla\u2019s autopilot probed by government after crash kills driver. https:\/\/money.cnn.com\/2016\/06\/30\/technology\/tesla-autopilot-death\/index.html."},{"volume-title":"Data augmentation for improving deep learning in image classification problem. In 2018 international interdisciplinary PhD workshop (IIPhDW)","author":"Miko\u0142ajczyk Agnieszka","key":"e_1_3_2_1_32_1","unstructured":"Agnieszka Miko\u0142ajczyk and Micha\u0142 Grochowski . 2018. Data augmentation for improving deep learning in image classification problem. In 2018 international interdisciplinary PhD workshop (IIPhDW) . IEEE , 117\u2013122. Agnieszka Miko\u0142ajczyk and Micha\u0142 Grochowski. 2018. Data augmentation for improving deep learning in image classification problem. In 2018 international interdisciplinary PhD workshop (IIPhDW). IEEE, 117\u2013122."},{"key":"e_1_3_2_1_33_1","unstructured":"Gaspard Monge. 1781. M\u00e9moire sur la th\u00e9orie des d\u00e9blais et des remblais. Histoire de l\u2019Acad\u00e9mie Royale des Sciences de Paris (1781).  Gaspard Monge. 1781. M\u00e9moire sur la th\u00e9orie des d\u00e9blais et des remblais. Histoire de l\u2019Acad\u00e9mie Royale des Sciences de Paris (1781)."},{"key":"e_1_3_2_1_34_1","volume-title":"Predicting the future\u2014big data, machine learning, and clinical medicine. The New England journal of medicine 375, 13","author":"Obermeyer Ziad","year":"2016","unstructured":"Ziad Obermeyer and Ezekiel\u00a0 J Emanuel . 2016. Predicting the future\u2014big data, machine learning, and clinical medicine. The New England journal of medicine 375, 13 ( 2016 ), 1216. Ziad Obermeyer and Ezekiel\u00a0J Emanuel. 2016. Predicting the future\u2014big data, machine learning, and clinical medicine. The New England journal of medicine 375, 13 (2016), 1216."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_1_36_1","volume-title":"Gaussian mixture models.Encyclopedia of biometrics 741, 659-663","author":"Reynolds A","year":"2009","unstructured":"Douglas\u00a0 A Reynolds . 2009. Gaussian mixture models.Encyclopedia of biometrics 741, 659-663 ( 2009 ). Douglas\u00a0A Reynolds. 2009. Gaussian mixture models.Encyclopedia of biometrics 741, 659-663 (2009)."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"key":"e_1_3_2_1_38_1","unstructured":"Burr Settles. 2009. Active learning literature survey. (2009).  Burr Settles. 2009. Active learning literature survey. (2009)."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416621"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0197-0"},{"key":"e_1_3_2_1_41_1","volume-title":"Testing deep neural networks. arXiv preprint arXiv:1803.04792","author":"Sun Youcheng","year":"2018","unstructured":"Youcheng Sun , Xiaowei Huang , Daniel Kroening , James Sharp , Matthew Hill , and Rob Ashmore . 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792 ( 2018 ). Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792 (2018)."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380400"},{"key":"e_1_3_2_1_43_1","volume-title":"A Survey on Optimal Transport for Machine Learning: Theory and Applications. arXiv preprint arXiv:2106.01963","author":"Torres Luis\u00a0Caicedo","year":"2021","unstructured":"Luis\u00a0Caicedo Torres , Luiz\u00a0Manella Pereira , and M\u00a0Hadi Amini . 2021. A Survey on Optimal Transport for Machine Learning: Theory and Applications. arXiv preprint arXiv:2106.01963 ( 2021 ). Luis\u00a0Caicedo Torres, Luiz\u00a0Manella Pereira, and M\u00a0Hadi Amini. 2021. A Survey on Optimal Transport for Machine Learning: Theory and Applications. arXiv preprint arXiv:2106.01963 (2021)."},{"key":"e_1_3_2_1_44_1","volume-title":"Visualizing data using t-SNE.Journal of machine learning research 9, 11","author":"Maaten Laurens Van\u00a0der","year":"2008","unstructured":"Laurens Van\u00a0der Maaten and Geoffrey Hinton . 2008. Visualizing data using t-SNE.Journal of machine learning research 9, 11 ( 2008 ). Laurens Van\u00a0der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of machine learning research 9, 11 (2008)."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"}],"event":{"name":"Internetware 2023: 14th Asia-Pacific Symposium on Internetware","acronym":"Internetware 2023","location":"Hangzhou China"},"container-title":["Proceedings of the 14th Asia-Pacific Symposium on Internetware"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3609437.3609457","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3609437.3609457","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:24Z","timestamp":1750178784000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3609437.3609457"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,4]]},"references-count":45,"alternative-id":["10.1145\/3609437.3609457","10.1145\/3609437"],"URL":"https:\/\/doi.org\/10.1145\/3609437.3609457","relation":{},"subject":[],"published":{"date-parts":[[2023,8,4]]},"assertion":[{"value":"2023-10-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}