{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T22:08:37Z","timestamp":1750889317475,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":37,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T00:00:00Z","timestamp":1697846400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key R&D Plan","award":["No.2022YFC3303302"],"award-info":[{"award-number":["No.2022YFC3303302"]}]},{"name":"National Natural Science Foundation of China under Grant","award":["No.61976204"],"award-info":[{"award-number":["No.61976204"]}]},{"DOI":"10.13039\/501100005090","name":"Beijing Nova Program","doi-asserted-by":"publisher","award":["Z201100006820062"],"award-info":[{"award-number":["Z201100006820062"]}],"id":[{"id":"10.13039\/501100005090","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Youth Innovation Promotion Association CAS"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,10,21]]},"DOI":"10.1145\/3583780.3615161","type":"proceedings-article","created":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T07:45:26Z","timestamp":1697874326000},"page":"3918-3922","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Leveraging Post-Click User Behaviors for Calibrated Conversion Rate Prediction Under Delayed Feedback in Online Advertising"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2245-505X","authenticated-orcid":false,"given":"Yuyao","family":"Guo","sequence":"first","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences & University of Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9633-8361","authenticated-orcid":false,"given":"Xiang","family":"Ao","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences & University of Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3879-0576","authenticated-orcid":false,"given":"Qiming","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences & University of Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8833-5398","authenticated-orcid":false,"given":"Qing","family":"He","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences & University of Chinese Academy of Sciences, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2023,10,21]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_3_2_1_1_1","DOI":"10.1007\/978-3-030-67667-4_13"},{"key":"e_1_3_2_1_2_1","volume-title":"Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al.","author":"Gawlikowski Jakob","year":"2021","unstructured":"Jakob Gawlikowski , Cedrique Rovile Njieutcheu Tassi , Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. 2021 . A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342 (2021). Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. 2021. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342 (2021)."},{"key":"e_1_3_2_1_3_1","volume-title":"Weinberger","author":"Guo Chuan","year":"2017","unstructured":"Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q . Weinberger . 2017 . On Calibration of Modern Neural Networks. In ICML (Proceedings of Machine Learning Research , Vol. 70). PMLR, 1321-- 1330 . Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In ICML (Proceedings of Machine Learning Research, Vol. 70). PMLR, 1321--1330."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_4_1","DOI":"10.1145\/3511808.3557557"},{"key":"e_1_3_2_1_5_1","first-page":"3711","article-title":"Distribution-free binary classification: prediction sets, confidence intervals and calibration","volume":"33","author":"Gupta Chirag","year":"2020","unstructured":"Chirag Gupta , Aleksandr Podkopaev , and Aaditya Ramdas . 2020 . Distribution-free binary classification: prediction sets, confidence intervals and calibration . Advances in Neural Information Processing Systems , Vol. 33 (2020), 3711 -- 3723 . Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. 2020. Distribution-free binary classification: prediction sets, confidence intervals and calibration. Advances in Neural Information Processing Systems , Vol. 33 (2020), 3711--3723.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_6_1","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition. 7482--7491","author":"Kendall Alex","year":"2018","unstructured":"Alex Kendall , Yarin Gal , and Roberto Cipolla . 2018 . Multi-task learning using uncertainty to weigh losses for scene geometry and semantics . In Proceedings of the IEEE conference on computer vision and pattern recognition. 7482--7491 . Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7482--7491."},{"key":"e_1_3_2_1_7_1","volume-title":"Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12","author":"Kingma Diederik","year":"2014","unstructured":"Diederik Kingma and Jimmy Ba . 2014 . Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014). Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014)."},{"key":"e_1_3_2_1_8_1","volume-title":"Deepak Dilipkumar, Ferenc Husz\u00e1 r, Steven Yoo, and Wenzhe Shi.","author":"Ktena Sofia Ira","year":"2019","unstructured":"Sofia Ira Ktena , Alykhan Tejani , Lucas Theis , Pranay Kumar Myana , Deepak Dilipkumar, Ferenc Husz\u00e1 r, Steven Yoo, and Wenzhe Shi. 2019 . Addressing delayed feedback for continuous training with neural networks in CTR prediction. In RecSys. ACM , 187--195. Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Husz\u00e1 r, Steven Yoo, and Wenzhe Shi. 2019. Addressing delayed feedback for continuous training with neural networks in CTR prediction. In RecSys. ACM, 187--195."},{"key":"e_1_3_2_1_9_1","volume-title":"Telmo de Menezes e Silva Filho, and Peter A. Flach","author":"Kull Meelis","year":"2017","unstructured":"Meelis Kull , Telmo de Menezes e Silva Filho, and Peter A. Flach . 2017 . Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In AISTATS (Proceedings of Machine Learning Research , Vol. 54). PMLR, 623-- 631 . Meelis Kull, Telmo de Menezes e Silva Filho, and Peter A. Flach. 2017. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In AISTATS (Proceedings of Machine Learning Research, Vol. 54). PMLR, 623--631."},{"key":"e_1_3_2_1_10_1","volume-title":"Hao Song, and Peter A. Flach.","author":"Kull Meelis","year":"2019","unstructured":"Meelis Kull , Miquel Perell\u00f3 - Nieto , Markus K\"a ngsepp , Telmo de Menezes e Silva Filho , Hao Song, and Peter A. Flach. 2019 . Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. CoRR , Vol. abs\/ 1910 .12656 (2019). Meelis Kull, Miquel Perell\u00f3 -Nieto, Markus K\"a ngsepp, Telmo de Menezes e Silva Filho, Hao Song, and Peter A. Flach. 2019. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. CoRR , Vol. abs\/1910.12656 (2019)."},{"unstructured":"Ananya Kumar Percy Liang and Tengyu Ma. 2019. Verified Uncertainty Calibration. (2019) 3787--3798. Ananya Kumar Percy Liang and Tengyu Ma. 2019. Verified Uncertainty Calibration. (2019) 3787--3798.","key":"e_1_3_2_1_11_1"},{"key":"e_1_3_2_1_12_1","volume-title":"Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems","author":"Lakshminarayanan Balaji","year":"2017","unstructured":"Balaji Lakshminarayanan , Alexander Pritzel , and Charles Blundell . 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , Vol. 30 ( 2017 ). Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , Vol. 30 (2017)."},{"key":"e_1_3_2_1_13_1","volume-title":"Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909.13550","author":"Laves Max-Heinrich","year":"2019","unstructured":"Max-Heinrich Laves , Sontje Ihler , Karl-Philipp Kortmann , and Tobias Ortmaier . 2019. Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909.13550 ( 2019 ). Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, and Tobias Ortmaier. 2019. Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909.13550 (2019)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_14_1","DOI":"10.1145\/3404835.3463045"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_15_1","DOI":"10.1145\/3539618.3591747"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_16_1","DOI":"10.1145\/3209978.3210104"},{"key":"e_1_3_2_1_17_1","volume-title":"Proc. icml","volume":"30","author":"Maas Andrew L","year":"2013","unstructured":"Andrew L Maas , Awni Y Hannun , Andrew Y Ng , 2013 . Rectifier nonlinearities improve neural network acoustic models . In Proc. icml , Vol. 30 . Citeseer, 3. Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, Vol. 30. Citeseer, 3."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_18_1","DOI":"10.1109\/TMI.2020.3006437"},{"key":"e_1_3_2_1_19_1","volume-title":"Dokania","author":"Mukhoti Jishnu","year":"2020","unstructured":"Jishnu Mukhoti , Viveka Kulharia , Amartya Sanyal , Stuart Golodetz , Philip H. S. Torr , and Puneet K . Dokania . 2020 . Calibrating Deep Neural Networks using Focal Loss . (2020). Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet K. Dokania. 2020. Calibrating Deep Neural Networks using Focal Loss. (2020)."},{"key":"e_1_3_2_1_20_1","volume-title":"Hinton","author":"Rafael M\u00fc","year":"2019","unstructured":"Rafael M\u00fc ller, Simon Kornblith , and Geoffrey E . Hinton . 2019 . When does label smoothing help? (2019), 4696--4705. Rafael M\u00fc ller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? (2019), 4696--4705."},{"volume-title":"Obtaining Well Calibrated Probabilities Using Bayesian Binning","author":"Naeini Mahdi Pakdaman","unstructured":"Mahdi Pakdaman Naeini , Gregory F. Cooper , and Milos Hauskrecht . 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning . In AAAI. AAAI Press , 2901--2907. Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI. AAAI Press, 2901--2907.","key":"e_1_3_2_1_21_1"},{"key":"e_1_3_2_1_22_1","volume-title":"Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions. In WWW. ACM \/ IW3C2, 729--739.","author":"Pan Feiyang","year":"2020","unstructured":"Feiyang Pan , Xiang Ao , Pingzhong Tang , Min Lu , Dapeng Liu , Lei Xiao , and Qing He . 2020 . Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions. In WWW. ACM \/ IW3C2, 729--739. Feiyang Pan, Xiang Ao, Pingzhong Tang, Min Lu, Dapeng Liu, Lei Xiao, and Qing He. 2020. Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions. In WWW. ACM \/ IW3C2, 729--739."},{"key":"e_1_3_2_1_23_1","volume-title":"Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548","author":"Pereyra Gabriel","year":"2017","unstructured":"Gabriel Pereyra , George Tucker , Jan Chorowski , \u0141ukasz Kaiser , and Geoffrey Hinton . 2017. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 ( 2017 ). Gabriel Pereyra, George Tucker, Jan Chorowski, \u0141ukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 (2017)."},{"doi-asserted-by":"crossref","unstructured":"John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers Vol. 10 3 (1999) 61--74. John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers Vol. 10 3 (1999) 61--74.","key":"e_1_3_2_1_24_1","DOI":"10.7551\/mitpress\/1113.003.0008"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_25_1","DOI":"10.1109\/CVPR.2019.00924"},{"unstructured":"Shuyang Sun Jie-Neng Chen Ruifei He Alan Yuille Philip Torr and Song Bai. 2022. LUMix: Improving Mixup by Better Modelling Label Uncertainty. arxiv: 2211.15846 [cs.CV] Shuyang Sun Jie-Neng Chen Ruifei He Alan Yuille Philip Torr and Song Bai. 2022. LUMix: Improving Mixup by Better Modelling Label Uncertainty. arxiv: 2211.15846 [cs.CV]","key":"e_1_3_2_1_26_1"},{"doi-asserted-by":"crossref","unstructured":"Sunil Thulasidasan Gopinath Chennupati Jeff A. Bilmes Tanmoy Bhattacharya and Sarah Michalak. 2019. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. (2019) 13888--13899. Sunil Thulasidasan Gopinath Chennupati Jeff A. Bilmes Tanmoy Bhattacharya and Sarah Michalak. 2019. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. (2019) 13888--13899.","key":"e_1_3_2_1_27_1","DOI":"10.2172\/1525811"},{"key":"e_1_3_2_1_28_1","volume-title":"Heteroscedastic calibration of uncertainty estimators in deep learning. arXiv preprint arXiv:1910.14179","author":"Venkatesh Bindya","year":"2019","unstructured":"Bindya Venkatesh and Jayaraman J Thiagarajan . 2019. Heteroscedastic calibration of uncertainty estimators in deep learning. arXiv preprint arXiv:1910.14179 ( 2019 ). Bindya Venkatesh and Jayaraman J Thiagarajan. 2019. Heteroscedastic calibration of uncertainty estimators in deep learning. arXiv preprint arXiv:1910.14179 (2019)."},{"doi-asserted-by":"crossref","unstructured":"Penghui Wei Weimin Zhang Ruijie Hou Jinquan Liu Shaoguo Liu Liang Wang and Bo Zheng. 2022. Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. (2022) 2645--2649. Penghui Wei Weimin Zhang Ruijie Hou Jinquan Liu Shaoguo Liu Liang Wang and Bo Zheng. 2022. Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. (2022) 2645--2649.","key":"e_1_3_2_1_29_1","DOI":"10.1145\/3477495.3531911"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_30_1","DOI":"10.1145\/3404835.3463053"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_31_1","DOI":"10.1145\/3397271.3401443"},{"key":"e_1_3_2_1_32_1","volume-title":"Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems","author":"Wilson Andrew G","year":"2020","unstructured":"Andrew G Wilson and Pavel Izmailov . 2020. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems , Vol. 33 ( 2020 ), 4697--4708. Andrew G Wilson and Pavel Izmailov. 2020. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems , Vol. 33 (2020), 4697--4708."},{"doi-asserted-by":"crossref","unstructured":"Jia-Qi Yang Xiang Li Shuguang Han Tao Zhuang De-Chuan Zhan Xiaoyi Zeng and Bin Tong. 2021. Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling. (2021) 4582--4589. Jia-Qi Yang Xiang Li Shuguang Han Tao Zhuang De-Chuan Zhan Xiaoyi Zeng and Bin Tong. 2021. Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling. (2021) 4582--4589.","key":"e_1_3_2_1_33_1","DOI":"10.1609\/aaai.v35i5.16587"},{"unstructured":"Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML. Morgan Kaufmann 609--616. Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML. Morgan Kaufmann 609--616.","key":"e_1_3_2_1_34_1"},{"doi-asserted-by":"crossref","unstructured":"Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In KDD. ACM 694--699. Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In KDD. ACM 694--699.","key":"e_1_3_2_1_35_1","DOI":"10.1145\/775047.775151"},{"key":"e_1_3_2_1_36_1","volume-title":"ICML (Proceedings of Machine Learning Research","volume":"11128","author":"Zhang Jize","year":"2020","unstructured":"Jize Zhang , Bhavya Kailkhura , and Thomas Yong-Jin Han . 2020 . Mix-n-Match : Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning . In ICML (Proceedings of Machine Learning Research , Vol. 119). PMLR, 11117-- 11128 . Jize Zhang, Bhavya Kailkhura, and Thomas Yong-Jin Han. 2020. Mix-n-Match : Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. In ICML (Proceedings of Machine Learning Research, Vol. 119). PMLR, 11117--11128."},{"key":"e_1_3_2_1_37_1","volume-title":"Confidence calibration for convolutional neural networks using structured dropout. arXiv preprint arXiv:1906.09551","author":"Zhang Zhilu","year":"2019","unstructured":"Zhilu Zhang , Adrian V Dalca , and Mert R Sabuncu . 2019. Confidence calibration for convolutional neural networks using structured dropout. arXiv preprint arXiv:1906.09551 ( 2019 ). Zhilu Zhang, Adrian V Dalca, and Mert R Sabuncu. 2019. Confidence calibration for convolutional neural networks using structured dropout. arXiv preprint arXiv:1906.09551 (2019)."}],"event":{"sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"],"acronym":"CIKM '23","name":"CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management","location":"Birmingham United Kingdom"},"container-title":["Proceedings of the 32nd ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583780.3615161","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583780.3615161","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:43Z","timestamp":1750178203000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583780.3615161"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,21]]},"references-count":37,"alternative-id":["10.1145\/3583780.3615161","10.1145\/3583780"],"URL":"https:\/\/doi.org\/10.1145\/3583780.3615161","relation":{},"subject":[],"published":{"date-parts":[[2023,10,21]]},"assertion":[{"value":"2023-10-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}