{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T11:23:35Z","timestamp":1778757815569,"version":"3.51.4"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2024,5,25]],"date-time":"2024-05-25T00:00:00Z","timestamp":1716595200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,25]],"date-time":"2024-05-25T00:00:00Z","timestamp":1716595200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Active learning can improve the efficiency of training prediction models by identifying the most informative new labels to acquire. However, non-response to label requests can impact active learning\u2019s effectiveness in real-world contexts. We conceptualise this degradation by considering the type of non-response present in the data, demonstrating that biased non-response is particularly detrimental to model performance. We argue that biased non-response is likely in contexts where the labelling process, by nature, relies on user interactions. To mitigate the impact of biased non-response, we propose a cost-based correction to the sampling strategy\u2013the <jats:italic>Upper Confidence Bound of the Expected Utility (UCB-EU)<\/jats:italic>\u2013that can, plausibly, be applied to any active learning algorithm. Through experiments, we demonstrate that our method successfully reduces the harm from labelling non-response in many settings. However, we also characterise settings where the non-response bias in the annotations remains detrimental under UCB-EU for specific sampling methods and data generating processes. Finally, we evaluate our method on a real-world dataset from an e-commerce platform. We show that UCB-EU yields substantial performance improvements to conversion models that are trained on clicked impressions. Most generally, this research serves to both better conceptualise the interplay between types of non-response and model improvements via active learning, and to provide a practical, easy-to-implement correction that mitigates model degradation.<\/jats:p>","DOI":"10.1007\/s10618-024-01026-x","type":"journal-article","created":{"date-parts":[[2024,5,25]],"date-time":"2024-05-25T07:01:46Z","timestamp":1716620506000},"page":"2117-2140","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Active learning with biased non-response to label requests"],"prefix":"10.1007","volume":"38","author":[{"given":"Thomas S.","family":"Robinson","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Niek","family":"Tax","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard","family":"Mudd","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ido","family":"Guy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,5,25]]},"reference":[{"key":"1026_CR1","unstructured":"Amin K, DeSalvo G, Rostamizadeh A (2021) Learning with labeling induced abstentions. In: Advances in Neural Information Processing Systems, pp 12576\u201312586"},{"issue":"2","key":"1026_CR2","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/1964897.1964906","volume":"12","author":"J Attenberg","year":"2011","unstructured":"Attenberg J, Provost F (2011) Inactive learning? difficulties employing active learning in practice. ACM SIGKDD Explorations Newsl 12(2):36\u201341","journal-title":"ACM SIGKDD Explorations Newsl"},{"key":"1026_CR3","unstructured":"Audibert JY, Bubeck S, Munos R (2010) Best arm identification in multi-armed bandits. In: COLT, pp 41\u201353"},{"key":"1026_CR4","doi-asserted-by":"crossref","unstructured":"Barbieri N, Silvestri F, Lalmas M (2016) Improving post-click user engagement on native ads via survival analysis. In: Proceedings of the 25th International Conference on World Wide Web, pp 761\u2013770","DOI":"10.1145\/2872427.2883092"},{"issue":"4","key":"1026_CR5","doi-asserted-by":"publisher","first-page":"967","DOI":"10.1287\/moor.2014.0663","volume":"39","author":"G Bart\u00f3k","year":"2014","unstructured":"Bart\u00f3k G, Foster DP, P\u00e1l D et al (2014) Partial monitoring-classification, regret bounds, and algorithms. Math Oper Res 39(4):967\u2013997","journal-title":"Math Oper Res"},{"key":"1026_CR6","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1007\/s41060-018-0116-z","volume":"5","author":"F Carcillo","year":"2018","unstructured":"Carcillo F, Le Borgne YA, Caelen O et al (2018) Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization. Int J Data Sci Anal 5:285\u2013300","journal-title":"Int J Data Sci Anal"},{"key":"1026_CR7","unstructured":"Cortes C, DeSalvo G, Gentile C, et\u00a0al (2018) Online learning with abstention. In: International conference on machine learning, pp 1059\u20131067"},{"key":"1026_CR8","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1016\/j.cosrev.2016.05.002","volume":"20","author":"M Elahi","year":"2016","unstructured":"Elahi M, Ricci F, Rubens N (2016) A survey of active learning in collaborative filtering recommender systems. Comput Sci Rev 20:29\u201350","journal-title":"Comput Sci Rev"},{"key":"1026_CR9","unstructured":"Fang M, Zhu X, Zhang C (2012) Active learning from oracle with knowledge blind spot. In: Twenty-Sixth AAAI Conference on Artificial Intelligence"},{"key":"1026_CR10","unstructured":"Farquhar S, Gal Y, Rainforth T (2021) On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665"},{"issue":"2\u20133","key":"1026_CR11","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1023\/A:1007330508534","volume":"28","author":"Y Freund","year":"1997","unstructured":"Freund Y, Seung HS, Shamir E et al (1997) Selective sampling using the query by committee algorithm. Mach Learn 28(2\u20133):133","journal-title":"Mach Learn"},{"key":"1026_CR12","unstructured":"Gardner J, Pleiss G, Weinberger KQ, et\u00a0al (2018) GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. In: Advances in neural information processing systems"},{"issue":"236","key":"1026_CR13","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1080\/01621459.1946.10501894","volume":"41","author":"MH Hansen","year":"1946","unstructured":"Hansen MH, Hurwitz WN (1946) The problem of non-response in sample surveys. J Am Stat Assoc 41(236):517\u2013529","journal-title":"J Am Stat Assoc"},{"issue":"10","key":"1026_CR14","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1109\/TPAMI.2014.2307881","volume":"36","author":"SJ Huang","year":"2014","unstructured":"Huang SJ, Jin R, Zhou ZH (2014) Active learning by querying informative and representative examples. IEEE Trans Pattern Anal Mach Intell 36(10):1936\u20131949","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"1026_CR15","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.ins.2022.10.066","volume":"616","author":"Q Jin","year":"2022","unstructured":"Jin Q, Yuan M, Li S et al (2022) Cold-start active learning for image classification. Inf Sci 616:16\u201336","journal-title":"Inf Sci"},{"issue":"1","key":"1026_CR16","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1017\/S0003055401000235","volume":"95","author":"G King","year":"2001","unstructured":"King G, Honaker J, Joseph A et al (2001) Analyzing incomplete political science data: an alternative algorithm for multiple imputation. Am Polit Sci Rev 95(1):49\u201369","journal-title":"Am Polit Sci Rev"},{"issue":"2","key":"1026_CR17","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1017\/pan.2020.49","volume":"30","author":"R Lall","year":"2022","unstructured":"Lall R, Robinson T (2022) The midas touch: Accurate and scalable missing-data imputation with deep learning. Polit Anal 30(2):179\u2013196","journal-title":"Polit Anal"},{"key":"1026_CR18","doi-asserted-by":"publisher","DOI":"10.1017\/9781108571401","volume-title":"Bandit algorithms","author":"T Lattimore","year":"2020","unstructured":"Lattimore T, Szepesv\u00e1ri C (2020) Bandit algorithms. Cambridge University Press"},{"key":"1026_CR19","doi-asserted-by":"crossref","unstructured":"Lewis DD (1995) A sequential algorithm for training text classifiers: Corrigendum and additional data. In: ACM SIGIR Forum, pp 13\u201319","DOI":"10.1145\/219587.219592"},{"key":"1026_CR20","doi-asserted-by":"crossref","unstructured":"Lin C, Mausam M, Weld D (2016) Re-active learning: Active learning with relabeling. In: Proceedings of the AAAI Conference on Artificial Intelligence","DOI":"10.1609\/aaai.v30i1.10315"},{"key":"1026_CR21","doi-asserted-by":"crossref","unstructured":"Lin X, Chen X, Song L, et\u00a0al (2023) Tree based progressive regression model for watch-time prediction in short-video recommendation. arXiv preprint arXiv:2306.03392","DOI":"10.1145\/3580305.3599919"},{"key":"1026_CR22","volume-title":"Statistical analysis with missing data,","author":"RJ Little","year":"2019","unstructured":"Little RJ, Rubin DB (2019) Statistical analysis with missing data, vol 793. John Wiley & Sons"},{"key":"1026_CR23","doi-asserted-by":"crossref","unstructured":"Ma X, Zhao L, Huang G, et\u00a0al (2018) Entire space multi-task model: An effective approach for estimating post-click conversion rate. In: Proceedings of the International ACM SIGIR Conference on Research & Development in Information Retrieval, pp 1137\u20131140","DOI":"10.1145\/3209978.3210104"},{"key":"1026_CR24","unstructured":"McCallum A, Nigam K, et\u00a0al (1998) Employing EM and pool-based active learning for text classification. In: ICML, pp 350\u2013358"},{"key":"1026_CR25","unstructured":"Mohan K, Pearl J, Tian J (2013) Graphical models for inference with missing data"},{"key":"1026_CR26","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1016\/j.neucom.2021.11.027","volume":"471","author":"CV Nguyen","year":"2022","unstructured":"Nguyen CV, Ho LST, Xu H et al (2022) Bayesian active learning with abstention feedbacks. Neurocomputing 471:242\u2013250","journal-title":"Neurocomputing"},{"key":"1026_CR27","doi-asserted-by":"crossref","unstructured":"Nguyen VA, Shi P, Ramakrishnan J, et\u00a0al (2020) CLARA: confidence of labels and raters. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 2542\u20132552","DOI":"10.1145\/3394486.3403304"},{"key":"1026_CR28","doi-asserted-by":"crossref","unstructured":"Rosales R, Cheng H, Manavoglu E (2012) Post-click conversion modeling and analysis for non-guaranteed delivery display advertising. In: Proceedings of the fifth ACM international conference on Web search and data mining, pp 293\u2013302","DOI":"10.1145\/2124295.2124333"},{"issue":"3","key":"1026_CR29","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1093\/biomet\/63.3.581","volume":"63","author":"DB Rubin","year":"1976","unstructured":"Rubin DB (1976) Inference and missing data. Biometrika 63(3):581\u2013592","journal-title":"Biometrika"},{"key":"1026_CR30","unstructured":"Settles B (2009) Active learning literature survey technical report. University of Wisconsin-Madison Department of Computer Sciences"},{"key":"1026_CR31","doi-asserted-by":"crossref","unstructured":"Settles B (2012) Uncertainty sampling. In: Active Learning. Springer, p 11\u201320","DOI":"10.1007\/978-3-031-01560-1_2"},{"key":"1026_CR32","doi-asserted-by":"crossref","unstructured":"Seung HS, Opper M, Sompolinsky H (1992) Query by committee. In: Proceedings of the fifth annual workshop on Computational learning theory, pp 287\u2013294","DOI":"10.1145\/130385.130417"},{"key":"1026_CR33","doi-asserted-by":"crossref","unstructured":"Sheng VS, Provost F, Ipeirotis PG (2008) Get another label? improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining, pp 614\u2013622","DOI":"10.1145\/1401890.1401965"},{"issue":"1","key":"1026_CR34","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1093\/bioinformatics\/btr597","volume":"28","author":"DJ Stekhoven","year":"2012","unstructured":"Stekhoven DJ, B\u00fchlmann P (2012) Missforest-non-parametric missing value imputation for mixed-type data. Bioinformatics 28(1):112\u2013118","journal-title":"Bioinformatics"},{"key":"1026_CR35","doi-asserted-by":"crossref","unstructured":"Tax N, de Vries KJ, de Jong M et al (2021) (2021) Machine learning for fraud detection in e-commerce: A research agenda. Deployable Machine Learning for Security Defense: Second International Workshop, MLHat 2021, Virtual Event, August 15. Springer, pp 30\u201354","DOI":"10.1007\/978-3-030-87839-9_2"},{"key":"1026_CR36","unstructured":"Tianchi (2018) Ad display\/click data on taobao.com. https:\/\/tianchi.aliyun.com\/dataset\/dataDetail?dataId=56"},{"key":"1026_CR37","unstructured":"Tifrea A, Clarysse J, Yang F (2023) Margin-based sampling in high dimensions: When being active is less efficient than staying passive. In: International Conference on Machine Learning (ICML), vol 202. PMLR, pp 34222\u201334262"},{"key":"1026_CR38","doi-asserted-by":"crossref","unstructured":"Yan S, Chaudhuri K, Javidi T (2015) Active learning from noisy and abstention feedback. In: 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp 1352\u20131357","DOI":"10.1109\/ALLERTON.2015.7447165"},{"key":"1026_CR39","unstructured":"Yan S, Chaudhuri K, Javidi T (2016) Active learning from imperfect labelers. In: Advances in Neural Information Processing Systems"},{"key":"1026_CR40","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1016\/j.patcog.2018.06.004","volume":"83","author":"Y Yang","year":"2018","unstructured":"Yang Y, Loog M (2018) A benchmark and comparison of active learning for logistic regression. Pattern Recogn 83:401\u2013415","journal-title":"Pattern Recogn"},{"key":"1026_CR41","doi-asserted-by":"crossref","unstructured":"Zhao L, Sukthankar G, Sukthankar R (2011) Incremental relabeling for active learning with noisy crowdsourced annotations. In: 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, IEEE, pp 728\u2013733","DOI":"10.1109\/PASSAT\/SocialCom.2011.193"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-024-01026-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-024-01026-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-024-01026-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T10:40:24Z","timestamp":1722336024000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-024-01026-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,25]]},"references-count":41,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["1026"],"URL":"https:\/\/doi.org\/10.1007\/s10618-024-01026-x","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,25]]},"assertion":[{"value":"5 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial\/non-financial interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interest"}},{"value":"There are no ethical issues. This research was considered and approved by the London School of Economics\u2019 research ethics procedure (ref. 183654).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}}]}}