{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T23:09:32Z","timestamp":1777417772098,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,11,3]],"date-time":"2019-11-03T00:00:00Z","timestamp":1572739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"the National Key R&D Program of China","award":["[2018YFB1004703]"],"award-info":[{"award-number":["[2018YFB1004703]"]}]},{"name":"the Shanghai Science and Technology Fund","award":["[17510740200]"],"award-info":[{"award-number":["[17510740200]"]}]},{"name":"the Huawei Innovation Research Program","award":["[HO2018085286]"],"award-info":[{"award-number":["[HO2018085286]"]}]},{"name":"the State Key Laboratory of Air Traffic Management System and Technology","award":["[SKLATM20180X]"],"award-info":[{"award-number":["[SKLATM20180X]"]}]},{"name":"the CCF-Huawei Database System Innovation Research Plan","award":["[CCF-Huawei DBIR2019002A]"],"award-info":[{"award-number":["[CCF-Huawei DBIR2019002A]"]}]},{"name":"the National Natural Science Foundation of China","award":["[61672353]"],"award-info":[{"award-number":["[61672353]"]}]},{"name":"the National Natural Science Foundation of China","award":["[61872238]"],"award-info":[{"award-number":["[61872238]"]}]},{"name":"the Tencent Social Ads Rhino-Bird Focused Research Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,11,3]]},"DOI":"10.1145\/3357384.3358027","type":"proceedings-article","created":{"date-parts":[[2019,11,4]],"date-time":"2019-11-04T14:11:35Z","timestamp":1572876695000},"page":"1633-1641","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Reinforcement Learning with Sequential Information Clustering in Real-Time Bidding"],"prefix":"10.1145","author":[{"given":"Junwei","family":"Lu","sequence":"first","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chaoqi","family":"Yang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaofeng","family":"Gao","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liubin","family":"Wang","sequence":"additional","affiliation":[{"name":"Tencent, Shenzhen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Changcheng","family":"Li","sequence":"additional","affiliation":[{"name":"Tencent, Shenzhen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guihai","family":"Chen","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,11,3]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Development of the PID controller","author":"Bennett Stuart","year":"1993","unstructured":"Stuart Bennett . 1993. Development of the PID controller . IEEE control systems , Vol. 13 , 6 ( 1993 ), 58--62. Stuart Bennett. 1993. Development of the PID controller. IEEE control systems , Vol. 13, 6 (1993), 58--62."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Christian Borgs Jennifer Chayes Nicole Immorlica Kamal Jain Omid Etesami and Mohammad Mahdian. 2007. Dynamics of bid optimization in online advertisement auctions. In WWW. ACM 531--540.  Christian Borgs Jennifer Chayes Nicole Immorlica Kamal Jain Omid Etesami and Mohammad Mahdian. 2007. Dynamics of bid optimization in online advertisement auctions. In WWW. ACM 531--540.","DOI":"10.1145\/1242572.1242644"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Andrei Broder Evgeniy Gabrilovich Vanja Josifovski George Mavromatis and Alex Smola. 2011. Bid generation for advanced match in sponsored search. In WSDM. ACM 515--524.  Andrei Broder Evgeniy Gabrilovich Vanja Josifovski George Mavromatis and Alex Smola. 2011. Bid generation for advanced match in sponsored search. In WSDM. ACM 515--524.","DOI":"10.1145\/1935826.1935901"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Han Cai Kan Ren Weinan Zhang Kleanthis Malialis Jun Wang Yong Yu and Defeng Guo. 2017. Real-Time Bidding by Reinforcement Learning in Display Advertising. In WSDM. ACM 661--670.  Han Cai Kan Ren Weinan Zhang Kleanthis Malialis Jun Wang Yong Yu and Defeng Guo. 2017. Real-Time Bidding by Reinforcement Learning in Display Advertising. In WSDM. ACM 661--670.","DOI":"10.1145\/3018661.3018702"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Junxuan Chen Baigui Sun Hao Li Hongtao Lu and Xian-Sheng Hua. 2016. Deep ctr prediction in display advertising. In ACM MM. ACM 811--820.  Junxuan Chen Baigui Sun Hao Li Hongtao Lu and Xian-Sheng Hua. 2016. Deep ctr prediction in display advertising. In ACM MM. ACM 811--820.","DOI":"10.1145\/2964284.2964325"},{"key":"e_1_3_2_1_6_1","volume-title":"Mats Brorsson, and Omar Cherkaoui.","author":"Du Manxing","year":"2017","unstructured":"Manxing Du , Redouane Sassioui , Georgios Varisteas , Radu State , Mats Brorsson, and Omar Cherkaoui. 2017 . Improving Real-Time Bidding Using a Constrained Markov Decision Process. In ADMA. 711--726. Manxing Du, Redouane Sassioui, Georgios Varisteas, Radu State, Mats Brorsson, and Omar Cherkaoui. 2017. Improving Real-Time Bidding Using a Constrained Markov Decision Process. In ADMA. 711--726."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Junqi Jin Chengru Song Han Li Kun Gai Jun Wang and Weinan Zhang. 2018. Real-time bidding with multi-agent reinforcement learning in display advertising. In CIKM. ACM 2193--2201.  Junqi Jin Chengru Song Han Li Kun Gai Jun Wang and Weinan Zhang. 2018. Real-time bidding with multi-agent reinforcement learning in display advertising. In CIKM. ACM 2193--2201.","DOI":"10.1145\/3269206.3272021"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Yuchin Juan Yong Zhuang Wei-Sheng Chin and Chih-Jen Lin. 2016. Field-aware factorization machines for CTR prediction. In RecSys. ACM 43--50.  Yuchin Juan Yong Zhuang Wei-Sheng Chin and Chih-Jen Lin. 2016. Field-aware factorization machines for CTR prediction. In RecSys. ACM 43--50.","DOI":"10.1145\/2959100.2959134"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.2946664"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2854283"},{"key":"e_1_3_2_1_11_1","unstructured":"Jiwei Li Will Monroe Tianlin Shi S\u00e9bastien Jean Alan Ritter and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In EMNLP . 2157--2169.  Jiwei Li Will Monroe Tianlin Shi S\u00e9bastien Jean Alan Ritter and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In EMNLP . 2157--2169."},{"key":"e_1_3_2_1_12_1","unstructured":"Shiau Hong Lim Huan Xu and Shie Mannor. 2013. Reinforcement learning in robust markov decision processes. In NIPS. 701--709.  Shiau Hong Lim Huan Xu and Shie Mannor. 2013. Reinforcement learning in robust markov decision processes. In NIPS. 701--709."},{"key":"e_1_3_2_1_13_1","volume-title":"OpenAI Pieter Abbeel, and Igor Mordatch","author":"Lowe Ryan","year":"2017","unstructured":"Ryan Lowe , Yi Wu , Aviv Tamar , Jean Harb , OpenAI Pieter Abbeel, and Igor Mordatch . 2017 . Multi-agent actor-critic for mixed cooperative-competitive environments. In NIPS . 6379--6390. Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. In NIPS . 6379--6390."},{"key":"e_1_3_2_1_14_1","unstructured":"Travis Mandel Yun-En Liu Emma Brunskill and Zoran Popovic. 2016. Efficient Bayesian Clustering for Reinforcement Learning.. In IJCAI . 1830--1838.  Travis Mandel Yun-En Liu Emma Brunskill and Zoran Popovic. 2016. Efficient Bayesian Clustering for Reinforcement Learning.. In IJCAI . 1830--1838."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Shie Mannor Ishai Menache Amit Hoze and Uri Klein. 2004. Dynamic abstraction in reinforcement learning via clustering. In ICML. ACM 71.  Shie Mannor Ishai Menache Amit Hoze and Uri Klein. 2004. Dynamic abstraction in reinforcement learning via clustering. In ICML. ACM 71.","DOI":"10.1145\/1015330.1015355"},{"key":"e_1_3_2_1_16_1","volume-title":"Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.","author":"Mnih Volodymyr","year":"2016","unstructured":"Volodymyr Mnih , Adria Puigdomenech Badia , Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016 . Asynchronous methods for deep reinforcement learning. In ICML. 1928--1937. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In ICML. 1928--1937."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Claudia Perlich Brian Dalessandro Rod Hook Ori Stitelman Troy Raeder and Foster Provost. 2012. Bid optimizing and inventory scoring in targeted online advertising. In SIGKDD. ACM 804--812.  Claudia Perlich Brian Dalessandro Rod Hook Ori Stitelman Troy Raeder and Foster Provost. 2012. Bid optimizing and inventory scoring in targeted online advertising. In SIGKDD. ACM 804--812.","DOI":"10.1145\/2339530.2339655"},{"key":"e_1_3_2_1_18_1","volume-title":"Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542","author":"Pinto Lerrel","year":"2017","unstructured":"Lerrel Pinto , Marcin Andrychowicz , Peter Welinder , Wojciech Zaremba , and Pieter Abbeel . 2017. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542 ( 2017 ). Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. 2017. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542 (2017)."},{"key":"e_1_3_2_1_19_1","unstructured":"Satinder P Singh Tommi Jaakkola and Michael I Jordan. 1995. Reinforcement learning with soft state aggregation. In NIPS. 361--368.  Satinder P Singh Tommi Jaakkola and Michael I Jordan. 1995. Reinforcement learning with soft state aggregation. In NIPS. 361--368."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICTAI.2012.101"},{"key":"e_1_3_2_1_21_1","unstructured":"Harm Van Seijen Mehdi Fatemi Joshua Romoff Romain Laroche Tavian Barnes and Jeffrey Tsang. 2017. Hybrid reward architecture for reinforcement learning. In NIPS. 5392--5402.  Harm Van Seijen Mehdi Fatemi Joshua Romoff Romain Laroche Tavian Barnes and Jeffrey Tsang. 2017. Hybrid reward architecture for reinforcement learning. In NIPS. 5392--5402."},{"key":"e_1_3_2_1_22_1","volume-title":"Position auctions. international Journal of industrial Organization","author":"Varian Hal R","year":"2007","unstructured":"Hal R Varian . 2007. Position auctions. international Journal of industrial Organization , Vol. 25 , 6 ( 2007 ), 1163--1178. Hal R Varian. 2007. Position auctions. international Journal of industrial Organization , Vol. 25, 6 (2007), 1163--1178."},{"key":"e_1_3_2_1_23_1","volume-title":"LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions. arXiv preprint arXiv:1708.05565","author":"Wang Yu","year":"2017","unstructured":"Yu Wang , Jiayi Liu , Yuxiang Liu , Jun Hao , Yang He , Jinghe Hu , Weipeng Yan , and Mantian Li . 2017 . LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions. arXiv preprint arXiv:1708.05565 (2017). Yu Wang, Jiayi Liu, Yuxiang Liu, Jun Hao, Yang He, Jinghe Hu, Weipeng Yan, and Mantian Li. 2017. LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions. arXiv preprint arXiv:1708.05565 (2017)."},{"key":"e_1_3_2_1_24_1","volume-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning","author":"Williams Ronald J","year":"1992","unstructured":"Ronald J Williams . 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , Vol. 8 , 3--4 ( 1992 ), 229--256. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , Vol. 8, 3--4 (1992), 229--256."},{"key":"e_1_3_2_1_25_1","unstructured":"Jian Xu Kuang-chih Lee Wentong Li Hang Qi and Quan Lu. 2015. Smart pacing for effective online ad campaign optimization. In SIGKDD. ACM 2217--2226.  Jian Xu Kuang-chih Lee Wentong Li Hang Qi and Quan Lu. 2015. Smart pacing for effective online ad campaign optimization. In SIGKDD. ACM 2217--2226."},{"key":"e_1_3_2_1_26_1","volume-title":"A Nonparametric Delayed Feedback Model for Conversion Rate Prediction. arXiv preprint arXiv:1802.00255","author":"Yoshikawa Yuya","year":"2018","unstructured":"Yuya Yoshikawa and Yusaku Imai . 2018. A Nonparametric Delayed Feedback Model for Conversion Rate Prediction. arXiv preprint arXiv:1802.00255 ( 2018 ). Yuya Yoshikawa and Yusaku Imai. 2018. A Nonparametric Delayed Feedback Model for Conversion Rate Prediction. arXiv preprint arXiv:1802.00255 (2018)."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501040.2501980"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Weinan Zhang Yifei Rong Jun Wang Tianchi Zhu and Xiaofan Wang. 2016. Feedback control of real-time display advertising. In WSDM. ACM 407--416.  Weinan Zhang Yifei Rong Jun Wang Tianchi Zhu and Xiaofan Wang. 2016. Feedback control of real-time display advertising. In WSDM. ACM 407--416.","DOI":"10.1145\/2835776.2835843"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Weinan Zhang Shuai Yuan and Jun Wang. 2014. Optimal real-time bidding for display advertising. In SIGKDD. ACM 1077--1086.  Weinan Zhang Shuai Yuan and Jun Wang. 2014. Optimal real-time bidding for display advertising. In SIGKDD. ACM 1077--1086.","DOI":"10.1145\/2623330.2623633"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Jun Zhao Guang Qiu Ziyu Guan Wei Zhao and Xiaofei He. 2018. Deep Reinforcement Learning for Sponsored Search Real-time Bidding. In SIGKDD. ACM 1021--1030.  Jun Zhao Guang Qiu Ziyu Guan Wei Zhao and Xiaofei He. 2018. Deep Reinforcement Learning for Sponsored Search Real-time Bidding. In SIGKDD. ACM 1021--1030.","DOI":"10.1145\/3219819.3219918"}],"event":{"name":"CIKM '19: The 28th ACM International Conference on Information and Knowledge Management","location":"Beijing China","acronym":"CIKM '19","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 28th ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357384.3358027","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3357384.3358027","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:18Z","timestamp":1750203858000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357384.3358027"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,3]]},"references-count":30,"alternative-id":["10.1145\/3357384.3358027","10.1145\/3357384"],"URL":"https:\/\/doi.org\/10.1145\/3357384.3358027","relation":{},"subject":[],"published":{"date-parts":[[2019,11,3]]},"assertion":[{"value":"2019-11-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}