{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T07:48:38Z","timestamp":1744444118333,"version":"3.37.3"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,19]],"date-time":"2022-04-19T00:00:00Z","timestamp":1650326400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,19]],"date-time":"2022-04-19T00:00:00Z","timestamp":1650326400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100013297","name":"Eurostars","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100013297","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Intell Syst"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Real time bidding is one of the most popular ways of selling impressions in online advertising, where online ad publishers allocate some blocks in their websites to sell in online auctions. In real time bidding, ad networks connect publishers and advertisers. There are many available ad networks for publishers to choose from. A possible approach for selecting ad networks and sending ad requests is called Waterfall Strategy, in which ad networks are selected sequentially. The ordering of the ad networks is very important for publishers, and finding the ordering that will provide maximum revenue is a hard problem due to the highly dynamic environment. In this paper, we propose a dynamic ad network ordering method to find the best ordering of ad networks for publishers that opt for Waterfall Strategy to select ad networks. This method consists of two steps. The first step is a prediction model that is trained on real time bidding historical data and provides an estimation of revenue for each impression. These estimations are used as initial values for the Q-table in the second step. The second step is based on Reinforcement Learning and improves the output of the prediction model. By calculating the revenue of our method and comparing that with the revenue of a fixed and predefined ordering method, we show that our proposed dynamic ad network ordering method increases publishers\u2019 revenue.<\/jats:p>","DOI":"10.1007\/s44196-022-00077-6","type":"journal-article","created":{"date-parts":[[2022,4,19]],"date-time":"2022-04-19T15:21:18Z","timestamp":1650381678000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Dynamic Ad Network Ordering Method Using Reinforcement Learning"],"prefix":"10.1007","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1558-8380","authenticated-orcid":false,"given":"Reza Refaei","family":"Afshar","sequence":"first","affiliation":[]},{"given":"Yingqian","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Uzay","family":"Kaymak","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,4,19]]},"reference":[{"key":"77_CR1","doi-asserted-by":"crossref","unstructured":"Afshar, R.R., Zhang, Y., Firat, M., Kaymak, U.: A reinforcement learning method to select ad networks in waterfall strategy. In: Proceedings of the 11th International Conference on Agents and Artificial Intelligence. SCITEPRESS-Science and Technology Publications (2018)","DOI":"10.5220\/0007395502560265"},{"key":"77_CR2","doi-asserted-by":"crossref","unstructured":"Afshar, R.R., Zhang, Y., Firat, M., Kaymak, U.: A decision support method to increase the revenue of ad publishers in waterfall strategy. In: IEEE Conference on Computational Intelligence for Financial Engineering and Economics (CIFEr) (2019)","DOI":"10.1109\/CIFEr.2019.8759106"},{"key":"77_CR3","doi-asserted-by":"crossref","unstructured":"Afshar, R.R., Zhang, Y., Firat, M., Kaymak, U., Metin, A.I., Tarak\u00e7\u0131o\u011flu, G.S., Ba\u015f, C.: Reserve price optimization with header bidding and ad exchange. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp. 830\u2013835 (2020)","DOI":"10.1109\/SMC42975.2020.9283479"},{"key":"77_CR4","doi-asserted-by":"crossref","unstructured":"Austin, D., Seljan, S., Monello, J., Tzeng, S.: Reserve price optimization at scale. In: 2016 IEEE 3rd International Conference on Data Science and Advanced Analytics (DSAA). IEEE, pp. 528\u2013536 (2016)","DOI":"10.1109\/DSAA.2016.32"},{"key":"77_CR5","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-319-25023-6","volume-title":"The Programmatic Advertising Principle","author":"O Busch","year":"2016","unstructured":"Busch, O.: The Programmatic Advertising Principle, pp. 3\u201315. Springer, Cham (2016)"},{"key":"77_CR6","doi-asserted-by":"crossref","unstructured":"Cai, H., Ren, K., Zhang, W., Malialis, K., Wang, J., Yu, Y., Guo, D.: Real-time bidding by reinforcement learning in display advertising. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, pp. 661\u2013670 (2017)","DOI":"10.1145\/3018661.3018702"},{"key":"77_CR7","doi-asserted-by":"publisher","first-page":"875","DOI":"10.1007\/978-0-387-09823-4_45","volume-title":"In Data Mining and Knowledge Discovery Handbook","author":"NV Chawla","year":"2009","unstructured":"Chawla, N.V.: Data mining for imbalanced datasets: an overview. In: In Data Mining and Knowledge Discovery Handbook, pp. 875\u2013886. Springer, Berlin (2009)"},{"key":"77_CR8","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321\u2013357 (2002)","journal-title":"J. Artif. Intell. Res."},{"key":"77_CR9","unstructured":"Dinodia, P.: Header bidding vs waterfall: How the two revenue optimisation hacks differ (2017)"},{"issue":"8","key":"77_CR10","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","author":"T Fawcett","year":"2006","unstructured":"Fawcett, T.: An introduction to roc analysis. Pattern Recogn. Lett. 27(8), 861\u2013874 (2006)","journal-title":"Pattern Recogn. Lett."},{"key":"77_CR11","doi-asserted-by":"publisher","DOI":"10.16997\/book23","volume-title":"The Online Advertising Tax as the Foundation of a Public Service Internet","author":"C Fuchs","year":"2018","unstructured":"Fuchs, C.: The Online Advertising Tax as the Foundation of a Public Service Internet. University of Westminster Press, London (2018)"},{"key":"77_CR12","volume-title":"Learning Scikit-Learn: Machine Learning in Python","author":"R Garreta","year":"2013","unstructured":"Garreta, R., Moncecchi, G.: Learning Scikit-Learn: Machine Learning in Python. Packt Publishing Ltd, Birmingham (2013)"},{"key":"77_CR13","volume-title":"Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems","author":"A G\u00e9ron","year":"2017","unstructured":"G\u00e9ron, A.: Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O\u2019Reilly Media, Inc., Newton (2017)"},{"key":"77_CR14","doi-asserted-by":"crossref","unstructured":"Ghosh, A., McAfee, P., Papineni, K., Vassilvitskii, S.: Bidding for representative allocations for display advertising. In: International Workshop on Internet and Network Economics. Springer, Berlin, pp. 208\u2013219 (2009)","DOI":"10.1007\/978-3-642-10841-9_20"},{"key":"77_CR15","unstructured":"Graham, R.: A brief history of digital ad buying and selling (2010)"},{"issue":"1","key":"77_CR16","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1080\/10641734.2008.10505236","volume":"30","author":"L Ha","year":"2008","unstructured":"Ha, L.: Online advertising research in advertising journals: a review. J. Curr. Issues Res. Advert. 30(1), 31\u201348 (2008)","journal-title":"J. Curr. Issues Res. Advert."},{"key":"77_CR17","volume-title":"Digital Design and Computer Architecture","author":"D Harris","year":"2010","unstructured":"Harris, D., Harris, S.: Digital Design and Computer Architecture. Morgan Kaufmann, Burlington (2010)"},{"key":"77_CR18","doi-asserted-by":"crossref","unstructured":"Jin, J., Song, C., Li, H., Gai, K., Wang, J., Zhang, W.: Real-time bidding with multi-agent reinforcement learning in display advertising. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, pp. 2193\u20132201 (2018)","DOI":"10.1145\/3269206.3272021"},{"key":"77_CR19","unstructured":"Kveton, B., Mahdian, S., Muthukrishnan, S., Wen, Z., Xian, Y.: Waterfall bandits: learning to sell ads online (2019). arXiv preprint arXiv:1904.09404"},{"issue":"2","key":"77_CR20","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1109\/TCSS.2018.2831234","volume":"5","author":"J Li","year":"2018","unstructured":"Li, J., Ni, X., Yuan, Y.: The reserve price of ad impressions in multi-channel real-time bidding markets. IEEE Trans. Comput. Soc. Syst. 5(2), 583\u2013592 (2018)","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"issue":"4","key":"77_CR21","first-page":"em0123","volume":"5","author":"C Loebbecke","year":"2020","unstructured":"Loebbecke, C., Cremer, S., Richter, M.: Header bidding as smart service for selling ads in the digital era. J. Inf. Syst. Eng. Manag. 5(4), em0123 (2020)","journal-title":"J. Inf. Syst. Eng. Manag."},{"issue":"1","key":"77_CR22","first-page":"1","volume":"264","author":"A McAndrew","year":"2004","unstructured":"McAndrew, A.: An introduction to digital image processing with Matlab notes for scm2511 image processing. Sch. Comput. Sci. Math. Vic. Univ. Technol. 264(1), 1\u2013264 (2004)","journal-title":"Sch. Comput. Sci. Math. Vic. Univ. Technol."},{"key":"77_CR23","doi-asserted-by":"crossref","unstructured":"Muthukrishnan, S.: Ad exchanges: research issues. In: WINE \u201909 Proceedings of the 5th International Workshop on Internet and Network Economics, pp. 1\u201312 (2009)","DOI":"10.1007\/978-3-642-10841-9_1"},{"key":"77_CR24","doi-asserted-by":"crossref","unstructured":"Pachilakis, M., Papadopoulos, P., Markatos, E.P., Kourtellis, N.: No more chasing waterfalls: a measurement study of the header bidding ad-ecosystem. In: Proceedings of the Internet Measurement Conference on\u2014IMC \u201919, pp. 280\u2013293 (2019)","DOI":"10.1145\/3355369.3355582"},{"key":"77_CR25","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011)","journal-title":"J. Mach. Learn. Res."},{"key":"77_CR26","volume-title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","author":"ML Puterman","year":"2014","unstructured":"Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (2014)"},{"key":"77_CR27","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1007\/978-3-030-37494-5_2","volume-title":"Agents and Artificial Intelligence","author":"R Refaei Afshar","year":"2019","unstructured":"Refaei Afshar, R., Zhang, Y., Firat, M., Kaymak, U.: Reinforcement learning method for ad networks ordering in real-time bidding. In: van den Herik, J., Paula Rocha, A., Steels, L. (eds.) Agents and Artificial Intelligence, pp. 16\u201336. Springer, Cham (2019)"},{"key":"77_CR28","doi-asserted-by":"crossref","unstructured":"Rhuggenaath, J., Akcay, A., Zhang, Y., Kaymak, U.: A PSO-based algorithm for reserve price optimization in online ad auctions. In: 2019 IEEE Congress on Evolutionary Computation (IEEE CEC) (2019)","DOI":"10.1109\/CEC.2019.8789915"},{"key":"77_CR29","doi-asserted-by":"crossref","unstructured":"Rhuggenaath, J., Akcay, A., Zhang, Y., Kaymak, U.: Fuzzy logic based pricing combined with adaptive search for reserve price optimization in online ad auctions. In: IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) (2019)","DOI":"10.1109\/FUZZ-IEEE.2019.8858975"},{"key":"77_CR30","doi-asserted-by":"crossref","unstructured":"Rhuggenaath, J., Akcay, A., Zhang, Y., Kaymak, U.: Optimizing reserve prices for publishers in online ad auctions. In: 2019 IEEE Conference on Computational Intelligence for Financial Engineering and Economics (CIFEr) (2019)","DOI":"10.1109\/CIFEr.2019.8759123"},{"key":"77_CR31","first-page":"85","volume-title":"Digital Display Advertising","author":"KM Ryan","year":"2014","unstructured":"Ryan, K.M., Graham, R.S.: Digital Display Advertising, pp. 85\u2013100. Palgrave Macmillan US, New York (2014)"},{"issue":"4","key":"77_CR32","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1287\/mksc.2017.1083","volume":"37","author":"A Sayedi","year":"2018","unstructured":"Sayedi, A.: Real-time bidding in online display advertising. Mark. Sci. 37(4), 553\u2013568 (2018)","journal-title":"Mark. Sci."},{"issue":"7587","key":"77_CR33","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","volume":"529","author":"D Silver","year":"2016","unstructured":"Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den\u00a0Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484 (2016)","journal-title":"Nature"},{"key":"77_CR34","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT, Cambridge (2018)"},{"issue":"1","key":"77_CR35","first-page":"1","volume":"4","author":"C Szepesv\u00e1ri","year":"2010","unstructured":"Szepesv\u00e1ri, C.: Algorithms for reinforcement learning. Synth. Lect. Artif. Intell. Mach. Learn. 4(1), 1\u2013103 (2010)","journal-title":"Synth. Lect. Artif. Intell. Mach. Learn."},{"issue":"7","key":"77_CR36","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/j.future.2008.02.012","volume":"24","author":"D Vengerov","year":"2008","unstructured":"Vengerov, D.: A gradient-based reinforcement learning approach to dynamic pricing in partially-observable environments. Futur. Gener. Comput. Syst. 24(7), 687\u2013693 (2008)","journal-title":"Futur. Gener. Comput. Syst."},{"key":"77_CR37","doi-asserted-by":"crossref","unstructured":"Wang, J., Chen, B.: Selling futures online advertising slots via option contracts. In: Proceedings of the 21st International Conference on World Wide Web. ACM, pp. 627\u2013628 (2012)","DOI":"10.1145\/2187980.2188160"},{"issue":"4\u20135","key":"77_CR38","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1561\/1500000049","volume":"11","author":"J Wang","year":"2017","unstructured":"Wang, J., Zhang, W., Yuan, S., et al.: Display advertising with real-time bidding (rtb) and behavioural targeting. Found. Trends\u00ae Inf. Retriev. 11(4\u20135), 297\u2013435 (2017)","journal-title":"Found. Trends\u00ae Inf. Retriev."},{"key":"77_CR39","doi-asserted-by":"crossref","unstructured":"Wu, D., Chen, X., Yang, X., Wang, H., Tan, Q., Zhang, X., Xu, J., Gai, K.: Budget constrained bidding by model-free reinforcement learning in display advertising. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, pp. 1443\u20131451 (2018)","DOI":"10.1145\/3269206.3271748"},{"key":"77_CR40","doi-asserted-by":"crossref","unstructured":"Xie, Z., Lee, K.C., Wang, L.: Optimal reserve price for online ads trading based on inventory identification. In: Proceedings of the ADKDD\u201917. ACM, pp.\u00a06 (2017)","DOI":"10.1145\/3124749.3124760"},{"key":"77_CR41","doi-asserted-by":"crossref","unstructured":"Yuan, S., Wang, J., Zhao, X.: Real-time bidding for online advertising: measurement and analysis. In: Proceedings of the Seventh International Workshop on Data Mining for Online Advertising. ACM, p.\u00a03 (2013)","DOI":"10.1145\/2501040.2501980"},{"key":"77_CR42","doi-asserted-by":"crossref","unstructured":"Yuan, S., Wang, J., Chen, B., Mason, P., Seljan, S.: An empirical study of reserve price optimisation in real-time bidding. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp. 1897\u20131906 (2014)","DOI":"10.1145\/2623330.2623357"},{"key":"77_CR43","doi-asserted-by":"crossref","unstructured":"Zhang, W., Yuan, S., Wang, J.: Optimal real-time bidding for display advertising. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp. 1077\u20131086 (2014)","DOI":"10.1145\/2623330.2623633"}],"container-title":["International Journal of Computational Intelligence Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-022-00077-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44196-022-00077-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-022-00077-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,19]],"date-time":"2022-04-19T15:25:25Z","timestamp":1650381925000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44196-022-00077-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,19]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["77"],"URL":"https:\/\/doi.org\/10.1007\/s44196-022-00077-6","relation":{},"ISSN":["1875-6883"],"issn-type":[{"type":"electronic","value":"1875-6883"}],"subject":[],"published":{"date-parts":[[2022,4,19]]},"assertion":[{"value":"2 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declaration"}},{"value":"The author(s) declare(s) that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"27"}}