{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T16:57:22Z","timestamp":1776445042842,"version":"3.51.2"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,9,25]],"date-time":"2023-09-25T00:00:00Z","timestamp":1695600000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>We employ deep reinforcement learning (RL) to train an agent to successfully translate a high-frequency trading signal into a trading strategy that places individual limit orders. Based on the ABIDES limit order book simulator, we build a reinforcement learning OpenAI gym environment and utilize it to simulate a realistic trading environment for NASDAQ equities based on historic order book messages. To train a trading agent that learns to maximize its trading return in this environment, we use Deep Dueling Double Q-learning with the APEX (asynchronous prioritized experience replay) architecture. The agent observes the current limit order book state, its recent history, and a short-term directional forecast. To investigate the performance of RL for adaptive trading independently from a concrete forecasting algorithm, we study the performance of our approach utilizing synthetic alpha signals obtained by perturbing forward-looking returns with varying levels of noise. Here, we find that the RL agent learns an effective trading strategy for inventory management and order placing that outperforms a heuristic benchmark trading strategy having access to the same signal.<\/jats:p>","DOI":"10.3389\/frai.2023.1151003","type":"journal-article","created":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T06:07:46Z","timestamp":1695708466000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Asynchronous Deep Double Dueling Q-learning for trading-signal execution in limit order book markets"],"prefix":"10.3389","volume":"6","author":[{"given":"Peer","family":"Nagy","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jan-Peter","family":"Calliess","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefan","family":"Zohren","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,9,25]]},"reference":[{"key":"B1","article-title":"\u201cAdaptive market making via online learning,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Abernethy","year":"2013"},{"key":"B2","doi-asserted-by":"publisher","first-page":"5","DOI":"10.21314\/JOR.2001.041","article-title":"Optimal execution of portfolio transactions","volume":"3","author":"Almgren","year":"2001","journal-title":"J. Risk"},{"key":"B3","doi-asserted-by":"crossref","DOI":"10.1145\/3490354.3494433","article-title":"\u201cABIDES-gym: gym environments for multi-agent discrete event simulation and application to financial markets,\u201d","volume-title":"Proceedings of the Second ACM International Conference on AI in Finance","author":"Amrouni","year":"2021"},{"key":"B4","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1512\/iumj.1957.6.56038","article-title":"A Markovian decision process","volume":"6","author":"Bellman","year":"1957","journal-title":"J. Math. Mech"},{"key":"B5","article-title":"Deep reinforcement learning for active high frequency trading","author":"Briola","year":"2021","journal-title":"arXiv preprint arXiv:2101.07107"},{"key":"B6","article-title":"Openai gym","author":"Brockman","year":"2016","journal-title":"arXiv preprint arXiv:1606.01540"},{"key":"B7","author":"Byrd","year":""},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1145\/3384441.3395986","article-title":"\u201cAbides: Towards high-fidelity multi-agent market simulation,\u201d","author":"Byrd","year":"","journal-title":"Proceedings of the 2020 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/1350486X.2018.1434009","article-title":"Enhancing trading strategies with order book signals","volume":"25","author":"Cartea","year":"2018","journal-title":"Appl. Mathem. Finance"},{"key":"B10","doi-asserted-by":"publisher","first-page":"542","DOI":"10.1111\/mafi.12181","article-title":"Trading co-integrated assets with price impact","volume":"29","author":"Cartea","year":"2019","journal-title":"Mathem. Finan"},{"key":"B11","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1007\/s11579-016-0162-z","article-title":"Incorporating order-flow into optimal execution","volume":"10","author":"Cartea","year":"2016","journal-title":"Mathem. Finan. Econ"},{"key":"B12","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1080\/1350486X.2020.1726783","article-title":"Spoofing and price manipulation in order-driven markets","volume":"27","author":"Cartea","year":"2020","journal-title":"Appl. Mathem. Finan"},{"key":"B13","article-title":"Generating long sequences with sparse transformers","author":"Child","year":"2019","journal-title":"arXiv preprint arXiv:1904.10509"},{"key":"B14","article-title":"Conditional generators for limit order book environments: Explainability, challenges, and robustness","author":"Coletta","year":"2023","journal-title":"arXiv preprint arXiv:2306.12806"},{"key":"B15","doi-asserted-by":"crossref","DOI":"10.1145\/3533271.3561753","article-title":"\u201cLearning to simulate realistic limit order book markets from data as a world agent,\u201d","volume-title":"Proceedings of the Third ACM International Conference on AI in Finance","author":"Coletta","year":"2022"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1145\/3490354.3494411","article-title":"\u201cTowards realistic market simulations: a generative adversarial networks approach,\u201d","volume-title":"Proceedings of the Second ACM International Conference on AI in Finance","author":"Coletta","year":"2021"},{"key":"B17","volume-title":"Deep Execution-Value and Policy Based Reinforcement Learning for Trading and Beating Market Benchmarks","author":"Dab\u00e9rius","year":"2019"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1709","DOI":"10.1080\/14697688.2013.803148","article-title":"Limit order books","volume":"13","author":"Gould","year":"2013","journal-title":"Quant. Finan"},{"key":"B19","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1111\/mafi.12382","article-title":"Recent advances in reinforcement learning in finance","volume":"33","author":"Hambly","year":"2023","journal-title":"Mathem. Finan"},{"key":"B20","volume-title":"Trading and Exchanges: Market Microstructure for Practitioners","author":"Harris","year":"2003"},{"key":"B21","article-title":"\u201cDouble Q-learning,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Hasselt","year":"2010"},{"key":"B22","article-title":"Distributed prioritized experience replay","author":"Horgan","year":"2018","journal-title":"arXiv preprint arXiv:1803.00933"},{"key":"B23","volume-title":"Lobster: Limit Order Book Reconstruction System","author":"Huang","year":"2011"},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1145\/3383455.3422570","article-title":"\u201cMulti-agent reinforcement learning in a realistic limit order book market simulation,\u201d","volume-title":"Proceedings of the First ACM International Conference on AI in Finance","author":"Karpe","year":"2020"},{"key":"B25","article-title":"\u201cMachine learning for market microstructure and high frequency trading,\u201d","volume-title":"High Frequency Trading: New Realities for Traders, Markets, and Regulators","author":"Kearns","year":"2013"},{"key":"B26","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"B27","volume-title":"Deep Order Flow Imbalance: Extracting Alpha at Multiple Horizons From the Limit Order Book","author":"Kolm","year":"2021"},{"key":"B28","first-page":"1892","article-title":"\u201cDeep reinforcement learning for market making,\u201d","volume-title":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems","author":"Kumar","year":"2020"},{"key":"B29","article-title":"Playing atari with deep reinforcement learning","author":"Mnih","year":"2013","journal-title":"arXiv preprint arXiv:1312.5602"},{"key":"B30","unstructured":"Algorithmic trading market share, size &trading statistics2022"},{"key":"B31","article-title":"\u201cMachine learning for microstructure data-driven execution algorithms,\u201d","volume-title":"The Handbook on AI and Big Data Applications in Investments","author":"Nagy","year":"2023"},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.1145\/1143844.1143929","article-title":"\u201cReinforcement learning for optimized trade execution,\u201d","volume-title":"Proceedings of the 23rd International Conference on Machine Learning","author":"Nevmyvaka","year":"2006"},{"key":"B33","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1080\/1350486X.2022.2077783","article-title":"Double deep Q-learning for optimal execution","volume":"28","author":"Ning","year":"2021","journal-title":"Appl. Mathem. Finan"},{"key":"B34","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1016\/S0927-0507(05)80172-0","article-title":"Markov decision processes","volume":"2","author":"Puterman","year":"1990","journal-title":"Handb. Operat. Res. Manage. Sci"},{"key":"B35","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1016\/j.ejor.2021.04.050","article-title":"Deep reinforcement learning for the optimal placement of cryptocurrency limit orders","volume":"296","author":"Schnaubelt","year":"2022","journal-title":"Eur. J. Operat. Res"},{"key":"B36","article-title":"Proximal policy optimization algorithms","author":"Schulman","year":"2017","journal-title":"arXiv preprint arXiv:1707.06347"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1449","DOI":"10.1080\/14697688.2019.1622295","article-title":"Universal features of price formation in financial markets: perspectives from deep learning","volume":"19","author":"Sirignano","year":"2019","journal-title":"Quantit. Finan"},{"key":"B38","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"B39","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v30i1.10295","article-title":"\u201cDeep reinforcement learning with double Q-learning,\u201d","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Van Hasselt","year":"2016"},{"key":"B40","article-title":"\u201cAttention is all you need,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"B41","first-page":"1995","article-title":"\u201cDueling network architectures for deep reinforcement learning,\u201d","volume-title":"International Conference on Machine Learning","author":"Wang","year":"2016"},{"key":"B42","article-title":"Model-based reinforcement learning for predictions and control for limit order books","author":"Wei","year":"2019","journal-title":"arXiv preprint arXiv:1910.03743"},{"key":"B43","article-title":"Model-based deep reinforcement learning for dynamic portfolio optimization","author":"Yu","year":"2019","journal-title":"arXiv preprint arXiv:1901.08740"},{"key":"B44","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1080\/1350486X.2021.1967767","article-title":"Deep learning for market by order data","volume":"28","author":"Zhang","year":"2021","journal-title":"Appl. Mathem. Finan"},{"key":"B45","article-title":"Multi-horizon forecasting for limit order books: Novel deep learning approaches and hardware acceleration using intelligent processing units","author":"Zhang","year":"2021","journal-title":"arXiv preprint arXiv:2105.10430"},{"key":"B46","doi-asserted-by":"publisher","first-page":"3001","DOI":"10.1109\/TSP.2019.2907260","article-title":"DeepLOB: Deep convolutional neural networks for limit order books","volume":"67","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Signal Proc"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1151003\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T06:07:58Z","timestamp":1695708478000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1151003\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,25]]},"references-count":46,"alternative-id":["10.3389\/frai.2023.1151003"],"URL":"https:\/\/doi.org\/10.3389\/frai.2023.1151003","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,25]]},"article-number":"1151003"}}