{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:12:21Z","timestamp":1760242341715,"version":"build-2065373602"},"reference-count":41,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2017,5,10]],"date-time":"2017-05-10T00:00:00Z","timestamp":1494374400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The bilateral spot electricity market is very complicated because all generation units and demands must strategically bid in this market. Considering renewable resource penetration, the high variability and the non-dispatchable nature of these intermittent resources make it more difficult to model and simulate the dynamic bidding process and the equilibrium in the bilateral spot electricity market, which makes developing fast and reliable market modeling approaches a matter of urgency nowadays. In this paper, a Gradient Descent Continuous Actor-Critic algorithm is proposed for hour-ahead bilateral electricity market modeling in the presence of renewable resources because this algorithm can solve electricity market modeling problems with continuous state and action spaces without causing the \u201ccurse of dimensionality\u201d and has low time complexity. In our simulation, the proposed approach is implemented on an IEEE 30-bus test system. The adequate performance of our proposed approach\u2014such as reaching Nash Equilibrium results after enough iterations of training are tested and verified, and some conclusions about the relationship between increasing the renewable power output and participants\u2019 bidding strategy, locational marginal prices, and social welfare\u2014is also evaluated. Moreover, the comparison of our proposed approach with the fuzzy Q-learning-based electricity market approach implemented in this paper confirms the superiority of our proposed approach in terms of participants\u2019 profits, social welfare, average locational marginal prices, etc.<\/jats:p>","DOI":"10.3390\/a10020053","type":"journal-article","created":{"date-parts":[[2017,5,10]],"date-time":"2017-05-10T12:04:20Z","timestamp":1494417860000},"page":"53","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Application of Gradient Descent Continuous Actor-Critic Algorithm for Bilateral Spot Electricity Market Modeling Considering Renewable Power Penetration"],"prefix":"10.3390","volume":"10","author":[{"given":"Huiru","family":"Zhao","sequence":"first","affiliation":[{"name":"School of Economics and Management, North China Electric Power University, Beijing 102206, China"}]},{"given":"Yuwei","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Economics and Management, North China Electric Power University, Beijing 102206, China"}]},{"given":"Mingrui","family":"Zhao","sequence":"additional","affiliation":[{"name":"School of Economics and Management, North China Electric Power University, Beijing 102206, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4113-9925","authenticated-orcid":false,"given":"Chuyu","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Business Administration, China University of Petroleum-Beijing, Beijing 102249, China"}]},{"given":"Qingkun","family":"Tan","sequence":"additional","affiliation":[{"name":"School of Economics and Management, North China Electric Power University, Beijing 102206, China"}]}],"member":"1968","published-online":{"date-parts":[[2017,5,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Alikhanzadeh, A., and Irving, M. (2012, January 22\u201326). Combined oligopoly and oligopsony bilateral electricity market model using CV equilibria. Proceedings of the 2012 IEEE Power and Energy Society General Meeting, San Diego, CA, USA.","DOI":"10.1109\/PESGM.2012.6345293"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1172","DOI":"10.1016\/j.rser.2015.12.020","article-title":"Application of fuzzy Q-learning for electricity market modeling by considering renewable power penetration","volume":"56","author":"Mohammad","year":"2016","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1049\/iet-gtd.2011.0816","article-title":"Integrated renewable-conventional generation expansion planning using multi objective framework","volume":"6","author":"Aghaei","year":"2012","journal-title":"IET Gener. Transm. Distrib."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Yu, N., Liu, C.C., and Tesfatsion, L. (2007, January 5\u20138). Modeling of Suppliers' Learning Behaviors in an Electricity Market Environment. Proceedings of the 2007 International Conference on Intelligent Systems, Niigata, Japan.","DOI":"10.1109\/ISAP.2007.4441590"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1016\/j.rser.2014.12.019","article-title":"Variability assessment and forecasting of renewables: A review for solar, wind, wave and tidal resources","volume":"44","author":"Carpman","year":"2015","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1109\/JSYST.2011.2162895","article-title":"Impacts of Large-Scale Integration of Intermittent Resources on Electricity Markets: A Supply Function Equilibrium Approach","volume":"6","author":"Buygi","year":"2012","journal-title":"IEEE Syst. J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2886","DOI":"10.1016\/j.energy.2010.03.019","article-title":"Supply curve bidding of electricity in constrained power networks","volume":"35","year":"2010","journal-title":"Energy"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.ijepes.2014.10.053","article-title":"Optimal bidding strategy for GENCOs based on parametric linear programming considering incomplete information","volume":"66","author":"Gao","year":"2015","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1727","DOI":"10.1016\/j.epsr.2009.07.011","article-title":"Influence of feasibility constrains on the bidding strategy selection in a day-ahead electricity market session","volume":"79","author":"Borghetti","year":"2009","journal-title":"Electr. Power Syst. Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1016\/j.asoc.2014.03.027","article-title":"Generation bidding strategy in a pool based electricity market using Shuffled Frog Leaping Algorithm","volume":"21","author":"Kumar","year":"2014","journal-title":"Appl. Soft Comput."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3459","DOI":"10.1016\/j.energy.2011.03.050","article-title":"An evolutionary game approach to analyzing bidding strategies in electricity markets with elastic demand","volume":"36","author":"Wang","year":"2011","journal-title":"Energy"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1016\/j.energy.2013.03.060","article-title":"Game-theory-based generation maintenance scheduling in electricity markets","volume":"55","author":"Min","year":"2013","journal-title":"Energy"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"881","DOI":"10.1016\/j.renene.2015.05.024","article-title":"An environmental\/techno-economic approach for bidding strategy in security-constrained electricity markets by a bi-level harmony search algorithm","volume":"83","author":"Shivaie","year":"2015","journal-title":"Renew. Energy"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"524","DOI":"10.3182\/20120902-4-FR-2032.00092","article-title":"Equilibrium Calculation in Electricity Market Modeled as a Two-stage Stochastic Game using competitive Coevolutionary Algorithms","volume":"45","author":"Ladjici","year":"2012","journal-title":"IFAC Proc. Vol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/j.apenergy.2014.01.003","article-title":"A game theoretic framework for a next-generation retail electricity market with high penetration of distributed residential electricity suppliers","volume":"119","author":"Su","year":"2014","journal-title":"Appl. Energy"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.epsr.2007.01.009","article-title":"Supplier\u2019s optimal bidding strategy in electricity pay-as-bid auction: Comparison of the Q-learning and a model-based approach","volume":"78","author":"Rahimiyan","year":"2008","journal-title":"Electr. Power Syst. Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1016\/j.epsr.2011.03.004","article-title":"An agent-based FTR auction simulator","volume":"81","author":"Ziogos","year":"2011","journal-title":"Electr. Power Syst. Res."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Santos, G., Fernandes, R., Pinto, T., Praa, I., Vale, Z., and Morais, H. (2015, January 11\u201316). MASCEM: EPEX SPOT Day-Ahead market integration and simulation. Proceedings of the 2015 18th International Conference on Intelligent System Application to Power Systems (ISAP), Porto, Portugal.","DOI":"10.1109\/ISAP.2015.7325554"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"696","DOI":"10.1016\/j.ijepes.2012.05.056","article-title":"Multi-agent based experimental analysis on bidding mechanism in electricity auction markets","volume":"43","author":"Liu","year":"2012","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, H., and Tesfatsion, L. (2009, January 26\u201330). The AMES wholesale power market test bed: A computational laboratory for research, teaching, and training. Proceedings of the 2009 IEEE Power & Energy Society General Meeting, Calgary, AB, Canada.","DOI":"10.1109\/PES.2009.5275969"},{"key":"ref_21","unstructured":"Conzelmann, G., Boyd, G., Koritarov, V., and Veselka, T. (2005, January 12\u201316). Multi-agent power market simulation using EMCAS. Proceedings of the IEEE Power Engineering Society General Meeting, San Francisco, CA, USA."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1009","DOI":"10.1016\/j.eneco.2009.12.004","article-title":"An agent-based approach equipped with game theory: Strategic collaboration among learning agents during a dynamic market change in the California electricity crisis","volume":"32","author":"Sueyoshi","year":"2010","journal-title":"Energy Econ."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"6367","DOI":"10.1016\/j.energy.2011.09.037","article-title":"Optimal bidding strategy in a competitive electricity market based on agent-based approach and numerical sensitivity analysis","volume":"36","author":"Mahvi","year":"2011","journal-title":"Energy"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"725","DOI":"10.3390\/en9090725","article-title":"Application of gradient descent continuous actor-critic algorithm for double-side day-ahead electricity market modeling","volume":"9","author":"Zhao","year":"2016","journal-title":"Energies"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Lau, A.Y.F., Srinivasan, D., and Reindl, T. (2013, January 16\u201319). A reinforcement learning algorithm developed to model GenCo strategic bidding behavior in multidimensional and continuous state and action spaces. Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), Singapore.","DOI":"10.1109\/ADPRL.2013.6614997"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/j.enconman.2014.05.002","article-title":"Strategic bidding for wind power producers in electricity markets","volume":"86","author":"Sharma","year":"2014","journal-title":"Energy Convers. Manag."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.apenergy.2013.11.055","article-title":"Wind power bidding in electricity markets with high wind penetration","volume":"118","author":"Vilim","year":"2014","journal-title":"Appl. Energy"},{"key":"ref_28","first-page":"16","article-title":"Renewable energy trading in electricity market: Review and prospect","volume":"10","author":"Kang","year":"2016","journal-title":"South. Power Syst. Technol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3370","DOI":"10.1016\/j.rser.2012.02.019","article-title":"Grid integration of intermittent renewable energy sources using price-responsive plug-in electric vehicles","volume":"16","author":"Dallinger","year":"2012","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1016\/j.enpol.2013.06.003","article-title":"Development of a virtual power market model to investigate strategic and collusive behavior of market players","volume":"61","author":"Moghaddam","year":"2013","journal-title":"Energy Policy"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Reeg, M., Hauser, W., Wassermann, S., and Weimer-Jehle, W. (2012, January 3\u20137). AMIRIS: An Agent-Based Simulation Model for the Analysis of Different Support Schemes and Their Effects on Actors Involved in the Integration of Renewable Energies into Energy Markets. Proceedings of the 2012 23rd International Workshop on Database and Expert Systems Applications, Vienna, Austria.","DOI":"10.1109\/DEXA.2012.54"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Haring, T., Andersson, G., and Lygeros, J. (2012, January 10\u201312). Evaluating market designs in power systems with high wind penetration. Proceedings of the 2012 9th International Conference on the European Energy Market (EEM), Florence, Italy.","DOI":"10.1109\/EEM.2012.6254715"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Soares, T., Santos, G., Pinto, T., Morais, H., Pinson, P., and Vale, Z. (2015, January 11\u201316). Analysis of strategic wind power participation in energy market using MASCEM simulator. Proceedings of the 2015 18th International Conference on Intelligent System Application to Power Systems (ISAP), Porto, Portugal.","DOI":"10.1109\/ISAP.2015.7325552"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/s11067-014-9272-4","article-title":"Integrating Intermittent Renewable Wind Generation-A Stochastic Multi-Market Electricity Model for the European Electricity Market","volume":"15","author":"Abrell","year":"2015","journal-title":"Netw. Spat. Econ."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1109\/TSTE.2015.2504561","article-title":"Control and Bidding Strategy for Virtual Power Plants with Renewable Generation and Inelastic Demand in Electricity Markets","volume":"7","author":"Zhao","year":"2016","journal-title":"IEEE Trans. Sustain. Energy"},{"key":"ref_36","unstructured":"Zou, P., Chen, Q., Xia, Q., Kang, C., He, G., and Chen, X. (2015, January 26\u201330). Modeling and algorithm to find the economic equilibrium for pool-based electricity market with the changing generation mix. Proceedings of the 2015 IEEE Power & Energy Society General Meeting, Denver, CO, USA."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.tej.2016.05.001","article-title":"Wholesale electricity market design with increasing levels of renewable generation: Incentivizing flexibility in system operations","volume":"29","author":"Ela","year":"2016","journal-title":"Electr. J."},{"key":"ref_38","unstructured":"Liuhui, W., Xian, W., and Shanghua, Z. (2016, January 25\u201328). Electricity market equilibrium analysis for strategic bidding of wind power producer with demand response resource. Proceedings of the 2016 IEEE PES Asia-Pacific Power and Energy Engineering Conference, Xi\u2019an, China."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3211","DOI":"10.1016\/j.enpol.2009.07.038","article-title":"Market behaviour with large amounts of intermittent generation","volume":"38","author":"Green","year":"2010","journal-title":"Energy Policy"},{"key":"ref_40","unstructured":"Chen, G. (2014). Research on Value Function Approximation Methods in Reinforcement Learning. [Master\u2019s Thesis, Soochow University]."},{"key":"ref_41","unstructured":"Chen, Z. (2007). Study on Locational Marginal Prices and Congestion Management Algorithm. [Ph.D. Thesis, North China Electric Power University]."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/10\/2\/53\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:35:16Z","timestamp":1760207716000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/10\/2\/53"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,5,10]]},"references-count":41,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2017,6]]}},"alternative-id":["a10020053"],"URL":"https:\/\/doi.org\/10.3390\/a10020053","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2017,5,10]]}}}