{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T00:39:43Z","timestamp":1768437583016,"version":"3.49.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2019,12,31]],"date-time":"2019-12-31T00:00:00Z","timestamp":1577750400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,31]],"date-time":"2019-12-31T00:00:00Z","timestamp":1577750400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Wireless Netw"],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>To address the problems of the slow convergence and inefficiency in the existing adaptive PID controllers, we propose a new adaptive PID controller using the asynchronous advantage actor\u2013critic (A3C) algorithm. Firstly, the controller can train the multiple agents of the actor\u2013critic structures in parallel exploiting the multi-thread asynchronous learning characteristics of the A3C structure. Secondly, in order to achieve the best control effect, each agent uses a multilayer neural network to approach the strategy function and value function to search the best parameter-tuning strategy in continuous action space. The simulation results indicate that our proposed controller can achieve the fast convergence and strong adaptability compared with conventional controllers.<\/jats:p>","DOI":"10.1007\/s11276-019-02225-x","type":"journal-article","created":{"date-parts":[[2019,12,31]],"date-time":"2019-12-31T16:02:27Z","timestamp":1577808147000},"page":"3537-3547","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Design and application of adaptive PID controller based on asynchronous advantage actor\u2013critic learning method"],"prefix":"10.1007","volume":"27","author":[{"given":"Qifeng","family":"Sun","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chengze","family":"Du","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Youxiang","family":"Duan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"Ren","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hongqiang","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,12,31]]},"reference":[{"key":"2225_CR1","doi-asserted-by":"crossref","unstructured":"Adel, T., & Abdelkader, C. (2013). A particle swarm optimization approach for optimum design of PID controller for nonlinear systems. In International conference on electrical engineering and software applications (pp. 1\u20134). IEEE.","DOI":"10.1109\/ICEESA.2013.6578478"},{"issue":"5","key":"2225_CR2","doi-asserted-by":"crossref","first-page":"2658","DOI":"10.1016\/j.asoc.2012.11.021","volume":"13","author":"A Savran","year":"2013","unstructured":"Savran, A. (2013). A multivariable predictive fuzzy PID control system. Applied Soft Computing, 13(5), 2658\u20132667.","journal-title":"Applied Soft Computing"},{"issue":"3","key":"2225_CR3","first-page":"1","volume":"5","author":"D Jiang","year":"2018","unstructured":"Jiang, D., Wang, W., Shi, L., & Song, H. (2018). A compressive sensing-based approach to end-to-end network traffic reconstruction. IEEE Transactions on Network Science and Engineering, 5(3), 1\u201312.","journal-title":"IEEE Transactions on Network Science and Engineering"},{"issue":"5","key":"2225_CR4","first-page":"1","volume":"13","author":"D Jiang","year":"2018","unstructured":"Jiang, D., Huo, L., & Li, Y. (2018). Fine-granularity inference and estimations to network traffic for SDN. PLoS ONE, 13(5), 1\u201323.","journal-title":"PLoS ONE"},{"key":"2225_CR5","first-page":"1","volume":"5","author":"X Zhang","year":"2014","unstructured":"Zhang, X., Bao, H., Du, J., & Wang, C. (2014). Application of a new membership function in nonlinear fuzzy PID controllers with variable gains. Information and Control, 5, 1\u20137.","journal-title":"Information and Control"},{"issue":"1","key":"2225_CR6","first-page":"167","volume":"32","author":"Li Caocang","year":"2015","unstructured":"Caocang, Li, & Cuifang, Zhang. (2015). Adaptive neuron PID control based on minimum resource allocation network. Application Research of Computers, 32(1), 167\u2013169.","journal-title":"Application Research of Computers"},{"issue":"S2","key":"2225_CR7","first-page":"134","volume":"35","author":"X Sheng","year":"2015","unstructured":"Sheng, X., Jiang, T., Wang, J., et al. (2015). Speed-feed-forward PID controller design based on BP neural network. Journal of Computer Applications, 35(S2), 134\u2013137.","journal-title":"Journal of Computer Applications"},{"issue":"1","key":"2225_CR8","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/S1006-1266(07)60009-1","volume":"17","author":"XS Wang","year":"2007","unstructured":"Wang, X. S., Cheng, Y. H., & Wei, S. (2007). A proposal of adaptive PID controller based on reinforcement learning. Journal of China University of Mining and Technology, 17(1), 40\u201344.","journal-title":"Journal of China University of Mining and Technology"},{"issue":"2","key":"2225_CR9","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1016\/j.compeleceng.2017.09.009","volume":"66","author":"L Huo","year":"2018","unstructured":"Huo, L., Jiang, D., & Lv, Z. (2018). Soft frequency reuse-based optimization algorithm for energy efficiency of multi-cell networks. Computers and Electrical Engineering, 66(2), 316\u2013331.","journal-title":"Computers and Electrical Engineering"},{"issue":"2","key":"2225_CR10","first-page":"20","volume":"7","author":"A Akbarimajd","year":"2015","unstructured":"Akbarimajd, A. (2015). Reinforcement learning adaptive PID controller for an under-actuated robot arm. International Journal of Integrated Engineering, 7(2), 20\u201327.","journal-title":"International Journal of Integrated Engineering"},{"issue":"8","key":"2225_CR11","first-page":"1187","volume":"28","author":"XS Chen","year":"2011","unstructured":"Chen, X. S., & Yang, Y. M. (2011). A novel adaptive PID controller based on actor\u2013critic learning. Control Theory and Applications, 28(8), 1187\u20131192.","journal-title":"Control Theory and Applications"},{"key":"2225_CR12","unstructured":"Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., et al. (2016). An actor\u2013critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086."},{"key":"2225_CR13","unstructured":"Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., et al. (2016). Sample efficient actor\u2013critic with experience replay. arXiv preprint arXiv:1611.01224."},{"key":"2225_CR14","unstructured":"Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., et al. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning (pp. 1928\u20131937)."},{"issue":"10","key":"2225_CR15","doi-asserted-by":"crossref","first-page":"3305","DOI":"10.1109\/TITS.2017.2778939","volume":"19","author":"D Jiang","year":"2018","unstructured":"Jiang, D., Huo, L., Lv, Z., Song, H., & Qin, W. (2018). A joint multi-criteria utility-based network selection approach for vehicle-to-infrastructure networking. IEEE Transactions on Intelligent Transportation Systems, 19(10), 3305\u20133319.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"issue":"01","key":"2225_CR16","first-page":"1","volume":"41","author":"Q Liu","year":"2018","unstructured":"Liu, Q., Zhai, J. W., Zhang, Z. Z., & Zhong, S. (2018). A survey on deep reinforcement learning. Chinese Journal of Computers, 41(01), 1\u201327.","journal-title":"Chinese Journal of Computers"},{"issue":"9","key":"2225_CR17","first-page":"1588","volume":"43","author":"R Qin","year":"2015","unstructured":"Qin, R., Zeng, S., Li, J. J., & Yuan, Y. (2015). Parallel enterprises resource planning based on deep reinforcement learning. Zidonghua Xuebao\/Acta Automatica Sinica, 43(9), 1588\u20131596.","journal-title":"Zidonghua Xuebao\/Acta Automatica Sinica"},{"key":"2225_CR18","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2019.2930226","author":"D Jiang","year":"2019","unstructured":"Jiang, D., Wang, Y., Lv, Z., Qi, S., & Singh, S. (2019). Big data analysis-based network behavior insight of cellular networks for industry 4.0 applications. IEEE Transactions on Industrial Informatics. https:\/\/doi.org\/10.1109\/TII.2019.2930226.","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"2225_CR19","first-page":"136","volume":"2","author":"HC Tang","year":"2005","unstructured":"Tang, H. C., Li, Z. X., Wang, Z. T., et al. (2005). A fuzzy PID control system. Electric Machines and Control, 2, 136\u2013138.","journal-title":"Electric Machines and Control"},{"key":"2225_CR20","first-page":"1991","volume":"S3","author":"JP Sun","year":"2006","unstructured":"Sun, J. P., Yan, L., Li, Y., et al. (2006). Design of fuzzy PID controllers based on improved genetic algorithms. Chinese Journal of Scientific Instrument, S3, 1991\u20131992.","journal-title":"Chinese Journal of Scientific Instrument"},{"issue":"12","key":"2225_CR21","first-page":"2732","volume":"23","author":"YH Zhu","year":"2011","unstructured":"Zhu, Y. H., Xue, L. Y., & Huang, W. (2011). Design of fuzzy PID controller based on self-organizing adjustment factors. Journal of System Simulation, 23(12), 2732\u20132737.","journal-title":"Journal of System Simulation"},{"key":"2225_CR22","first-page":"1711","volume":"07","author":"FF Liao","year":"2005","unstructured":"Liao, F. F., & Xiao, J. (2005). Research on self-tuning of PID parameters based on BP neural networks. Acta Simulata Systematica Sinica, 07, 1711\u20131713.","journal-title":"Acta Simulata Systematica Sinica"},{"key":"2225_CR23","first-page":"43","volume":"1","author":"GY Li","year":"2008","unstructured":"Li, G. Y., & Chen, X. L. (2008). Neural network self-learning PID controller based on real-coded genetic algorithm. Micromotors Servo Technique, 1, 43\u201345.","journal-title":"Micromotors Servo Technique"},{"key":"2225_CR24","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.procs.2015.06.023","volume":"54","author":"R Patel","year":"2015","unstructured":"Patel, R., & Kumar, V. (2015). Multilayer neuro PID controller based on back propagation algorithm. Procedia Computer Science, 54, 207\u2013214.","journal-title":"Procedia Computer Science"},{"issue":"6","key":"2225_CR25","doi-asserted-by":"crossref","first-page":"1091","DOI":"10.1002\/oca.2314","volume":"38","author":"SK Nie","year":"2017","unstructured":"Nie, S. K., Wang, Y. J., Xiao, S., & Liu, Z. (2017). An adaptive chaos particle swarm optimization for tuning parameters of PID controller. Optimal Control Applications and Methods, 38(6), 1091\u20131102.","journal-title":"Optimal Control Applications and Methods"},{"issue":"8","key":"2225_CR26","doi-asserted-by":"crossref","first-page":"2389","DOI":"10.1007\/s13369-015-1752-4","volume":"40","author":"A Aziz Khater","year":"2015","unstructured":"Aziz Khater, A., El-Bardini, M., & El-Rabaie, N. M. (2015). Embedded adaptive fuzzy controller based on reinforcement learning for DC motor with flexible shaft. Arabian Journal for Science and Engineering, 40(8), 2389\u20132406.","journal-title":"Arabian Journal for Science and Engineering"},{"issue":"3","key":"2225_CR27","first-page":"579","volume":"52","author":"Z Liu","year":"2015","unstructured":"Liu, Z., Zeng, X., Liu, H., & Chu, R. (2015). A heuristic two-layer reinforcement learning algorithm based on BP neural networks. Journal of Computer Research and Development, 52(3), 579\u2013587.","journal-title":"Journal of Computer Research and Development"},{"issue":"4","key":"2225_CR28","doi-asserted-by":"crossref","first-page":"2375","DOI":"10.1109\/JIOT.2017.2759728","volume":"5","author":"J Zhu","year":"2018","unstructured":"Zhu, J., Song, Y., Jiang, D., & Song, H. (2018). A new deep-Q-learning-based transmission scheduling mechanism for the cognitive Internet of things. IEEE Internet of Things Journal, 5(4), 2375\u20132385.","journal-title":"IEEE Internet of Things Journal"},{"key":"2225_CR29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2013.08.037","volume":"261","author":"X Xu","year":"2014","unstructured":"Xu, X., Zuo, L., & Huang, Z. (2014). Reinforcement learning algorithms with function approximation: Recent advances and applications. Information Sciences, 261, 1\u201331.","journal-title":"Information Sciences"},{"issue":"1","key":"2225_CR30","first-page":"1","volume":"1","author":"D Jiang","year":"2018","unstructured":"Jiang, D., Huo, L., & Song, H. (2018). Rethinking behaviors and activities of base stations in mobile cellular networks based on big data analysis. IEEE Transactions on Network Science and Engineering, 1(1), 1\u201312.","journal-title":"IEEE Transactions on Network Science and Engineering"},{"issue":"1","key":"2225_CR31","first-page":"196","volume":"7","author":"F Wang","year":"2019","unstructured":"Wang, F., Jiang, D., & Qi, S. (2019). An adaptive routing algorithm for integrated information networks. China Communications, 7(1), 196\u2013207.","journal-title":"China Communications"},{"issue":"4","key":"2225_CR32","first-page":"1","volume":"23","author":"L Huo","year":"2019","unstructured":"Huo, L., & Jiang, D. (2019). Stackelberg game-based energy-efficient resource allocation for 5G cellular networks. Telecommunication System, 23(4), 1\u201311.","journal-title":"Telecommunication System"},{"issue":"6","key":"2225_CR33","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.1109\/JIOT.2016.2613111","volume":"3","author":"D Jiang","year":"2016","unstructured":"Jiang, D., Zhang, P., Lv, Z., & Song, H. (2016). Energy-efficient multi-constraint routing algorithm with load balancing for smart city applications. IEEE Internet of Things Journal, 3(6), 1437\u20131447.","journal-title":"IEEE Internet of Things Journal"},{"issue":"2017","key":"2225_CR34","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1016\/j.neucom.2016.07.056","volume":"220","author":"D Jiang","year":"2017","unstructured":"Jiang, D., Li, W., & Lv, H. (2017). An energy-efficient cooperative multicast routing in multi-hop wireless networks for smart medical applications. Neurocomputing, 220(2017), 160\u2013169.","journal-title":"Neurocomputing"},{"key":"2225_CR35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11227-018-2671-4","volume":"75","author":"F Wang","year":"2019","unstructured":"Wang, F., Jiang, D., Wen, H., & Song, H. (2019). Adaboost-based security level classification of mobile intelligent terminals. The Journal of Supercomputing, 75, 1\u201319.","journal-title":"The Journal of Supercomputing"},{"key":"2225_CR36","doi-asserted-by":"crossref","first-page":"22013","DOI":"10.1109\/ACCESS.2017.2760885","volume":"5","author":"M Sun","year":"2017","unstructured":"Sun, M., Jiang, D., Song, H., & Liu, Y. (2017). Statistical resolution limit analysis of two closely-spaced signal sources using Rao test. IEEE Access, 5, 22013\u201322022.","journal-title":"IEEE Access"}],"container-title":["Wireless Networks"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11276-019-02225-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11276-019-02225-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11276-019-02225-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,3]],"date-time":"2021-07-03T06:29:31Z","timestamp":1625293771000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11276-019-02225-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,31]]},"references-count":36,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["2225"],"URL":"https:\/\/doi.org\/10.1007\/s11276-019-02225-x","relation":{},"ISSN":["1022-0038","1572-8196"],"issn-type":[{"value":"1022-0038","type":"print"},{"value":"1572-8196","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12,31]]},"assertion":[{"value":"31 December 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}