{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T10:21:39Z","timestamp":1783938099923,"version":"3.55.0"},"reference-count":38,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,4,28]],"date-time":"2023-04-28T00:00:00Z","timestamp":1682640000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Science and Technology Innovation 2030\u2014\u201cThe New 34 Generation of Artificial Intelligence\u201d","award":["2018AAA0100901"],"award-info":[{"award-number":["2018AAA0100901"]}]},{"name":"Science and Technology Innovation 2030\u2014\u201cThe New 34 Generation of Artificial Intelligence\u201d","award":["2020BD003"],"award-info":[{"award-number":["2020BD003"]}]},{"name":"PKU-Baidu Fund","award":["2018AAA0100901"],"award-info":[{"award-number":["2018AAA0100901"]}]},{"name":"PKU-Baidu Fund","award":["2020BD003"],"award-info":[{"award-number":["2020BD003"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Games have long been benchmarks and testbeds for AI research. In recent years, with the development of new algorithms and the boost in computational power, many popular games played by humans have been solved by AI systems. Mahjong is one of the most popular games played in China and has been spread worldwide, which presents challenges for AI research due to its multi-agent nature, rich hidden information, and complex scoring rules, but it has been somehow overlooked in the community of game AI research. In 2020 and 2022, we held two AI competitions of Official International Mahjong, the standard variant of Mahjong rules, in conjunction with a top-tier AI conference called IJCAI. We are the first to adopt the duplicate format in evaluating Mahjong AI agents to mitigate the high variance in this game. By comparing the algorithms and performance of AI agents in the competitions, we conclude that supervised learning and reinforcement learning are the current state-of-the-art methods in this game and perform much better than heuristic methods based on human knowledge. We also held a human-versus-AI competition and found that the top AI agent still could not beat professional human players. We claim that this game can be a new benchmark for AI research due to its complexity and popularity among people.<\/jats:p>","DOI":"10.3390\/a16050235","type":"journal-article","created":{"date-parts":[[2023,5,1]],"date-time":"2023-05-01T12:10:03Z","timestamp":1682943003000},"page":"235","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Official International Mahjong: A New Playground for AI Research"],"prefix":"10.3390","volume":"16","author":[{"given":"Yunlong","family":"Lu","sequence":"first","affiliation":[{"name":"School of Computer Science, Peking University, Beijing 100871, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenxin","family":"Li","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University, Beijing 100871, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenlong","family":"Li","sequence":"additional","affiliation":[{"name":"Mahjong International League, 1002 Lausanne, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,28]]},"reference":[{"key":"ref_1","first-page":"21","article-title":"Chinook the world man-machine checkers champion","volume":"17","author":"Schaeffer","year":"1996","journal-title":"AI Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/S0004-3702(01)00129-1","article-title":"Deep blue","volume":"134","author":"Campbell","year":"2002","journal-title":"Artif. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"508","DOI":"10.1126\/science.aam6960","article-title":"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker","volume":"356","author":"Schmid","year":"2017","journal-title":"Science"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1126\/science.aao1733","article-title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals","volume":"359","author":"Brown","year":"2018","journal-title":"Science"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"885","DOI":"10.1126\/science.aay2400","article-title":"Superhuman AI for multiplayer poker","volume":"365","author":"Brown","year":"2019","journal-title":"Science"},{"key":"ref_9","unstructured":"Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki, W.M., Dudzik, A., Huang, A., Georgiev, P., and Powell, R. (2019). Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind Blog, 2."},{"key":"ref_10","unstructured":"Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., and Hesse, C. (2019). Dota 2 with large scale deep reinforcement learning. arXiv."},{"key":"ref_11","first-page":"621","article-title":"Towards playing full moba games with deep reinforcement learning","volume":"33","author":"Ye","year":"2020","journal-title":"ADvances Neural Inf. Process. Syst."},{"key":"ref_12","unstructured":"Li, J., Koyamada, S., Ye, Q., Liu, G., Wang, C., Yang, R., Zhao, L., Qin, T., Liu, T.Y., and Hon, H.W. (2020). Suphx: Mastering mahjong with deep reinforcement learning. arXiv."},{"key":"ref_13","unstructured":"Copeland, B.J. (2023, April 28). The Modern History of Computing. Available online: https:\/\/plato.stanford.edu\/entries\/computing-history\/."},{"key":"ref_14","unstructured":"Wikipedia (2023, February 08). World Series of Poker\u2014Wikipedia, The Free Encyclopedia. Available online: http:\/\/en.wikipedia.org\/w\/index.php?title=World%20Series%20of%20Poker&oldid=1133483344."},{"key":"ref_15","first-page":"112","article-title":"The annual computer poker competition","volume":"34","author":"Bard","year":"2013","journal-title":"AI Mag."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lu, Y., and Li, W. (2022). Techniques and Paradigms in Modern Game AI Systems. Algorithms, 15.","DOI":"10.3390\/a15080282"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1109\/TSSC.1968.300136","article-title":"A formal basis for the heuristic determination of minimum cost paths","volume":"4","author":"Hart","year":"1968","journal-title":"IEEE Trans. Syst. Sci. Cybern."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1016\/0004-3702(79)90016-X","article-title":"A minimax algorithm better than alpha-beta?","volume":"12","author":"Stockman","year":"1979","journal-title":"Artif. Intell."},{"key":"ref_19","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_20","unstructured":"Heinrich, J., Lanctot, M., and Silver, D. (2015, January 7\u20139). Fictitious self-play in extensive-form games. Proceedings of the International Conference on Machine Learning. PMLR, Lille, France."},{"key":"ref_21","unstructured":"McMahan, H.B., Gordon, G.J., and Blum, A. (2003, January 21\u201324). Planning in the presence of cost functions controlled by an adversary. Proceedings of the 20th International Conference on Machine Learning (ICML-03), Washington, DC, USA."},{"key":"ref_22","first-page":"4193","article-title":"A unified game-theoretic approach to multiagent reinforcement learning","volume":"30","author":"Lanctot","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_23","unstructured":"Wikipedia (2023, February 08). Mahjong solitaire\u2014Wikipedia, The Free Encyclopedia. Available online: http:\/\/en.wikipedia.org\/w\/index.php?title=Mahjong%20solitaire&oldid=1129612325."},{"key":"ref_24","unstructured":"Osborne, M.J., and Rubinstein, A. (1994). A Course in Game Theory, MIT Press."},{"key":"ref_25","unstructured":"Sloper, T. (2023, February 08). FAQ 10: Simplified Rules for Mah-Jongg\u2014sloperama.com. Available online: https:\/\/sloperama.com\/mjfaq\/mjfaq10.html."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhou, H., Zhang, H., Zhou, Y., Wang, X., and Li, W. (2018, January 2\u20134). Botzone: An online multi-agent competitive platform for ai education. Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, Larnaca, Cyprus.","DOI":"10.1145\/3197091.3197099"},{"key":"ref_27","unstructured":"(2023, February 08). IJCAI 2020 Mahjong AI Competition\u2014botzone.org.cn. Available online: https:\/\/botzone.org.cn\/static\/gamecontest2020a.html."},{"key":"ref_28","unstructured":"(2023, February 08). IJCAI 2022 Mahjong AI Competition\u2014botzone.org.cn. Available online: https:\/\/botzone.org.cn\/static\/gamecontest2022a.html."},{"key":"ref_29","unstructured":"(2023, February 08). GitHub\u2014ailab-pku\/PyMahjongGB: Python Fan Calculator for Chinese Standard Mahjong\u2014github.com. Available online: https:\/\/github.com\/ailab-pku\/PyMahjongGB."},{"key":"ref_30","unstructured":"Yata (2023, February 08). SuperJong. Available online: https:\/\/botzone.org.cn\/static\/IJCAI2020MahjongPPT\/03-SuperJong-yata.pdf."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Xie, S., Girshick, R., Doll\u00e1r, P., Tu, Z., and He, K. (2017, January 21\u201326). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.634"},{"key":"ref_33","unstructured":"Heinrich, J., and Silver, D. (2016). Deep reinforcement learning from self-play in imperfect-information games. arXiv."},{"key":"ref_34","unstructured":"Zhao, E., Yan, R., Li, J., Li, K., and Xing, J. (March, January 22). AlphaHoldem: High-Performance Artificial Intelligence for Heads-Up No-Limit Texas Hold\u2019em from End-to-End Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event."},{"key":"ref_35","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_37","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 19\u201324). Asynchronous methods for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, PMLR, New York City, NY, USA."},{"key":"ref_38","unstructured":"Guan, Y., Liu, M., Hong, W., Zhang, W., Fang, F., Zeng, G., and Lin, Y. (2022). PerfectDou: Dominating DouDizhu with Perfect Information Distillation. arXiv."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/5\/235\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:26:14Z","timestamp":1760124374000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/5\/235"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,28]]},"references-count":38,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["a16050235"],"URL":"https:\/\/doi.org\/10.3390\/a16050235","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,28]]}}}