Solving the playing strategy of Dou Dizhu using convolutional neural network: A residual learning approach

Abstract

Poker is the typical game of incomplete information, and remains a longstanding challenge problem in artificial intelligence (AI). The game of Dou Dizhu has been viewed as a thorny topic in AI since it is featured with hidden information and large branching factors, and the cooperation and competition should also be handled. In this article, deep learning is adopted to train a supervised learning playing strategy network (PSN) for Dou Dizhu directly from expert human playing. Through experiments, it was found that the sample design with the appropriate historical playing hand sequence and more features of the playing situation, can help the PSN learn more competitive and accurate playing strategies faster. In the online game platform, the strategy network-based game agent reaches an average winning rate of 52.22% against the human players. In addition, the analysis of the gameplay data against human players shows that the playing strategy network has learned the rules of playing and the characteristics of card recognition and reasonable demolition, cooperation and reasoning. Finally, we improve the performance of the PSN in the aspect of sample design. Then, the experimental results show that with proper marking of the number of remaining hands, the performance of the PSN can be enhanced.

Keywords

Supervised learning convolutional neural networks playing strategy incomplete information Dou Dizhu

Get full access to this article

View all access options for this article.

References

Silver

Schrittwieser

Simonyan

Antonoglou

Huang

and Guez

, Mastering the game of Go without human knowledge, Nature 550(7676) (2017), 354.

Schaeffer

Culberson

Treloar

Knight

and Szafron

, A world championship caliber checkers program, Artificial Intelligence 53(2–3) (1992), 273–289.

Lai

, Giraffe: Using deep reinforcement learning to play chess, arXiv preprint arXiv:1509.01549, 2015.

Moravčík

Schmid

Burch

Lisý

Morrill

Bard

et al., Deepstack: Expert-level artificial intelligence in heads-up no-limit poker, Science 356(6337) (2017), 508–513.

Schaeffer

Hlynka

and Jussila

, Temporal difference learning applied to a high-performance game-playing program, in Proc. 2001 International joint conference on Artificial intelligence, 2001.

Guo

Singh

Lee

Lewis

R.-L.

and Wang

, Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning, in Advances in neural information processing systems, 2014.

Silver

Huang

Maddison

C.-J.

Guez

Sifre

and Van Den Driessche

, Mastering the game of Go with deep neural networks and tree search, Nature 529(7587) (2016), 484.

Liu

and Ding

, The design and implementation of a computer game algorithm of Dou Dizhu. In 2017 29th Chinese Control And Decision Conference (CCDC), 2017, pp. 7612–7617.

Whitehouse

Powley

E.-J.

and Cowling

P.-I.

, Determinization and information set Monte Carlo tree search for the card game Dou Di Zhu, in 2011 IEEE Conference on Computational Intelligence and Games (CIG’11), 2011.

10.

Ginsberg

M.-L.

, GIB: Imperfect information in a computationally challenging game, Journal of Artificial Intelligence Research 14 (2001), 303–358.

11.

Buro

Long

J.-R.

Furtak

and Sturtevant

, Improving state evaluation, inference, and search in trick-based card games, in Twenty-First International Joint Conference on Artificial Intelligence, 2009.

12.

Schäfer

Buro

and Hartmann

, The UCT algorithm applied to games with imperfect information, Diploma, Otto-Von-Guericke Univ. Magdeburg, Magdeburg, Germany 2008.

13.

Frank

and Basin

, Search in games with incomplete information: A case study using bridge card play, Artificial Intelligence 100(1–2) (1998), 87–123.

14.

Cowling

P.-I.

Powley

E.J.

and Whitehouse

, Information set monte carlo tree search, IEEE Transactions on Computational Intelligence and AI in Games 4(2) (2012), 120–143.

15.

LeCun

Bengio

and Hinton

, Deep learning, Nature 521(7553) (2015), 436.

16.

Krizhevsky

Sutskever

and Hinton

G.-E.

, Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems (2012), 1097–1105.

17.

Sermanet

Eigen

Zhang

Mathieu

Fergus

and LeCun

, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229, 2013.

18.

Zeiler

M.-D.

and Fergus

, Visualizing and understanding convolutional networks, in European conference on computer vision (2014), 818–833.

19.

Guo

and Lai

, A survey on still image based human action recognition, Pattern Recognition 47(10) (2014), 3343–3361.

20.

Presti

L.-L.

and La Cascia

, 3D skeleton-based human action classification: A survey, Pattern Recognition 53 (2016), 130–147.

21.

Zhang

Ogunbona

P.-O.

Wang

and Tang

, RGB-D-based action recognition datasets: A survey, Pattern Recognition 60 (2016), 86–105.

22.

Kim

, Convolutional neural networks for sentence classification, Empirical Methods in Natural Language Processing (2014), 1746–1751.

23.

Mnih

Kavukcuoglu

Silver

Rusu

A.-A.

Veness

and Bellemare

M.-G.

, Human-level control through deep reinforcement learning, Nature 518 (7540) (2015), 529–533.

24.

Lin

Chen

and Yan

, Network in network, arXiv: Neural and Evolutionary Computing, 2013.

25.

Szegedy

Liu

Jia

Sermanet

Reed

Anguelov

et al., Going deeper with convolutions, in: Proc. 2015 IEEE conference on computer vision and pattern recognition, 2015.

26.

Ioffe

and Szegedy

, Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167, 2015.

27.

Szegedy

Vanhoucke

Ioffe

Shlens

and Wojna

, Rethinking the inception architecture for computer vision, in: Proc. 2016 IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.

28.

Zhang

Ren

and Sun

, Deep residual learning for image recognition, in: Proc. 2016 IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

29.

Sutton

R.-S.

and Barto

A.-G.

, Reinforcement learning: An introduction: MIT press, 2018.

30.

Zeiler

M.-D.

, ADADELTA: an adaptive learning rate method, arXiv preprint arXiv:1212.5701, 2012.

31.

Haynes

Corns

and Venayagamoorthy

G.-K.

, An exponential moving average algorithm, in 2012 IEEE Congress on Evolutionary Computation, 2012.

32.

Rules of Card Games: Dou Di Zhu, [Online]. Available: https://www.pagat.com/climbing/doudizhu.html.

33.

Manuel

L.-M.

Carro

and Antonio

S.-E.

, Application of deep reinforcement learning to intrusion detection for supervised problems, Expert Systems with Applications 141(112963) (2020).