The predictions of classical game theory for one-shot and finitely repeated play of many 2x2 simulta neous games do not correspond to human behavior observed in laboratory experiments. The promis ing results of learning models in tracking human behavior coupled with the growing electronic market and the number of e-commerce applications has resulted in an increased interest in studying the behavior of adaptive artificial agents in different economic games. We model agents with a reinforce ment learning algorithm and analyze cooperative behavior in a sequential prisoner's dilemma game. Our results demonstrate the ability of artificial agents to learn cooperative behavior even in sequential games where defection is the subgame perfect Nash equilibrium. We attribute the reciprocal-like behavior to the structural flow of information, which reduces the risks of exploitation faced by the second-mover. Additionally, we analyze the impact of the second-mover's temptation payoff and pay off risks on the rate of cooperative behavior.
Andreoni, J. , & Miller, J. H. (1993). Rational cooperation in the finitely repeated prisoner's dilemma: Experimental evidence . The Economic Journal, 103, 570–585 .
2.
Ashlock, D. , Smucker, M. D. , Stanley, E. A. , & Tesfatsion, L. (1996). Preferential partner selection in an evolutionary study of prisoner's dilemma . BioSystems, 37(1–2), 99–125 .
3.
Axelrod, R. (1984). The evolution of cooperation. New York: Basic Books .
4.
Axelrod, R. , & Hamilton, W. D. (1981). The evolution of cooperation . Science, 211, 1390–1396 .
5.
Bolle, F. , & Ockenfels, P. (1990). Prisoner's dilemma as a game with incomplete information . Journal of Economic Psychology, 11, 69–84 .
6.
Camerer, C. , & Ho, T. (1999). Experience-weighted attraction learning in normal form games . Econometrica, 67(4), 827–874 .
7.
Clark, K. , & Sefton, M. (2001). The sequential prisoner's dilemma: Evidence on reciprocation . The Economic Journal, 111, 51–68 .
8.
Cox, J. C. , & Deck, C. A. (2005). On the nature of reciprocal motives . Economic Inquiry, 43(3), 623–635 .
9.
Duffy, J. (2001). Learning to speculate: Experiments with artificial and real agents . Journal of Economic Dynamics and Control, 25(3–4), 295–319 .
10.
Erev, I. , & Roth, A. (1998). Predicting how people play games: Reinforcement learning in games with unique strategy equilibrium . American Economic Review, 88, 848–881 .
11.
Fang, C. , Kimbrough, S. O. , Pace, S. , Valluri, A. , & Zheng, Z. (2002). On adaptive emergence of trust behavior in the game of stag hunt . Journal of Group Decision and Negotiation, 11(6), 449–467 .
12.
Feltovich, N. (2000). Reinforcement-based vs. beliefs-based learning in experimental asymmetric-information games . Econometrica, 68, 605–641 .
13.
Grossman, S. J. , & Hart, O. D. (1983). An analysis of principal– agent problem . Econometrica, 51, 7–45 .
14.
Guth, W. (1995). On ultimatum bargaining experiments—a personal review . Journal of Economic Behavior and Organization, 27, 329–344 .
15.
Holmstrom, B. (1979). Moral hazard and observability . The Bell Journal of Economics, 10(1), 74–91 .
16.
Huck, S. , & Miller, W. (2000). Perfect versus imperfect observability—an experimental test of Bagwell's result . Games and Economic Behavior, 31, 174–190 .
17.
Kaelbling, L. P. , Littman, M. L. , & Moore, A. W. (1996). Reinforcement learning: A survey . Journal of Artificial Intelligence Research, 4, 237–285 .
18.
Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. Proceedings of the Eleventh International Conference on Machine Learning(pp. 157–163). San Francisco: Morgan Kaufmann .
19.
Littman, M. L. (2001). Friend-or-foe Q-learning in general-sum games. Proceedings of the Eighteenth International Conference on Machine Learning (pp. 322–328). San Francisco: Morgan Kaufmann .
20.
McCabe, K. , & Smith, V. (2000). A two-person trust game played by naïve and sophisticated subjects . Proceedings of the National Academy of Sciences, 97(7), 3777–3781 .
21.
McKelvey, R. , & Palfrey, T. (1992). An experimental study of the centipede game . Econometrica, 60(4), 803–836 .
22.
Meidinger, C. , & Terracol, A. (2002). Reciprocation and reinforcement learning model in the investment game . Paper presented at the 19th Conference on Journees de Micro-economic Appliquee.
23.
Miller, J. H. (1996). The coevolution of automata in the repeated prisoner's dilemma . Journal of Economic Behavior and Organization, 29(1), 87–112 .
24.
von Neumann, J. , & Morgenstern, O. (1944). Theory of games and economics behavior. Princeton, NJ: Princeton University Press .
25.
Nowak, M. A. (1990). Stochastic Strategies in the Prisoner's Dilemma . Theoretical Population Biology, 38, 93–112
26.
Nowak, M. A. , & May, R. M. (1992). Evolutionary games and spatial chaos . Nature, 359, 826–829 .
27.
Ochs, J. , & Roth, A. E. (1989). An experimental study of sequential bargaining . American Economic Review, 79, 355–384 .
28.
Oskamp, S. (1971). Effects of programmed strategies on cooperation in the prisoner's dilemma and other mixed-motive games . Journal of Conflict Resolution, 15, 225–229 .
29.
Poundstone, W. (1993). Prisoner's dilemma: John von Neumann, game theory, and the puzzle of the bomb. Oxford, UK: Oxford University Press .
30.
Radner, R. (1981). Monitoring cooperative agreements in a repeated principal–agent relationship . Econometrica, 49, 1127–1148 .
31.
Radner, R. (1985). Repeated principal–agent games with discounting . Econometrica, 53(5), 1173–1198 .
32.
Radner, R. (1986). Repeated partnership games with imperfect monitoring and no discounting . The Review of Economic Studies, 53(1), 43–57 .
33.
Ross, S. (1973). The economic theory of agency: The principal's problem . American Economic Review, 63, 134–139 .
34.
Rubinstein, A. (1986). Finite automata in the repeated prisoner's dilemma . Journal of Economic Theory, 39, 83–96 .
35.
Sandholm, T. W. , & Crites, R. H. (1996) Multi-agent reinforcement learning in the iterated prisoner's dilemma , BioSystems, 37(1–2), 47–166 .
36.
Selten, R. , & Stoecker, R. (1986). End behavior in sequences of finite prisoners' dilemma super-games . Journal of Economic Behavior and Organization, 7, 47–70 .
37.
Shoham, Y. , & Tennenholtz, M. (1993). Co-learning and the evolution of coordinated multi-agent activity.
38.
Shubik, M. (1970). Game theory, behavior, and the paradox of the prisoner's dilemma: Three solutions . Journal of Conflict Resolution, 14, 181–194 .
39.
Suguwara, T. , & Lesser, V. (1993). On-line learning of coordination plans. Computer Science Technical Report93–27, Amherst, MA: University of Massachusetts .
40.
Sutton, R. S. , & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press .
41.
Tesauro, G. J. (1992). Practical issues in temporal difference learning . Machine Learning, 8, 257–277 .
42.
Tesfatsion, L. (2002). Agent-based computational economics: Growing economies from the bottom up . Artificial Life, 9(1), 55–82 .
43.
Watkins, C. J. C. H. (1989). Learning from delayed rewards. PhD Thesis, Cambridge University, UK.
44.
Wu, D. J. , Kimbrough, S. , & Zhong, F. (2002). Artificial agents play the mad mex trust game: A computational approach. In R. H. Sprague, Jr. (Ed.), Proceedings of the Thirty-fifth Annual Hawaii International Conference on System Sciences (HICSS-35, CD ROM), Los Alamitos, CA: IEEE Computer Society Press (pp. 335–347).
45.
Yang, C. , Weimann, J. , & Mitropoulos, A. (2001). Game structure and bargaining power in sequential mini-games: An experiment.
46.
Zhong, F. , Kimbrough, S. O. , & Wu, D. J. (2002). Cooperative agent systems: Artificial agents play the ultimatum game . Journal of Group Decision and Negotiation, 11(6), 433–447 .
