Multi-agent systems can be seen as an apparatus for testing the performance of real distributed systems. One problem encountered in multi-agent systems with the learning capability is credit assignment. This paper presents two methods for solving this problem. The first method assigns credit to the agents according to the history of the interaction while the second method assigns credit to the agents according to the knowledge of agents, and thus the shares of the agents are extracted from the feedback of the environment. The computer experiments show that critic learning has a positive impact in credit assignment problem.
SkinnerB.F., Operant behavior, American Psychologist18(8) (1963), 503.
2.
KaelblingL.P., LittmanM.L. and MooreA.W., Reinforcement learning: A survey, Journal of Artificial Intelligence Research (1996), 237–285.
3.
AgoginoA.K. and TumerK., Unifying temporal and structural credit assignment problems, in Proceedings of the third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume22004, pp. 980–987.
4.
SuttonR.S., Temporal credit assignment in reinforcement learning, Ph.D. dissertation, University of Massachusetts –Amherst, 1984.
5.
WatkinsC.J. and DayanP., Q-learning, Machine Learning8(3-4) (1992), 279–292.
6.
PanaitL. and LukeS., Cooperative multi-agent learning: The state of the art, Autonomous Agents and Multi-Agent Systems11(3) (2005), 387–434.
7.
MaoW. and GratchJ., The social credit assignment problem, in Proceedings of the Fourth International Working Conference on Intelligent Virtual Agents, Springer, 2003, pp. 39–47.
8.
KelleyH.H., Attribution theory in social psychology, in Nebraska symposium on motivation, University of Nebraska Press,
1967.
9.
KimuraH. and KobayashiS., An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value function, in International Conference on Machine Learning, 1998, pp. 278–286.
10.
SuttonR.S. and BartoA.G., Reinforcement learning: An introduction, MIT Press Cambridge1(1) (1998).
11.
HaratiA., AhmadabadiM.N. and AraabiB.N., Knowledgebased multiagent credit assignment: A study on task type and critic information, IEEE Systems Journal1(1) (2007), 55–67.
12.
AgoginoA., TumerK. and MiikkulainenR., Efficient credit assignment through evaluation function decomposition, in Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, 2005, pp. 1309–1316.
13.
KhareV.R., YaoX. and SendhoffB., Credit assignment among neurons in co-evolving populations, in Parallel Problem Solving from Nature-PPSN VIII, Springer, 2004, pp. 882–891.
14.
FriedrichJ., UrbanczikR. and SennW., Spatio-temporal credit assignment in neuronal population learning, PLoS Comput Biol7(6) (2011), e1002092.
MataricM.J., Reward functions for accelerated learning, in Proceedings of the Eleventh International Conference on Machine Learning, 1994, pp. 181–189.
17.
WeissG., Multiagent systems: A modern approach to distributed artificial intelligence, MIT Press, 1999.
18.
AraiS. and SycaraK., Credit assignment method for learning effective stochastic policies in uncertain domains, 2001, pp. 815–822.
19.
WalshM.M. and AndersonJ.R., Neural correlates of temporal credit assignment, in Proceedings of the 10th International Conference on Cognitive Modeling, pp. 2010, 256–270.
20.
JonesJ. and GoelA.K., Structural credit assignment in hierarchical classification, in Proceedings of International Conference on Artificial Intelligence, pp. 2007, 378–384.
21.
JonesJ. and GoelA., Hierarchical judgement composition: Revisiting the structural credit assignment problem, in Proceedings of the AAAIWorkshop on Challenges in Game AISan Jose, CA, USA, 2004, pp. 67–71.
22.
RoelfsemaP.R. and van OoyenA., Attention-gated reinforcement learning of internal representations for classification, Neural Computation17(10) (2005), 2176–2214.
23.
JonesJ. and GoelA., Knowledge organization and structural credit assignment, Reasoning, Representation, and Learning in Computer Games, 2005, p. 37.
24.
SaitoH., KatahiraK., OkanoyaK. and OkadaM., Statistical mechanics of structural and temporal credit assignment effects on learning in neural networks, Physical Review E83(5) (2011), 051125.
25.
MatarićM.J., Learning in behavior-based multi-robot systems: Policies, models, and other agents, Cognitive Systems Research2(1) (2001), 81–93.
26.
FuW.-T. and AndersonJ.R., From recurrent choice to skill learning: A reinforcement-learning model, Journal of Experimental Psychology: General135(2) (2006), 184.
27.
PashaeiK., TaghiyarehF. and BadieK., A negotiation-based genetic framework for multi-agent credit assignment, in Multiagent System Technologies, Springer, 2014, pp. 72–89.
28.
BianchiD., Learning grammatical rules from examples using a credit assignment algorithm, in Proceedings of the First Online Workshop on Soft Computing (WSC1), 1996, pp. 113–118.
29.
AgoginoA.K. and TumerK., Quicr-learning for multi-agent coordination, in Proceedings of the National Conference on Artificial Intelligence, vol. 21, no. 2. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006, p. 1438.
30.
BednarJ., Credit assignment and federal encroachment, Supreme Court Economic Review15(1) (2007), 285–308.
31.
FrandsenT.F. and NicolaisenJ., What is in a name? credit assignment practices in different disciplines, Journal of Informetrics4(4) (2010), 608–617.
32.
FranklinJ.A. and ManfrediV.U., Nonlinear credit assignment for musical sequences, in Second International Workshop on Intelligent Systems Design and Application, 2002, pp. 245–250.
33.
WhitacreJ.M., PhamT.Q. and SarkerR.A., Credit assignment in adaptive evolutionary algorithms, in Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, 2006, pp. 1353–1360.
34.
ZhuD. and KongM., A fuzzy cmac neural network model based on credit assignment, International Journal of Information Technology12(6) (2006), 1–8.
35.
SuS.-F., LeeZ.-J. and WangY.-P., Robust and fast learning for fuzzy cerebellar model articulation controllers, Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on36(1) (2006), 203–208.
36.
YehM.-F. and ChangK.-C., A self-organizing cmac network with gray credit assignment, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics36(3) (2005), 623–635.
37.
MataricM.J., Learning to behave socially, in Proceedings of Third International Conference on Simulation of Adaptive Behavior617 (1994), 453–462.
38.
RaeisyB., HaghighiS.G. and SafaviA.A., Active noise control system via multi-agent credit assignment, Journal of Intelligent and Fuzzy Systems26(2) (2014), 1051–1063.
39.
WolpertD.H., WheelerK.R. and TumerK., General principles of learning-based multi-agent systems, in Proceedings of the Third Annual Conference on Autonomous Agents, 1999, pp. 77–83.
40.
OntañónS. and PlazaE., A bartering approach to improve multiagent learning, pp, in Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 1 (2002), 386–393.
41.
TangamchitP., DolanJ.M. and KhoslaP.K., The necessity of average rewards in cooperative multirobot learning, in IEEE International Conference on Robotics and Automation, 2002, Proceedings ICRA’02, vol. 2, (2002), pp. 1296–1301.
42.
FuW.-T. and AndersonJ.R., Solving the credit assignment problem: Explicit and implicit learning of action sequences with probabilistic outcomes, Psychological Research72(3) (2008), 321–330.
43.
SamejimaK., DoyaK. and KawatoM., Inter-module credit assignment in modular reinforcement learning, Neural Networks16(7) (2003), 985–994.
44.
YasiniS., SitaniM.B.N. and KiramporA., Reinforcement learning and neural networks for multi-agent nonzerosum games of nonlinear constrained-input systems, International Journal of Machine Learning and Cybernetics, 2014, pp. 1–14.
45.
ShojaiemehrB. and RafsanjaniM.K., A multi-agent based model for collective purchasing in electronic commerce, International Journal of Machine Learning and Cybernetics4(4) (2013), 409–417.
46.
LangJ. and JiangQ., Artificial intelligence based profitsharing algorithm in multi-agent systems, Journal of Networks9(1) (2014), 48–55.
47.
YliniemiL. and TumerK., Multi-objective multiagent credit assignment through difference rewards in reinforcement learning, in Simulated Evolution and Learning, Springer, 2014, pp. 407–418.
48.
BengioY., How auto-encoders could provide credit assignment in deep networks via target propagation, arXiv preprint arXiv:1407.7906, 2014.
49.
TaghizadehN. and BeigyH., A novel graphical approach to automatic abstraction in reinforcement learning, Robotics and Autonomous Systems61(8) (2013), 821–835.
50.
RahaieZ. and BeigyH., Expertness framework in multi-agent systems and its application in credit assignment problem, Intelligent Data Analysis18(3) (2014), 511–528.
51.
AraabiB.N., MastoureshghS. and AhmadabadiM.N., A study on expertise of agents and its effects on cooperativelearning, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics37(2) (2007), 398–409.
