A multi-agent reinforcement learning algorithm with fuzzy approximation for Distributed Stochastic Unit Commitment

Abstract

This paper proposes a novel multi-agent unit commitment model under Smart Grid (SG) environment to minimize the demand satisfaction error and production cost. This is a distributed solution applicable in non-deterministic environments with stochastic parameters intending to solve Distributed Stochastic Unit Commitment (DSUC) problem. We use multi-agent reinforcement learning (RL) in which agents learn as independent learners to cooperatively satisfy the demand profile. The learning mechanism proceeds using a reward signal, which is given based on the performance of the entire system as well as the impact of the joint action of the agents. The learning agent utilizes a novel multi-agent version of Fuzzy Least Square Policy Iteration (FLSPI) as a model-free RL algorithm to approximate Q-function. Based on this approximation, the agent makes the best decision to achieve the goals while considering the constraints governing the system. Uncertainty sources in our definition of the problem are fluctuations in the predicted demand function, random productions of clean energy generators and the possibility of accidental failure in power generators. Training for one time interval (i.e. one season or one year) consisting of several time intervals (i.e. days) can be simultaneously conducted by one trial in our method. We have conducted our experiment in two different frameworks. These frameworks are defined based on the problem complexity in terms of the number of generators, the uncertainties in the environment and the system constraints. The results show that the learning agent learns to satisfy the demand profile as well as other constrains.

Keywords

Multi-agent reinforcement learning Stochastic Unit Commitment fuzzy approximation

Get full access to this article

View all access options for this article.

References

Agrawal , Overview of doe microgrid activities. In: Symposium on Microgrid, Montreal, June, 23 (2006).

S.M.

Amin and

B.F.

Wollenberg , Toward a smart grid: Power delivery for the 21st century, IEEE Power and Energy Magazine 3(5) (2005), 34–41.

T.R.

Ayodele , Determination of probability distribution function for modelling global solar radiation: Case study of ibadan, Nigeria, International Journal of Applied Science and Engineering 13(3) (2015), 233–245.

P.P.

Barker and

R.W.

De Mello , Determining the impact of distributed generation on power systems. i. radial distribution systems. In: Power Engineering Society Summer Meeting, 2000. IEEE, IEEE 3 (2000), 1645–1656.

Bellman , Dynamic programming. Courier Corporation (2013).

Boutilier , Sequential optimality and coordination in multiagent systems. In: IJCAI, 99 (1999), 478–485.

Claus and

Boutilier , The dynamics of reinforcement learning in cooperative multiagent systems, AAAI/IAAI 1998 (1998), 746–752.

Dibangoye ,

Doniec ,

Fakham ,

Colas and

Guillaud , Distributed economic dispatch of embedded generation in smart grids, Engineering Applications of Artificial Intelligence 44 (2015), 64–78.

Fang ,

Misra ,

Xue and

Yang , Smart grid—the new and improved power grid: A survey, IEEE Communications Surveys & Tutorials 14(4) (2012), 944–980.

10.

Ghorbani ,

Derhami and

Afsharchi , Fuzzy least square policy iteration and its mathematical analysis, International Journal of Fuzzy Systems 19(3) (2017), 849–862.

11.

Hawkes and

Leach , Modelling high level system design and unit commitment for a microgrid, Applied Energy 86(7) (2009), 1253–1265.

12.

Logenthiran ,

Srinivasan ,

Khambadkone and

Aung , Multi-agent system (mas) for short-term generation scheduling of a microgrid. In: Sustainable Energy Technologies (ICSET), 2010 IEEE International Conference on, IEEE (2010), pp. 1–6.

13.

Logenthiran ,

Srinivasan ,

Khambadkone and

Aung , Scalable multi-agent system (mas) for operation of a microgrid in islanded mode. In: Power Electronics, Drives and Energy Systems (PEDES) & 2010 Power India, 2010 Joint International Conference on, IEEE, (2010), pp. 1–6.

14.

Mannion ,

Mason ,

Devlin ,

Duggan and

Howley , Dynamic economic emissions dispatch optimisation using multi-agent reinforcement learning. In: Proceedings of the Adaptive and Learning Agents workshop (at AAMAS 2016) (2016).

15.

Nagata ,

Ohono ,

Kubokawa ,

Sasaki and

Fujita , A multi-agent approach to unit commitment problems. In: Power Engineering Society Winter Meeting, 2002. IEEE, IEEE, 1 (2002), pp. 64–69.

16.

Nikovski and

Zhang , Factored markov decision process models for stochastic unit commitment. In: Innovative Technologies for an E_cient and Reliable Electricity Supply (CITRES), 2010 IEEE Conference on, IEEE, (2010), pp. 28–35.

17.

M.P.

Nowak and

Römisch , Stochastic lagrangian relaxation applied to power scheduling in a hydro-thermal system under uncertainty, Annals of Operations Research 100(1-4) (2000), 251–272.

18.

U.A.

Ozturk ,

Mazumdar and

B.A.

Norman , A solution to the stochastic unit commitment problem using chance constrained programming, IEEE Transactions on Power Systems 19(3) (2004), 1589–1598.

19.

Papavasiliou and

S.S.

Oren , Multiarea stochastic unit commitment for high wind penetration in a transmission constrained network, Operations Research 61(3) (2013), 578–592.

20.

Saravanan ,

Das ,

Sikri and

Kothari , A solution to the unit commitment problem–a review, Frontiers in Energy 7(2) (2013), 223.

21.

Soltani ,

Ghaljehei ,

Gharehpetian and

Aalami , Integration of smart grid technologies in stochastic multi-objective unit commitment: An economic emission analysis, International Journal of Electrical Power & Energy Systems 100 (2018), 565–590.

22.

Stone and

Veloso , Multiagent systems: A survey from a machine learning perspective, Autonomous Robots 8(3) (2000), 345–383.

23.

R.S.

Sutton and

A.G.

Barto , Reinforcement learning, Journal of Cognitive Neuroscience 11(1) (1999), 126–134.

24.

Takriti ,

J.R.

Birge and

Long , A stochastic model for the unit commitment problem, IEEE Transactions on Power Systems 11(3) (1996), 1497–1508.

25.

Wang ,

Wang and

Guan , Stochastic unit commitment with uncertain demand response, IEEE Transactions on Power Systems 28(1) (2013), 562–563.

26.

Wang ,

Xia and

Kang , A novel security stochastic unit commitment for wind-thermal system operation. In: Electric Utility Deregulation and Restructuring and Power Technologies (DRPT), 2011 4th International Conference on, IEEE, (2011), pp. 386–393.

27.

Xiao ,

Dai ,

Pengy ,

Wang and

H.V.

Poor , Reinforcement learning-based energy trading for microgrids. arXiv preprint arXiv:180106285 (2018).

28.

N.Y.

YÜrÜşen , and

J.J.

Melero , Probability density function selection based on the characteristics of wind speed data. In: Journal of Physics: Conference Series, IOP Publishing 753 (2016), 032067.