The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.
LiXGuoXTangH, et al. Survey of integrated flexible job shop scheduling problems. Comput Ind Eng2022; 174: 108786.
2.
ZhengPWangHSangZ, et al. Smart manufacturing systems for Industry 4.0: conceptual framework, scenarios, and future perspectives. Front Mech Eng2018; 13: 137–150.
3.
ZhangJDingGZouY, et al. Review of job shop scheduling research and its new perspectives under Industry 4.0. J Intell Manuf2019; 30: 1809–1830.
WorknehADMahaG.Scheduling algorithms: challenges towards smart manufacturing. Int J Electr Comput Eng Syst2022; 13(7): 587–600.
6.
GaoKCaoZZhangL, et al. A review on swarm intelligence and evolutionary algorithms for solving flexible job shop scheduling problems. IEEE/CAA J Autom Sin2019; 6(4): 904–916.
7.
DemirYYilmazH.An efficient priority rule for flexible job shop scheduling problem. J Eng Res Appl Sci2021; 10(2): 1906–1918.
8.
KatochSChauhanSSKumarV.A review on genetic algorithm: past, present, and future. Multimedia Tools Appl2021; 80: 8091–8126.
9.
HanYChenXXuM, et al. A multi-objective flexible job-shop cell scheduling problem with sequence-dependent family setup times and intercellular transportation by improved NSGA-II. Proc IMechE, Part B: J Engineering Manufacture2022; 236(5): 540–556.
10.
NingTJinHSongX, et al. An improved quantum genetic algorithm based on MAGTD for dynamic FJSP. J Ambient Intell Hum Comput2018; 9: 931–940.
11.
ChenZZouJWangW.Digital twin oriented multi-objective flexible job shop scheduling model and its hybrid particle swarm optimization. Proc IMechE, Part B: J Engineering Manufacture2023; 237(8): 1269–1282.
12.
XiaWWuZ.An effective hybrid optimization approach for multi-objective flexible job-shop scheduling problems. Comput Ind Eng2005; 48(2): 409–425.
13.
ZhangXWangSYiL, et al. An integrated ant colony optimization algorithm to solve job allocating and tool scheduling problem. Proc IMechE, Part B: J Engineering Manufacture2018; 232(1): 172–182.
14.
YangCHaiboSDalianC.Differential evolution algorithm with dynamic multi-population applied to flexible job shop schedule. Eng Optim2022; 54(3): 387–408.
15.
OrduMDerO.Polymeric materials selection for flexible pulsating heat pipe manufacturing using a comparative hybrid MCDM approach. Polymers2023; 15(13): 2933.
16.
IkuobaseEOkpakoSO.Application of MCDM method in material selection for optimal design: a review. Results Mater2020; 7: 100115.
17.
BaiYLvY. Reinforcement learning-based job shop scheduling for remanufacturing production. In: IEEE international conference on industrial engineering and engineering management, Kuala Lumpur, Malaysia, 7–10 December 2022, pp.246–251. New York: IEEE.
18.
MousaviSSSchukatMHowleyE. Deep reinforcement learning: an overview. In: Proceedings of SAI intelligent systems conference, London, UK, 21–22 September 2016, pp.426–440. Cham: Springer.
19.
PutermanML. Markov decision processes. In: PutermanML (ed.) Handbooks in operations research and management science. Elsevier. 1990, pp.331–434.
20.
WuCZhouFTsaiC, et al. A deep learning approach for the dynamic dispatching of unreliable machines in re-entrant production systems. Int J Prod Res2020; 58(9): 2822–2840.
21.
WuZFanHSunY, et al. Efficient multi-objective optimization on dynamic flexible job shop scheduling using deep reinforcement learning approach. Processes2023; 11(7): 2018.
22.
ShaoKTangZZhuY, et al. A survey of deep reinforcement learning in video games. arXiv Preprint arXiv:1912.10944, 2019.
23.
McKenzieMCMcDonnellMD.Modern value based reinforcement learning: a chronological review. IEEE Access2022; 10: 134704–134725.
24.
GanXZuoYYangG, et al. Dynamic scheduling for dual-objective job shop with machine breakdown by reinforcement learning. Proc IMechE, Part B: J Engineering Manufacture2024; 238(1–2): 3–17.
25.
LiuCLChangCCTsengCJ.Actor-critic deep reinforcement learning for solving job shop scheduling problems. IEEE Access2020; 8: 71752–71762.
26.
YangSXuZWangJ.Intelligent decision-making of scheduling for dynamic permutation flowshop via deep reinforcement learning. Sensors2021; 21: 1019.
27.
GuiYTangDZhuH, et al. Dynamic scheduling for flexible job shop using a deep reinforcement learning approach. Comput Ind Eng2023; 180: 109255.
28.
BouteRNGijsbrechtsJVanJW, et al. Deep reinforcement learning for inventory control: a roadmap. Eur J Oper Res2022; 298(2): 401–412.
29.
VanHHGuezASilverD.Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, Phoenix, AZ, 12–17 February 2016.
30.
ZhaoYWangYTanY, et al. Dynamic jobshop scheduling algorithm based on deep Q network. IEEE Access2021; 9: 122995–123011.
31.
TurgutYBozdagCE. Deep Q-network model for dynamic job shop scheduling problem based on discrete event simulation. In: Winter simulation conference, Orlando, FL, 14–18 December 2020, pp.1551–1559. New York: IEEE.
32.
XuZChangDLuoT, et al. Intelligent scheduling of double-deck traversable cranes based on deep reinforcement learning. Eng Optim2023; 55(12): 2034–2050.
33.
LuoS.Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Appl Soft Comput2020; 91: 106208.
34.
LuoSZhangLFanY.Dynamic multi-objective scheduling for flexible job shop by deep reinforcement learning. Comput Ind Eng2021; 59: 107489.
35.
ChangJYuDHuY, et al. Deep reinforcement learning for dynamic flexible job shop scheduling with random job arrival. Processes2022; 10(4): 760.
36.
ChuaHS.Dynamic flexible job shop scheduling using double deep Q-network. Singapore: Final Year Project, Nanyang Technological University, 2021.
37.
WangZSchaulTHesselM, et al. Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on machine learning, New York, NY, 19–24 June 2016, pp.1995–2003.
38.
QinZDazzleJLuY.Dynamic production scheduling towards self-organizing mass personalization: a multi-agent dueling deep reinforcement learning approach. J Manuf Syst2023; 68: 242–257.
39.
HanBYangJ.Research on adaptive job shop scheduling problems based on dueling double DQN. IEEE Access2020; 8: 186474–186495.
40.
HuangYWeiGLWangYX. Vd d3qn: the variant of double deep q-learning network with dueling architecture. In: 37th Chinese control conference, Wuhan, China, 25–27 July 2018, pp.9130–9135.
41.
ShiDFanWXiaoY, et al. Intelligent scheduling of discrete automated production line via deep reinforcement learning. Int J Prod Res2020; 58: 3362–3380.
42.
ShiueYLeeKSuC.Real-time scheduling for a smart factory using a reinforcement learning approach. Comput Ind Eng2018; 125: 604–614.
