Material handling is one of the most important components in manufacturing industries. Conveyor belts belong to one of the material handling equipment categories that move objects and connect distant sections of a manufacturing process. This paper proposes an integrated approach of deep-learning object recognition in a packaging conveyor for real-time automatic object counting, shape detection, and color description. The deep neural network model was built in TensorFlow by using the Single-Shot Detection (SSD) method and Feature Pyramid Network (FPN) extraction. The model was improved by implementing k-means clustering and k-nearest neighbors algorithms in the object detection and color description. The novelty of the research is the implementation of deep learning with three integrated features in a manufacturing-related process. A prototype of a packaging conveyor was constructed to function as a miniature manufacturing process. The proposed system's accuracy was evaluated in terms of object counting, shape detection, and color description. Test objects varied in 2 different shapes (cube and box) and three different colors (blue, orange, and green). The average accuracy of the proposed system is 93.7% in object counting, 97.2% in shape detection, and 77.5% in color description, with an overall average accuracy of 90.0%. The color description was found to be sensitive to illumination level. Future improvements for the color descriptor's accuracy include creating a specific color data set and covering data with various illumination levels.
BahaghighatM.AkbariL.XinQ. (2019). A machine learning-based approach for counting blister cards within drug packages. IEEE Access, 7, 83785–83796. https://doi.org/10.1109/ACCESS.2019.2924445
2.
BendowskaK.ZawadzkiP. (2023). Development and verification of a simulation model of an automated assembly line. Applied Sciences, 13(18), 10142. https://doi.org/10.3390/app131810142
3.
FujitaH.ItagakiM.IchikawaK.HooiY. K.KawaharaK.SarlanA. (2020). Fine-tuned surface object detection applying pre-trained mask R-CNN models. In International conference on computational intelligence (ICCI) (pp. 19–22). IEEE Press.
4.
GkioxariG.GirshickR.MalikJ. (2015). Contextual action recognition with R* CNN. In IEEE International conference on computer vision (ICCV) (pp. 1080–1088). IEEE Press.
5.
GoodfellowI.BengioY.CourvilleA. (2016). Deep learning, edition (pp. 224–270). MIT Press.
6.
HaoJ.HoT. K. (2019). Machine learning made easy: A review of Scikit-learn package in python programming language. Journal of Education and Behavioral Statistics, 44(3), 348–361. https://doi.org/10.3102/1076998619832248
7.
HeK.ZhangX.RenS.SunJ. (2016). Deep residual learning for image recognition. In IEEE Conference on computer vision and pattern recognition (CVPR) (pp. 770–778). IEEE Press.
8.
HouL.ChenC.WangS.WuY.ChenX. (2022). Multi-object detection method in construction machinery swarm operations based on the improved YOLOv4 model. Sensors, 22(19), 7294. https://doi.org/10.3390/s22197294
9.
HuangZ.YangS.ZhouM., et al. (2022). Making accurate object detection at the edge: Review and new approach. Artificial Intelligence Review, 55(3), 2245–2274. https://doi.org/10.1007/s10462-021-10059-3
10.
HuntV. D.PugliaA.PugliaM. (2007). RFID: A guide to radio frequency identification, 1st edn (pp. 25–32). John Wiley & Sons.
11.
KapusiT. P.ErdeiT. I.HusiG.HajduA. (2022). Application of deep learning in the deployment of an industrial SCARA machine for real-time object detection. Robotics, 11(4), 69. https://doi.org/10.3390/robotics11040069
12.
KhanF. S.AnwerR. M.van de WeijerJ.BagdanovA. D.VanrellM.LopezA. M. (2012). Color attributes for object detection. In IEEE Conference on computer vision and pattern recognition (CVPR) (pp. 3306–3313). IEEE Press.
13.
KimK.KimJ.KangS.KimJ.LeeJ. (2012). Object recognition for cell manufacturing system. In 9th International conference on ubiquitous robots and ambient intelligence (URAI) (pp. 512–514). IEEE Press.
14.
KoudelkováJ.PospíšilR. (2017). Introduction identification technology in manufacturing companies and their impact on business economics. Global Economic Observer, 5(1), 193–200.
15.
LiM.PiD.QinS. (2023). An efficient single shot detector with weight-based feature fusion for small object detection. Scientific Reports, 13(1), 9883. https://doi.org/10.1038/s41598-023-36972-x
16.
LinT.-Y.DollárP.GirshickR.HeK.HariharanB.BelongieS. (2017). Feature pyramid networks for object detection. In IEEE Conference on computer vision and pattern recognition (CVPR) (pp. 936–944). IEEE Press.
17.
LiuJ.LouL.HuangD.ZhengY.XiaW. (2018). Lane detection based on straight line model and K-means clustering. In In IEEE 7th data driven control and learning systems conference (DDCLS) (pp. 527–532). IEEE Press.
18.
LiuW.AnguelovD.ErhanD., et al. (2016). SSD: Single shot multibox detector. In European conference on computer vision (ECCV) (pp. 21–37). Springer.
19.
MajinJ. J.ValenciaY. M.StivanelloM. E.StemmerM. R.SalazarJ. D. (2021). A novel deep learning based method for detection and counting of vehicles in urban traffic surveillance systems. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 43, 793–800. https://doi.org/10.5194/isprs-archives-XLIII-B2-2021-793-2021
20.
MoonJ.LimS.LeeH.YuS.LeeK.-B. (2022). Smart count system based on object detection using deep learning. Remote Sensing, 14(15), 3761. https://doi.org/10.3390/rs14153761
21.
MuslimA.BukhoriI.SuhartomoA. (2020). A design of autonomous remote control car using convolutional neural network and haar-like features classifier. In IEEE International conference on sustainable engineering and creative computing (ICSECC) (pp. 119–124). IEEE Press.
PangY.WangT.AnwerR. M.KhanF. S.ShaoL. (2019). Efficient featurized image pyramid network for single shot detector. In IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 7328–7336). IEEE Press.
24.
PardameanB.AbidF.CenggoroT. W.ElwirehardjaG. N.MuljoH. H. (2022). Counting people inside a region-of-interest in CCTV footage with deep learning. PeerJ Computer Science, 8, 1067. https://doi.org/10.7717/peerj-cs.1067
25.
QiangB.ChenR.ZhouM.PangY.ZhaiY.YanM. (2020). Convolutional neural networks-based object detection algorithm by jointing semantic segmentation for images. Sensors, 20(18), 5080–5094. https://doi.org/10.3390/s20185080
26.
Ricolfe-VialaC.CorrecherA.BlanesC. (2023). Detection of bad stapled nails in wooden packages. Applied Science (Basel), 13(9), 5644. https://doi.org/10.3390/app13095644
27.
RochaL. F.FerreiraM.SantosV.MoreiraA. P. (2014). Object recognition and pose estimation for industrial applications: A cascade system. Robotics and Computer-Integrated Manufacturing, 30(6), 605–621. https://doi.org/10.1016/j.rcim.2014.04.005
28.
SandlerM.HowardA.ZhuM.ZhmoginovA.ChenL.-C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on computer vision and pattern recognition (CVPR) (pp. 4510–4520). IEEE Press.
29.
SimK.LeeG. (2021). Neural-network-based object recognition and RANSAC-based three-dimensional posture estimation for solving the pickup problem of occluded parts. Journal of the Korean Society of Manufacturing Technology Engineers, 30(5), 372–381. https://doi.org/10.7735/ksmte.2021.30.5.372
30.
SitompulE. (2019). System modelling using neural networks with external recurrence and exponential quadratic cost function. In International conference on sustainable engineering and creative computing (ICSECC) (pp. 149–154). IEEE Press.
31.
SitompulE.BunawanA. (2011). Control of traffic light using fuzzy logic and harmony search algorithm. In 3rd International conference on machine learning and computing (ICMLC) (pp. 112–116). IEEE Press.
32.
SitompulE.SetiawanV. L.TariganH. J.GalinaM. (2024). Image classification of fabric defects using ResNet50 deep transfer learning in FastAI. Bulletin of Electrical Engineering and Informatics, 13(5), 3255–3267. https://doi.org/10.11591/eei.v13i5.8218
33.
SteffensC. R.HuttnerV.MessiasL. R. V.DrewsP. L. J.BotelhoS. S. C.GuerraR. S. (2019). CNN-based luminance and color correction for ILL-exposed images. In IEEE International conference on image processing (ICIP) (pp. 3252–3256). IEEE Press.
34.
TsaiC.-M.ShihF. Y.HsiehJ.-W. (2022). Real-time vehicle counting by deep-learning networks. In International conference on machine learning and cybernetics (ICMLC) (pp. 175–181). IEEE Press.
35.
WangJ.ZhangX.YangX. L.ZhaoZ. (2023). Study on vehicle color detection and recognition based on deep learning. In Optoelectronic imaging and multimedia technology IX (pp. 123170S). SPIE Digital Library.
36.
WeijerJ. V. D.SchmidC. (2006). Coloring local feature extraction. In European conference on computer vision (ECCV) (pp. 334–348). Springer.
37.
WuS.WangX.GuoC. (2023). Application of feature pyramid network and feature fusion single shot multibox detector for real-time prostate capsule detection. Electronics (Basel), 12(4), 1060.
38.
YanX.MelkoteS.MishraA. K.RajagopalanS. (2023). A digital apprentice for chatter detection in machining via human–machineuh-bree-vee-ay-shun interaction. Journal of Intelligent Manufacturing, 34(1), 3039–3052. https://doi.org/10.1007/s10845-022-01992-3
39.
YatskivN.YatskivS.VasylykA. (2020). Method of robotic process automation in software testing using artificial intelligence. In 10th International conference on advanced computer information technologies (ACIT) (pp. 501–504). IEEE Press.
40.
YehC.-H.LinC.-H.KangL.-W.HuangC.-H.LinM.-H.ChangC.-Y.WangC.-C. (2022). Lightweight deep neural network for joint learning of underwater object detection and color conversion. IEEE Transactions on Neural Networks and Learning Systems, 33(11), 6129–6143. https://doi.org/10.1109/TNNLS.2021.3072414
41.
YunH.KimE.KimD. M.ParkH. W.JunM. B.-G. (2023). Machine learning for object recognition in manufacturing applications. International Journal of Precision Engineering and Manufacturing, 24(4), 683–712. https://doi.org/10.1007/s12541-022-00764-6
42.
ZhangR. F.UrbanowiczR. J. (2020). A scikit-learn compatible learning classifier system. In Genetic and evolutionary computation conference companion (pp. 1816–1823). ACM Digital Library.
43.
ZhaoQ., et al. (2019). M2Det: A single-shot object detector based on multi-level feature pyramid network. In 33rd AAAI international conference on artificial intelligence (pp. 9259–9266). AAAI, USA.
