The first major contribution of the paper is the proposal of using an improved DEtection Transformer network (named R2N-DETR) and Kinect-V2 camera for detecting multiple-size peaches under orchards with varied illumination and fruit occlusion. R2N-DETR model first employed Res2Net-50 to extract a fused low-high level feature map containing fine spatial features and precise semantic information of multi-size peaches from Red-Green-Blue-Depth (RGB-D) images. Second, the encoder-decoder was performed on the feature map to obtain the global context. Finally, all detected objects were detected according to each object’s global context. For the detection of 1101 RGB-D images (imaged from two orchards over three years), the R2N-DETR model achieves an average precision of 0.944 and an average detecting time of 53 ms for each image. The developed system could provide precise visual guidance for robotic picking and contribute to improving yield prediction by providing accurate fruit counting.
LiY. and WangL., Genetic resources, breeding programs in China, and gene mining of peach: A review, Horticultural Plant Journal6 (2020), 205–215. doi: 10.1016/j.hpj.2020.06.001.
2.
SaediS.I. and KhosraviH., A deep neural network approach towards real-time on-branch fruit recognition for precision horticulture, Expert Systems with Applications159 (2020), 113594. doi: 10.1016/j.eswa.2020.113594.
3.
WangX. and YangW., Water quality monitoring and evaluation using remote sensing techniques in China: A systematic review, Ecosystem Health and Sustainability5 (2019), 47–56. doi: 10.1080/20964129.2019.1571443.
4.
KapachK.BarneaE.MaironR.EdanY. and Ben-ShaharO., Computer vision for fruit harvesting robots–state of the art and challenges ahead, International Journal of Computational Vision and Robotics3 (2012), 4–34. doi: 10.1504/IJCVR..
5.
WangT.ChenB.ZhangZ.LiH. and ZhangM., Applications of machine vision in agricultural robot navigation: A review, Computers and Electronics in Agriculture198 (2022), 107085. doi: 10.1016/j.compag.2022.107085.
6.
WangM.LiuW.ChenM.HuangX. and HanW., A band selection approach based on a modified gray wolf optimizer and weight updating of bands for hyperspectral image, Applied Soft Computing112 (2021), 107805. doi: 10.1016/j.asoc.2021.107805.
7.
FuL.MajeedY.ZhangX.KarkeeM. and ZhangQ., Faster R–CNN–based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting, Biosystems Engineering197 (2020), 245–256. doi: 10.1016/j.biosystemseng.2020.07.007.
8.
BarneaE.MaironR. and Ben-ShaharO., Colour-agnostic shape-based 3D fruit detection for crop harvesting robots, Biosystems Engineering146 (2016), 57–70. doi: 10.1016/j.biosystemseng.2016.01.013.
9.
ZouZ.ShiZ.GuoY. and YeJ., Object detection in 20 years: A survey, arXiv preprint arXiv:1905.05055, 2019. doi: 10.48550/arXiv.1905.05055.
10.
ZhaoZ.-Q.ZhengP.XuS.-T. and WuX., Object detection with deep learning: A review, IEEE Transactions on Neural Networks and Learning Systems30 (2019), 3212–3232. doi: 10.1109/TNNLS.2018.2876865.
11.
DiwanT.AnirudhG. and TembhurneJ.V., Object detection using YOLO: challenges, architectural successors, datasets and applications, Multimedia Tools and Applications (2022), 1–33. doi: 10.1007/s11042-022-13644-y.
12.
VoX.-T. and JoK.-H., A review on anchor assignment and sampling heuristics in deep learning-based object detection, Neurocomputing506 (2022), 96–116. doi: 10.1016/j.neucom.2022.07.003.
13.
TuS.XueY.ZhengC.QiY.WanH. and MaoL., Detection of passion fruits and maturity classification using Red-Green-Blue Depth images, Biosystems Engineering175 (2018), 156–167. doi: 10.1016/j.biosystemseng.2018.09.004.
14.
WuG.LiB.ZhuQ.HuangM. and GuoY., Using color and 3D geometry features to segment fruit point cloud and improve fruit recognition accuracy, Computers and Electronics in Agriculture174 (2020), 105475. doi: 10.1016/j.compag.2020.105475.
15.
SongZ.FuL.WuJ.LiuZ.LiR. and CuiY., Kiwifruit detection in field images using Faster R-CNN with VGG16, IFAC-PapersOnLine52 (2019), 76–81. doi: 10.1016/j.ifacol.2019.12.500.
16.
ChuP.LiZ.LammersK.LuR. and LiuX., Deep learning-based apple detection using a suppression mask R-CNN, Pattern Recognition Letters147 (2021), 206–211. doi: 10.1016/j.patrec.2021.04.022.
17.
TianY.YangG.WangZ.WangH.LiE. and LiangZ., Apple detection during different growth stages in orchards using the improved YOLO-V3 model, Computers and Electronics in Agriculture157 (2019), 417–426. doi: 10.1016/j.compag.2019.01.012.
18.
MirhajiH.SoleymaniM.AsakerehA. and MehdizadehS.A., Fruit detection and load estimation of an orange orchard using the YOLO models through simple approaches in different imaging and illumination conditions, Computers and Electronics in Agriculture191 (2021), 106533. doi: 10.1016/j.compag.2021.106533.
19.
ZhengM.GaoP.ZhangR.LiK.WangX.LiH. and DongH., End-to-end object detection with adaptive clustering transformer, arXiv preprint arXiv:2011.09315, 2020. doi: 10.48550/arXiv.2011.09315.
20.
HeoB.YunS.HanD.ChunS.ChoeJ. and OhS.J., Rethinking spatial dimensions of vision transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, 11936–11945.
21.
CarionN.MassaF.SynnaeveG.UsunierN.KirillovA. and ZagoruykoS., End-to-end object detection with transformers, in: European Conference on Computer Vision, Springer, 2020, 213–229.
22.
LinT.-Y.MaireM.BelongieS.HaysJ.PeronaP.RamananD.DollárP. and ZitnickC.L., Microsoft coco: Common objects in context, in: European Conference on Computer Vision, Springer, 2014, 740–755.
23.
ZhuX.SuW.LuL.LiB.WangX. and DaiJ., Deformable detr: Deformable transformers for end-to-end object detection, arXiv preprint arXiv:2010.04159, 2020. doi: 10.48550/arXiv.2010.04159.
24.
FuL.FengY.WuJ.LiuZ.GaoF.MajeedY.Al-MallahiA.ZhangQ.LiR. and CuiY., Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model, Precision Agriculture22 (2021), 754–776. doi: 10.1007/.
25.
GaoS.-H.ChengM.-M.ZhaoK.ZhangX.-Y.YangM.-H. and TorrP., Res2net: A new multi-scale backbone architecture, IEEE Transactions on Pattern Analysis and Machine Intelligence43 (2019), 652–662. doi: 10.1109/TPAMI.2019.2938758.
26.
VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA.N.KaiserŁ. and PolosukhinI., Attention is all you need, Advances in Neural Information Processing Systems30 (2017), 5998–6008.
27.
BelloI.ZophB.VaswaniA.ShlensJ. and LeQ.V., Attention augmented convolutional networks, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3286–3295.
RoyA.M.BoseR. and BhaduriJ., A fast accurate fine-grain object detection model based on YOLOv4 deep neural network, Neural Computing and Applications34 (2022), 3895–3921. doi: 10.1007/s00521-021-06651-x.
30.
QiuH.MaY.LiZ.LiuS. and SunJ., Borderdet: Border feature for dense object detection, in: European Conference on Computer Vision, Springer, 2020, pp. 549–564.
31.
SongG.LiuY. and WangX., Revisiting the sibling head in object detector, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11563–11572.
32.
YangJ.LiC.ZhangP.DaiX.XiaoB.YuanL. and GaoJ., Focal attention for long-range interactions in vision transformers, Advances in Neural Information Processing Systems34 (2021), 30008–30022.
33.
ZhangW.LiH.LiY.LiuH.ChenY. and DingX., Application of deep learning algorithms in geotechnical engineering: a short critical review, Artificial Intelligence Review54 (2021), 5633–5673. doi: 10.1007/s10462-021-09967-1.
34.
ZhengC.ChenP.PangJ.YangX.ChenC.TuS. and XueY., A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard, Biosystems Engineering206 (2021), 32–54. doi: 10.1016/j.biosystemseng.2021.03.012.
35.
BochkovskiyA.WangC.-Y. and LiaoH.-Y.M., Yolov4: Optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.10934, 2020. doi: 10.48550/arXiv.2004.10934.
36.
MaiX.ZhangH.JiaX. and MengM.Q.-H., Faster R-CNN with classifier fusion for automatic detection of small fruits, IEEE Transactions on Automation Science and Engineering17 (2020), 1555–1569. doi: 10.1109/TASE.2020.2964289.
37.
WanS. and GoudosS., Faster R-CNN for multi-class fruit detection using a robotic vision system, Computer Networks168 (2020), 107036. doi: 10.1016/j.comnet.2019.107036.
38.
GongC.WangD.LiM.ChandraV. and LiuQ., Keepaugment: A simple information-preserving data augmentation approach, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1055–1064.
39.
YangC.WangY.ZhangJ.ZhangH.WeiZ.LinZ. and YuilleA., Lite vision transformer with enhanced self-attention, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11998–12008.
40.
DaiX.ChenY.YangJ.ZhangP.YuanL. and ZhangL., Dynamic detr: End-to-end object detection with dynamic attention, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2988–2997.
