Research Progress on Vision System and Manipulator of Fruit Picking Robot

doi:10.3778/j.issn.1002-8331.2209-0183

Abstract

Abstract: Fruit picking robot is of great significance to the realization of automatic intelligence of fruit equipment. In this paper, the research work on the key technologies of fruit-picking robot at home and abroad in recent years is summarized, firstly, the key technologies of fruit-picking robot vision system, such as traditional image segmentation methods based on fruit features, such as threshold method, edge detection method, clustering algorithm based on color features and region-based image segmentation algorithm, are discussed, the object recognition algorithm based on depth learning and the target fruit location are analyzed and compared, and the state-of-the-art of fruit picking robot manipulator and end-effector is summarized, finally, the development trend and direction of fruit-picking robot in the future are prospected, which can provide reference for the related research of fruit-picking robot.

Key words: picking robot, fruit harvest, machine vision, neural network, manipulator

摘要： 水果采摘机器人对实现水果装备自动化智能化具有重要意义。对近年国内外水果采摘机器人领域关键技术研究工作进行总结，根据水果采摘机器人的重要组成结构与关键技术，对水果采摘机器人视觉系统的关键技术：传统基于水果特征的图像分割方法如阈值法、边缘检测法、基于颜色特征的聚类算法与基于区域的图像分割算法，基于深度学习的目标识别算法以及目标果实的定位等进行分析和对比；总结了水果采摘机器人机械臂与末端执行器的技术发展现状，指出水果采摘机器人存在的问题；对未来水果采摘机器人的发展趋势及方向进行了展望，可为水果采摘机器人相关研究提供参考。

关键词: 采摘机器人, 水果收获, 机器视觉, 神经网络, 机械手

GOU Yuanmin, YAN Jianwei, ZHANG Fugui, SUN Chengyu, XU Yong. Research Progress on Vision System and Manipulator of Fruit Picking Robot[J]. Computer Engineering and Applications, 2023, 59(9): 13-26.

苟园旻, 闫建伟, 张富贵, 孙成宇, 徐勇. 水果采摘机器人视觉系统与机械手研究进展[J]. 计算机工程与应用, 2023, 59(9): 13-26.

References

[1] 卢军，王贤锋，后德家.水果采摘机器人视觉系统研究进展[J].湖北农业科学，2012，51（21）：4705-4708.
LU J，WANG X F，HOU D J.Development of machine vision system for fruit harvesting robots[J].Hubei Agricultural Sciences，2012，51（21）：4705-4708.
[2] 李涛，邱权，赵春江，等.矮化密植果园多臂采摘机器人任务规划[J].农业工程学报，2021，37（2）：1-10.
LI T，QIU Q，ZHAO C J，et al.Task planning of multi-arm harvesting robots for high-density dwarf orchards[J].Transactions of the Chinese Society of Agricultural Engineering，2021，37（2）：1-10.
[3] SILWAL A，DAVIDSON J R，KARKEE M，et al.Design，integration，and field evaluation of a robotic apple harvester[J].Journal of Field Robotics，2017，34（6）：1140-1159.
[4] PRETER A D，ANTHONIS J，BAERDEMAEKER J D.Development of a robot for harvesting strawberries[J].IFAC-Papers OnLine，2018，51（17）：14-19.
[5] WILLIAMS H A M，JONES M H，NEJATI M，et al.Robotic kiwifruit harvesting using machine vision，convolutional neural networks，and robotic arms[J].Biosystems Engineering，2019，181：140-156.
[6] XIONG Y，PENG C，GRIMSTAD L，et al.Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper[J].Computers and Electronics in Agriculture，2019，157：392-402.
[7] VROCHIDOU E，TZIRIDIS K，NIKOLAOU A，et al.An autonomous grape-harvester robot：integrated system architecture[J].Electronics，2021，10（9）.
[8] YU Y，ZHANG K，LIU H，et al.Real-time visual localization of the picking points for a ridge-planting strawberry harvesting robot[J].IEEE Access，2020，8：116556-116568.
[9] 赵德安，吴任迪，刘晓洋，等.气电混合驱动全天候苹果收获机器人设计与试验[J].农业机械学报，2020，51（2）：21-28.
ZHAO D A，WU R D，LIU X Y，et al.Design and experiment of apple harvesting robot based on gas-electric hybrid drive[J].Transactions of the Chinese Society for Agricultural Machinery，2020，51（2）：21-28.
[10] BU L X，CHEN C K，HU G R，et al.Design and evaluation of a robotic apple harvester using optimized picking patterns[J].Computers and Electronics in Agriculture，2022，198：107092.
[11] YU X J，FAN Z M，WANG X D，et al.A lab-customized autonomous humanoid apple harvesting robot[J].Computers & Electrical Engineering，2021，96.
[12] 程祥云，宋欣.果蔬采摘机器人视觉系统研究综述[J].浙江农业科学，2019，60（3）：490-493.
CHENG X Y，SONG X.Review on visual system for picking robots of fruits and vegetables[J].Journal of Zhejiang Agricultural Sciences，2019，60（3）：490-493.
[13] 李大华，赵相飞，许亮，等.基于纹理特征与HSI空间的苹果识别与标定[J].图学学报，2016，37（5）：688-693.
LI D H，ZHAO X F，XU L，et al.Apple identification and calibration based on the texture characteristics and HSI space[J].Journal of Graphics，2016，37（5）：688-693.
[14] LIN G H，TANG Y H，ZOU X J，et al.Color-，depth-，and shape-based 3D fruit detection[J].Precision Agriculture，2020，21（1）：1-17.
[15] MIZUSHIMA A，LU R F.An image segmentation method for apple sorting and grading using support vector machine and Otsu’s method[J].Computers and Electronics in Agriculture，2013，94：29-37.
[16] 郭艾侠，邹湘军，邹海鑫.基于二次阈值的荔枝果与结果母枝图像识别[J].计算机工程与设计，2014，35（2）：557-561.
GUO A X，ZOU X J，ZOU H X.Recognition on image of litchi fruits and their main fruit bearing based on twin-threshold method[J].Computer Engineering and Design，2014，35（2）：557-561.
[17] LIN G C，TANG Y C，ZOU X J，et al.Fruit detection in natural environment using partial shape matching and probabilistic Hough transform[J].Precision Agriculture，2020，21（1）：160-177.
[18] 宋荣杰，宁纪锋，刘秀英，等.基于纹理特征和SVM的QuickBird影像苹果园提取[J].农业机械学报，2017，48（3）：188-197.
SONG R J，NING J F，LIU X Y，et al.Apple orchard extraction with quickbird imagery based on texture features and support vector machine[J].Transactions of the Chinese Society for Agricultural Machinery，2017，48（3）：188-197.
[19] YU L，XIONG J，FANG X，et al.A litchi fruit recognition method in a natural environment using RGB-D images[J].Biosystems Engineering，2021，204：50-63.
[20] 黄小玉，李光林，马驰，等.基于改进判别区域特征融合算法的近色背景绿色桃子识别[J].农业工程学报，2018，34（23）：142-148.
HUANG X Y，LI G L，MA C，et al.Green peach recognition based on improved discriminative regional feature integration algorithm in similar background[J].Transactions of the Chinese Society of Agricultural Engineering，2018，34（23）：142-148.
[21] GAN H，LEE W S，ALCHANATIS V，et al.Immature green citrus fruit detection using color and thermal images[J].Computers and Electronics in Agriculture，2018：152.
[22] 徐惠荣，叶尊忠，应义斌.基于彩色信息的树上柑橘识别研究[J].农业工程学报，2005，21（5）：98-101.
XU H R，YE Z Z，YING Y B.Identification of citrus fruit in a tree canopy using color information[J].Transactions of the Chinese Society of Agricultural Engineering，2005，21（5）：98-101.
[23] 谢国俊，曹其新，刘建政，等.基于多方位视觉的果实形状特征的提取研究[J].农业工程学报，2007，23（7）：127-132.
XIE G J，CAO Q X，LIU J Z，et al.Method for fruit shape feature acquisit ion based on multidirectional vision[J].Transactions of the CSAE，2007，23（7）：127-132.
[24] 李国利，姬长英，顾宝兴.基于单目视觉与超声检测的振荡果实采摘识别与定位[J].农业机械学报，2015，46（11）：1-8.
LI G L，JI C Y，GU B X.Recognition and location of oscillating fruit based on monocular vision and ultrasonic testing[J].Transactions of the Chinese Society for Agricultural Machinery，2015，46（11）：1-8.
[25] MALDONADO W，BARBOSA J C.Automatic green fruit counting in orange trees using digital images[J].Computers and Electronics in Agriculture，2016，127.
[26] LI Y H，WANG H J，ZHOU W L，et al.Monocular vision and calculation of regular three-dimensional target pose based on Otsu and Haar-feature AdaBoost classifier[J].International Journal of Agricultural and Biological Engineering，2020，13（5）：171-180.
[27] WU G，LI B，ZHU Q，et al.Using color and 3D geometry features to segment fruit point cloud and improve fruit recognition accuracy[J].Computers and Electronics in Agriculture，2020，174.
[28] 初广丽，张伟，王延杰，等.基于机器视觉的水果采摘机器人目标识别方法[J].中国农机化学报，2018，39（2）：83-88.
CHU G L，ZHANG W，WANG Y J，et al.A method of fruit picking robot target identification based on machine vision[J].Journal of Chinese Agricultural Mechanization，2018，39（2）：83-88.
[29] TAN K，LEE W S，GAN H，et al.Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes[J].Biosystems Engineering，2018，176：59-72.
[30] 徐黎明，吕继东.基于同态滤波和K均值聚类算法的杨梅图像分割[J].农业工程学报，2015，31（14）：202-208.
XU L M，LV J D.Bayberry image segmentation based on homomorphic filtering and K-means clustering algorithm[J].Transactions of the Chinese Society of Agricultural Engineering，2015，31（14）：202-208.
[31] 熊俊涛，林睿，刘振，等.夜间自然环境下荔枝采摘机器人识别技术[J].农业机械学报，2017，48（11）：28-34.
XUN J T，LIN R，LIU Z，et al.Visual technology of picking robot to detect litchi at nighttime under natural environment[J].Transactions of the Chinese Society for Agricultural Machinery，2017，48（11）：28-34.
[32] JI W，ZHAO D，CHENG F Y，et al.Automatic recognition vision system guided for apple harvesting robot[J].Computers & Electrical Engineering，2012，38（5）：1186-1195.
[33] LINKER R，COHEN O，NAOR A.Determination of the number of green apples in RGB images recorded in orchards[J].Computers and Electronics in Agriculture，2011，81.
[34] 赵德安，刘晓洋，陈玉，等.苹果采摘机器人夜间识别方法[J].农业机械学报，2015，46（3）：15-22.
ZHAO D A，LIU X Y，CHEN Y，et al.Image recognition at night for apple picking robot[J].Transactions of the Chinese Society for Agricultural Machinery，2015，46（3）：15-22.
[35] 吕继东，赵德安，姬伟，等.采摘机器人振荡果实动态识别[J].农业机械学报，2012，43（5）：173-178.
LV J D，ZHAO D A，JI W，et al.Dynamic recognition of oscillating fruit for harvesting robot[J].Transactions of the Chinese Society for Agricultural Machinery，2012，43（5）：173-178.
[36] LI H，LEE W，WANG K.Identifying blueberry fruit of different growth stages using natural outdoor color images[J].Computers and Electronics in Agriculture，2014，106：91-101.
[37] 熊俊涛，邹湘军，彭红星，等.扰动柑橘采摘的实时识别与采摘点确定技术[J].农业机械学报，2014，45（8）：38-43.
XIONG J T，ZOU X J，PENG H X，et al.Real-time identification and picking point localization of disturbance citrus picking[J].Transactions of the Chinese Society for Agricultural Machinery，2014，45（8）：38-43.
[38] 罗陆锋，邹湘军，杨洲，等.基于改进人工蜂群模糊聚类的葡萄图像快速分割方法[J].农业机械学报，2015，46（3）：23-28.
LUO L F，ZOU X J，YANG Z，et al.Grape image fast segmentation based on improved artificial bee colony and fuzzy clustering[J].Transactions of the Chinese Society for Agricultural Machinery，2015，46（3）：23-28.
[39] 闫建伟，赵源，张乐伟，等.改进Faster-RCNN自然环境下识别刺梨果实[J].农业工程学报，2019，35（18）：144-151.
YAN J W，ZHAO Y，ZHANG L W，et al.Recognition of rosa roxbunghii in natural environment based on improved Faster-RCNN[J].Transactions of the Chinese Society of Agricultural Engineering，2019，35（18）：144-151.
[40] FU L，MAJEED Y，ZHANG X，et al.Faster R-CNN-based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting[J].Biosystems Engineering，2020，197：245-256.
[41] TIAN Y，YANG G，WANG Z，et al.Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J].Computers and Electronics in Agriculture，2019，157：417-426.
[42] 吕石磊，卢思华，李震，等.基于改进YOLOv3-LITE轻量级神经网络的柑橘识别方法[J].农业工程学报，2019，35（17）：205-214.
LV S L，LU S H，LI Z，et al.Orange recognition method using improved YOLOv3-LITE lightweight neural network[J].Transactions of the Chinese Society of Agricultural Engineering，2019，35（17）：205-214.
[43] 彭红星，黄博，邵园园，等.自然环境下多类水果采摘目标识别的通用改进SSD模型[J].农业工程学报，2018，34（16）：155-162.
PENG H X，HUANG B，SHAO Y Y，et al.General improved SSD model for picking object recognition of multiple fruits in natural environment[J].Transactions of the Chinese Society of Agricultural Engineering，2018，34（16）：155-162.
[44] 肖德琴，蔡家豪，林思聪，等.基于IFSSD卷积神经网络的柚子采摘目标检测模型[J].农业机械学报，2020，51（5）：28-35.
XIAO D Q，CAI J H，LIN S C，et al.Grapefruit detection model based on IFSSD convolution network[J].Transactions of the Chinese Society of Agricultural Engineering，2020，51（5）：28-35.
[45] 罗会兰，张云.基于深度网络的图像语义分割综述[J].电子学报，2019，47（10）：2211-2220.
LUO H L，ZHANG Y.A survey of image semantic segmentation based on deep network[J].Acta Electronica Sinica，2019，47（10）：2211-2220.
[46] MAJEED Y，ZHANG J，ZHANG X，et al.Deep learning based segmentation for automated training of apple trees on trellis wires[J].Computers and Electronics in Agriculture，2020，170.
[47] 薛君蕊，王昱潭，曲爱丽，等.基于改进FCN-8s的灵武长枣图像分割方法[J].农业工程学报，2021，37（5）：191-197.
XUE J R，WANG Y T，QU A L，et al.Image segmentation method for Lingwu long jujubes based on improved FCN-8s[J].Transactions of the Chinese Society of Agricultural Engineering，2021，37（5）：191-197.
[48] 梁新宇，林洗坤，权冀川，等.基于深度学习的图像实例分割技术研究进展[J].电子学报，2020，48（12）：2476-2486.
LIANG X Y，LIN X K，QUAN J C，et al.Research on the progress of image instance segmentation based on deep learning[J].Acta Electronica Sinica，2020，48（12）：2476-2486.
[49] PEREZ-BORRERO I，MARIN-SANTOS D，GEGUNDEZ-ARIAS M E，et al.A fast and accurate deep learning method for strawberry instance segmentation[J].Computers and Electronics in Agriculture，2020，178.
[50] ZHENG C，CHEN P，PANG J，et al.A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard[J].Biosystems Engineering，2021，206：32-54.
[51] GAO F，FU L，ZHANG X，et al.Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN[J].Computers and Electronics in Agriculture，2020，176.
[52] LIANG C，XIONG J，ZHENG Z，et al.A visual detection method for nighttime litchi fruits and fruiting stems[J].Computers and Electronics in Agriculture，2020，169.
[53] 王金鹏，高凯，姜洪喆，等.基于改进的轻量化卷积神经网络火龙果检测方法[J].农业工程学报，2020，36（20）：218-225.
WANG J P，GAO K，JIANG H Z，et al.Method for detecting dragon fruit based on improved lightweight convolutional neural network[J].Transactions of the Chinese Society of Agricultural Engineering，2020，36（20）：218-225.
[54] 黄豪杰，段先华，黄欣辰.基于深度学习水果检测的研究与改进[J].计算机工程与应用，2020，56（3）：127-133.
HUANG H J，DUAN X H，HUANG X C.Research and improvement of fruits detection based on deep learning[J].Computer Engineering and Applications，2020，56（3）：127-133.
[55] LIN G C，TANG Y C，WANG X J，et al.Three-dimensional reconstruction of guava fruits and branches using instance segmentation and geometry analysis[J].Computers and Electronics in Agriculture，2021，184.
[56] YU Y，ZHANG K，YANG L，et al.Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN[J].Computers and Electronics in Agriculture，2019，163.
[57] MEHTA S S，BURKS T F.Vision-based control of robotic manipulator for citrus harvesting[J].Computers and Electronics in Agriculture，2014，102：146-158.
[58] CHEN C，LI B，LIU J X，et al.Monocular positioning of sweet peppers：an instance segmentation approach for harvest robots[J].Biosystems Engineering，2020，196：15-28.
[59] 刘洁，周典卓，李燕，等.基于目标像素变化的柚果单目测距算法[J].农业工程学报，2021，37（19）：183-191.
LIU J，ZHOU D Z，LI Y，et al.Monocular distance measurement algorithm for pomelo fruit based on target pixels change[J].Transactions of the Chinese Society of Agricultural Engineering，2021，37（19）：183-191.
[60] 司永胜，乔军，刘刚，等.苹果采摘机器人果实识别与定位方法[J].农业机械学报，2010，41（9）：148-153.
SI Y S，QIAO J，LIU G，et al.Recognition and location of fruits for apple harvesting robot[J].Transactions of the Chinese Society for Agricultural Machinery，2010，41（9）：148-153.
[61] XIONG J，HE Z，LIN R，et al.Visual positioning technology of picking robots for dynamic litchi clusters with disturbance[J].Computers and Electronics in Agriculture，2018，151：226-237.
[62] MEHTA S S，TON C，ASUNDI S，et al.Multiple camera fruit localization using a particle filter[J].Computers and Electronics in Agriculture，2017，142：139-154.
[63] CHEN M，TANG Y，ZOU X，et al.Three-dimensional perception of orchard banana central stock enhanced by adaptive multi-vision technology[J].Computers and Electronics in Agriculture，2020，174（C）.
[64] 刘兆祥，刘刚，乔军.苹果采摘机器人三维视觉传感器设计[J].农业机械学报，2010，41（2）：171-175.
LIU Z X，LIU G，QIAO J.Development of a 3-dimension vision sensor in apple harvesting robot[J].Transactions of the Chinese Society for Agricultural Machinery，2010，41（2）：171-175.
[65] GENE-MOLA J，GREGORIO E，GUEVARA J，et al.Fruit detection in an apple orchard using a mobile terrestrial laser scanner[J].Biosystems Engineering，2019，187：171-184.
[66] GUO N，ZHANG B，ZHOU J，et al.Pose estimation and adaptable grasp configuration with point cloud registration and geometry understanding for fruit grasp planning[J].Computers and Electronics in Agriculture，2020，179.
[67] NGUYEN T T，VANDEVOORDE K，WOUTERS N，et al.Detection of red and bicoloured apples on tree with an RGB-D camera[J].Biosystems Engineering，2016，146：33-44.
[68] 杨长辉，刘艳平，王毅，等.自然环境下柑橘采摘机器人识别定位系统研究[J].农业机械学报，2019，50（12）：14-22.
YANG C H，LIU Y P，WANG Y，et al.Research and experiment on recognition and location system for citrus picking robot in natural environment[J].Transactions of the Chinese Society for Agricultural Machinery，2019，50（12）：14-22.
[69] ZHAO Y，GONG L，HUANG Y，et al.A review of key techniques of vision-based control for harvesting robot[J].Computers and Electronics in Agriculture，2016，127：311-323.
[70] 李国利，姬长英，顾宝兴，等.多末端苹果采摘机器人机械手运动学分析与试验[J].农业机械学报，2016，47（12）：14-21.
LI G L，JI C Y，GU B X，et al.Kinematics analysis and experiment of apple harvesting robot manipulator with multiple end-effectors[J].Transactions of the Chinese Society for Agricultural Machinery，2016，47（12）：14-21.
[71] 张凯良，杨丽，王粮局，等.高架草莓采摘机器人设计与试验[J].农业机械学报，2012，43（9）：165-172.
ZHANG K L，YANG L，WANG L J，et al.Design and experiment of elevated substrate culture strawberry picking robot[J].Transactions of the Chinese Society for Agricultural Machinery，2012，43（9）：165-172.
[72] 李伟，李吉，张俊雄，等.苹果采摘机器人机械臂优化设计及仿真[J].北京工业大学学报，2009，35（6）：721-726.
LI W，LI J，ZHANG J X，et al.Optimization design and simulations of the apple-picking-robot arm[J].Journal of Beijing University of Technology，2009，35（6）：721-726.
[73] 林用满，管卫华，甘莉莉.六自由度水果采摘机械臂结构设计与试验[J].中国农机化学报，2019，40（2）：62-71.
LIN Y M，GUAN W H，GAN L L.Structure design and test of six-DOF fruit-picking mechanical arm[J].Journal of Chinese Agricultural Mechanization，2019，40（2）：62-71.
[74] 刘继展，李萍萍，李智国.番茄采摘机器人末端执行器的硬件设计[J].农业机械学报，2008，39（3）：109-112.
LIU J Z，LI P P，LI G Z.Hardware design of the end-effector for tomato-harvesting robot[J].Transactions of the Chinese Society for Agricultural Machinery，2008，39（3）：109-112.
[75] GRAHAM S S，ZONG W，FENG J，et al.Design and testing of a kiwifruit harvester end-effector[J].Transactions of the ASABE，2018，61（1）：45-51.
[76] 成亮，王立杰，张国庆.一种自适应多功能水果采摘装置[J].林业工程学报，2020，5（2）：130-136.
CHENG L，WANG L J，ZHANG G Q.SeIf?adaptive and multi?functional fruit picking device[J].Journal of Forestry Engineering，2020，5（2）：130-136.
[77] 卢伟，宋爱国，蔡健荣，等.柑橘采摘机器人结构设计及运动学算法[J].东南大学学报（自然科学版），2011，41（1）：95-100.
LU W，SONG A G，CAI J R，et al.Structural design and kinematics algorithm research for orange harvesting robot[J].Journal of Southeast University（Natural Science Edition），2011，41（1）：95-100.
[78] WANG Y，YANG Y，YANG C，et al.End-effector with a bite mode for harvesting citrus fruit in random stalk orientation environment[J].Computers and Electronics in Agriculture，2019，157：454-470.
[79] 李道义，陈雷，尚小龙，等.菠萝采摘机械手结构设计[J].农业工程，2019，9（2）：1-5.
LI D Y，CHEN L，SHANG X L，et al.Structure design of pineapple picking manipulator[J].Agricultural Engineering，2019，9（2）：1-5.
[80] ZHANG K X，LAMMERS K，CHU P Y，et al.System design and control of an apple harvesting robot[J].Mechatronics，2021，79.
[81] 王海峰，李斌，刘广玉，等.菠萝采摘机械手的设计与试验[J].农业工程学报，2012，28（2）：42-46.
WANG H F，LI B，LIU G Y，et al.Design and test of pineapple harvesting manipulator[J].Transactions of the Chinese Society of Agricultural Engineering，2012，28（2）：42-46.
[82] 陈军，王虎，蒋浩然，等.猕猴桃采摘机器人末端执行器设计[J].农业机械学报，2012，43（10）：151-154.
CHEN J，WANG H，JIANG H R，et al.Design of end-effector for kiwifruit harvesting robot[J].Transactions of the Chinese Society for Agricultural Machinery，2012，43（10）：151-154.
[83] CUI Y，GEJIMA Y，KOBAYASHI T，et al.Study on cartesian-type strawberry-harvesting robot[J].Sensor Letters：A Journal Dedicated to all Aspects of Sensors in Science，Engineering，and Medicine，2013，11（6/7）：1223-1228.
[84] FENG Q C，ZOU W，FAN P F，et al.Design and test of robotic harvesting system for cherry tomato[J].International Journal of Agricultural and Biological Engineering，2018，11（1）：96-100.