กลยุทธ์ในการควบคุมหุ่นยนต์เก็บเกี่ยวผลไม้ด้วยการรับรู้ภาพ

Abstract

Recently, there has been development of several agriculture robots to solve the problem of declined farm worker population. Agriculture robots offers some advantages over human worker in terms of precision and attentiveness. To design these systems, one must consider main components to suit certain environment that the robots need to operate. Such components include steering system, manipulator system, end effector and sensory systems. Due to various types of the environment, robotic vision can be considered as one of the most crucial elements. The purpose of this research is to develop vision-based strategies for controlling and planning the motion of a fruit-harvesting robot. The Mask-RCNN, a deep learning convolutional neural network, is used to generate an object detection model that returns the object label, bounding box, and masked area. From experimental tests, the developed model can detect mangos with 97% accuracy in the case full view of mangos is available and 78% accuracy when the mangos’ view is partially occluded. The object detection model together with point-cloud data from stereo camera are used to determine the three-dimensional position and size of the mangos. In this study, two models based on: 1) pinhole method and 2) ellipsoid fitting method have been developed for such task. Statistical model has also been developed to predict whether and which side the mango may be obstructed by other objects, in order to plan the trajectory of the robot arm. The performances of the models were tested in a laboratory. A six degree of freedom UR5 robot manipulator were used in the experiment. In the case that the mangos can be fully seen from the camera image, the pinhole modeling method can determine the 3-D position with the average errors (x,y,z) of 2.4, 1.2, -4.5 mm, and the size with the average errors (width and height) of 1.9, 4.1 mm. For the model based on the ellipsoid fitting method, the average errors are: 4.3, 6.8, -6.9 mm for the position and -9.4, 2.5 mm for the size. For the occluded case, the pinhole model method was used to predict the position and the size. Based on the bounding box data, the average errors are 1.0, 2.5, -10.6 mm for the position and -0.9, -1.8 for the size prediction. Based on the object mask data, the average errors are 1.0, 2.5, -9.5 mm for the position and 1.2, 2.4 mm for the size prediction. As expected, predictions for the position and size of the mango for the full-view cases have higher accuracy than those for the occluded cases. The masked data were used to create a statistical model to predict the occluded side of the mango. In the experiments, the model could predict with 100 percent accuracy in the case of no occlusion. When the view of the mangos is obstructed by objects such as leaves, the model could predict the occluded side with 97 percent accuracy. The results of this work show that it is feasible to employ stereo cameras which are relatively inexpensive compared to other 3D type sensors, such as 3D LiDAR to predict the size and position of targeted object for use in agricultural robotic applications.