In orchard 3D sensing for crop identification and localization in a virtual environment : a thesis presented in partial fulfilment of the requirements for the degree of Master of Engineering in School of Engineering at Massey University, Palmerston North, New Zealand

Abstract

Due to recent events of the global pandemic, as well as decreasing labour available in the agricultural sector in New Zealand, there has been an overall labour shortage in this multi-million dollar industry. With the advancement and accessibility of new technologies, this is an industry in which these can be applied effectively to increase efficiency and potentially be done remotely. This can be a key benefit for the industry, as it would help avoid labour shortages with the ability to undertake some orchard work remotely, as well as providing job opportunities to those who may not normally be able to undertake this sort of physical work. Another resulting benefit in having the ability to potentially work remotely means that the need for travel to and around the orchard is decreased, which would also have sustainability benefits. One such technology that has been looked at, with developments being made over the recent years is robotic harvesting. A key element in this process is the remote sensing and vision systems used to identify, and subsequently localise, the targeted crop to be harvested.

Currently, most of the research that has been done in this area has been using expensive, high precision 3D scanning tools. However, with the advancement in this area of technology, cheaper and more practical alternatives may be suitable for the application.

The primary focus of this project was to evaluate and quantify the accuracy and therefore suitability of multiple remote sensing systems to be used in an orchard environment. For this project, consumer grade RGBD(Red, Green, Blue, Depth)camera systems were used to identify and localise apples, with the resulting depth and image data being interacted with in a virtual reality environment in the form of point clouds as well as mesh bodies, in order to identify and localise the apples in 3D space in a virtual environment.

Multiple metrics were evaluated for these systems including depth accuracy, point density, external factor effects such as sunlight, positional accuracy in 3D space and error in the localization of objects. After initial investigation, it was found that it would be possible to use these devices for this application, however refinements would need to be made. The depth data for both forms of 3D sensing was very accurate and effective in an indoor environment, with high depth precision and point density, however there was more variation when data was collected outdoors, mainly due to interference from external factors such as sunlight. As expected, point density decreased with increasing distance, resulting in an optimal operating range of 1-2m being established. The resultant point clouds collected were also used to interact in both a 2D and 3D environment, using CloudCompare and virtual reality, respectively, in order to quantify the error in manual localisation of objects in 3D space. The error in the manual localisation of objects scanned in the test orchard setup, in this case apples, was also quantified by comparing results from a purely virtual environment using virtual 3D scanned apples to the point cloud data collected from the orchard setup. An interesting finding from this was that most of the variation in results came from the error in manual localisation, rather than inaccurate depth data.

Overall, this project was successful in analysing and quantifying the characteristics of multiple remote sensing methods, finding that the concept would work for the application based on the results collected, but would benefit with future development and refinement in order to be commercially viable. Another key area that would need subsequent research and development is the data transfer between the environment being scanned, and the environment in which this data is interacted with, as to be practical for real-world use, this would need to be efficient and automated. With further development, filters could be applied to the data to more efficiently target the crop, with a suitable algorithm being applied to automatically identify and localize the crop, which could lead to increased efficiency, quality and accuracy of the ideal application of robotic harvesting.