Synthetic hyperspectral reflectance data augmentation by generative adversarial network to enhance grape maturity determination

Abstract

Non-destructive and rapid grape maturity detection is important for the wine industry. The ongoing development of hyperspectral imaging techniques and deep learning methods has greatly helped in non-destructive assessing of grape quality and maturity, but the performance of deep learning methods depends on the volume and the quality of labeled data for training. Building non-destructive grape quality or maturity testing datasets requires damaging grapes for chemical analysis to produce labels which are time consuming and resource intensive. To solve this problem, this study proposed a conditional Wasserstain Generative Adversarial Network (WGAN) with the gradient penalty data augmentation technique to generate synthetic hyperspectral reflectance data of two grape maturity categories (ripe and unripe) and different Total Soluble Solids (TSS) values. The conditional WGAN with the gradient penalty was trained for a range of epochs: 500, 1000, 2000, 8000, 10,000, and 20,000. After training of 10,000 epochs, synthetic hyperspectral reflectance data were very similar to real spectra for each maturity category and different TSS values. Thereafter, contextual deep three-dimensional CNN (3D-CNN), Spatial Residual Network (SSRN) and Support Vector Machine (SVM) are trained on original training and syn- thetic + original training datasets to classify grape maturity. The synthetic hyperspectral reflectance data, incrementally added to the original training set in steps of 250, 500, 1000, 1500, and 2000 samples, consistently resulted in higher model performance compared to training solely on the original dataset. The best results were achieved by augmenting the training dataset with 2000 synthetic samples and training with a 3D-CNN, yielding a classification accuracy of 91 % on the testing set. To better assess the effectiveness of GAN-based data augmentation methods, two widely used regression models: Partial Least Squares Regression (PLSR) and one-dimensional CNN (1D-CNN) were used based on same data augmentation method. The best result was achieved by adding 250 synthetic samples to the original training set when training 1D-CNN model, yielding an R2 of 0.78, RMSE of 0.63 ◦Brix, and RPIQ of 3.36 on the testing set. This study indicated that deep learning models combined with conditional WGAN with the gradient penalty data augmentation technique had a good application prospect in the grape maturity assessment.