Research on apple epidermal damage grading based on lightweight convolutional neural network

Abstract: 【Objective】In the process of apple selling, the damage of its epidermis will directly affect the economic value of the fruit. The presence and severity of apple surface damage directly affect the sales link, and customers often care about the epidermis damage when choosing apples. At present, most studies focus on apple size, color and appearance classification, and the use of high- end instruments to detect the damage inside the apple, while the study on the direct classification of surface damage is rare. The camera was used to collect apple epidermis damage images, classify and preprocess the acquired images, and conduct a direct classification on apple epidermis damage based on transfer learning method, which can provide a theoretical basis for improving the classification efficiency of apple epidermis damage and guiding the classification and apple sale after harvesting.【Methods】Firstly, the camera was used to collect the top, side and bottom images of Fuji and Danxia apples to form the firststage data set. Then, 11 batch of operations, such as contrast adjustment, rotation, flip and noise addition, were carried out to expand the data set to 9360 pieces to form the second- stage data set. At thesame time, the expanded sample set was uniformly adjusted to 224×224 pixels to form the final data set. According to the ratio of 7∶3∶3, the preprocessed data set was divided into training set, verification set and test set. Five lightweight convolutional neural networks less than 20 MB, MobileNet- v2, SqueezeNet, ShuffleNet, NASNet-Mobile and EfficientNet-b0, were selected for initial training, introduction of migration learning training and migration learning under the same super-parameter Settings On a Bottleneck basis, and three methods were added for detailed freezing network layer weights (the MobileNet-v2 network structure is specifically divided into 21 modules for freezing training, which contain 3 convolutional modules, 1 average pooling module, and 17 Bottleneck modules).【Results】The test accuracy of the five kinds of networks after initial training was only 56.32%-71.98%. The final training accuracy of MobileNet-v2 model based on transfer learning was 99.04%, 18.79% higher than that of the worst EfficientNet-b0 model among lightweight convolutional neural networks. After freezing different module parameters on the basis of the MobileNet-v2 model were based on transfer learning, it was concluded that models Bottleneck 3-1, when they select to freeze to the first convolutional module, can shorten model training time and improve model validation accuracy. When Bottleneck 3-1 module was frozen, the training time for Bottleneck 3-1 was shortened by 29.32% compared to MobileNet-v2 model based on transfer learning, the verification accuracy increased by 0.93%, and the test accuracy increased by 1.12 percentage points to 91.58%. The average time for detecting a single image was 0.14 s. The network size was 8.15 MB, which can meet the requirements of fast identification. The final training loss value of the MobileNet-v2 model based on transfer learning was less than 0.04, which was 0.5 lower than that of the worst performing EfficientNet-b0 model in lightweight convolutional neural networks. The test results showed the recall rate and precision rate of MobileNet-v2 confusion matrix diagram based on transfer learning and five kinds of lightweight convolutional neural networks were based on transfer learning in the test set. Among them, the MobileNet-v2 model based on transfer learning had the best performance, and the recall rate of 6 types of data in the test set ranged from 89.40% to 100%. The precision ranged from 53.52% to 99.78%. The Grad-CAM visualization comparison of the trained network showed that the SqueezeNet model based on transfer learning had the worst visualization effect and the lowest recognition accuracy. The visualization effect of NASNet- Mobile model based on transfer learning was poor. It can only display a large range of concern areas, and the recognition degree of some pictures was not high. The visualization effect of the MobileNet-v2 model based on transfer learning was obviously better than the previous two models, but the key areas identified by the model were different from the reality. A MobileNet- v2 model based on transfer learning tended to have the best visualization effect on a network that was Bottleneck 3-1 when it was frozen to a Bottleneck 3-1 module, and the key areas identified by the model had the highest compatibility with the actual situation.【Conclusion】In this study, five kinds of lightweight models with Bottleneck 3-1 were selected for initialization training and transfer learning training, and it was concluded that MobileNet-v2 model with transfer learning had the best effect. Then, the freezing strategy was used for hierarchical training. The verification accuracy reached 92.23% when Bottleneck 3-1 was frozen. The test accuracy was 91.58% , the average recognition time was 0.14 s, and the network size was 8.15 MB, which can provide technical reference for mobile terminals and embedded devices in the direct classification of apple fruit damage.