A research team from Skoltech and Saint-Petersburg State University of Aerospace Instrumentation presented a paper, where they pioneered an alternative method for detecting decayed and moldy apples at the post-harvest stage, when fruits are stored and then delivered to the customer. A computer vision system will determine different defects at the early stage, when they can be invisible to the human eye. The paper is published in the Entropy journal.

Despite the pervasive automatization, the quality of fruits and vegetables at the post-harvest stage, especially when grading, is mostly controlled by the human. When products are delivered, they can get decayed or damaged. Some areas of decay can be overlooked or missed out, while an agriculturalist can’t always be available to determine the type of disease or damage.

Using apples, the team studied two types of defects: decay and molding. If, for example, apples are densely packed, they hit each other, and thus they decay faster in those areas. Mold results from a radical violation of storage conditions or when it was overlooked while harvesting.

For detecting defects, specialists use infrared light, but, according to the researchers, it requires multi- and hyperspectral devices, which are very expensive and not always user-friendly. The study aimed to offer an alternative to those cameras, which use deep-learning-based models that can generate infrared images. The authors emphasize that they didn’t intend to replace traditional methods and only tried to put forward a more affordable and innovative way.   

“We employed two types of neural networks: generative adversarial network and convolutional neural network. The former allows converting one type of image into the other. In our case, we got infrared images from RGB, meaning visible photos. But that’s not enough for detecting defects because generative adversarial networks don’t classify images. Here, convolutional neural networks come into play. They help detect and segment objects of the needed classes in the photos,” said the leading author of the study, Nikita Stasenko, a junior research engineer at Skoltech’s Agro Center.

Experiments involved several stages. The team started with collecting and processing data from visible images. To that end, the authors selected 16 apples of four different kinds. The apples had different treatment: an apple with no treatment, a thoroughly washed and wiped apple, a mechanically damaged apple, and a shock-frozen apple supercooled under −20°С.

“When we collected data, we checked several models based on generative adversarial networks — Pix2Pix, CycleGAN, and Pix2PixHD — and compared generated infrared images with the original ones. According to the quality metrics, images that were generated by Pix2PixHD turned out to be as close to the original as possible,” added Nikita Stasenko.

The second stage used the Mask R-CNN convolutional neural network. In the previous research, this model was the most effective. To train it, the team collected another dataset from infrared images and annotated them: they marked healthy apples and those with areas of decay and molding.

At the third stage, the authors used Jetson Nano — a special embedded system, which allows running trained neural networks. In the future, this system will give rise to a real device for detecting crop defects. Also, the team is planning to scale results on other types of crop and test other neural models.

Among researchers who also contributed to the study are Islomjon Shukhratov and Dmitry Shadrin from the Applied AI Center, Andrey Somov from the Engineering Center, and their colleagues from other institutions.