Nikita Belyakov and Svetlana Illarionova, researchers from the Skoltech AI Center, presented a new method for semantic segmentation of multispectral data, which can be used to recognize clouds, shadows, and snow patches in satellite images. This approach will increase accuracy of recognizing complex climatic structures in images without an additional human involvement in data annotation. The research results are presented in Advances in Space Research. The code and examples are available on GitHub. 

Convolutional neural networks have become one of the best tools for image and video recognition. To accurately segment objects, they need a large amount of high-quality training data that requires human preparation. To enhance segmentation quality, different approaches are employed, such as data augmentation techniques.

The new research, supported by a grant from the Russian Science Foundation, seeks to improve the accuracy of recognition and classification of rare or difficult-to-analyze objects in satellite images, such as clouds, their shadows, and snow patches, at the preliminary stage of satellite data preparation for solving environmental analysis tasks. The authors proposed an approach called CSIA — Climate Structures Inpainting Augmentations. With it, additional climatic structures are “completed” in the original images. Realistic fragments generated by neural networks are added to areas where such objects are absent, which artificially increases the amount of training data.

“The main feature of our approach is that we ‘complete’ realistic climatic structures — clouds, their shadows, and snow patches — and embed them in satellite images without the need for additional manual data annotation,” says Nikita Belyakov, a PhD student from the Skoltech’s Computational and Data Science and Engineering program.

“We artificially expand the sample and teach the neural network not to get confused when it encounters rare or difficult-to-segment objects. Our method helps models better understand the geometry and optics of climate objects, which is especially important when analyzing large regions and rare weather phenomena,” commented Svetlana Illarionova, who heads the research group at the Skoltech AI Center.

Experiments have shown that CSIA significantly improves segmentation of clouds and shadows on Landsat-8 data and in the SPARCS dataset. By combining the U-Net++ architecture with the Model Soups approach, accuracy is enhanced even further by averaging multiple models. The authors claim that this combined solution enables computer vision to learn from heterogeneous data more efficiently and reliably recognize complex classes.

The study opens up opportunities for more accurate segmentation in a wide variety of applications, from climate monitoring of vast regions to environmental projects and agricultural tasks. For example, the solution facilitates the effective analysis of the forest area, its characteristics, and changes, even in northern regions with a high percentage of clouds, while considering the impact of climatic conditions on the images. The researchers intend to keep developing the method by adapting it to other types of remote sensing data and introducing more generation mechanisms that are adapted to seasonal and weather changes.