3.1.

Feature Encoder–Decoder Module

The feature encoder and decoder form a symmetrical U-shaped structure, which is the backbone of the E-UNET. The spatial feature encoder is divided into three layers, each layer contains two convolution filters, the second convolution filter is followed by a max-pooling kernel for down-sampling operations. As shown by the gray dashed arrows in Fig. 3, the output of each encoder layer and each MSD path is connected to the corresponding decoder input via a skip connection. During the feature decoding process, three cascade up-sampling operations are carried out to restore the size of the merged spatial and spectral feature maps to the same size as the input image. At the bottom of the U-shaped structure, an MSP block is embedded to improve the segmentation performance by including global context information.

3.2.

Multi-spectral 3D Convolution Path Module

Although the classical U-Net¹² can also handle multi-spectral images, its two-dimensional (2D) convolution filters can only use features extracted from the spatial dimensions of each band.²² Therefore, we add an MSD path module to capture the nonlinear relationships of adjacent pixels between different spectral bands, which are neglected by the 2D convolution filters in the classical U-Net.¹² Table 1 shows the size and number of filters in the 3D convolution layers (from C1 to R3-C6 in Fig. 3) and in the max-pooling layers (from P1 to P2 in Fig. 3), as well as the output size of each MSD path. The size of the 3D convolution filters is $5\times 5\times 5$. The max-pooling kernels down-sample the output spectral features of each 3D convolution filter, so the spectral feature maps are aligned in the cross-sectional direction with the spatial feature maps extracted by the encoder layers in the U-shaped structure. The spectral feature maps of each size are then sent to the corresponding decoders in the U-shaped structure through the skip connections.

Table 1

Details of the MSD path module of E-UNET. (C, P, and R stand for convolution, max-pooling, and reshaping filters, respectively).

To balance the weights between the spectral and spatial features and prevent the network from giving too much weight to the spectral features, a $1\times 1$ sized convolution filter is added after each MSD path to reduce the dimensionality and computational cost of the spectral features. With the help of the MSD path module, the E-UNET automatically extracts the spectral features from adjacent pixels by the 3D convolution filters and combines them with the spatial features extracted by the U-shaped encoder–decoder module through the skip connections. Thus, E-UNET effectively improves the accuracy of multi-spectral image segmentation.

3.3.

Multi-scale Pooling Block

In the PV semantic segmentation task, it is a big challenge to cope with the substantial variation in sizes of different PV facilities. In the Sentinel-2 multi-spectral images with the finest resolution of 10 m used in this study, the typical width of the total outline of large and continuously aligned PV facilities is about 100 to 200 pixels. Meanwhile, the outline of small and scattered PV facilities or the gap between PV panels is only a few to tens of pixels wide. In the classical U-Net¹² deep network, the maximum pooling uses only one fixed-size pooling kernel. Therefore, the classical U-Net¹² only perceives the context within a fixed-size receptive field and does not fully integrate important multi-scale spatial information.

Inspired by the pyramid pooling structure,²³ we add an MSP block at the bottom of the U-shaped structure, i.e., below the third encoder layer. As shown in Fig. 4, the MSP block uses four sizes of pooling kernels, namely, $32\times 32$, $16\times 16$, $8\times 8$, and $4\times 4$, to divide the spatial feature map into sub-regions of different sizes to perceive contextual relationships and information at different spatial scales.²⁴ To balance the number of features from different pooling kernels and reduce the computational cost, we add a $1\times 1$ sized convolution filter after each pooling kernel to reduce the dimensionality of the spatial features extracted by each pooling kernel to $1/N$of its original dimensionality. We then use up-sampling operations to map the spatial features at different sub-region scales back to the same size as the original spatial features. Finally, the spatial features at different sub-regional scales are cascaded to form a feature pyramid of the MSP block, as shown in Fig. 4.

Fig. 4

Diagram of the MSP block.

4.1.

Dataset

We use sliding cropping to cut the 137 images containing PV facilities into patches with a repetition rate of 0.01. To meet the E-UNET’s requirements for input data, the size of each patch is set to $256\times 256\mathrm{pixels}$. In addition, we perform data augmentation by flipping the patches vertically and horizontally to expand the dataset and prevent over-fitting of the training model. Then, we divide the entire dataset into training, validation, and test sets in a ratio of roughly 8:1:1. Therefore, we use 1746, 262, and 230 patches to train, validate, and test our E-UNET, respectively.

4.2.

Experimental Environment

Keras 2.2.0 with a Tensorflow-gpu 1.7.0 backend is used as the deep learning framework in the experiments. The experiments are conducted on a server with two Intel(R) Xeon(R) Gold 5218R CPUs @ 2.10 GHz with a total of 40 cores, 125 GB RAM, and an NVIDIA Tesla T4 graphics card. The server’s operating system is Ubuntu 18.04.5 LTS.

4.3.

Experimental Design

The experiments are divided into two types: architecture ablation experiments and performance comparison experiments. We first optimize the architecture and parameters of the E-UNET through the architecture ablation experiments; we then analyze and evaluate the performance of the E-UNET in PV semantic segmentation task through the comparative experiments.

4.4.

Evaluation Metrics

We use five metrics, namely, overall accuracy (OA), recall rate, $F1$, Matthews correlation coefficient (MCC),³³ and kappa coefficient,³⁴ defined by Eqs. (26), to evaluate the PV detection performance of each model. These evaluation metrics are calculated from a confusion matrix constructed from the number of pixels that are false negative (FN), false positive (FP), true positive (TP), and true negative (TN).

4.5.

Uncertainty Analysis

We randomly generate five image sets for training, validation, and test from the entire image dataset in a ratio of roughly 8:1:1. We use each of these five image sets to train and evaluate the PV detection performance of all models. Following the strategy of Gu et al.,²⁴ we use the mean and variance of the PV detection performance of each model on these five image sets to analyze the uncertainty of the models.

5.1.

Results of the Architecture Ablation Experiments

Table 2 lists the PV detection performance and uncertainty of the six models in the architecture ablation experiments. Experimental results show that the U-Net-MSP-MSD-k5 model architecture formed by adding the MSP module and the MSP module with 3D convolution filters of size $5\times 5\times 5$ to the U-Net+ structure has the best PV detection performance.

Table 2

PV detection performance and uncertainty of each model in the architecture ablation experiments. The best-performing model and its metrics are highlighted in bold.

The comparison of experimental results of the U-Net+ and the U-Net-MSP confirms that adding the MSP module capable of aggregating spatial information at different scales to the U-Net+ structure improves the PV detection performance.

The comparison of experimental results of the U-Net+, the U-Net-MSD-k1, and the U-Net-MSD-k5 confirms that adding the MSD module capable of extracting spectral features from multi-spectral images to the U-Net+ structure improves the PV detection performance. The comparison also shows that the U-Net-MSD-k5 model, which extracts spectral features from five adjacent spectral bands of each pixel, improves the PV detection performance more than the U-Net-MSD-k1 model, which only extracts spectral features from a single spectral band of each pixel.

As shown in Fig. 5, for images with similar spectral features in PV and background areas, a good PV detection performance cannot be achieved by adding only the MSD module or only the MSP module to the U-Net+ structure. The red and blue boxes in Fig. 5(a) indicate the area with PV panels installed and the background area without PV panels, respectively. The average spectral values of the pixels in the red and blue boxes are shown in Fig. 5(g). The average spectral values of these two regions are very similar. Figure 5(b) shows the manual labeling results of the PV pixels in Fig. 5(a), which are used as the true values to evaluate the PV detection performance of the models.

Fig. 5

PV detection results of an image containing PV areas and background areas with similar spectral features: (a) RGB image synthesized from Sentinel-2 multi-spectral image; (b) Manual labeling results of PV panels in (a); (c) U-Net-MSP-MSD-k5; (d) U-Net-MSD-k5; (e) U-Net-MSP; and (f) U-Net+; and (g) average spectral values of the pixels in the red and blue boxes of (a).

Figures 5(c)–5(f) show the PV detection results of the four models, namely, U-Net-MSP-MSD-k5, U-Net-MSD-k5, U-Net-MSP, and U-Net+, respectively. Compared with the true values in Fig. 5(b), it is obvious that the U-Net-MSD-k5, the U-Net-MSP, and the U-Net+ do not fully and accurately detect the PV panels in and around the area indicated by the red box in Fig. 5(a). The U-Net+ misses a large area of PV panels as indicated by the green box in Fig. 5(f). Because the U-Net-MSD-k5 and the U-Net-MSP add the MSD module for sensing spectral features and the MSP module for sensing spatial features at different scales to the U-Net+ structure, respectively, their PV detection results for the same area are much better than that of the U-Net+. As shown in Fig. 5(c), only the U-Net-MSP-MSD-k5, which is formed by adding both the MSD and the MSP modules to the U-Net+ structure, nearly completely detects the PV panels in the area indicated by the green box.

The experimental results shown in Fig. 5 confirm that the simultaneous use of spectral and spatial features extracted at different scales effectively improves the accuracy of PV detection from multi-spectral images. Therefore, we select the U-Net-MSP-MSD-k5 as the final E-UNET model.

5.2.

Results of the Comparative Experiments

5.2.1.

E-UNET versus other deep learning models

Table 3 lists the PV detection performance and uncertainty of the E-UNET, U-Net+, U-Net,¹² and four state-of-the-art deep-learning models, namely SegNet,²⁷ FCN,²⁸ HRNet,²⁹ and PSPNet,²³ in the comparative experiments.

Table 3

PV detection performance and uncertainty of the seven models in the comparative experiments. The best-performing model and its metrics are highlighted in bold.

Model	OA	MCC	F1	Kappa coefficient	Recall	Training time
E-UNET	0.989 ± 0.000	0.862 ± 0.022	0.869 ± 0.022	0.934 ± 0.008	0.875 ± 0.023	9 h
U-Net+	0.987 ± 0.001	0.847 ± 0.024	0.855 ± 0.024	0.919 ± 0.009	0.856 ± 0.023	3 h
U-Net	0.982 ± 0.001	0.822 ± 0.026	0.830 ± 0.027	0.870 ± 0.016	0.839 ± 0.024	2.5 h
SegNet	0.973 ± 0.001	0.788 ± 0.022	0.803 ± 0.023	0.845 ± 0.009	0.811 ± 0.030	2 h
FCN	0.976 ± 0.001	0.801 ± 0.025	0.815 ± 0.026	0.860 ± 0.011	0.822 ± 0.026	5 h
PSPNet	0.981 ± 0.001	0.831 ± 0.022	0.843 ± 0.024	0.896 ± 0.008	0.845 ± 0.023	5.5 h
HRNet	0.986 ± 0.001	0.842 ± 0.023	0.850 ± 0.023	0.910 ± 0.008	0.848 ± 0.024	4 h

The experimental results show that the E-UNET achieves the highest values in all five detection performance evaluation metrics. Compared with the U-Net+, which has the second-best detection performance, the E-UNET’s OA, MCC, $F1$, kappa coefficient, and recall metrics improve by 0.2%, 1.5%, 1.4%, 1.5%, and 1.9%, respectively.

Figure 6 shows boxplots of the recall rate of all seven models in the five uncertainty experiments with different sets of training, validation, and test images. The E-UNET has the highest median recall rate and the smallest boxplot height, indicating that the E-UNET achieves the best PV detection performance with the least performance fluctuations in the experiments.

Fig. 6

Boxplots of the recall rate of E-UNET, U-Net+, U-Net, SegNet, FCN, PSPNet, and HRNet on five sets of test images. The outliers are indicated by +. The solid lines within the boxes represent the median value. The lower and upper whiskers represent the minimum and maximum values, respectively. The lower and upper edges of the boxes represent the first (Q1) and third quartiles (Q3), respectively.

Figure 7 shows the PV detection results of the seven models for images containing dark backgrounds, roads, and vegetation with texture features similar to PV facilities.

Fig. 7

Four examples of the PV detection results of E-UNET, U-Net +, U-Net, SegNet, PSPNet, FCN, and HRNet. (a) Containing roads; (b) containing vegetation; and (c) and (d) dark backgrounds.

The SegNet²⁷ and the PSPNet²³ have many incorrect detections for images containing vegetation or dark backgrounds. In the detection results of FCN,²⁸ there are many irregular burrs of different sizes at the edges of the PV panels. Both the U-Net¹² and the HRNet²⁹ obtain relatively good PV detection results for the images in Figs. 7(a)–7(c), but for the image containing dark backgrounds in Fig. 7(d), they both mis-detect a large area of the background as PV panels.

The U-Net+ obtains relatively complete PV detection results, but the PV panel edges and the gaps between PV panels in its detection results are more blurred than those in the detection results of the E-UNET. It also does not detect the PV panels within the area marked by the red box in Fig. 7(d), whereas the E-UNET accurately detects them.

The experimental results also indicate that the E-UNET outperforms other networks in detecting the overall contour and edge details of PV panels from multi-spectral images. The multiple connections and the complementary spatial-spectral information at different scales between the encoder, the decoder, the MSD path module, and the MSP block in the E-UNET prevent the problem of irregular PV contours when the decoder recovers the image size from partial features that have lost some detailed information in the multiple down-sampling operations. In addition, the MSP module in the E-UNET utilizes multi-scale spatial features captured by multiple receptive fields at different scales to enable the detection of super-large PV facilities and achieve more stable and reliable PV segmentation results.

5.2.2.

E-UNET versus the pixel-based RF classifier

Table 4 lists the PV detection performance and uncertainty of the E-UNET and the pixel-based RF classifier³⁰ in the comparative experiments. The experimental results indicate that the E-UNET performs better than the pixel-based RF classifier.³⁰ Some examples of the PV detection results of these two models are shown in Fig. 8.

Table 4

PV detection performance and uncertainty of the E-UNET and the pixel-based RF classifier in the comparative experiments. The best-performing model and its metrics are highlighted in bold.

Model	OA	MCC	F1	Kappa coefficient	Recall	Training time
E-UNET	0.989 ± 0.000	0.862 ± 0.022	0.869 ± 0.022	0.934 ± 0.008	0.875 ± 0.023	9 h
RF	0.981 ± 0.001	0.827 ± 0.025	0.834 ± 0.026	0.879 ± 0.019	0.822 ± 0.026	4 h

Fig. 8

Four examples of the PV detection results of the E-UNET and the pixel-based RF classifier. (a) Containing vegetation; (b) Dark backgrounds; (c)–(d) Photovoltaic array with many texture details.

Although the pixel-based RF classifier³⁰ may be more accurate in detecting some small-scale details of PV panels, it does not take advantage of the information provided by neighboring pixels, which is usually strongly correlated, resulting in a lot of scattered and fragmented PV panels and backgrounds in its detection results, as shown in Figs. 8(a), 8(c), and 8(d).

Furthermore, as shown in Fig. 8(b), for images in which the backgrounds and PV panels have similar spectral or spatial texture features, the pixel-based RF classifier³⁰ is more likely to mis-detect PV panels as backgrounds than the E-UNET.