ISSN: 2167-0587
+44-77-2385-9429
Research Article - (2013) Volume 3, Issue 1
Fusing high-spatial resolution panchromatic and high-spectral resolution multispectral images with complementary characteristics provides basis for complex land-use and land-cover type classifications. In this research, we investigated how well different pan sharpening algorithms perform when applied to single-sensor single-date and multi-senor multi–date images that encompass the Horton Plains national park (HPNP), a highly fragile eco-region that has been experiencing severe canopy depletion since 1970s, in Sri Lanka. Our aim was to deliver resolution-enhanced multitemporal images from multiple earth observation (EO) data sources in support of long-term dieback monitoring in the HPNP. We selected six candidate fusion algorithms: Brovey transform, Ehlers fusion algorithm, high-pass filter (HPF) fusion algorithm, modified intensity-hue-saturation (MIHS) fusion algorithm, principal component analysis (PCA) fusion algorithm, and the wavelet-PCA fusion algorithm. These algorithms were applied to eight different aerial and satellite images taken over the HPNP during last five decades. Fused images were assessed for spectral and spatial fidelity using fifteen quantitative quality indicators and visual inspection methods. Spectral quality metrics include correlation coefficient, root-mean-square-error (RMSE), relative difference to mean, relative difference to standard deviation, spectral discrepancy, deviation index, peak signal-to-noise ratio index, entropy, mean structural similarity index, spectral angle mapper, and relative dimensionless global error in synthesis. The spatial integrity of fused images was assessed using Canny edge correspondence, high-pass correlation coefficient, RMSE of Sobel-filtered edge images, and Fast Fourier Transform correlation. The Wavelet-PCA algorithm exhibited the worst spatial improvement while the Ehlers.MIHS and PCA fusion algorithms showed mediocre results. With respect to our multidimensional quality assessment,the HPF emerged as the best performing algorithm for single-sensor single-date and multi-sensor multi-date data fusion.We further examined the effect of fusion in the object-based image analysis framework. Our subjective analysis showed the improvement of image object candidates when panchromatic images’ high-frequency information is injected to low resolution multispectral images.
Keywords: Image fusion; Fusion evaluation; Ecosystem monitoring; Canopy dieback; Horton plains; Sri Lanka
Forest ecosystems in developing countries are being depleted at alarming rates [1,2]. Sri Lanka is classified as one of the 25 biodiversity hotspots in the world. The country harbors two world-heritage nature reserves designated by the United Nations Educational, Scientific and Cultural Organization (UNESCO). Sri Lanka has been experiencing severe depletion of its biodiversity owing to overwhelming anthropogenic stresses acting on forest ecosystems. During last century, Sri Lanka’s total close-canopy forest cover has been decreased from about 84% of the total area to about 30% [3-5].
The Horton Plains National Park (HPNP) is a UNESCO designated world heritage nature reserve, which is located in the Central Highlands of Sri Lanka. This fragile eco-region provides habitats for nearly half of Sri Lanka’s endemic flowering plants and endemic vertebrates [6,7]. Studies reveal that some selected sites of HPNP are represented by 57 species of vascular plants belonging to 44 genera and 31 families [8]. Of these, 18 species are only seen in montane forests in Sri Lanka and India [7]. Apart from invaluable ecological richness, HPNP’s serene landscape has made an inextricable link to Sri Lanka’s tourism industry.
The HPNP has been received greater attention during last three decades owing to the sever canopy diebacks reported in certain parts of the park. Since the initial documentation occurred in late 1970s [9,10], nearly 37 plant species have been susceptible to dieback and 26 among them are endemic to Sri Lanka [11,12]. Through ground-based inventories of canopy cover and health status, investigators noted that approximately 17.2% of forested areas (~655 Ha) in the national park have been subjected to severe dieback [12,13]. Families like Lauraceae, Simplocaceae, and Myrtaceae have shown a greater vulnerability to forest dieback. Syzigium rotundifolium, Ilex walkeri, Euodia lunuankenda, Symplocos bractealis serve as the dominant species susceptible to forest dieback [11]. Ediriweera et al. [7] noted that the susceptibility to dieback gradually increases as the DBH class increases. Owing to HPNP’s high ecological and cultural values, there has been a growing interest on understanding factors associated with the canopy depletion. Several theories have been put forwarded such as, acid rain [14], climate change [15], elevated total nutrient content [16], diseases [11], sambur damage, and heavy metal contamination [6,12,17], however, the etiology of the forest dieback remains unexplained.
The utility of Earth Observation (EO) data in complex land cover mapping applications is a well addressed research problem, There is a plethora of literature on how air- and space-born data with varying spatial (coarse, moderate, high and very-high resolution), spectral, and radiometric resolutions assist in multi-scale vegetation information extraction, ranging from forest stand to individual tree canopies. However, we believe that the full strength of EO data and advanced image processing techniques are weakly exploited in relation to ecological applications in Sri Lanka. Remote sensing serves as a cost effective tool for developing countries [18]. Excluding very high resolution commercial satellite images, many other civilian-use sensors provide image data through public domains at no cost. For example, current and archived images of LandSat MSS/TM/ETM+, EO-1 ALI/ Hyparion, ASTER, and MODIS can be freely downloaded from the United States Geological Survey (USGS) Earth Resources Observation and Science (EROS) Center (https://www.earthexplorer.usgs.gov), global data explorer (https://www.gdex.cr.usgs.gov), and University of Maryland’s Global Land Cover Facility (GLCF) (https://www.glcf. umiacs.umd.edu). These images cover large geographical areas and offer the possibility of time series analysis given the large quantity of archived data spanning many years. Low spatial resolution of these images (e.g. LandSat MSS 60m) stands as the main disadvantage because accurate vegetation mapping also requires high frequency information. In this context, fusing multi-platform data types with complimentary characteristics serve as one of the most viable and cost effective solution.
Moderate and very-high resolution sensors typically record image data in a low resolution multispectral (MS) mode and high resolution panchromatic (PAN) mode (e.g., EO-1 ALI: PAN = 10m, MS = 30m; SPOT-5: PAN 5m, MS = 10m; IKONOS: PAN = 1m, MS = 4m, WorlView-2: PAN = 0.46cm, MS = 1.84m) due to the limited on-board storage capacity and data transmission rates from space-born platforms to the ground stations [19-21]. The high spatial resolution is needed to accurately describe the shapes of features and structures, and the high spectral resolution is needed to classify complex land-use and landcover types [22-24]. Fusing PAN and MS images with complementary characteristics can provide a better visualization of the observed area [22,23]. Image fusion can be applied to various types of data sets, such as single-sensor single-/ multi-date (e.g. PAN and MS images of IKONIOS, SAR multi-temporal images), multi-sensor single- /multidate (e.g. high and low resolution images of SPOT and LandSat, VIR and SAR multi-temporal images), single-data from multiple sensors (e.g. ERS-1 and ERS-2), and RS data with ancillary data (e.g. fusion of images with topographic maps). Many image-fusion algorithms were developed for combining complimentary characteristics of PAN and MS images to produce an enhanced multispectral image of high spatial resolution. Several classifications for grouping fusion algorithms have been proposed in literature [20,22,24-27]. In general, fusion techniques can be grouped as spectral substitution methods, arithmetic merging, and spatial-domain methods.
A fusion algorithm that preserves the spectral properties of the MS data and the spatial properties of the PAN data would be ideal, but there is always compromise [28,29]. Many studies report the problems and limitations associated with different fusion techniques [30,31]. The most-encountered problem in fusion algorithms is that the fused image exhibits a notable deviation in visual appearance and spectral values from the original MS image [32]. Spectral distortions including spatial artifacts affect both manual and automated classifications because any error in the synthesis of the spectral signatures at the highest spatial resolution incurs an error in the decision [23]. Qualitative comparison of the fused image and the original MS and PAN images for color preservation and spatial improvements is the most simple but effective way of benchmarking different fusion algorithms [28,33]; however, visual inspection methods are subjective and largely depend on the experience of the interpreter [24,34].
A number of objective metrics have been proposed to quantify spectral and spatial distortions incurred during the fusion process. Most widely used metrics for evaluating spectral fidelity are two-dimensional Correlation Coefficient (CC), Root Mean Squared Error (RMSE), relative difference of means, relative variation, deviation index, and band discrepancy. Workers like Vijayaraj et al. [35], Karathanassi et al. [36], Yakhdani and Azizi [27], and Witharana et al. [29] utilized Peak- Signal-to-Noise Ratio (PSNR) and entropy as spectral quality metrics in addition to common indicators. Wald [22] proposed the ERGAS metric (from its French acronym: erreur relatif globale adimensionnelle de synthe`se, which means relative dimensionless global error in synthesis), which aims to provide a quick but accurate measure of the overall quality of a fused product. Few workers used the spectral angle mapper (SAM) to assess the overall spectral quality of fused images. Wang et al. (2004) proposed another metric called Mean Structure Similarity Index (MSSIM), which was developed based on the findings of Wang and Bovik (2002). Compared to spectral quality indicators, only few metrics are available to evaluate the spatial fidelity of fused images [29,37], Ehlers et al. [24], Gangkofner et al. [20], Klonus and Ehlers [34], Yakhdani and Azizi [27], and Witharana [28] used highpass correlation and edge detection using filters like Canny, Sobel, and Perwitte.
This study serves a corner stone of our ongoing effort on introducing Geographic Object-Based Image Analysis (GEOBIA, also called OBIA) framework to the vegetation mapping efforts in the HPNP aiming on two foci: 1) forest dieback and 2) invasive plant species. GEOBIA (or OBIA) is a novel conceptualization of image understating that mimics innate cognition abilities of humans. Unlike pixel-based paradigm that is solely driven on spectral signatures of individual pixels, GEOBIA integrates spectral, spatial, and contextual properties into image classification workflows (Balschke 2010). Thus, in case of GEOBIA, spatial properties of images cannot be overlooked and injection of high frequency information is necessary for better image segmentation results. The central objective of this research is to investigate how well different fusion algorithms when applied to singlesensor single-date and multi-senor multi–date images taken over the Horton Plains national park representing crucial time intervals. The spectral and spatial fidelity of fused images were assessed using a variety of quantitative quality indicators and visual inspection methods. The quantitative indicators include eleven spectral quality metrics and three spatial quality metrics. A novel spatial metric based on Fourier transform was also integrated into our spatial quality budget. We made few preliminary quality assessments on image segmentation results to demonstrate the importance of data fusion in segmentation workflows.
The remainder of this paper is structured as follows. Section 2 describes study areas, image data, fusion algorithms, and evaluation methods. Section 3 reports the spatial and spectral fidelity of fused products in terms of quantitative indices and visual inspections. Section 4 contains a discussion explaining the results based on the performances of fusion algorithms. Finally, conclusions are drawn in Section 5.
Study area and data
The Horton Plains national park encompasses 3,200 Ha in Central Highlands of Sri Lanka (Figure 1). The park comprises upper montane rain forest (cloud forests) and wet patana grasslands and characterized by undulating terrain of rolling hills and valleys with a network of streams. The annual rainfall in the area ranges 2000 mm - 5000 mm. We selected a representative study area from the south west corner of the park comprising major land cover types and observable canopycover changes occurred over the time.
Image scenes used in this study belong to two different platforms: 1) air-borne and 2) space-borne. The former group entails images from two different aerial missions commissioned in year 1956 and 1986. The latter comprises images acquired by four different satellite sensors ranging from moderate spatial resolution to very-high spatial resolution (Figure 2). Table 1 summarizes general characteristics of the source data. The images are spatially registered to the Universal Transverse Mercator (UTM) coordinate system on the WGS 84 datum.
Platform | Sensor | Acquisition date | Properties | Source | ||
---|---|---|---|---|---|---|
spatial/scale | spectral | radiometric | ||||
Aerial | Unknown | 1956 | 1:40,000 | PAN | Scanned and stored as 8 bit data | Department of Geology, University of Peradeniya, Sri Lanka |
Unknown | 1986 | 1:25,000 | PAN | Mahaweli Authority, Nawalapitiya, Sri Lanka |
||
Satellite | LandSat Multispectral scanner (MSS) | 1972 | 60m | 4 bands | 8 bit | USGS EROS (http://www.earthexplorer.usgs.gov) |
LandSat Thematic Mapper scanner (TM) | 1986 | 30 m | 7 bands | 8 bit | ||
LandSat Enhanced Thematic Mapper scanner (ETM+) | 2000 | 28.5m (MS) 14.5 m (PAN) |
8 bands | 8 bit | ||
LandSat Enhanced Thematic Mapper scanner (ETM+) | 2003 | |||||
EO-1 Advanced Land Imager (ALI) | 2004 | 10m (PAN) 30 m (MS) |
10 bands | 12 bit | ||
IKONOS-2 | 2005 | 1 m (PAN) 4 m (MS) |
4 bands | 11 bit | Purchased |
Table 1: General characteristics of image data.
Methods
We selected a 2km x 2km subset as the focal Area of Interest (AOI). The selection of the subset was made focusing on the land cover types that are most likely to be extracted (e.g., water, grassland, forest, and riverine vegetation) and distinct changes occurred over the time (e.g., depleted forest cover) .Aerial images acquired in 1956 and 1986 were scanned using 600 dots-per-inch (dpi) resolution and stored as 8-bit data. The images were then ortho-rectified using 90 meter Shuttle Radar Topographic Mission (SRTM) Digital Elevation Model (DEM) and co-registered with the IKONOS (2005) image. The scales of these images were known but lines-per-inch (lpi) count was unavailable, we therefore set spatial resolution of pre-processed images to 2m resolution. All other moderate resolution satellite images were coregistered with the IKONOS image to maintain the spatial consistency. We aimed to maintain the maximum spatial resolution ratio between PAN and MS as 1:4. Thus, high spatial resolution images were resampled as necessary to maintain 1:4 ratios. In case of the 1956 aerial (2m) and the 1972 LandSat MSS (60m) fusion, the aerial image was degraded to 15m resolution. When fusing the 1986 aerial (2m) and the 1992 LandSat TM (28.5m) images, the former was down sampled to 7m resolution.
We tested six fusion algorithms that are commonly encountered in the literature and built into image processing software packages: 1) Brovey (EH) transform, Ehlers (EH) fusion algorithm, High-Pass Filter (HPF) fusion algorithm, Modified Intensity-Hue-Saturation (MIHS) fusion algorithm, Principal Component Analysis (PCA) fusion algorithm, and the Wavelet-PCA (WV-PCA) fusion algorithm. Discussion of theoretical basis of these candidate algorithms is beyond our scope and we refer readers to relevant literature listed in Table 2. We used ERDAS Imagine 2011 to implement fusion algorithms. Some of the candidate fusion algorithms are proprietary (e.g., Ehlers fusion - ERDAS Imagine). Unlike the Brovey transform algorithm, which produce three-band fused images (B, G, and R or G, R, and NIR); other candidate algorithms are capable of accepting more than three bands at a time and producing four-band fused images in a single iteration. Therefore, we produced true- and false-color composites of BT algorithm and layer-stacked them to create fourband pan sharpened images. Fusion results were assessed using a series of quality metrics along with detailed visual inspection procedures to evaluate the spectral and spatial fidelity of fused products compared to their original MS and PAN images. Objective metrics were calculated independently for each subset and separately for each band (except for ERGAS and SAM). Subsequently, mean values were calculated for all bands. Use of eleven spectral and three spatial metrics, totaling 14 objective quality indicators in our evaluation procedure, might be questionable because these metrics. However, our justification is that it is important to employ a full complement of objective quality indicators and reexamine their stability and redundancy, and investigate the dependency of the ranking of fusion algorithms on quality metrics. These metrics’ mathematical and statistical bases are well addressed in literature; we therefore refer readers to Table 3 for relevant references. Beyond commonly found spatial quality indicators, we tested a new metric based on the Fast Fourier Transform (FFT) to assess the spatial fidelity which was initially proposed by Civco et al. [38]. In our recent work [37], we further investigated the discriminative capacity of this metric. Our argument is the original PAN image and the fused image should resemble in the Fourier domain if high frequency information is inject from the PAN image to the MS image during fusion. The fusion-evaluation workflow is depicted in Figure 3.
Algorithm | Reference |
---|---|
Brovey transform (BT) | [25,33,36,41-45] |
Ehlers fusion (EH) | [24,32,34,46] |
High-pass filter (HPF) | [20,30,47,48] |
Modified intensity hue saturation (MIHS) |
[24,33,34,49] |
Principle component analysis (PCA) |
[25,30,36,44,47,50] |
Wavelet Transform (WV) | [20,27,35,36,43] |
Table 2: Candidate fusion methods and related literature.
Quality metric | Addressed issue/domain/expected value | Reference | |
---|---|---|---|
Spectral | Correlation coefficient (CC) | •Quantifies the spectral correspondence between the original MS and fused images. •domain [-1,1] •As close to 1 as possible |
[19,20,36] |
Root-mean -square- error (RMSE) | •Measures the average amount of spectral distortion in each pixel •domain [0,inf) •Lower value |
[22,23,34] | |
Relative difference to mean (RDM) | •Measure the changes in the shape of the histogram of fused image compared to original MS image. •domain (-inf, inf) •As close to 0 as possible |
[21,35,36] | |
Relative difference to standard deviation (RDS) | [21,23,36] | ||
Spectral discrepancy (SD) | •Band-wise measure of the spectral quality of the fuse image •domain [0,inf] •As close to 0 as possible |
[27,32,51] | |
Deviation index (DI) | •Quantifies the normalized absolute difference of the fused image with the original MS image. •domain [0,inf] •As close to 0 as possible |
[32,34,36] | |
Peak signal-to-noise ratio (PSNR) | •Indicates the radiometric distortion of the fused image compared to the original MS image. •The highest possible PSNR |
[36,52] | |
Entropy (E) | •Measures the additional information (spectral and spatial) available in the fused image compared to the original MS image. •The smallest possible entropy difference with the original MS image. |
[35,36] | |
Mean structural similarity index (MSSIM) | •Reveals the spectral and structural similarity between the fused and original MS image by luminance, contrast, and structure and applying to a moving window. •domain [0,1] •As close to 0 as possible |
[19,24,32] | |
Spectral angle mapper (SAM) | •Pixel-wise comparison of fused image and original MS image. The value 0 indicates low resemblance while 1 indicates a high resemblance. •domain [0,1] •As close to 0 as possible |
[44,53,54] | |
Relative dimensionless global error in synthesis (ERGAS) | •A global indicator that calculates the amount of spectral distortion. •domain [0,inf) •Lower value (< 3) |
[22,23,54] | |
Spatial | Canny edge correspondence (CEC) | •A band-wise comparison of edges detected in the original PAN and the fused image. CES measured in percent. •domain [0,100]% •as close to 100 as possible |
[24,27] |
High-pass(HP) correlation coefficient (HP-CC) | •Quantifies the correlation between the HP filtered bands of fused image and the HP-filtered PAN image. •domain [-1,1] •as close to 1 as possible |
[20,24,55,56] | |
RMSE of Sobel filtered Pan and fused images (Sobel-RMSE) | •Measures the average amount of spatial distortion in each pixel •domain [0,inf) •Lower value |
[19,34] |
Table 3: Summary of quantitative quality metrics.
In order to demonstrate the value of injection of spatial structures into MS images in GEOBIA framework, we introduced fused product of 1956 aerial and 1972 LandSat MS fusion and the original LandSat MS image to the eCognition Developer’s Multi resolution Segmentation Algorithm (MRS). The quality of image segments (also called image object candidates [39]) of fused and non-fused images were compared. With the capability eCognition Developer’s Cognition Network Language (CNL), an exemplar classification was done by applying a class-modeling approach [40] where object candidates were refined in cyclic and adaptive manner to represent meaningful target.
Fusion evaluation
Visual assessment: To inspect the color preservation and spatial improvement, fused images were compared to the original MS and PAN images, respectively. We selected false-color composites (bands 2, 3, and 4) for visual inspections because this band combination is widely used for many remote sensing applications. However, we had to use a true-color composite for the ALI (2004) single-sensor fusion Fused images along with their original images were inspected by two photo-interpretation experts to identify any spectral distortions, (e.g., brightness reversions, saturation, a complete change of spectral characteristics, unnatural/artificial colors) and spatial improvement. Although we inspected all fused images, only four scenarios are presented, i.e., aerial (1956) - LandSat MSS (1972), aerial (1986) - LandSat TM (1992), LandSat ETM (2003) - ALI (2004), and IKONOS (2005) in Figures 4-7, respectively. Based on expert evaluations, fused products were ranked and the results (best and worst fusion algorithms) are listed in Table 4.
Fusion | Spectral similarity | Spatial similarity | ||
---|---|---|---|---|
Best | Worst | Best | Worst | |
Aerial (1956)-LandSat MSS (1972) | HP filter | Brovey | Brovey | Wavelet |
Aerial (1986)-LandSat TM (1992) | HP filter | Brovey | Ehlers | Wavelet |
LandSat ETM+ (2000) | HP filter | Brovey | PC | Wavelet |
LandSat ETM+ (2003)-EO-ALI (2004) | HP filter | Brovey/MIHS | Ehlers | Wavelet |
EO-ALI (2004) | Brovey | MIHS | Brovey | Wavelet |
IKONOS (205) | HP filter /Ehlers | Brovey | Ehlers | Wavelet |
Table 4: Objective evaluation of fused images by experts.
Quantitative assessment: We corroborated visual assessment with eleven spectral metrics and three spatial metrics. In order to give a detailed picture, band-wise scores of CC and PSNR and global scores of ERGAS and SAM are shown in Figure 8. Tables 5-10 summarize the mean scores (averaged over bands) reported by quality metrics for the six fusion scenarios. Fusion algorithms in each table are ranked by their correlation coefficient scores. The best value reported to a given metric is highlighted in gray while the worst value is in bold font. The spatial fidelity of fused images was further analyzed using a new metric, which is based on Fast Fourier Transform (FFT). We selected few fusion scenarios for demonstration purposes. Figure 9 and 10 depict exemplar Fourier-magnitude images of the original PAN and three fused images which showed best, worst and average spatial and spectral improvement with respect to the other quality indicators (i.e., Tables 5-10). The former represent aerial (1956) - LandSat (1972) fusion while the latter pertains to IKONOS (2005) single-sensor fusion. We plotted Digital Numbers (DN) of Fourier-magnitude images of original PAN images and those of selected fused images. Figure 11 shows scatter plots constructed for two multi-sensor data fusion scenarios.
Image segmentation: As a test run, we selected a crucial multisensor fusion scenario (aerial (1956) - LandSat MS (1972)) and applied eCognition Developer’s MRS algorithm to the fused image and the original LandSat MS image. The resulting image objects and the extracted canopy cover are shown in Figure 12.
Figure 12: The quality of image objects candidates derived from the original LandSat MSS image and the fused image (i.e., after injecting spatial structures from the high resolution aerial image using High-pass filter fusion algorithm). (a) Image objects of the original and the fused images draped over the high resolution aerial image, (b) Classified image objects as forest (in green), and (c) After applying advanced object fusion algorithm to merge forest’ objects.
Fusion algorithm | Spectral metric | Spatial metric | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CC | RMSE | RDM | RDS | SD | DI | PSNR | ENT | MSSIM | ERGAS | SAM | Canny edge | HPF-CC | Sobel-RMSE | |
HP Filter | 0.814 | 2.04 | 0.031 | -0.094 | 1.549 | 0.073 | 42.848 | 0.208 | 0.999 | 2.349 | 1.736 | 84.187 | 0.755 | 13.039 |
Wavelet | 0.739 | 2.949 | 0.011 | 0.138 | 1.558 | 0.068 | 40.999 | 0.128 | 0.995 | 2.987 | 2.274 | 79.771 | 0.642 | 12.304 |
Ehlers | 0.622 | 2.733 | 0.003 | -0.095 | 2.073 | 0.095 | 40.185 | 0.635 | 0.997 | 3.144 | 2.11 | 83.815 | 0.73 | 13.052 |
PC | 0.503 | 3.33 | 0.013 | -0.104 | 2.657 | 0.109 | 39.577 | 0.326 | 0.996 | 3.427 | 2.79 | 87.246 | 0.808 | 12.579 |
MIHS | 0.487 | 3.256 | 0.014 | -0.087 | 2.521 | 0.115 | 38.727 | 0.202 | 0.996 | 3.712 | 3.012 | 83.891 | 0.754 | 12.914 |
Brovey | 0.38 | 3.584 | 0.013 | -0.099 | 2.849 | 0.126 | 37.999 | 0.244 | 0.995 | 3.971 | 2.775 | 94.696 | 0.989 | 8.571 |
Table 5: Reported scores of spectral and spatial quality metrics for the fusion of aerial image (1956) and LandSat MSS image (1972).
Fusion Algorithm | Spectral metric | Spatial metric | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CC | RMSE | RDM | RDS | SD | DI | PSNR | ENT | MSSIM | ERGAS | SAM | Canny edge | HPF-CC | Sobel-RMSE | |
HP filter | 0.691 | 4.69 | -0.011 | 0.3 | 2.557 | 0.064 | 35.572 | 0.154 | 0.993 | 2.844 | 2.525 | 84.837 | 0.792 | 22.213 |
Wavelet | 0.609 | 5.063 | 0.006 | 0.113 | 3.146 | 0.076 | 39.112 | 0.18 | 0.992 | 3.123 | 2.719 | 80.719 | 0.624 | 21.469 |
MIHS | 0.486 | 5.862 | -0.009 | 0.213 | 3.751 | 0.093 | 34.111 | 0.083 | 0.989 | 3.734 | 3.091 | 87.261 | 0.762 | 22.065 |
Ehlers | 0.371 | 7.504 | -0.007 | 0.564 | 3.916 | 0.097 | 31.176 | 0.566 | 0.984 | 4.869 | 2.884 | 83.717 | 0.76 | 22.1 |
PC | 0.37 | 8.593 | -0.01 | 0.853 | 4.268 | 0.106 | 30.404 | 0.035 | 0.979 | 5.516 | 4.218 | 88.291 | 0.888 | 21.42 |
Brovey | 0.327 | 8.277 | 3.306 | -0.013 | 0.704 | 4.376 | 0.104 | 30.732 | 0.036 | 3.924 | 3.184 | 94.844 | 0.989 | 14.934 |
Table 6: Reported scores of spectral and spatial quality metrics for the fusion of aerial image (1986) and LandSat TM image (1992).
Fusion Algorithm | Spectral metric | Spatial metric | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CC | RMSE | RDM | RDS | SD | DI | PSNR | ENT | MSSIM | ERGAS | SAM | Canny edge | HPF-CC | Sobel-RMSE | |
HP filter | 0.912 | 2.295 | 0.004 | 0.003 | 1.316 | 0.025 | 42.056 | 0.212 | 0.998 | 2.328 | 1.072 | 84.496 | 0.923 | 1.191 |
Wavelet | 0.864 | 3.377 | 0.008 | 0.119 | 1.334 | 0.025 | 38.92 | 0.125 | 0.998 | 3.647 | 0.603 | 79 | 0.598 | 3.111 |
Ehlers | 0.656 | 5.579 | 0.006 | 0.316 | 2.36 | 0.045 | 32.644 | 0.167 | 0.995 | 6.001 | 1.433 | 84.948 | 0.919 | 1.43 |
PC | 0.616 | 5.489 | 0.008 | 0.31 | 2.835 | 0.054 | 33.937 | 0.34 | 0.993 | 5.757 | 2.177 | 86.251 | 0.935 | 0.968 |
MIHS | 0.608 | 6.653 | 0.009 | 0.435 | 2.728 | 0.052 | 32.644 | 0.167 | 0.992 | 7.411 | 2.119 | 84.234 | 0.912 | 1.317 |
Brovey | 0.47 | 5.787 | 0.009 | 0.2 | 3.422 | 0.065 | 33.057 | 1.14 | 0.992 | 5.805 | 2.458 | 87.383 | 0.923 | 1.491 |
Table 7: Reported scores of spectral and spatial quality metrics for the fusion of PAN and MS bands of the LandSat ETM+ image (2000).
Fusion Algorithm | Spectral Metric | Spatial Metric | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CC | RMSE | RDM | RDS | SD | DI | PSNR | ENT | MSSIM | ERGAS | SAM | Canny edge | HPF-CC | Sobel-RMSE | |
Wavelet | 0.902 | 233.861 | -0.001 | 0.079 | 116.729 | 0.039 | 25.068 | 1.595 | 0.862 | 3.630 | 1.702 | 78.018 | 0.568 | 242.994 |
HP filter | 0.859 | 247.971 | -0.002 | -0.019 | 171.730 | 0.053 | 23.482 | 1.597 | 0.721 | 4.165 | 1.038 | 82.477 | 0.912 | 172.918 |
Ehlers | 0.761 | 297.307 | -0.005 | -0.021 | 217.913 | 0.071 | 21.050 | 1.598 | 0.554 | 5.189 | 1.182 | 85.837 | 0.917 | 211.651 |
PC | 0.737 | 350.045 | 0.003 | -0.022 | 273.701 | 0.080 | 21.218 | 1.603 | 0.601 | 5.587 | 1.537 | 88.763 | 0.922 | 162.310 |
MIHS | 0.622 | 356.016 | 0.000 | -0.010 | 269.346 | 0.094 | 18.933 | 1.597 | 0.495 | 6.797 | 1.830 | 86.992 | 0.891 | 210.725 |
Brovey | 0.524 | 320.422 | -0.001 | -0.020 | 243.866 | 0.090 | 18.856 | 3.272 | 0.517 | 5.995 | 1.347 | 76.487 | 0.604 | 3.474 |
Table 8: Reported scores of spectral and spatial quality metrics for the fusion of the PAN image of LandSat ETM+ image (2003) and the MS image of EO-1 ALI image (2004).
Fusion Algorithm | Spectral Metric | Spatial Metric | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CC | RMSE | RDM | RDS | SD | DI | PSNR | ENT | MSSIM | ERGAS | SAM | Canny edge | HPF-CC | Sobel-RMSE | |
Ehlers | 0.913 | 56.499 | -0.002 | -0.018 | 38.394 | 0.035 | 32.025 | 3.259 | 0.828 | 1.392 | 0.991 | 90.645 | 0.918 | 30.187 |
Brovey | 0.888 | 65.119 | -0.002 | -0.002 | 43.530 | 0.039 | 30.825 | 2.587 | 0.795 | 1.609 | 1.260 | 93.871 | 0.961 | 34.835 |
HP filter | 0.888 | 65.473 | -0.007 | -0.031 | 44.609 | 0.040 | 30.838 | 2.699 | 0.816 | 1.625 | 1.062 | 93.059 | 0.975 | 32.285 |
PC | 0.824 | 88.928 | -0.008 | -0.001 | 60.775 | 0.054 | 28.832 | 2.623 | 0.688 | 2.272 | 1.820 | 83.135 | 0.769 | 40.072 |
Wavelet | 0.784 | 106.411 | -0.008 | 0.052 | 67.196 | 0.062 | 27.687 | 2.628 | 0.629 | 2.810 | 2.466 | 81.107 | 0.328 | 81.044 |
MIHS | 0.777 | 102.401 | -0.009 | 0.004 | 69.801 | 0.062 | 27.799 | 2.870 | 0.627 | 2.660 | 2.254 | 83.026 | 0.722 | 43.684 |
Table 9: Reported scores of spectral and spatial quality metrics for the fusion of PAN image and visible bands of EO-1 ALI image (2004).
Fusion Algorithm | Spectral Metric | Spatial Metric | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CC | RMSE | RDM | RDS | SD | DI | PSNR | ENT | MSSIM | ERGAS | SAM | Canny edge | HPF-CC | Sobel-RMSE | |
Wavelet | 0.891 | 28.886 | -0.007 | 0.001 | 21.182 | 0.077 | 39.578 | 1.999 | 0.916 | 2.601 | 2.282 | 77.619 | 0.802 | 27.576 |
HP filter | 0.867 | 30.929 | -0.008 | -0.005 | 22.398 | 0.083 | 38.456 | 2.001 | 0.890 | 2.839 | 2.734 | 87.912 | 0.970 | 22.326 |
PC | 0.851 | 34.584 | -0.007 | -0.005 | 25.177 | 0.090 | 38.700 | 2.007 | 0.870 | 3.105 | 3.120 | 82.981 | 0.791 | 23.166 |
Ehlers | 0.751 | 36.516 | -0.008 | -0.001 | 26.268 | 0.099 | 36.157 | 2.053 | 0.855 | 3.281 | 3.007 | 83.433 | 0.924 | 20.246 |
MIHS | 0.704 | 40.565 | -0.007 | -0.007 | 29.036 | 0.108 | 35.438 | 1.995 | 0.831 | 3.615 | 2.307 | 83.295 | 0.890 | 22.303 |
Brovey | 0.578 | 41.941 | -0.007 | -0.002 | 31.559 | 0.125 | 34.143 | 2.081 | 0.847 | 4.136 | 2.859 | 90.787 | 0.959 | 23.095 |
Table 10: Reported scores of spectral and spatial quality metrics for the fusion of PAN image and MS bands of IKONOS image (2005).
From the point of visual inspections, no single algorithm was able to produce superior results by simultaneously preserving spectral and spatial properties of the original MS and PAN images. In most cases, the High-pass filter algorithm exhibited mediocre fusion results with respect to color similarity and spatial improvement. Visual inspections are necessary but alone are not sufficient; our contention is that they should always be corroborated with objective quality indices.
With respect to band-wise variations of correlation coefficient and peak-signal-to-noise ratio (Figure 8), the High-pass filter outperformed the other five algorithms in most cases. For example, in case of multiplatform scenarios (e.g., aerial (1956) - LandSat MSS (1972)), the High-pass filter algorithm reported consistently high values for CC and PSNR for all the bands and lowest values for SAM and ERGAS. This emphasizes the HPF algorithm’s ability to inject spatial structures from the high resolution aerial image to the low resolution MS image while preserving spectral and radiometric information of the MS image. When fusing PAN image of LandSat ETM+ (2003) and the MS image of ALI (2004), the Wavelet-PCA fusion algorithm exhibited high CC and PSNR values compared the HPF algorithm. In terms of SAM and ERGAS, the Wavelet-PCA algorithm was spectrally superior to the HPF algorithm. However, in general, all fusing algorithms reported notably low CC and PSNR values for NIR1 and NIR2 bands. As stated earlier, ALI sensor’s PAN image is restricted to the visible part of the spectrum (480nm - 690 nm). This limits the fusion of ALI sensor’s NIR and SWIR bands with its 10m resolution PAN image. However, the PAN image of the LandSat ETM+ (520 nm - 900 nm) expands over the visible and NIR bands of the ALI sensor. Thus, the design goal of ETM+ - ALI fusion scenario (i.e., LandSat ETM+ (2003) and ALI (2004)) was to inject high frequency information from the LandSat ETM+ image to ALI image and produce a five-band multispectral image (B,G,R,NIR1,NIR2) with 15m spatial resolution. This kind of fusion can be confronted mainly due to lack of archived data and cloud cover. We suspect that differences in sensor characteristics and radiometric resolutions of these two images might have attributed to the poor spectral quality of the fused products. When fusing PAN image and MS bands (B2, G, and R) of ALI image, Brovey transform algorithm, Ehlers algorithm, and High-pass filter algorithm exhibited equal performances for bandwise metrics and two global indicators (Figure 8). It is interesting note that the BT fusion algorithm’s improvement when only three bands are involved in fusion process. In case of IKONOS image, HPF algorithm, PC algorithm, and Wavelet-PCA, algorithm achieved notably high band-wise CC and PSNR values.
With respect to mean scores of the objective spectral quality indicators (Tables 5-10), HPF algorithm exhibited best values (see values highlighted in gray) for the majority of metrics in aerial-LandSat data fusion scenarios (Table 5 and 6) and the single-sensor fusion of LandSat ETM+ (Table 7) The Wavelet-PCA algorithm proven to be the best candidates in terms of spectral metrics in the multi-sensor fusion of LandSat ETM+ and ALI images and the single-sensor fusion IKONOS image (Table 9 and 10). The Ehlers fusion emerged as the best candidate when applied to ALI image (single-sensor fusion). The BT fusion algorithm reported the worst values for spectral metrics for the five for the six fusion scenario. This observation further emphasises the failure of BT algorithm when more than three bands are involved.
Regarding spatial quality assessment (Tables 5-10), despite the superior performances with respect to spectral similarity, wavelet- PCA algorithm exhibited poor spatial improvement while HPF and Ehlers fusion algorithms a showed mediocre spatial fidelity. Unlike for spectral quality metrics, the BT fusion algorithm achieved the best scores for spatial quality indicators. The poor spatial improvement of the wavelet-PCA algorithm is highlighted in both multi-sensor and single-sensor data fusion. Comparison of Fourier magnitude images of the original PAN and fused images further support the superiority and the inferiority of the HP fusion algorithm and the wavelet-PCA algorithm, respectively. It should be noted that the BT fusion algorithm exhibited the best scores for spatial metrics at the expense of sever spectral distortion. These observations emphasize the necessity of a combined approach (i.e., spectral and spatial fidelity) for benchmarking fusion results because the best color preservation of an algorithm can be observed even if no pan sharpening is performed; on the other hand, a fusion algorithm can achieve the best spatial improvement while producing results with the worst color preservation. Overall, scores reported for our spectral budget clearly demonstrated the superiority of spatial-domain methods (i.e., HPF algorithm and Ehlers fusion algorithm) compared to popular spectral substitution fusion techniques such as Brovey transform, MIHS, and PC.
We emphasized the importance of spatial information in the GEOBIA framework because the image segmentation process is not solely driven on per-pixel spectra but also integrates spatial and contextual characteristics when producing non-overlapping homogeneous image objects. The quality of image object candidates affects subsequent classification workflows. Figure 12 demonstrates the improvement of the quality of image objects when high frequency information of the aerial images is injected to the low resolution LandSat MSS image. Our understanding is that the fusions of two aerial images (1956 and 1986) with LandSat MSS (1972) and LandSat TM (1986) are of high value due to several reasons. The forest dieback was first documented in late 1970s, thus the fusion of aerial (1956) image and the LandSat MSS (1972) produces a high resolution MS image (15m) representing pre-dieback or early- stage dieback conditions of the HPNP. We could have spatially improved the LandSat MSS image to 10m resolution instead of 15m because the resolution ratio between the PAN image and MS image can reach up to 1:6. The second fusion scenario provides a 7m resolution MS image capturing a postdieback condition of the park. The most important reason is that we used archived data from public domains and produced useful spatially enhanced images for pre-IKONOS era (i.e., before 1999) time periods.
Fusion of two images with 20 year time difference might be questionable because in single-sensor multi-date and multi-sensor multi-date data fusion, near- contemporaneous images are desired. Due to the scarcity of decadal aerial surveys in Sri Lanka, the 1956 aerial image emerged as the best candidate to spatially enhance the 1972 LandSat MSS image. We also explored other high-spatial resolution data sources as an alternative to the 1956 aerial image. Especially KH-series declassified military intelligence imagery that is now available in public domains. KH-7 Surveillance System and the KH-9 Mapping System declassified satellite imagery consists of approximately photographic 50,000 images that were taken from 1963 to1980 of various locations around the world. Most of these images are found to be near-contemporaneous with the 1970 LandSat MSS data. However, we had to disqualify these images due to the heavy cloud cover over the HPNP.
We applied six fusion algorithms to single-sensor single-date and multi-senor multi–date images taken over the Horton Plains national park. Benchmarking of fusion algorithms was conducted visually and quantitatively, the latter based on eleven spectral and four spatial indices. From our multidimensional quality assessment, there is no fusion method that exhibited superior performances simultaneously for color preservation and spatial improvement. The HPF emerged as the best performing algorithm for single-sensor single-date and multi-sensor multi-date data fusion. Fusing high-spatial resolution panchromatic and high-spectral resolution multispectral images with complementary characteristics improves the quality of image objects and better delineates complex land cover types. Our findings shed new light on how multiple earth observation data with complimentary characteristics can be transformed into useful products in support of long-term ecosystem motoring applications.
This project is support by the ASPRS (American Society of Photogrammetry and Remote Sensing) GeoEye Award and the Alexander Goetz Award.