Prof. Begüm Demir Profile

Prof. Begüm Demir

Professor

SPIE Involvement:

Conference Program Committee | Author

Proceedings Article | 26 October 2022 Presentation + Paper

Multi-modal fusion transformer for visual question answering in remote sensing

Tim Siebert, Kai Norman Clasen, Mahdyar Ravanbakhsh, Begüm Demir

Proceedings Volume 12267, 122670L (2022) https://doi.org/10.1117/12.2636276

KEYWORDS: Remote sensing, Transformers, Feature extraction, Image analysis, Multispectral sensing, Artificial intelligence, Machine learning

Read Abstract +

With the new generation of satellite technologies, the archives of remote sensing (RS) images are growing very fast. To make the intrinsic information of each RS image easily accessible, visual question answering (VQA) has been introduced in RS. VQA allows a user to formulate a free-form question concerning the content of RS images to extract generic information. It has been shown that the fusion of the input modalities (i.e., image and text) is crucial for the performance of VQA systems. Most of the current fusion approaches use modalityspecific representations in their fusion modules instead of joint representation learning. However, to discover the underlying relation between both the image and question modality, the model is required to learn the joint representation instead of simply combining (e.g., concatenating, adding, or multiplying) the modality-specific representations. We propose a multi-modal transformer-based architecture to overcome this issue. Our proposed architecture consists of three main modules: i) the feature extraction module for extracting the modality-specific features; ii) the fusion module, which leverages a user-defined number of multi-modal transformer layers of the VisualBERT model (VB); and iii) the classification module to obtain the answer. In contrast to recently proposed transformer-based models in RS VQA, the presented architecture (called VBFusion) is not limited to specific questions, e.g., questions concerning pre-defined objects. Experimental results obtained on the RSVQAxBEN and RSVQA-LR datasets (which are made up of RGB bands of Sentinel-2 images) demonstrate the effectiveness of VBFusion for VQA tasks in RS. To analyze the importance of using other spectral bands for the description of the complex content of RS images in the framework of VQA, we extend the RSVQAxBEN dataset to include all the spectral bands of Sentinel-2 images with 10m and 20m spatial resolution. Experimental results show the importance of utilizing these bands to characterize the land-use land-cover classes present in the images in the framework of VQA. The code of the proposed method is publicly available at https://git.tu-berlin.de/rsim/multimodal- fusion-transformer-for-vqa-in-rs.

Proceedings Article | 26 October 2022 Presentation + Paper

Advanced deep learning architectures for accurate detection of subsurface tile drainage pipes from remote sensing images

Tom Lukas Breitkopf, Leonard Hackel, Mahdyar Ravanbakhsh, Anne-Karin Cooke, Sandra Willkommen, Stefan Broda, Begüm Demir

Proceedings Volume 12267, 122670O (2022) https://doi.org/10.1117/12.2636263

KEYWORDS: Remote sensing, Transformers, Artificial intelligence, Image segmentation

Read Abstract +

Proceedings Article | 15 October 2019 Paper

Approximating JPEG 2000 wavelet representation through deep neural networks for remote sensing image scene classification

Akshara Preethy Byju, Gencer Sumbul, Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 11155, 111550S (2019) https://doi.org/10.1117/12.2534643

KEYWORDS: Wavelets, Remote sensing, Scene classification, Image compression, Image classification, Neural networks, Deconvolution, Image resolution, Binary data, Image storage

Read Abstract +

Proceedings Article | 9 October 2018 Presentation + Paper

A novel active learning technique for multi-label remote sensing image scene classification

Bayable Teshome Zegeye, Begüm Demir

Proceedings Volume 10789, 107890B (2018) https://doi.org/10.1117/12.2500191

KEYWORDS: Remote sensing, Image classification, Scene classification, Image analysis

Read Abstract +

This paper presents a novel multi-label active learning (MLAL) technique in the framework of multi-label remote sensing (RS) image scene classification problems. The proposed MLAL technique is developed in the framework of the multi-label SVM classifier (ML-SVM). Unlike the standard AL methods, the proposed MLAL technique redefines active learning by evaluating the informativeness of each image based on its multiple land-cover classes. Accordingly, the proposed MLAL technique is based on the joint evaluation of two criteria for the selection of the most informative images: i) multi-label uncertainty and ii) multi-label diversity. The multi-label uncertainty criterion is associated to the confidence of the multi-label classification algorithm in correctly assigning multi-labels to each image, whereas multi-label diversity criterion aims at selecting a set of un-annotated images that are as more diverse as possible to reduce the redundancy among them. In order to evaluate the multi-label uncertainty of each image, we propose a novel multi-label margin sampling strategy that: 1) considers the functional distances of each image to all ML-SVM hyperplanes; and then 2) estimates the occurrence on how many times each image falls inside the margins of ML-SVMs. If the occurrence is small, the classifiers are confident to correctly classify the considered image, and vice versa. In order to evaluate the multi-label diversity of each image, we propose a novel clustering-based strategy that clusters all the images inside the margins of the ML-SVMs and avoids selecting the uncertain images from the same clusters. The joint use of the two criteria allows one to enrich the training set of images with multi-labels. Experimental results obtained on a benchmark archive with 2100 images with their multi-labels show the effectiveness of the proposed MLAL method compared to the standard AL methods that neglect the evaluation of the uncertainty and diversity on multi-labels.

Proceedings Article | 9 October 2018 Presentation + Paper

A novel coarse-to-fine remote sensing image retrieval system in JPEG-2000 compressed domain

Akshara Preethy Byju, Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 10789, 107890T (2018) https://doi.org/10.1117/12.2327051

KEYWORDS: Image retrieval, Remote sensing, Image compression, Feature extraction, Wavelets, Image processing

Read Abstract +

Proceedings Article | 4 October 2017 Presentation + Paper

A novel class sensitive hashing technique for large-scale content-based remote sensing image retrieval

Thomas Reato, Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 10427, 1042712 (2017) https://doi.org/10.1117/12.2279407

KEYWORDS: Image retrieval, Remote sensing, Feature extraction, Image storage, Binary data

Read Abstract +

Proceedings Article | 18 October 2016 Presentation + Paper

Domain adaptation based on deep denoising auto-encoders for classification of remote sensing images

Emanuele Riz, Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 10004, 100040K (2016) https://doi.org/10.1117/12.2241982

KEYWORDS: Remote sensing, Image classification, Denoising, Multispectral imaging, Computer programming, Statistical analysis, Detection and tracking algorithms, Error analysis, Image resolution, Solid state lighting

Read Abstract +

Proceedings Article | 15 October 2015 Paper

Compressed histogram attribute profiles for the classification of VHR remote sensing images

Romano Battiti, Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 9643, 96430R (2015) https://doi.org/10.1117/12.2195988

KEYWORDS: Image classification, Image filtering, Remote sensing, Quantization, Image compression, Feature extraction, Buildings, Image resolution, Data modeling, Filtering (signal processing)

Read Abstract +

This paper presents a novel compressed histogram attribute profile (CHAP) for classification of very high resolution remote sensing images. The CHAP characterizes the marginal local distribution of attribute filter responses to model the texture information of each sample with a small number of image features. This is achieved based on a three steps algorithm. The first step is devoted to provide a complete characterization of spatial properties of objects in a scene. To this end, the attribute profile (AP) is initially built by the sequential application of attribute filters to the considered image. Then, to capture complete spatial characteristics of the structures in the scene a local histogram is calculated for each sample of each image in the AP. The local histograms of the same pixel location can contain redundant information since: i) adjacent histogram bins can provide similar information; and ii) the attributes obtained with similar attribute filter threshold values lead to redundant features. In the second step, to point out the redundancies the local histograms of the same pixel locations in the AP are organized into a 2D matrix representation, where columns are associated to the local histograms and rows represents a specific bin in all histograms of the considered sequence of filtered attributes in the profile. This representation results in the characterization of the texture information of each sample through a 2D texture descriptor. In the final step, a novel compression approach based on a uniform 2D quantization strategy is applied to remove the redundancy of the 2D texture descriptors. Finally the CHAP is classified by a Support Vector Machine classifier with histogram intersection kernel that is very effective for high dimensional histogram-based feature representations. Experimental results confirm the effectiveness of the proposed CHAP in terms of computational complexity, storage requirements and classification accuracy when compared to the other AP-based methods.

Proceedings Article | 15 October 2014 Paper

A semisupervised support vector regression method to estimate biophysical parameters from remotely sensed images

Davide Castelletti, Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 9244, 924412 (2014) https://doi.org/10.1117/12.2069607

KEYWORDS: Statistical analysis, Solid state lighting, Biological research, Remote sensing, Detection and tracking algorithms, Statistical modeling, LIDAR, Copper, Soil science, Control systems

Read Abstract +

Proceedings Article | 8 November 2012 Paper

A novel active learning method for support vector regression to estimate biophysical parameters from remotely sensed images

Begüm Demir, Lorenzo Bruzzone

Proceedings Volume 8537, 85370L (2012) https://doi.org/10.1117/12.979666

KEYWORDS: Statistical analysis, Error analysis, Biological research, Remote sensing, LIDAR, Detection and tracking algorithms, Signal processing, Image classification, Image processing, Machine learning

Read Abstract +

This paper presents a novel active learning (AL) technique in the context of ε-insensitive support vector regression (SVR) to estimate biophysical parameters from remotely sensed images. The proposed AL method aims at selecting the most informative and representative unlabeled samples which have maximum uncertainty, diversity and density assessed according to the SVR estimation rule. This is achieved on the basis of two consecutive steps that rely on the kernel kmeans clustering. In the first step the most uncertain unlabeled samples are selected by removing the most certain ones from a pool of unlabeled samples. In SVR problems, the most uncertain samples are located outside or on the boundary of the ε-tube of SVR, as their target values have the lowest confidence to be correctly estimated. In order to select these samples, the kernel k-means clustering is applied to all unlabeled samples together with the training samples that are not SVs, i.e., those that are inside the ε-tube, (non-SVs). Then, clusters with non-SVs inside are rejected, whereas the unlabeled samples contained in the remained clusters are selected as the most uncertain samples. In the second step the samples located in the high density regions in the kernel space and as much diverse as possible to each other are chosen among the uncertain samples. The density and diversity of the unlabeled samples are evaluated on the basis of their clusters’ information. To this end, initially the density of each cluster is measured by the ratio of the number of samples in the cluster to the distance of its two furthest samples. Then, the highest density clusters are chosen and the medoid samples closest to the centers of the selected clusters are chosen as the most informative ones. The diversity of samples is accomplished by selecting only one sample from each selected cluster. Experiments applied to the estimation of single-tree parameters, i.e., tree stem volume and tree stem diameter, show the effectiveness of the proposed technique.

Proceedings Article | 27 October 2011 Paper

A change-detection-driven approach to active transfer learning for classification of image time series

Begüm Demir, Francesca Bovolo, Lorenzo Bruzzone

Proceedings Volume 8180, 81800E (2011) https://doi.org/10.1117/12.898596

KEYWORDS: Remote sensing, Image processing, Image classification, Target detection, Target acquisition, Data acquisition, Multispectral imaging, Statistical analysis, Actinium, Image analysis

Read Abstract +

SPIE Journal Paper | 1 July 2008

Downsampling-based multiple description image coding using optimal filtering

Yüksel Yapici, Begüm Demir, Sarp Ertürk, Oguzhan Urhan

JEI, Vol. 17, Issue 03, 033018, (July 2008) https://doi.org/10.1117/12.10.1117/1.2976420

KEYWORDS: Optimal filtering, Image compression, Image filtering, Receivers, Quantization, Brain-machine interfaces, Filtering (signal processing), Computer programming, Transform theory, Wavelets

Read Abstract +

Showing 5 of 12 publications

Conference Committee Involvement (11)

Artificial Intelligence and Image and Signal Processing for Remote Sensing XXX

16 September 2024 | Edinburgh, United Kingdom

Image and Signal Processing for Remote Sensing XXIX

4 September 2023 | Amsterdam, Netherlands

Image and Signal Processing for Remote Sensing XXVIII

5 September 2022 | Berlin, Germany

Image and Signal Processing for Remote Sensing XXVII

13 September 2021 | Online Only, Spain

Image and Signal Processing for Remote Sensing XXVI

21 September 2020 | Online Only, United Kingdom

Showing 5 of 11 Conference Committees

Keywords/Phrases

Search In:

Publication Years