1.
INTRODUCTION
Metadata was introduced in order to easily find and determine geospatial datasets coming from different sources such as local, regional or national governments, private sector, etc,1. Discovery metadata focuses on the bits of information necessary to allow catalogues to manage simple queries and make data known. However once data is discovered, the answer to the question “what dataset will be more appropriated for a specific purpose?”, is related with the data quality information included in the metadata2. Providers try to convey the information about data quality into a set of comparable quality indicators describing different aspects of the dataset, including the geometric dimension, the thematic dimension and the temporal dimension3. These dimensions combined with the diversity of geospatial products results in long lists of commonly used quality indicators. ISO 19157 tries to standardize these measurements in an Annex that describes about 80 indicators. This ISO defines the quality component that is going to be measured, the individual uncertainties or errors that are going to be collected or measured and the statistics or metrics that are going to be used to create the indicator. UncertML provides a list of uncertainties that can be used for individual measurements while QualityML, collecting the statistics and metrics from ISO19157 and other sources, is used to elaborate the indicators and organizing them in a rational way4. Therfore, QualityML is a dictionary of quality measurements and an encoding system to expose them.
Sometimes, the data quality description in the producers’ metadata is not enough for the final user, who will often need a more practical approach based on user centric descriptions on previous usage experiences and associated difficulties5. The Open Geospatial Consortium has published an standard to formalize the feedback of the users about geospatial artifacts called the Geospatial User Feedback (GUF)6.
In Citizen Science, users are considered feedback providers but also data producers. In this case, the production becomes distributed. Despite the lack of a central responsible party and the heterogeneity of the participants in data collection, some simple best practices can dramatically improve the quality of the data set. Simple filtering techniques can be applied to validate data input (e.g. syntactical validation done by the data input user interfaces or usage of controlled vocabularies). More complex procedures involve the creation of a social network that reviews the data and associate a level of perceived trustworthiness of some contributors7. The lack of data quality assessment measures conducted by the responsible of the repository combined with the lack of simple data visualization tools (that make errors prominent) can also reduce the confidence in citizen science datasets8.
1.1
Data quality measures for land-cover maps
Commonly, the focus of data quality is in the precision of the position of the objects measured as one can see by the number of measures detailed by the ISO 191157 standards for geospatial quality. Land-cover (LC) maps are different in nature. Typically they define a list of land-use or land-cover classes and they cover the territory in a continuous way assigning a class name to all the space. This way, the thematic quality is as important as the positional quality and sometimes both qualities cannot be decoupled. The procedures to quality control a land-use and land-cover (LULC) map have been clearly defined and they are mainly based on comparing the map with a set of measures considered correct and collected at the ground9. The basis practice is the application of a confusion or error matrix. This matrix provides a table where the names of the classes are the titles of both rows and columns. Columns express the ground truth values and rows the categories classified as in the map. The confusion matrix provides a great deal of information, not only in terms of the average quality of the map but also on the individual classes that are more sensible to mistakes because they are more often confused with other classes. In addition, the confusion matrix is relatively easy to create by the producer and relatively easy to interpret by the user. Precisely, one problem with the confusion matrix is its nature, becoming too verbose and detailed to be easy compared. As a remedy, several indicators that can be computed from the confusion matrix have been suggested. One of the best known is the Kappa index. Even if a recent publication suggested that Kappa indices are useless and misleading for the practical applications in remote sensing10, this publication and others still propose better indices based on the confusion matrix numbers11.
This paper acknowledges the fact that previous quality indicators (e.g. based on the Kappa index distributed by the producers) can be wrong and proposes a tool that allows users to actually create their own confusion matrices, calculate the newly proposed derivative indices for the area they are interested in and publish these indices linked to the QualityML definitions.
1.2
Creating land-cover maps from OSM
The democratization of high-quality location determination using low cost GPS hardware recently integrated in smart phones has made possible all sorts of crowdsourcing, Volunteered Geographic Information (VGI) and Citizen Science projects. In fact, the contributor can focus on providing good quality thematic attributes to the positions that the GPS records. One popular project that has been around since 2004 is OpenStreetMap (OSM). OSM’s aim is to create a set of map data that is free to use, editable, and licensed under open copyright schemes (Open Database License (ODbL))12. OSM provides an open source alternative to topographic maps in a single project that covers the whole world. Each feature must have at least one attribute (tag) describing it. A tag consists in a key-value-pair (e.g. landuse = forest), where keys describe a general topic or type of an attribute (e.g. landuse) and values give a value for that key (e.g. forest). These data tags are the bases for the map available at osmlanduse.org. Since OSM tags do not directly correspond to LULC classes, there are some gaps in the resulting product that can be covered by combining the OSM data with other data sources such as free remote sensing data13. The product is available here: https://osmlanduse.org/. A second team is developing OSM2LULC as part of the open source GeoData Algorithms for Spatial Problems (GASP) Python package. The method starts with line-based OSM features that are converted into polygons and associating OSM features to a LULC class14 using decision rules. The final product is available here: https://vgi.uc.pt/expvgi.
2.
IMPLEMENTATION
Some efforts were done to propose solutions that can be applied to the citizen science dataset reusing ISO 19157 and reference datasets to calculate indicators covering in different aspects of data quality, such as the architecture proposed in the COBWeb project15. In some cases these processes are offered as chainable Web Processing services (WPS)16, but setting up a WPS and use it in a WPS client is outside of the capacities of most users. In addition, when these estimations are calculated, this information is not distributed as part of the metadata accompanying that data and it might be lost in the process of data sharing. In the Ground Truth 2.0 project, our proposal was to provide a web map client that enables the user to assess the quality of different products by itself and to choose what project would fit better its purpose.
2.1
A technical approach in the client side
In this paper, we propose a strategy for map browsers consisting of implementing raster layer visualization as a Web Map Service (WMS) request that instead of dealing with static images in PNG or JPEG format for each layer, uses AJAX to do WMS requests that response binary arrays of a specific number of bytes (for example short integers of 16 bits). The proposed solution empowers the client as it is able to store the actual values of each band in memory and then it can use JavaScript code to operate with the data in many ways. First, the browser is able to build personalized views, enhance contrast, present histograms etc. at the client side. Secondly the user is able to perform spatial filters, generate animations and graphics of a time series, and perform complex calculations using several bands from the different available datasets17. Vector layers can be requested as Web Feature Service (WFS) or Sensor Observation Service (SOS) requests receiving a GeoJSON file as a result. In the Ground Truth 2.0 this approach was expanded to calculate some quality indicators of the data.
In the case of two raster datasets, one to assess quality and the other one considered the reference data, both are retrieved as two binary arrays. The process of creating a confusion matrix starts by creating a combination of the two layers in a single composite layer which pixels contain classes that correspond to all possible permutations of the legend of the two maps. The second step consists in creating a histogram as the count of the number of pixels in each combined class of the composite layer. This one dimensional histogram is rearranged in a 2 dimensional matrix that has the classes of the first layer as rows and the classes of the second layer as columns, resulting in a confusion matrix.
In the case of a raster dataset and one vector dataset that is composed by a list of ground truth point measurements, the confusion matrix is created by determining the value in the raster file of the geometrical position of each ground truth point and presenting these values in comparison with the vector file values in the form of a matrix.
From this confusion matrix, other derived indicators can be computed. They are presented as quality calculations in the QualityML format.
2.2
The Llobregat delta area in three different maps
To demonstrate the feasibility of this methodology, we selected an area of about 433 km2 in the delta of the Llobregat river in Catalonia (North East of Spain). This area presents a big variety of classes and our research group recently made a LULC map from remote sensing, giving us a dataset to compare with. We requested the OSMlanduse and the OSM2LULC products from the Heidelberg and the Coimbra teams respectively. Both OSM derived products were using a CORINE Land cover level 2 legend but our map had a legend based on other legacy maps. The first thing we needed to perform was harmonizing the legends of the three maps by reclassifying the remote sensing based one to the nomenclatures used by the other two LULC products.
Then, we included the three maps in the MiraMon WMS server and in the MiraMon web map browser. The visualization of the map is done dynamically on the client side so it is possible to present the same map in more than one style. This characteristic was used to actually resent the data with the CORINE land cover level 1 and the CORINE land cover level 2. In Fig 1 one can see the 3 LULC maps represented with the CORINE land cover level 1. The first thing that is appreciated is that OSM is an object based map that was not designed to cover the space in a continuous way. This results in many areas that are not classified. This effect is inexistent with the LULC remote sensing based map that provides a comprehensive coverage of the selected area. Secondly, visual differences between maps that will result in non-diagonal values in the confusion matrices are easily identified.
Fig 1.
The three land-use land-cover maps to compare. a) OSM2LULC b) OSMlanduse c) remote sensing map d) legend applied to the 3 maps.
The process of creating a confusion matrix starts by requesting the combination of both maps in a single layer. In Fig. 2, the land-use map based on OSM2LULC (Coimbra version) is compared with the remote sensing based map (CREAF-RS version). The result of the combination is shown in Fig. 3. In this example we only use the first level of the CORINE legend and although the number of combinations is 25, there are only 5 colors present, corresponding to the classes that are the same in both maps.
Once the combination is done, we can request a confusion matrix. The diagonal values of the matrix (represented in green) correspond to the areas that have the same value in both maps. The non-diagonal values are the areas that have different classes in both maps. We can also see some information about the most similar classes (artificial surfaces and forest and semi natural areas) as well as the Kappa coefficient that is 0.81 (in a Kappa coefficient, the closer to 1 the better).
Preliminary analysis of the data collected detects that in general the land-use maps provided by both teams have reasonably good overall accuracy of about 80%. There are classes that present more confusions than others. For example, the urban fabric class is frequently confused with Industrial, commercial and transport units. We also have observed that sometimes validating our land-use maps with land-cover classes detects discrepancies due to the intrinsic differences between the human interpretation of classes that tend to emphasize land-use aspects, while remote sensing classes are mainly showing the land cover. For example, a big park in the city is identified as artificial in the OSM Land-use classes while it is seen as a forest area from remote sensing due to its green land-cover aspects (see Fig. 5). Actually both interpretations can be considered as correct.
2.3
The Llobregat delta area compared to ground truth data
To do this we have used a web tool generated by IIASA called LACO-Wiki (laco-wiki.net). LACO-Wiki encapsulates the process of accuracy assessment and validation into a website that presents a set of four simple steps including uploading a land-cover map (see Fig. 6), creating a sample from the map (see Fig. 7), interpreting the sample with very high resolution imagery (see Fig. 8) and generating a report with accuracy measures18. The steps 1, 2 and 4 are done by the owner of the campaign but step 3 is done by volunteers. The graphical user interface is designed in a way that it is easy, clear and simple to respond to each validation. The interface offers only one point at a time on top of the selected imagery and reports on the class that has been associated with it. The volunteer can diagnose if the classification is correct or incorrect and in case it finds it incorrect, an alternative category can be provided (see Fig. 8). After the answer is provided a new point is presented. With some practice, a single point can be done in half a minute and a set of 100 points can be processed in 30 minutes.
Fig 7.
Procedure for creating a validation samples. Validation samples are creating random, stratified or systematic sampling.
Fig 8.
A point that is classified as Urban fabric is interpreted as incorrect and class Artificial, non-agricultural vegetated areas is proposed instead.
As a result, step 4 provides a point cloud containing information on the classified categories in each land-cover map and the actual class that volunteers have interpreted from the imagery that the LACOWiki offers as a background (see Fig. 9). In addition, LACOWiki also provides an Excel file with the confusion matrix generated by the points (see Fig. 10).
3.
DISCUSSION AND CONCLUSIONS
There is not much information about data quality in the metadata that producers provide. Most of the times producers prefer to release comprehensible data quality reports that require deep knowledge of the characteristics of the data and the production process. These reports are very specific and almost impossible to compare. The approach based on quality indicators proposed by ISO19157 provides a better approach in the future. In view of this situation we should provide users with the right tools to compare datasets. One possible approach is to generate Geospatial User Feedback based on experiences of the users and share them in a platform. This paper proposes an approach to enable users to run some of the ISO19157 quality measures by themselves and to create their own indicators in a web map browser at the same time that they are inspecting the data. The indicators have the special scope of the visible area, providing results adapted to the user area of interest.
The implementation is possible by combining some functionalities added in HTML5 such as AJAX, binary arrays and canvas. Using a WMS service in a new way, it is possible to transmit arrays of real values at screen resolution to the client, enabling the client to make some analytical operations, in particular, to determine quality indicators. The paper focuses on the implementation of the confusion matrix and the derived aggregated indicators and applies them to land-cover maps produced by transformation of OSM data and from remote sensing.
The comparison between OSM derived land-cover products and remote sensing derived land cover returns generally a good match. Most discrepancies are related to the different nature of the data capturing procedure. OSM based maps are more focused on interpretation of land use while remote sensing is only sensible to land cover characteristics. Urban usages are the most affected by these discrepancies. The results suggest that a combination of OSM and remote sensing should reduce the uncertainty of an integrated product as already suggested by other authors.
Calculated quality indicators are expressed in QualityML that characterizes the dimension of the quality indicator, the individual uncertainties or errors measured and the aggregation statistics or metrics. In a future development, the formal QualityML description obtained by a user will be shared with other users of the same map browser using the Geospatial User Feedback standards implemented in NiMMbus system. The GUF standard already considers the possibility to share quality assessment results in the form of quality reports.
The results described here are applicable to confusion matrices and thematic maps but the approach is of a general use. We will continue adding quality measures to the ones already implemented in the web map browser. In particular there are some quality measures that are particularly specific for citizen science that take advantage of the redundancy created by different users that should be useful. The open source code of the MiraMon web map browser presented here can be accessed from: https://github.com/joanma747/MiraMonMapBrowser
ACKNOWLEDGEMENTS
We thank the 2 providers of the OSM land-cover map Michael Schultz (University of Heidelberg) and Cidália Fonte and Joaquim Patriarca (University of Coïmbra) the provision of the OSM land-use land-cover maps and the LACO-Wiki responsible their support in the validation website. This work has been done under the H2020 project Ground Thuth 2.0 WeObserve and NextGEOSS projects. Ground Thuth 2.0, WeObserve and NextGEOSS projects have received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 689744, No 776740 and No 730329 respectively.
