Validation

Introduction

Uncertainty in modeling can originate from several sources, including incomplete knowledge of human-natural systems, misconception about the main system processes, lack of data, structural limits of the model itself, the impossibility of finding an optimal set of parameters able to cover multiple space-time resolutions and spans, level of stochasticity, and/or an incomplete statistical/heuristic learning process (Pontius Jr. and Spencer, 2005; Messina et al., 2008; Pontius and Neeti, 2010; Pontius Jr. and Petrova, 2010).

The terms verification and validation have often been confused and used interchangeably (Oreskes et al., 1994; Rykiel, 1996). Verification checks the ability of a model to meet the needs of the final user and attempts to identify internal logical/computational errors, while validation measures a model’s ability to represent cause-effect relations in site-specific contexts (Coquillard and Hill, 1997).

Validation refers to the comparison between model predictions and observations stored in the dataset that were not used to train the model. Thus, during validation the calibrated model is fed with new values, independent from the training dataset. Training and validation datasets are commonly created by randomly splitting observational data into two sub-sets, e.g. 80% of data for training and 20% for validation (Chung and Fabbri, 2003; Villa-Vialaneix et al., 2012), or by using time partitioning (i.e. calibration employs older records or time span, while validation uses the most recent records or time span (Pontius Jr. and Petrova, 2010).

Choosing the best goodness-of-fit criterion for validation depends strictly on data type and quality, project goals, and required accuracy. Even if model output is consistent with the training and test datasets, there is no guarantee that the model will perform equally well when tested with a different dataset for predictive purposes. For example, information captured by historical data could contain errors (Pontius Jr. and Petrova, 2010), or may be outdated and thus unrepresentative of present and future system dynamics.  Alternatively, a paucity of data may require selection of calibration and validation time intervals much shorter than the future period the model aims to predict (Pontius and Neeti, 2010). Finally, uncertainty surrounds whether the employed validation method is truly measuring model accuracy (Hagen-Zankera and Lajoie, 2008).

Spatial models require comparison within a neighborhood context, because maps that do not match exactly cell-by-cell may still present similar spatial patterns and therefore spatial agreement within a given cell vicinity. To address this issue, several vicinity-based comparison methods have been developed. For example, Costanza (1989) introduced the multiple-resolution fitting procedure that compares map spatial fitness within increasing window sizes. Power et al. (2001) provided the Fuzzy Inference System, a method based on hierarchical fuzzy pattern matching. Pontius Jr. (2000, 2002) introduced Klocation, which differentiates errors due to location and quantity. Hagen et al. (2003) developed the Kfuzzy, considered equivalent to the Kappa statistic, and the Fuzzy Similarity, which accounts for fuzziness of location and category within a cell neighborhood. Almeida et al. (2008) modified the latter and named it Reciprocal Similarity Comparison, because this metric corresponds to the minimum fuzzy similarity between map 1 of changes versus map 2 of changes and vice versa (Soares-Filho et al. 2009), (Fig. 1). Van Vliet et al. (2011) developed the Kappa Simulation, which assesses the agreement between the simulated land-use map and the actual one, based on the original land-use map.

We evaluated six map comparison methods using a set of synthetic maps (Fig. 2).  The best methods are the ones that yield the highest contrast between the reference map (Final) and a randomly blurred pattern (neutral model), but also capture similitude between maps that closely resemble each others, such as the maps of line1 x line2 (table 1). The best method is the Reciprocal Similarity Comparison, with exponential decay function truncated at 19×19-window size. This method was chosen because it focuses only on the goodness-of-fit of the changes rather than the whole map, clearly distinguishes well-matched spatial patterns from a neutral model (a random map with same number of changes; probability for all cells =0.5), and is virtually independent of window size of comparison if large windows are employed.

Conclusion

Most comparison metrics are not mutually comparable, nor are the scores of a specific metric comparable when applied to different models, given their different contexts (Hagen-Zankera and Lajoie, 2008). Our selected metric intrinsically incorporates a neutral model of permanence and can be used against a neutral model of random allocation to test whether the model predicts better than chance. Yet, benchmark for accepting or rejecting a model remains an open question.