Abstract for: Evaluating Model Credibility (Quality, Validity): Concepts, Tools and Recent Developments

Typically called ‘model validity’ testing, model credibility evaluation has always been a controversial issue in system dynamics methodology, and any modeling method in general. We briefly discuss why this important notion is so controversial. To this end, we classify major types of models, particularly as they impact the notion of model credibility. The two major model types are i- Purely statistical forecasting (black box) models, and ii- Causal-descriptive policy (transparent) models. We also provide an overview of philosophical roots of the controversy and show in the process why validity is a poor choice of term. We then focus on what makes system dynamics model credibility and evaluation unique and quite different, compared to forecasting models. These properties stem from the fact that system dynamics models are causal-descriptive and they are designed for long-term policy evaluation objectives (rather than short-medium term forecasting). One important result is that system dynamics credibility consists of two different aspects: structural and behavioral. In system dynamics, establishing structure credibility must strictly precede behavior credibility; the latter has no value without the former. We thus discuss Structural tests and behavior pattern tests for system dynamics separately. Structure tests can further be classified into direct and indirect structure tests. We place special emphasis on indirect structure tests. These are special behavior tests that can provide information on potential structural flaws. Closely related with model credibility, we discuss main principles of building high quality models, so that model credibility is automatically established in modeling process. We next provide a quick overview of two different model testing software developed at our SESDYN Lab: i- SiS to assist with indirect structure testing, and ii- BTS II to facilitate behavior pattern testing. We also offer a preview of our new integrated model evaluation software. Finally we discuss some implementation issues in practice.