Section 2.10 presents a case study for applying VV&UQ methods to climate models. The introductory paragraph of that section reads,
The previous discussion noted that uncertainty is pervasive in models of real-world phenomena, and climate models are no exception. In this case study, the committee is not judging the validity or results of any of the existing climate models, nor is it minimining the successes of climate modeling. The intent is only to discuss how VVUQ methods in these models can be used to improve the reliability of the predictions that they yield and provide a much more complete picture of this crucial scientific arena.
As noted in the front-matter, since this is a pre-print it is still subject to editorial revision.
Section 5.7 is also interesting. The report notes that an "ensemble of [global climate] models is a sample of convenience, and dependencies between different computational models are typically not accounted for in the statistical modeling. Interestingly, the estimated prediction uncertainty typically decreases as the ensemble size increases. It is not at all clear that this should be the case. One can argue that even if infinitely many models could be sampled, future climate would still not be perfectly understood because of our limited knowledge of climate physics. However, the hierarchical modeling approach is a first step for developing more realistic prediction uncertainties using MMEs. Research is needed to establish the connection between model-to-model differences and model-to-reality differences." [Emphasis added.]
ReplyDeleteThe report suggests further research will result in "improved methods for constructing ensembles of models, analysis of interdependence among models, assessment of confidence in particular models and their predictive power, and use of information-theoretic and statistical means for developing robust and reliable methods for model comparison, selection, and averaging/pooling." [Emphasis added.]
I sincerely hope such polite skepticism and constructive recommendations will be positively received by the global climate model community.
Lots of the folks in the Acknowledgments section of this report were at the NDV&V workshop. We talked a bit (a little bit, because everyone was trying very hard to be polite) about climate models and how VV&UQ was being done (or not). Even had one presentation from a climate modeler; the response was polite, and he acknowledged the limitations of what has been done to date in the field as far as VV&UQ.
ReplyDeleteBTW George, I noticed that comments on your site are restricted now; problems with spam?
Thanks Josh. Somehow my Google Blogger settings had gotten all glitched up. Hopefully updating them was successful.
ReplyDeleteI found this last bullet from the conclusion slide ("VUQ Risks and Challenges") of a VVUQ presentation pretty interesting:
ReplyDelete* Social: perception that VUQ is adversarial
** Decision-makers require a different level of evidence of correctness than scientific peers