Description: The U.S. Environmental Protection Agency (EPA) is one of several federal agencies responsible for protecting Americans against significant risks to human health and the environment. As part of that mission, EPA estimates the nature, magnitude, and likelihood of risks to human health and the environment; identifies the potential regulatory actions that will mitigate those risks and protect public health1 and the environment; and uses that information to decide on appropriate regulatory action. Uncertainties, both qualitative and quantitative, in the data and analyses on which these decisions are based enter into the process at each step. As a result, the informed identification and use of the uncertainties inherent in the process is an essential feature of environmental decision making.
Tuesday, February 12, 2013
Environmental Decisions in the Face of Uncertainty
Thursday, January 3, 2013
National Strategy for Advancing Climate Modeling
NAP has a new report out on a national strategy for advancing climate modeling. I've just done a quick skim so far. Some tidbits below that touch on V&V and uncertainty.
Friday, March 30, 2012
Empirical Imperatives
There is a belief that you can simply read imperatives from ‘the evidence’, and to organise society accordingly, as if instructed by mother nature herself. And worse still, there is reluctance on behalf of many engaged in the debate to recognise that this very technocratic, naturalistic and bureaucratic way of looking at the world reflects very much a broader tendency in contemporary politics. To point any of these problems out is to ‘deny the science’. ‘Science’, then, is a gun to the head.Shrinking the Sceptics
Sunday, May 8, 2011
Storms of Our Grandfathers
Are we "rolling 13s" and getting thousand year storms every year?
 |
| NOAA April 2011 Precipitation Anomaly |
The contour plots below are taken from Theory of the hydraulic jump and backwater curves. These studies of historical storm records were used to inform design decisions for the Miami Valley Conservancy District's retarding basins and channel improvements following the 1913 floods. My previous post has pictures of the hydraulic jump below Huffman Dam in operation.
My question to Dr Curry about what value high-fidelity (read: relatively expensive to run and analyze) climate simulations have for decision makers was motivated by reading up on infrastructure projects like the retarding basins and channel improvements in the Miami Valley. I think it would be interesting to take a look at a historical project like this that included rudimentary analysis of climate (weather event frequency and magnitude) in its design, and say, "here's how it would be informed differently using modern tools."
The design philosophy taken by the engineers working for the Miami Valley Conservancy District was to design for the worst possible case (historical records from Europe were also considered since they went back further and more reliably) plus roughly twenty percent margin due to the inherent uncertainty in estimating the worst possible case.
If it were necessary to depend wholly on the records of storms which have occurred in the United States, it might be thought possible for moderately great storms to occur over a period of a few hundred years, and then to find, as an exception, a storm three or four times as great. Theoretically that is very improbable, simply because water vapor in sufficient quantities cannot be transported from the ocean or gulf fast and long enough to cause such exceptional storms. As stated in chapter XI, however, records were collected of the stages of rivers in Europe for long periods of time, and these furnish fairly conclusive proof that such great exceptional storms actually do not occur. On the Danube at Vienna, for instance, we have records since about the year 1000 A.D.; fairly accurate records are available for stages of floods in the Tiber at Rome for more than 2,000 years; and records have been made of floods on the Seine at Paris for a long period of years.
Relation of Great Storms to Maximum Possible
After making the extensive investigation of storms in the eastern United States, it is believed that the March, 1913, flood is one of the great floods of centuries in the Miami Valley. In the course of three or four hundred years, however, a flood 15 or 20 per cent greater may occur. We do not believe a flood will ever occur which is more than 20 or 25 per cent in excess of that of March 1913. There is a factor of ignorance, however, against which we must provide, and the only way to do this is arbitrarily to increase the size of the maximum flood to be provided for. If longer records were available a closer estimate could be made, but in planning works on which the protection of the Miami Valley depends, it is necessary to go beyond human judgment. This has been done on all the other phases of the design, and we believe it would not be good engineering practice to stop at our judgment on this phase. We must be able to say that the engineering works are absolutely safe in every respect. For this reason provision is made for a flood nearly 40 per cent greater than that of March 1913. This is 15 or 20 per cent in excess of what is believed to be the greatest possible flood that will ever occur.Would modern tools cut the design margin due to reduced uncertainty or would they indicate that the project is now under-designed due to projected climate change? The latter seems unlikely considering that the magnitude of the purported effects has been repeatably shown to be smaller than we can reliably detect given the length of our data record. Would there be any practically significant changes to the decisions and designs? If your system already has sufficient margin for projected changes in weather-event magnitude do projected changes in frequency matter?
Reasons for Choosing as a Basis for Design a Flood 40% Greater than that of March 1913
Friday, May 6, 2011
Technocrats and Philosopher Kings can Save our Impotent Polity
This metaphysical confusion runs throughout Mooney’s argument. For Mooney, ‘ideology’ is some insidious, toxic force, the antithesis to ‘truth’ itself. The thrust of his argument is that we need particular scientific institutions to ameliorate this intrinsic weakness of human nature. And as such, these institutions deserve elevated status above the reach of those prone to ideology. Otherwise, we would tend towards creationism, to MMR-scares, to climate-change denial. In other words, our flawed minds would create a catastrophe, and it is this possibility of catastrophe that seemingly legitimises the elevated position of scientific institutions. Mooney reinvents Plato’s city state administrated by Philosopher Kings, the main differences being that Mooney conceives of a global polity, and the wisdom of the Guardians only produces the possibility of mere survival, not even a better way of life. To bring this back the matter of trust, Mooney doesn’t trust humans. Their minds are flawed. Their ambitions and ideas are mere fictions. The institutions they create are accordingly founded on false premises, which, instituted and acted upon, will cause disaster. Even when humans are exposed to ‘the truth’, it is, on Mooney’s view, absorbed into the poisonous, ideological programmes of partisans: liars and cheats who distort it. But without a disaster looming, this instance of a politics of fear would collapse.
He simply can’t make a popular argument for his political idea, and so turns to ‘science’ to identify the necessity of such a programme — i.e. the crisis — and to identify reasons why conventional democratic processes cannot realise it...It's always a good day when you can throw a little Plato into the mix ; - )
Tuesday, February 15, 2011
Comments on Spatio-Temporal Chaos
If one is optimistic, one may hope that the asymptotic measures will play for dissipative systems the sort of role which the Gibbs ensembles have played for statistical mechanics. Even if that is the case, the difficulties encountered in statistical mechanics in going from Gibbs ensembles to a theory of phase transitions may serve as a warning that we are, for dissipative systems, not yet close to a real theory of turbulence.What Are the Measures Describing Turbulence?
Saturday, January 22, 2011
Recurrence, Averaging and Predictability
Motivation and Background
Yet another installment in the Lorenz63 series. This time motivated by a commenter on Climate Etc. Tomas Milanovic claims that time averages are chaotic too in response to the oft repeated claim that the predictability limitations of nonlinear dynamical systems are not a problem in the case of climate prediction. Lorenz would seem to agree, “most climatic elements, and certainly climatic means, are not predictable in the first sense at infinite range, since a non-periodic series cannot be made periodic through averaging [1].” We’re not going to just take his word on it. We’ll see if we can demonstrate this with our toy model.
That’s the motivation, but before we get to toy model results a little background discussion is in order. In this previous entry I illustrated the different types of functionals that you might be interested in depending on whether you are doing weather prediction or climate prediction. I also made the remark, “A climate prediction is trying to provide a predictive distribution of a time-averaged atmospheric state which is (hopefully) independent of time far enough into the future.” It was pointed out to me that this is a testable hypothesis [2], and that the empirical evidence doesn’t seem to support the existence of time-averages (or other functionals) describing the Earth’s climate system that are independent of time [3]. In fact, the above assumption was critiqued by none other than Lorenz in 1968 [4]. In that paper he states,
Questions concerning the existence and uniqueness of long-term statistics fall into the realm of ergodic theory. [...] In the case of nonlinear equations, the uniqueness of long-term statistics is not assured. From the way in which the problem is formulated, the system of equations, expressed in deterministic form, together with a specified set of initial conditions, determines a time-dependent solution extending indefinitely into the future, and therefore determines a set of long-term statistics. The question remains as to whether such statistics are independent of the choice of initial conditions.
He goes on to define a system as transitive if the long-term statistics are independent of initial condition, and intransitive if there are “two or more sets of long-term statistics, each of which has a greater-than-zero probability of resulting from randomly chosen initial conditions.” Since the concept of climate change has no meaning for statistics over infinitely long intervals, he then defines a system as almost intransitive if the statistics at infinity are unique, but the statistics over finite intervals depend (perhaps even sensitively) on initial conditions. In the context of policy relevance we are generally interested in behavior over finite time-intervals.
In fact, from what I’ve been able to find, different large-scale spatial averages (or coherent structures, which you could track by suitable projections or filtering) of state for the climate system face similar limits to predictability as un-averaged states. The predictability just decays at a slower rate. So instead of predictive limitations for weather-like functionals on the order of a few weeks, the more climate-like functionals become unpredictable on slower time-scales. There’s no magic here, things don’t suddenly become predictable a couple decades or a century hence because you take an average. It’s just that averaging or filtering may change the rate that errors for that functional grow (because in spatio-temporal chaos different structures, or state vectors, will have different error growth rates and reach saturation at different times). Again Lorenz puts it well, “the theory which assures us of ultimate decay of atmospheric predictability says nothing about the rate of decay” [1]. Recent work shows that initialization matters for decadal prediction, and that the predictability of various functionals decay at different rates [5]. For instance, sea surface temperature anomalies are predictable at longer forecast horizons than surface temperatures over land. Hind-casts of large spatial averages on decadal time-scales have shown skill in the last two decades of the past century (though they had trouble beating a persistence forecast for much of the rest of the century) [6].
I’ve noticed in on-line discussions about climate science that some people think that the problem of establishing long term statistics for nonlinear systems is a solved one. That is not the case for the complex, nonlinear systems we are generally most interested in (there are results for our toy though [7, 8]). I think this snippet sums things up well,
Atmospheric and oceanic forcings are strongest at global equilibrium scales of 107 m and seasons to millennia. Fluid mixing and dissipation occur at micorscales of 10-3 m and 10-3s, and cloud particulate transformations happen at 10-6 m or smaller. Observed intrinsic variability is spectrally broad band across all intermediate scales. A full representation for all dynamical degrees of freedom in different quantities and scales is uncomputable even with optimistically foreseeable computer technology. No fundamentally reliable reduction of the size of the AOS [atmospheric oceanic simulation] dynamical system (i.e., a statistical mechanics analogous to the transition between molecular kinetics and fluid dynamics) is yet envisioned. [9]
Here McWilliams is making a point similar to that made by Lorenz in [4] about establishing a statistical mechanics for climate. This would be great if it happened, because that would mean that the problem of turbulence would be solved for us engineers too. Right now the best we have (engineers interested in turbulent flows and climate scientists too) is empirically adequate models that are calibrated to work well in specific corners of reality.
Lorenz was responsible for another useful concept concerning predictability, that is predictability of the first and second kind [1]. If you care about the time-accurate evolution of the order of states then you are interested in predictability of the first kind. If, however, you do not care about the order, but only the statistics, then you are concerned with predictability of the second kind. Unfortunately, Lorenz’s concepts of first and second kind predictability have been morphed in to a claim that first kind predictability is about solving initial value problem (IVP)s and second kind predictability is about solving boundary value problem (BVP)s. For example, “Predictability of the second kind focuses on the boundary value problem: how predictable changes in the boundary conditions that affect climate can provide predictive power [5].” This is unsound. If you read Lorenz closely, you’ll see that the important open question he was exploring about whether the climate is transitive, intransitive or almost intransitive has been assumed away by the spurious association of kinds of predictability with kinds of problems [1]. Lorenz never made this mistake, he was always clear that the difference in kinds of predictability depends on the functionals you are interested in, not whether it is appropriate to solve an IVP or a BVP (what reason could you have for expecting meaningful frequency statistics from a solution to a BVP?). Those considerations depend on the sort of system you have. In an intransitive or almost intransitive system even climate-like functionals depend on the initial conditions.
A good early paper on applying information theory concepts to climate predictability is by Leung and North [10], and there is a more recent review article that covers the basic concepts by DelSole and Tippett [11].
Recurrence Plots
Recurrence plots are useful for getting a quick qualitative feel for the type of response exhibited by a time-series [12, 13]. First we run a little initial condition (IC) ensemble with our toy model. The computer experiment we’ll run to explore this question will consist of perturbations to the initial conditions (I chose the size of the perturbation so the ensemble would blow-up around t = 12). Rather than sampling from a distribution for the members of the ensemble, I chose them according a stochastic collocation (this helps in getting the same results every time too).
One thing that these two plots makes clear is that it doesn’t make much sense to compare individual trajectories with the ensemble mean. The mean is a parameter of a distribution describing a population of which the trajectories are members. While the trajectories are all orbits on the attractor, the mean is not.
Comparing the chaotic recurrence plots with the plots below of a periodic series and a stochastic series illustrates the qualitative differences in appearance.
Clearly, both the ensemble mean and the individual trajectory are chaotic series, sort of “between” periodic and stochastic in their appearance. Ensemble averaging doesn’t make our chaotic series non-chaotic, what about time averaging?
Predictability Decay
How does averaging affect the decay of predictability for the state of the Lorenz63 system, and can we measure this effect? We can track how the predictability of the future state decays given knowledge of the initial state by using the relative entropy. There are other choices for measures such as mutual information [10]. Since we’ve already got our ensemble though, we can just use entropy like we did before. Rather than just a simple moving average, I’ll be calculating an exponentially weighted one using an FFT-based approach, of course (there’s some edge effects we’d need to worry about if this were a serious analysis, but we’ll ignore that for now). The entropy for the ensemble is shown for three different smoothing levels in Figure 4 (the high entropy prior to t = 5 for the smoothed series is spurious because I didn’t pad the series and it’s calculated with the FFT).
While smoothing does lower the entropy of the ensemble (lower entropy for more smoothing / smaller λ), it still experiences the same sort of “blow-up” as the unsmoothed trajectory. This indicates problems for predictability even for our time-averaged functionals. Guess what? The recurrence plot indicates that our smoothed trajectory is still chaotic!
This result shouldn't be too surprising, moving averages or smoothing (of whatever type you fancy) are linear operations. It would probably take a pretty clever nonlinear transformation to turn a chaotic series into a non-chaotic one (think about how the series in this case is generated in the first place). I wouldn't expect any combination of linear transformations to accomplish that.
Conclusions
I’ll begin the end with another great point from McWilliams (though I’ve not heard of sub-grid fluctuations referred to as “computational noise,” that term makes me think of round-off error) that should serve to temper our demands of predictive capability from climate models[9]:
Among their other roles, parametrizations regularize the solutions on the grid scale by limiting fine-scale variance (also known as computational noise). This practice makes the choices of discrete algorithms quite influential on the results, and it removes the simulation from the mathematically preferable realm of asymptotic convergence with resolution, in which the results are independent of resolution and all well conceived algorithms yield the same answer.
If I had read this earlier, I wouldn’t have spent so much time searching for something that doesn’t exist.
Regardless of my tortured learning process, what do the toy models tell us? Our ability to predict the future is fundamentally limited. Not really an earth-shattering discovery; it seems a whole lot like common sense. Does this have any implication for how we make decisions? I think it does. Our choices should be robust with respect to these inescapable limitations. In engineering we look for broad optimums that are insensitive to design or requirements uncertainties. The same sort of design thinking applies to strategic decision making or policy design. The fundamental truism for us to remember in trying to make good decisions under the uncertainty caused by practical and theoretical constraints is that limits on predictability do not imply impotence.
References
[1] Lorenz, E. N., The Physical Basis of Climate and Climate Modeling, Vol. 16 of GARP publication series, chap. Climatic Predictability, World Meteorological Organization, 1975, pp. 132–136.
[2] Pielke Sr, R. A., “your query,” September 2010, electronic mail to the author.
[3] Rial, J. A., Pielke Sr, R. A., Beniston, M., Claussen, M., Canadell, J., Cox, P., Held, H., Noblet-Ducoudr, N. D., Prinn, R., Reynolds, J. F., and Salas, J. D., “Nonlinearities, Feedbacks And Critical Thresholds Within The EarthS Climate System,” Climatic Change, Vol. 65, No. 1-2, 2004, pp. 11–38.
[4] Lorenz, E. N., “Climatic Determinism,” Meteorological Monographs, Vol. 8, No. 30, 1968.
[5] Collins, M. and Allen, M. R., “Assessing The Relative Roles Of Initial And Boundary Conditions In Interannual To Decadal Climate Predictability,” Journal ofClimate, Vol. 15, No. 21, 2002, pp. 3104–3109.
[6] Lee, T. C., Zwiers, F. W., Zhang, X., and Tsao, M., “Evidence of Decadal Climate Prediction Skill Resulting from Changes in Anthropogenic Forcing,” Journal of Climate, Vol. 19, 2006.
[7] Tucker, W., The Lorenz Attractor Exists, Ph.D. thesis, Uppsala University, 1998.
[8] Kehlet, B. and Logg, A., “Long-Time Computability of the Lorenz System,”http://lorenzsystem.net/.
[9] McWilliams, J. C., “Irreducible Imprecision In Atmospheric And Oceanic Simulations,” Vol. 104 of National Academy of Sciences, National Academy of Sciences, pp. 8709 – 8713.
[10] Leung, L.-Y. and North, G. R., “Information Theory and Climate Prediction,”Journal of Climate, Vol. 3, 1990, pp. 5–14.
[11] DelSole, T. and Tippett, M. K., “Predictability: Recent insights from information theory,” Reviews of Geophysics, Vol. 45, 2007.
[12] Eckmann, J.-P., Kamphorst, S. O., and Ruelle, D., “Recurrence Plots of Dynamical Systems,” EPL (Europhysics Letters), Vol. 4, No. 9, 1987, pp. 973.
[13] Marwan, N., “A historical review of recurrence plots,” The European PhysicalJournal - Special Topics, Vol. 164, 2008, pp. 3–12, 10.1140/epjst/e2008-00829-1.
Wednesday, January 19, 2011
Empiricism and Simulation
There are two orthogonal ideas that seem to get conflated in discussions about climate modeling. One is the idea that you’re not doing science if you can’t do a controlled experiment, but of course we have observational sciences like astronomy. The other is that all this new-fangled computer-based simulation is untrustworthy, usually because “it ain’t the way my grandaddy did science.” Both are rather silly ideas. We can still weigh the evidence for competing models based on observation, and we can still find protection from fooling ourselves even when those models are complex.
What does it mean to be an experimental as opposed to an observational science? Do sensitivity studies, and observational diagnostics using sophisticated simulations count as experiments? Easterbrook claims that because climate scientists do these two things with their models that climate science is an experimental science [1]. It seems like there is a motivation to claim the mantle of experimental, because it may carry more rhetorical credibility than the merely observational (the critic Easterbrook is addressing certainly thinks so). This is probably because the statements we can make about causality and the strength of the inferences we can draw are usually greater when we can run controlled experiments than when we are stuck with whatever natural experiments fortune provisions for us (and there are sound mathematical reasons for this, having to do with optimality in experimental design rather than any label we may place on the source of the data). This seeming motivation demonstrated by Easterbrook to embrace the label of empirical is in sharp contrast to the denigration of the empirical by Tobis in his three part series [2, 3, 4]. As I noted on his site, the narrative Tobis is trying to create with those posts has already been pre-messed with by Easterbrook, his readers just pointed out the obvious weaknesses too. One good thing about blogging is the critical and timely feedback.
The confusions of these two climate warriors are an interesting point of departure. I think they are both saying more than blah blah blah, so it’s worth trying to clarify this issue. The figure below is based on a technical report from Sandia [5], which is a good overview and description of the concepts and definitions for model verification and validation as it has developed in the computational physics community over the past decade or so. I think this emerging body of work on model V&V places the relative parts, experiment and simulation, in a sound framework for decision making and reasoning about what models mean.
|
The process starts at the top of the flowchart with a “Reality of Interest”, from which a conceptual model is developed. At this point the path splits into two main branches. One based on “Physical Modeling” and the other based on “Mathematical Modeling”. Something I don’t think many people realize is that there is a significant tradition of modeling in science that isn’t based on equations. It is no coincidence that an aeronautical engineer might talk of testing ideas with a wind-tunnel model or a CFD model. Both models are simplifications of the reality of interest, which, for that engineer, is usually a full-scale vehicle in free flight.
Figure 2 is just a look at the V&V process through my Design of Experiments (DoE) colored glasses.
My distorted view of the V&V process is shown to emphasize that there’s plenty of room for experimentalists to have fun (maybe even a job [3]) in this, admittedly model-centric, sandbox. However, the transferability of the basic experimental design skills between “Validation Experiments” and “Computational Experiments” says nothing about what category of science one is practicing. The method of developing models may very well be empirical (and I think Professor Easterbrook and I would agree it is, and maybe even should be), but that changes nothing about the source of the data which is used for “Model Validation.”
The computational experiments highlighted in Figure 2 are for correctness checking, but those aren’t the sorts of computational experiments Easterbrook claimed made climate science an experimental science. Where do sensitivity studies and model-based diagnostics fit on the flowchart? I think sensitivity studies fit well in the activity called “Pre-test Calculations”, which, one would hope, inform the design of experimental campaigns. Diagnostics are more complicated.
Heald and Wharton have a good explanation for the use of the term “diagnostic” in their book on microwave-based plasma diagnostics: “The term ‘diagnostics,’ of course, comes from the medical profession. The word was first borrowed by scientists engaged in testing nuclear explosions about 15 years ago [c. 1950] to describe measurements in which they deduced the progress of various physical processes from the observable external symptoms” [6]. With a diagnostic we are using the model to help us generate our “Experimental Data”, so that would happen within the activity of “Experimentation” on this flowchart. This use of models as diagnostic tools is applied to data obtained from either experiment (e.g. laboratory plasma diagnostics) or observations (e.g. astronomy, climate science), so it says nothing about whether a particular science is observational or experimental. Classifying scientific activities as experimental or observational is of passing interest, but I think far too much emphasis is placed on this question for the purpose of winning rhetorical “points.”
The more interesting issue from a V&V perspective is introducing a new connection in the flowchart that shows how a dependency between model and experimental data could exist (Figure 3). Most of the time the diagnostic model, and the model being validated are different. However, this case where they are the same is an interesting and practically relevant one that is not addressed in the current V&V literature that I know of (please share links if you “know of”).
It should be noted that even though the same model may be used to make predictions and perform diagnostics, it will usually be run in a different way for those two uses. The significant changes between Figure 1 and Figure 3 are the addition of a “Experimental Diagnostic” box and the change to the mathematical cartoon in the “Validation Experiment” box. The change to the cartoon is to indicate that we can’t measure what we want directly (u), so we have to use a diagnostic model to estimate it based on the things we can measure (b). An example of when the model-based diagnostic is relatively independent of the model being validated might be using laser-based diagnostic for fluid flow. The equations describing propagation of the laser through the fluid are not the same as those describing the flow. An example of when the two codes might be connected would be if you were trying to use ultrasound to diagnose a flow. The diagnostic model and the predictive model could both be Navier-Stokes with turbulence closures. Establishing the validity of which is the aim of the investigation. I’d be interested in criticisms of how I explained this / charted this out.
Afterward
Attempt at Answering Model Questions
I’m not in the target population that professor Easterbrook is studying, but here’s my attempt at answering his questions about model validation[7].
- “If I understand correctly–a model is ’valid’ (is that a formal term?) if the code is written to correctly represent the best theoretical science at the time...”
I think you are using an STS flavored definition for “valid.” The IEEE/AIAA/ASME/US-DoE/US-DoD definition differs. “Valid” means observables you get out of your simulations are “close enough” to observables in the wild (experimental results). The folks from DoE tend to argue for a broader definition of valid than the DoD folks. They’d like to include as “validation” activities of a scientist comparing simulation results and experimental results without reference to an intended use.
- “– so then what do the results tell you? What are you modeling for–or what are the possible results or output of the model?”
Doing a simulation (running the implementation of a model) makes explicit the knowledge implicit in your modeling choices. The model is just the governing equations, you have to run a simulation to find solutions to those governing equations.
- “If the model tells you something you weren’t expecting, does that mean it’s invalid? When would you get a result or output that conflicts with theory and then assess whether the theory needs to be reconsidered?”
This question doesn’t make sense to me. How could you get a model output that conflicted with theory? The model is based on theory. Maybe this question is about how simplifying assumptions could lead to spurious results? For example, if a simulation result shows failure to conserve mass/momentum/energy in a specific calculation possibly due to a modeling assumption (more likely due to a more mundane error), I don’t think anyone but a perpetual-motion machine nutter would seriously reconsider the conservation laws.
- “Then is it the theory and not the model that is the best tool for understanding what will happen in the future? Is the best we can say about what will happen that we have a theory that adheres to what we know about the field and that makes sense based on that knowledge?”
This one doesn’t make sense to me either. You have a “theory,” but you can’t formulate a “model” of it and run a simulation, or just a pencil and paper calculation? I don’t think I’m understanding how you are using those words.
- “What then is the protection or assurance that the theory is accurate? How can one ‘check’ predictions without simply waiting to see if they come true or not come true?”
There’s no magic; the protection from fooling ourselves is the same as it has always been, only the names of the problems change.
Attempt at Understanding Blah Blah Blah
- “The trouble comes when empiricism is combined with a hypothesis that the climate is stationary, which is implicit in how many of their analyses work.” [8]
The irony of this statement is extraordinary in light of all the criticisms by the auditors and others of statistical methods in climate science. It would be a valid criticism, if it were supported.
- “The empiricist view has never entirely faded from climatology, as, I think, we see from Curry. But it’s essentially useless in examining climate change. Under its precepts, the only thing that is predictable is stasis. Once things start changing, empirical science closes the books and goes home. At that point you need to bring some physics into your reasoning.” [2]
So we’ve gone from what could be reasonable criticism of unfounded assumptions of stationarity to empiricism being unable to explain or understand dynamics. I guess the guys working on embedding dimension stuff, or analogy based predictions would be interested to know that.
- “See, empiricism lacks consilience. When the science moves in a particular direction, they have nothing to offer. They can only read their tea leaves. Empiricists live in a world which is all correlation, and no causation.” [3]
Lets try some definitions.
- empiricism
- knowledge through observation
- consilience
- unity of knowledge, non-contradiction
How can the observations contradict each other? Maybe a particular explanation for a set of observations is not consilient with another explanation for a different set of observations. This seems to be something that would get straightened out in short order though: it’s on this frontier that scientific work proceeds. I’m not sure how empiricism is “all correlation.” This is just a bald assertion with no support.
- “While empiricism is an insufficient model for science, while not everything reduces to statistics, empiricism offers cover for a certain kind of pseudo-scientific denialism. [...] This is Watts Up technique asea; the measurements are uncertain; therefore they might as well not exist; therefore there is no cause for concern!” [4]
Tobis: Empiricism is an insufficient model for science. Feynman: The test of all knowledge is experiment. Tobis: Not everything reduces to statistics. Jaynes: Probability theory is the logic of science. To be fair, Feynman does go on to say that you need imagination to think up things to test in your experiments, but I’m not sure that isn’t included in empiricism. Maybe it isn’t included in the empiricism Tobis is talking about.
So that’s what all this is about? You’re upset at Watts making a fallacious argument about uncertainty? What does empiricism have to do with this? It would be simple enough to just point out that uncertainty doesn’t mean ignorance.
Not quite blah blah blah, but the argument is still hardly thought out and poorly supported.
References
[1] Easterbrook, S., “Climate Science is an Experimental Science,”http://www.easterbrook.ca/steve/?p=1322, February 2010.
[2] Tobis, M., “The Empiricist Fallacy,” http://initforthegold.blogspot.com/2010/11/empiricist-fallacy.html, November 2010.
[3] Tobis, M., “Empiricism as a Job,”http://initforthegold.blogspot.com/2010/11/empiricism-as-job.html, November 2010.
[4] Tobis, M., “Pseudo-Empiricism and Denialism,”http://initforthegold.blogspot.com/2010/11/pseudo-empiricism-and-denialism.html, November 2010.
[5] Thacker, B. H., Doebling, S. W., Hemez, F. M., Anderson, M. C., Pepin, J. E., and Rodriguez, E. A., “Concepts of Model Verification and Validation,” Tech. Rep. LA-14167-MS, Los Alamos National Laboratory, Oct 2004.
[6] Heald, M. and Wharton, C., Plasma Diagnostics with Microwaves, Wiley series in plasma physics, Wiley, New York, 1965.
[7] Easterbrook, S., “Validating Climate Models,”http://www.easterbrook.ca/steve/?p=2032, November 2010.
[8] Tobis, M., “Empiricism,”http://initforthegold.blogspot.com/2010/11/empiricism.html, November 2010.
Thanks to George Crews and Dan Hughes for their critical feedback on portions of this.
[Update: George left a comment with suggestions on changing the flowchart. Here's my take on his suggested changes.
Tuesday, August 3, 2010
No Fluid Dynamicist Kings in Flight-Test
This was a guest post over on Pielke's site.
Dr Pielke's Honest Broker concepts resonate with me because of practical decision support experiences I've had, and this post is an attempt to share some of those from a realm pretty far removed from the geosciences. All the views and opinions expressed are my own and in no way represent the position or policy of the US Air Force, Department of Defense or US Government. I am writing as a simple student of good decision making. My background is not climate science. I am an Aeronautical Engineer with a background in computational fluid dynamics, flight test and weapons development. I got interested in the discussions of climate policy because the intersection of computational physics and decision making under uncertainty is an interesting one no matter what the subject area. The discussion in this area is much more public than the ones I'm accustomed to, so it makes a great target of opportunity. The decision support concepts Dr Pielke discusses make so much sense to me now, but I can see how hard they are for technical folks to grasp because I used to be a very linear thinker when I was a young engineer right out of school.
My journeyman's education in decision support came when I got the chance to lead a small team doing Live Fire Test and Evaluation for the Air Force (you may not be familiar with LFT&E, it is a requirement that grew out of the Army gaming testing of the Bradley fighting vehicle in the 1980s, a situation that was fairly accurately lampooned in the movie "Pentagon Wars"). The competing values of the different stakeholders (folks appointed by congress to ensure sufficient realistic testing compared to folks at the service level doing product development) was really an eye-opening education for a technical nerd like me. I initially thought, "if only everyone can agree on the facts, the proper course of action will be clear". How naive I was! Thankfully, the very experienced fellows working for me didn't mind training up a rash, newly-minted, young Captain.
It's tough for some technical specialists (engineers/scientists) to recognize worthy objectives their field of study doesn't encompass. The reaction I see from the more technically oriented folks like Tobis (see how he struggles) reminds me a lot of the reaction that engineers in product development offices would have to the role of my little Live Fire office. A difficulty we often encountered was the LFT&E oversight folks wanted to accomplish testing that didn't have direct payoff to narrower product development goals that concerned the engineers. "What those people want to do is wasteful and stupid!" This parallels the recent sand berm example. The preferred explanation from the technician's perspective is that the other guy is bat-shit crazy, and his views should be ridiculed and de-legitimized. The truth is usually closer to the other guy having different objectives that aren't contained within the realm of the technician's expertise. In fact, the other person is probably being quite rational, given their priors, utility function and state of knowledge.
In my little Live Fire Office we had lots of discussion about what to call the role we did, and how to best explain it to the program managers. I wish I had heard of Dr Pielke's book back then, because "Honest Broker" would have been an apt description for much of the role. We acted as a broker between the folks in the Pentagon with the mandate from congress for sufficient, realistic testing, and the Air Force level program office with the mandate for product development. The value we brought (as we saw it), was that we were separate from the direct program office chain of command (so we weren't advocates for their position), but we understood the technical details of the particular system, and we also understood the differing values of the folks in the Pentagon (which the folks in the program office loved to refuse to acknowledge as legitimate, sound familiar?). That position turns out to be a tough sell (program managers get offended if you seem to imply they are dishonest), so I can empathize with the virulent reaction Dr Pielke gets on applying the Honest Broker concepts to climate policy decision support. People love to take offense over their honor. That's a difficult snare to avoid while you try to make clear that, while there's nothing dishonest about advocacy, there remains significant value in honest brokering. Maybe Honest Broker wouldn't be the best title to assume though. The first reaction out of a tight-fisted program manager would likely be "I'm honest, why do I need you?"
One of the reason my little office existed was because of some "lessons learned" from the Tri-Service Standoff Missile debacle (all good things in defense acquisition must grow out of historical buffoonery). The broader Air Force leadership realized that it was counterproductive to have product development engineers and program managers constantly trying to de-legitimize the different values that the oversight stake-holders brought (the differences springing largely from different appetites for risk and priors for deception) by wrangling over largely inconsequential, technical nits (like tree rings in the Climate Wars). The wiser approach was to maintain an expertise whose sole job was to recognize and understand the legitimate concerns of the oversight folks and incorporate those into a decision that meets the service's constraints as quickly and efficiently as possible. Rather than wasting time arguing, product development folks could focus on product development.
The other area where I've seen this dynamic play out is in making flight test decisions. In that case though, the values of all the stake-holders tend to align more closely, so the separation between technical expertise and decision making is less contentious (Dr Pielke's Tornado analogy). In contrast to the climate realm where it's argued that science compels because we're in the Tornado mode, the flight-test engineers understand that the boss is taking personal responsibility for putting lives at risk based on their analysis. They tend to be respectful of their crucial, but limited, role in the broader risk management process. Computational fluid dynamics can't tell us if it's worth risking the life of an air crew to collect that flight test data. In that case there is no confusion about who is king, and over what questions the technical expert must "pass over in silence."
Wednesday, April 14, 2010
Explosively Formed Projectiles: An Impact of Climate Change
Saturday, March 13, 2010
The Social Ethic and Appeals for Technocracy
If we don't revisit the notion of collective responsibility and sobriety soon, our descendants will pay a heavy price.
-- Michael Tobis
What we lack is an ethical framework for inter-generational responsibilities (such as “pass on a habitable planet to our children”). Cost-benefit analysis avoids these ethical questions, at a time when we desperately need to address them.and claims of the failure of democracy or public discourse on the other,
-- Steve Easterbrook
They have to see the need for pain, to sense the danger of doing nothing. They have to lead their leaders as well as follow – once they switch off, nothing good happens easily, if at all.
Wanted: an eco prophet
Perhaps we have to accept that there is no simple solution to public disbelief in science. The battle over climate change suggests that the more clearly you spell the problem out, the more you turn people away. If they don’t want to know, nothing and no one will reach them.Setting aside for now the unsound conflation of scientific insight with political consensus, the sentiment at the base of this meme is just as troubling. It is basically an argument that our old ethical theories and extant systems of governance are incapable of solving the problems, real or perceived, facing modern civilization. W.H. Whyte already wrote the response to this line of thought more ably than I ever could (though his target was mainly the rise of bureaucracy in business, and associated societal changes, his critique seems topical in this case as well).
-- George Monbiot
My charge against the Social Ethic, then, is on precisely the grounds of contemporary usefulness it so venerates. It is not, I submit, suited to the needs of "modern man," but is instead reinforcing precisely that which least needs to be emphasized, and at the expense of that which does. Here is my bill of particulars
It is redundant. In some societies individualism has been carried to such extremes as to endanger the society itself, and there exist today examples of individualism corrupted into a narrow egoism which prevents effective co-operation. This is a danger, there is no question of that. But is it today as pressing a danger as the oberse -- a climate which inhibits individual initiative and imagination, and the courage to exercise it against group opinion? Society is itself an education in the extrovert values, and I think it can be rightfully argued that rarely has there been a society which has preached them so hard. No man is an island unto himself, but how John Donne would writhe to hear how often and for what reasons, the thought is so tiresomely repeated.
It is premature. To preah technique before content, the skills of getting along isolated from why and to what end the getting along is for, does not produce maturity. It produces a sort of permanent prematurity, and this is true not only of the child being taught life adjustment but of the organization man being taught well-roundedness. This is a sterile concept, and those who believe that they have mastered human relations can blind themselves to the true bases of co-operation. People don't co-operate just to co-operate; they co-operate for substantive reasons, to achieve certain goals, and unless these are comprehended the little manipulations for morale, team spirit, and such are fruitless.
And they can be worse than fruitless. Held up as the end-all of organization leadership, the skills of human relations easily tempt the new administrator into the practice of a tyranny more subtle and more pervasive than that which he means to supplant. No one wants to see the old authoritarian return, but at least it could be said of him that what he wanted primarily from you was your sweat. The new man wants your soul.
It is delusory. It is easy to fight obvious tyranny; it is not easy to fight benevolence, and few things are more calculated to rob the individual of his defenses than the idea that his interests and those of society can be wholly compatible. The good society is the one in which they are most compatible, but they can never be completely so, and one who lets The Organization be the judge ultimately sacrifices himself. Like the good society, the good organization encourages individual expression, and many have done so. But there always remains some conflict between individual and The Organization. Is The Organization to be the arbiter? The Organization will look to its own interests, but it will look to the individual's only as The Organization interprets them.
It is static. Organization of itself has no dynamic. The dynamic is in the individual and thus he must not only question how The Organization interprets his interests, he must question how it interprets its own. The bold new plan he feels is necessary, for example. He cannot trust that The Organization will recognize this. Most probably, it will not. It is the nature of a new idea to confound current consensus -- even the mildly new idea. It might be patently in order, but, unfortunately, the group has a vested interest in it miseries as well as its pleasures, and irrational as this may be, many a member of organization life can recall instances where the group clung to known disadvantages rather than risk the anarchies of change.
It is self-destructive. The quest for normalcy, as we have seen in suburbia, is one of the great breeders of neuroses, and the Social Ethic only serves to exacerbate them. What is normalcy? We practice a great mutual deception. Everyone knows that they themselves are different -- that they are shy in company, perhaps, or dislike many things most people seem to like -- but they are not sure that other people are different too. Like the norms of personality testing, they see about them the sum of efforts of people like themselves to seem as normal as others and possibly a little more so. It is hard enough to learn to live with our inadequacies, and we need not make ourselves more miserable by a spurious ideal of middle-class adjustment. Adjustment to what? Nobody really knows -- and the tragedy is that they don't realize that the so-confident-seeming other people don't know either.
[...]
Science and technology do not have to be antithetical to individualism. To hold that they must be antithetical, as many European intellectuals do, is a sort of utopianism in reverse. For a century Europeans projected their dreams into America; now they are projecting their fears, and in so doing they are falling into the very trap they accuse us of. Attributing a power to the machine that we have never felt, they speak of it almost as if it were animistic and had a will of its own over and above the control of man. Thus they see our failures as inevitable, and those few who are consistent enough to pursue the logic of their charge imply that there is no hope to be found except through a retreat to the past.
This is a hopelessly pessimistic view.
The Organization Man
Whyte goes on to dismiss the nostalgic and naive caricature of individualism bandied about by the right, but rather calls for a pragmatic recognition of the natural tension between the individual and society, and that "[t]he central ideal -- that the individual, rather than society, must be the paramount end [...] is as vital and as applicable today as ever", impending climate catastrophes notwithstanding.
“small random variations in solar input (not to mention butterflies)” [as what makes weather random over the long term]
Chaos as you have discussed it requires fixed control parameters (absolutely constant solar input) and no external sources of variation not accounted for in the equations (no butterflies). You gave zero attention in your supposed response to my comment to this central issue. Others here have been accused of being non-responsive, but I have to say that is pretty non-responsive on your part.
The fact is as soon as there is any external perturbation of a chaotic system not accounted for in the dynamical equations, you have bumped the system from one path in phase space to another. Earth’s climate is continually getting bumped by external perturbations small and large. The effect of these is to move the actual observed trajectory of the system randomly – yes randomly – among the different possible states available for given energy/control parameters etc.
The randomness comes not from the chaos, but from external perturbation. Chaos amplifies the randomness so that at a time sufficiently far in the future after even the smallest perturbation, the actual state of the system is randomly sampled from those available. That random sampling means it has real statistics. The “states available” are constrained by boundaries – solar input, surface topography, etc. which makes the climate problem – the problem of the statistics of weather – a boundary value problem (BVP). There are many techniques for studying BVP’s – one of which is simply to randomly sample the states using as physical a model as possible to get the right statistics. That’s what most climate models do. That doesn’t mean it’s not a BVP.
Tomas’ comments about the 3-body system being not even “predictable statistically (e.g you can not put a probability on the event “Mars will be ejected from the solar system in N years”” is true in the strict sense of the exact mathematics assuming no external perturbations. That’s simply because for a deterministic system something will either happen or it won’t, there’s no issue of probability about it at all. But as soon as you add any sort of noise, your perfect chaotic system becomes a mere stochastic one over long time periods, and probabilities really do apply.
A nice review of the relationships between chaos, probability and statistics is this article from 1992:
“Statistics, Probability and Chaos” by L. Mark Berliner, Statist. Sci. Volume 7, Number 1 (1992), 69-90.
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1177011444
and see some of the discussion that followed in that journal (comments linked on that Project Euclid page).
Where’s the meat? Where’s the results for the problems we care about? I can calculate results for logistic maps and Lorenz ’63 on my laptop (and the attractor for that particular toy exists).