Comments on Various Consequences: Discussion of V and V

"Just because it's intractable to prove t...

2011-10-16T21:22:38.667-04:00

"Just because it's intractable to prove the correctness of software in general doesn't mean we can't prove some useful limited or special cases."

The key point. IMHO, a sufficiently clever guy should be able to formally bound any approximation error and the big-O for solution times for *any* practical PDE-based model. (Or any practical NP-complete class problem for that matter.)

George, I agree with you for the most part. VV&am...

2011-10-16T15:07:50.497-04:00

George, I agree with you for the most part. VV&UQ, properly understood, is about risk management (or credibility building).

For certain codes (numerical PDE solvers for instance), we can say that we've established the correctness of the implementation by demonstrating the design order of convergence. Just because it's intractable to prove the correctness of software in general doesn't mean we can't prove some useful limited or special cases.

Hi Josh, Slide 8 in your link above gives the fol...

2011-10-16T13:51:50.817-04:00

Hi Josh,

Slide 8 in your link above gives the following definitions for V&V:

Verification:
-- The process of determining that a computational software implementation correctly represents a model of a physical process
-- The process of determining that the equations are solved correctly.

Validation:
-- The process of assessing the degree to which a computer model is an accurate representation of the real world from the perspective of the models intended applications
-- The process of determining that we are using the correct equations.

[Emphasis mine.]

Because of the use of the terms 'correct' and 'correctly', the above definitions are vulnerable to an Oreskes-type "V&V is logically impossible" argument. Implementations of complex models will always be, to some degree of risk, buggy. And the theory behind a model is always an abductive approximation and simplification of Nature. Thus, our confidence in models can never be 100% -- logical certainty. IMHO, insisting on formal correctness will preclude any possibility of discovering a formal process for forging an IV&V consensus.

Also, to more easily reach a consensus tailoring of V&V processes and criteria, a diversity of overlapping models should be allowed. (The link's Slide 14 uses the terminology "Robustness to uncertainty".)

I keep harping on this, but the purpose of IV&V is not correctness, but risk management. I also disagree with the link's Slide 9 which characterizes V&V as "a methodology of constraint [on the fantasies of human imagination]". It's a methodology of risk reduction for our boundless human propensity for error.

Haha, maybe it could be crow-sourced.

Interesting: How I Learned to Stop Worrying and Love [the DOE's approach to verifying and validating models of] the Bomb.

Minor nit: crow-sourced is probably crowd-sourced ...

2011-10-11T21:00:09.206-04:00

Minor nit: crow-sourced is probably crowd-sourced unless there is some kind of corn crop involved. (I'm just catching up on some blogs after spending two weeks in conferences myself.)

Hi Josh, I've used this SCOTUS-style technique when presenting the "consensus" IV&V process/requirements tailoring to stakeholders for their acceptance. Obfuscation is more difficult using this approach.

Quality hinges on its corrective action processes. By putting forward a diversity of opinion, later necessary corrective actions become easier. Yet the documented majority tailoring opinion provides the justification for moving forward in face of risk.

George, I think I'm familiar with the sort of ...

2011-10-02T15:08:04.299-04:00

George, I think I'm familiar with the sort of consensus process you are talking about. I think consensus on verification is achievable. One of the significant barriers to this working for validation is the lack of shared purpose, trust, and diverse risk appetites of the various stake-holders. As Jaynes points out, all of us good Bayesians shouldn't necessarily expect convergence of views when political motivations become significant.

I am interested in hearing more about your experience in consensus building though.

The cost of IV&V must be weighed against its b...

2011-10-02T01:07:33.342-04:00

The cost of IV&V must be weighed against its benefits (largely risk reduction/confidence building). Therefore, it is always necessary, based on the nature and usage of the software, to tailor the SQA requirements and processes (of which IV&V is an important component).

For the GCMs, you clearly point out that we run the full gamut for suggested tailoring -- the extremes being "any IV&V is impossible" to "formally prove the software correctly executes".

What are the reasons for this? Well, tailoring is subjective. So we can encounter issues if 1) there is incomplete understanding of the nature of IV&V and its role in SQA, or 2) there is ethically/politically/financially motivated obfuscation.

So a consensus tailoring will be difficult. But, IMHO, a big benefit for tackling the problem directly would be to draw the subjectivity away from the climate science itself, where it is having a corrosive effect. And I am confident this is the thing to do. I have had some experience with IV&V tailoring consensus building in a contentious area (nuclear) and know it can be done.

I meant to link the Models as Ink Blots post too.