Tuesday, January 13, 2015

Guidelines for Planning and Evidence for Assessing a Well-Designed Experiment

This paper is full of great guidance for planning a campaign of experimentation, or assessing the sufficiency of a plan that already exists. The authors break up the effort into four phases:
  1. Plan a Series of Experiments to Accelerate Discovery
    1. Design Alternatives to Span the Factor Space
    2. Decide on a Design Strategy to Control the Risk of Wrong Conclusions
  2. Execute the Test
  3. Analyze the Experimental Design
They give a handy checklist for each phase (reproduced below). The checklists are comprehensive (significantly more than my little list of questions) and I think they stand-alone, but the whole paper is well worth a read. Design of experiments is more than just math, as this paper stresses it is a strategy for discovery.