Butts, Jeffrey A. and John K. Roman (2018). Good Questions: Building Evaluation Evidence in a Competitive Policy Environment. Justice Evaluation Journal, 1(1), published online first, July 2018.
Criminal justice evaluations are funded and executed in a policy environment increasingly characterized by intense competition that creates a premium on studies utilizing experimental methods. Randomization, however, is not a universally suitable method for answering all questions of criminal justice policy and practice. Experimental designs are particularly ill-suited for addressing key analytical challenges and exploiting important opportunities in justice, including discontinuity effects, interventions that depend on the perceptions and beliefs of individuals, and models of general policy change. Experiments are also rarely able to reliably measure and isolate the effects of very complex justice interventions. Policymakers and practitioners in the justice sector should consider evaluation research as a portfolio of strategic investments in knowledge development. Randomized controlled trials are merely one asset in a broader investment strategy.