The Evidence Generation initiative views research and evaluation as key components of a developmental journey that learning organizations follow. The generation of evidence is an orientation to practice, part of one’s daily routine. It is not a professional service that organizations simply purchase from outside contractors or consultants, and it is not something that can be put off until an organization needs to apply for new funding. Evaluation evidence should be a key part of program management from the very beginning.
“Knowledge Changes”
The approach adopted by Evidence Generation was inspired in part by an essay written in 2009 by Akiva Liberman. See Advocating Evidence-Generating Policies: A Role for the ASC. The Criminologist- Official Newsletter of the American Society of Criminology, Vol. 34(1): 1, 3-5. The short essay takes a critical view of conventional evidence-based thinking. It reminds readers to heed the advice of Donald Campbell, who viewed all social policies as experimental and all expected outcomes as hypotheses. Justice agencies should pursue the continued generation of evidence and not simply follow the existing knowledge base as if it were fixed and finite.
Public safety practitioners must embrace the quest for better evidence, and they need to appreciate the limitations of existing evaluation research. There will never be enough funding or enough competent researchers to test every single aspect of public safety systems or to assess the value of every single practice, program, or policy.
Research funding is scarce and competitive. Only a few models have been fortunate enough to receive the sustained investments required to generate high-quality evaluation evidence. More importantly, research investments are not impartial and unbiased. Investments in research are shaped by budgetary limitations, administrative obstacles, evaluation complexities, political values, organizational self-interest, and simple marketing and promotion. In any field, the practices established by rigorous research may be only a small portion of the total operational environment.
The published findings of past evaluations are not a sufficient basis for making all of the choices involved in building and operating the justice system of the future. Researchers and practitioners must work together and collaborate in the search for better evidence, and the most urgent need is for evidence about innovative programs and practices that are often overlooked in the competitive struggle for grants and contracts.
If an agency is devoted to cognitive-based therapy or youth mentoring, for example, there are ample evaluations and plenty of evidence that may be used to design new programs. If, on the other hand, a program provides educational enhancements, employment readiness, or creative activities to support positive behavior, there is much less evidence — not because these are inherently bad ideas, but because they either haven’t been fully developed or they have yet to be tested by competent evaluators.
Moreover, some critical elements in a public safety system cannot be evaluated with recidivism outcomes. A policy reform effort might be designed simply to lower costs. A restorative justice program might be asked to increase the speed of victim compensation. A legal aid organization might want to measure its effectiveness in representing youth in court. There is no compendium of evidence-based practices for such issues. Researchers should be prepared to work with public safety entities on these areas as well as those related to recidivism.
This is the mission of the Evidence Generation initiative — to help innovative organizations in the public safety sector develop stronger capacities for data collection and measurement, and to prepare themselves for future evaluation opportunities.
Learn More
