John K. Roman, Jeffrey A. Butts, and Caterina Roman (2011). Evaluating Systems Change in a Juvenile Justice Reform Initiative. Children and Youth Services Review, 33: S41-S53.

Evaluating comprehensive, inter-agency initiatives to reform human services systems presents substantial challenges to traditional research models. Outcomes are observed at the system level rather than the individual level, and the validity of study results may be challenged on a variety of dimensions, particularly small sample sizes and measurement error. We report the results of a cross-site evaluation of the first phase of Reclaiming Futures, a five-year effort to improve services and interventions for justice-involved youth in ten communities across the United States. We present the evaluation findings and discuss implications for similar research endeavors. The methods used in this study may be useful for other system-level evaluation efforts. A social network analysis analyzes changes in the size, density, and cohesiveness of stakeholder networks. Bivariate models test for associations between Reclaiming Futures implementation and key stakeholder perceptions of effective system change. Multivariate models are used to explore selection effects on item response. In general, we find positive correlations between successful implementation of Reclaiming Futures and indicators of effective and efficient system change.