Update 1
by Sheyla Delgado, Wogod Alawlaqi, Richard Espinobarros, Laila Alsabahi, Anjelica Camacho, and Jeffrey A. Butts
Introduction
This update is the first in a series of reports about the evaluation of the New York City Mayor’s Action Plan for Neighborhood Safety (MAP). Using a comprehensive program of social supports and community improvements, MAP is designed to enhance the living conditions and safety of residents in 17 public housing developments operated by the New York City Housing Authority (NYCHA).* This update introduces methods used to assemble a range of outcome measures about public housing developments participating in MAP as well as the statistical procedures used to select a matched comparison group of NYCHA housing developments not participating in MAP. When the evaluation is complete in 2020, differences in outcomes between the 17 MAP and 17 non-MAP housing developments will serve as the statistical basis for estimating the overall success of MAP.
* The MAP initiative is often described as an intervention focused on 15 housing developments, but NYCHA considers three of those developments (Red Hook, Queensbridge, and Van Dyke) as comprising two distinct communities. Thus, this study defines MAP as involving 18 sites. One of those sites, however, is a community for elderly residents (Van Dyke II). It was excluded from the evaluation design after researchers confirmed there were very few public safety issues at Van Dyke II and it was the only senior-exclusive site in the MAP initiative. Thus, this analysis describes MAP as an initiative affecting 17 NYCHA developments.
Improving Public Housing
At a time when New York City is benefiting from historic crime declines, many public housing residents in the City still face high levels of crime and victimization. New York City was a leader in 20th Century affordable housing innovations. As one of the first cities to implement subsidized and below-market housing for low-income residents, New York City forged a path that other U.S. cities followed (Bloom and Lasner 2015). New York City was also one of the first to discover the challenges and contradictions faced by local governments when they attempt to embed publicly-subsidized, affordable housing in market-based economies (Austen 2018). Since the 1960s, city officials have experimented with policies to support affordable housing while acknowledging market forces. One type of policy focuses on government-facilitated, place-based initiatives to improve living conditions and safety in public housing.
Evaluating MAP
The Mayor’s Office of Criminal Justice (MOCJ) launched MAP in 2014 as an effort to increase services and community supports for residents and families living in NYCHA developments. The goal is to improve the social and physical environments of housing developments in ways that support public safety. A number of public and private agencies joined MOCJ to implement MAP, including the New York City Police Department (NYPD), the New York City Housing Authority (NYCHA), and the city’s Human Resources Administration (HRA).
Interventions sponsored by MAP rely on a problem-solving approach. Residents, police, and other agencies work to expand resident access to services and social supports and to monitor the community’s physical security and health. By strengthening community supports and marshaling the talents and energies of residents themselves, MAP is designed to prevent crime and disorder while building stronger communities.
John Jay College’s Research and Evaluation Center began working on MAP in 2017 with funding provided by the New York City government through MOCJ. Researchers at JohnJayREC designed the MAP evaluation in partnership with NORC at the University of Chicago, a nationally-respected public opinion and polling firm.
The quasi-experimental evaluation is designed to estimate differences in living conditions in MAP developments versus those of NYCHA developments not involved in MAP. Researchers followed each set of developments over time to detect improvements and to estimate the extent to which they may be attributable to MAP.
Researchers began by assembling administrative and programmatic data to monitor possible outcomes in each study area. Measures of resident activities, organizational meetings, and service participation were combined into indices of implementation. Public safety metrics were compiled from police reports and data from the health care system. Researchers also observed MAP-related activities directly whenever possible and conducted a series of interviews with key participants.
Key outcomes tracked by the evaluation team include those directly related to program activities, general social and economic well-being, and public safety (crimes reported, arrests, shootings, violent injuries, etc.). The evaluation design also included a survey strategy to measure the perceptions of local residents about neighborhood safety and community well-being. If MAP is effective, researchers should be able to detect improvements on a variety of outcomes and those improvements should be reflected in the attitudes and opinions of residents.
Identifying Comparison Areas
Evaluating the effects of any place-based social intervention requires a strategy to measure outcomes in areas not receiving the intervention as well as those that are receiving the intervention. Collecting data from areas unaffected by an intervention is what researchers call measuring the “counterfactual.” In other words, what might have happened in MAP communities if the MAP initiative had never occurred?
When researchers detect differences between two sets of communities and those differences are correlated with the presence of an intervention after controlling for a range of other possible explanations, one may legitimately infer the intervention had an effect. Before making such inferences, however, researchers must first establish the similarity of the intervention and non-intervention (comparison) areas.
In the MAP evaluation, researchers estimated the counterfactual by selecting a set of 17 comparison areas among all NYCHA developments not involved in MAP (Table 1). The study relied on the statistical method known as “propensity score analysis” (PSA) to select the comparison group. In addition to statistical analysis, the study team made in-person, walking tours of all the comparison sites to ensure their suitability and similarity to the MAP developments.
Selecting comparison sites with propensity score analysis allowed researchers to consider a wide range of factors in judging the similarity of research sites. The method is often used in quasi-experimental studies to approximate randomized experiments and to reduce selection bias (Rosenbaum 2002; Rosenbaum and Rubin 1983).
Randomization is sometimes described as the “gold standard” for scientific research, but randomized or experimental designs are rarely employed in applied research, especially place-based studies. The selection of treatment areas in such studies often occurs prior to the integration of an evaluation design within the overall initiative and there are usually insufficient numbers of areas for rigorous statistical analysis. Frequently, the selection treatment areas is also intentionally (and justifiably) biased in favor of areas most in need of intervention.
The PSA approach involves an array of predictive factors. Rather than matching the 17 MAP sites with 17 non-MAP sites only on demographics, population size, or on any other single factor, a PSA allows researchers to consider an entire set of possible matching factors and to arrive at the best overall set of matching properties. The strength of the match is assessed statistically by weighing the predictive power of the PSA. In other words, how well does an analysis replicate the factors that influenced the location of the MAP initiative in the original 17 NYCHA developments?
Researchers at JohnJayREC analyzed various social, economic, and demographic characteristics of NYCHA developments (Table 2). Relevant data about resident demographics and the physical characteristics of each housing development were organized for inclusion in the PSA model.
Next, the analysis examined crime trends using data compiled by NYPD and disseminated on the New York City Open Data portal. All reported crimes (called “complaints” in New York City) were geographically aggregated based on their proximity to the city’s 300+ NYCHA developments. The study accounted for all serious crimes (dangerous weapons, felony assault, robbery, burglary, grand larceny, grand larceny auto, petty larceny, petty larceny auto, arson, and possession of stolen property and drugs) reported within or adjacent to NYCHA developments between 2006 and 2013 (the last year before MAP).
The analysis examined a range of crimes, including serious felonies against persons or property, other felonies against persons or property, as well as serious misdemeanors involving persons or property and a seventh category called serious violations—consisting mostly of an NYC criminal offense similar to harassment of persons. These multi-offense indices will be used to track changes in crime across the city, but the PSA depended on the exponential moving average (EMA, crimes per 1,000 population) of serious felonies (Figure 1).
Using data from the New York Metropolitan Transportation Authority (MTA), researchers created a measure of “public transportation isolation,” or the distance between the geographic center of each NYCHA development and the nearest entrance to the subway system used by millions of New Yorkers every day. Communities were considered to be isolated if they were located more than 2,640 feet from a subway entrance (roughly 10 blocks). Previous research suggests that use of public transportation begins to decline after the nearest access point exceeds 1,312 feet (Crowley, Shalaby, and Zarei 2009). Given the broad accessibility of multiple public transit options in New York City (buses, trains, subways, etc.) and the relatively high utilization of each system, the research team decided to double the acceptable travel distance suggested by previous studies.
After compiling all available and relevant data, the research team employed the PSA method to identify the best non-MAP NYCHA developments to serve as comparison sites. To ensure strong results, the analysis excluded communities with fewer than 100 residents, resulting in a final pool of 287 possible matches.
Researchers relied on logistic regression to calculate the probability of any one NYCHA site being chosen for MAP itself using all variables in the model, producing a score between 0.0 and 1.0 for each site (i.e. higher scores indicating communities most likely to be chosen for MAP). Finally, the analysis used a one-to-one radius matching approach and selected the best comparison sites with a caliper of 0.06 (setting an upper limit on differences in scores). Calipers set the maximum acceptable difference between available matches to help reduce bias, a method encouraged in studies with limited numbers of potential matches (Lunt 2014).
Results from the PSA model identified one variable that was significantly correlated with the selection of MAP developments: total population. A number of other indicators in the model, while not statistically significant for predictive purposes, were still retained to generate stronger comparison pools. The model performed reasonably well (using McFadden’s “pseudo R-squared” of .38), explaining 38 percent of variance in the probability of a site being selected for MAP (McFadden 1974).
After propensity scores were generated by the logistic regression model, the matching algorithm sorted the list of MAP sites by their propensity scores and for each site located the best non-MAP site with the closest equivalent score (within the set caliper of 0.06). This one-to-one radius matching process (without replacement) ensured a balanced sample of intervention sites and comparison sites.
Conclusion
The JohnJayREC research team succeeded in selecting a group of 17 public housing communities to serve as non-MAP comparison sites for the evaluation’s estimation of the counterfactual—i.e. what would be the conditions in NYCHA developments without MAP? While some minor differences remained between the characteristics of the MAP group and the non-MAP group even after matching, the results of the PSA indicated no significant differences in most of the important key indicators. Thus, the 17 NYCHA developments identified in the PSA analysis represented a useful and robust comparison group for estimating outcomes generated by the New York City Mayor’s Action Plan for Neighborhood Safety.
References
Austen, Ben (2018). High-Risers: Cabrini-Green and the Fate of American Public Housing. New York: HarperCollins Publishers.
Bloom, Nicholas Dagen and Matthew Gordon Lasner, Editors (2015). Affordable Housing in New York: The People, Places, and Policies that Transformed a City. Princeton, NJ: Princeton University Press.
Crowley, David, Amer Shalaby, Hossein Zarei (2009). Access Walking Distance, Transit Use, and Transit-Oriented Development in North York City Center, Toronto, Canada. Transportation Research Record: Journal of the Transportation Research Board, 2110: 96–105.
Lunt, Mark (2014). Selecting an Appropriate Caliper Can Be Essential for Achieving Good Balance With Propensity Score Matching. American Journal of Epidemiology, 179(2): 226–235.
McFadden, Daniel (1974). Conditional Logit Analysis and Qualitative Choice Behavior. In Zarembka, Paul (Editor). Frontiers in Econometrics, Chapter 4, pp. 105-142. New York: Academic Press.
Rosenbaum, Paul R. (2002). Observational Studies, 2nd Edition. New York: Springer.
Rosenbaum, Paul R. and Donald B. Rubin (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70(1): 41–55.
________________
Notes
Funding for this report was provided by the New York City Mayor’s Office of Criminal Justice (MOCJ). Points of view or opinions contained within this document are those of the authors and do not necessarily represent the official position or policies of the City University of New York, John Jay College, or their funding partners.
The authors are grateful to the staff and leadership of the Mayor’s Office of Criminal Justice for their guidance and support during the development of the project. The authors are also grateful for the assistance received from current and former colleagues from the Research and Evaluation Center who contributed to the report: Nicole Alexander, Justice Banks, Kwan-Lamar Blount-Hill, Patricia Cobar, Shun Feng, Sebastian Hoyos-Torres, Gina Moreno, Rhoda Ramdeen, Angela Silletti, Victor St. John, Jason Szkola, Kathleen Tomberg, Anthony Vega, and Kevin Wolff.
Recommended Citation
Delgado, Sheyla, Wogod Alawlaqi, Richard Espinobarros, Laila Alsabahi, Anjelica Camacho, and Jeffrey A. Butts (2018). Quasi-Experimental Comparison Design for Evaluating the Mayor’s Action Plan for Neighborhood Safety. MAP Evaluation Update 1. New York, NY: Research and Evaluation Center, John Jay College of Criminal Justice, City University of New York.
Copyright
Research and Evaluation Center
524 59th Street, Suite 605BMW
New York, NY 10019
http://www.JohnJayREC.nyc
