1 slider, Evaluation, Gina Moreno, Jeffrey Butts, JohnJayREC Products, Monographs, NYC Mayor's Action Plan, Richard Espinobarros, Sheyla Delgado

Public Safety Trends in MAP Communities and Matched Comparison Areas


UPDATE 3

button_opendoc_hiresby Sheyla Delgado, Richard Espinobarros, Gina Moreno, and Jeffrey A. Butts

INTRODUCTION

In 2017, John Jay College’s Research and Evaluation Center (JohnJayREC) began an evaluation of the New York City Mayor’s Action Plan for Neighborhood Safety (MAP). With funding provided by the New York City government through the Mayor’s Office of Criminal Justice (MOCJ), researchers designed the evaluation in partnership with NORC at the University of Chicago, a nationally-respected public opinion and polling firm.


MAP is designed to improve the safety and well-being of residents in 17 public housing developments operated by the New York City Housing Authority (NYCHA).(1) JohnJayREC’s quasi-experimental evaluation compares outcomes from the 17 MAP developments with a matched set of developments not involved in MAP. The analysis tracks key outcomes and estimates the extent to which they may have changed beginning with the initial launch of MAP in July 2014.

1. The MAP initiative is often described as an intervention focused on 15 housing developments, but NYCHA considers three of those developments (Red Hook, Queensbridge, and Van Dyke) as comprising two distinct communities each. Thus, MAP could be defined as an effort involving 18 sites. One of those sites, however, is exclusively for older residents (Van Dyke II). It was excluded from the study. Thus, this evaluation conceptualizes MAP as an initiative affecting 17 NYCHA communities.

This report presents preliminary findings about changes in violent injuries and other crime-related outcomes being tracked by the study using administrative data from law enforcement and the health care system. In later reports, the study will conduct more robust analyses to test the effects of mediating and moderating factors.

Study Sites

Evaluating the effects of any place-based social intervention requires a strategy to measure outcomes in areas not receiving the intervention as well as those receiving the intervention. Collecting data from areas unaffected by an intervention is what researchers call measuring the “counterfactual.” Outcomes in comparison areas represent what might have happened if the intervention never occurred.

When researchers detect differences between two sets of communities and those differences are correlated with the presence and strength of an intervention after controlling for a range of other possible explanations, one may legitimately infer the intervention had an effect. Before making such an inference, however, other data analyses must establish the similarity of intervention and non-intervention (comparison) areas.

In the MAP evaluation, researchers selected 17 comparison areas from among all NYCHA developments not involved in MAP. Comparison areas were identified using a statistical method known as propensity score matching in which a selection probability model is calculated for treatment sites and that model is used to identify the best set of matching non-treatment sites.

The evaluation’s propensity score matching relied on crime-related data compiled by NYPD and housing characteristics disseminated by NYCHA. Researchers examined reported crimes (“complaints”) between 2006 and 2013 (one year before the MAP initiative began). All complaints were geographically aggregated based on their proximity to the city’s 300+ NYCHA developments. These complaints included felonies and misdemeanors against persons or property and other offenses. In addition, researchers personally visited candidate comparison areas to ensure their suitability and similarity to the NYCHA sites and neighborhoods involved in MAP. For more information about the study design and the site selection process see Evaluation Update 1.

Methods and Data Sources

Two sources of administrative data are used in this report to analyze the intended outcomes of MAP: 1) publicly available data from the city’s Open Data portal; and 2) non-public information obtained through data-sharing agreements with state and local agencies (Table 1).

Publicly Available Data

The evaluation includes an array of outcome measures related to public safety and resident well-being. One of the main sources of data is the NYC Open Data portal, a platform that serves as a city administrative data repository housing over 3,700 data files, including historical data from the New York City Police Department, non-emergency complaint data from the 311 system, and public safety and quality of life violations from the Office of Administrative Trials and Hearings (OATH).

The main source of crime-related data for the MAP evaluation comes from the NYPD historical complaint data set with over six million felony, misdemeanor, and violation records from 2006 through June 2018 (updated on a quarterly basis). Most records (97%) include geographic coordinates denoting the closest mid-block location where reported incidents occurred. The availability of this rich source of point-level data allows researchers to examine crime-related outcomes across space and time.

Non-Public Data

Through a data-sharing agreement between NYPD and JohnJayREC, the research team also receives non-public shooting and homicide victimization data and violent arrest data. Data are maintained by the NYPD Crime Data Warehouse and include mid-block level counts of violent and weapons-related incidents across all New York City communities from 2006 through 2018 (updated quarterly). Each record provides geographic coordinates, critical occurrence information, and perpetrator characteristics. The MAP evaluation uses these data only as of 2010 because geographic information is missing from some records in previous years.

JohnJayREC also participates in a data agreement with the New York State Department of Health to access health care system data from the Statewide Planning and Research Cooperative System (SPARCS) database. Established in 1979, SPARCS is a comprehensive information source reported by medical providers throughout New York State. The data system collects patient-level data, specifically patient characteristics, diagnoses and treatments, services, and charges for each hospital inpatient stay and outpatient visit for all facilities licensed under Article 28 of the Public Health Law (hospitals and nursing homes) as well as free-standing ambulatory surgery centers.

The SPARCS system uses codes from the International Classification of Diseases (ICD-9 and ICD-10), providing all diagnoses, symptoms, procedures, and external causes of injury. External cause of injury codes (E-codes) define the manner of injury (self-inflicted, intentional, etc.), the mechanism (gun, knife, etc.), and the place of occurrence. The system includes a mortality flag indicating whether an injury resulted in death within a year of the incident.

Outcome Measures in the MAP Evaluation

To measure the effects of MAP, the research team will examine changes in key outcomes and test their association with the implementation of MAP components. The evaluation focuses on data elements available for all study areas (MAP and non-MAP) that are geographically specific (i.e., coded at the level of census tracts or smaller) with reliable records at least several years before and several years after the 2014 launch of MAP.

Researchers will draw final conclusions at the end of the evaluation in early 2020, but this report presents preliminary findings. The analysis focuses on group-level differences between the 17 MAP sites and 17 comparison sites. The study team constructed 13 outcomes to assess MAP’s effectiveness. Eleven outcome measures were constructed from NYPD data. Two outcome measures relied on SPARCS data. A 50-foot geodesic distance buffer (to account for the curviature of the earth) was used to assign all crime and injury incidents to the nearest NYCHA development without overlap.

Crime-related indicators were created with complaint data as well as information about homicide and shooting victimizations (a person hit by gunfire). Two different sets of indices were created from the NYPD complaint data. One set replicates the crime categorizations used by NYPD’s CompStat process (e.g., “7 Major Felonies”). A second set categorizes offenses into person-related felonies and misdemeanors, property-related felonies and misdemeanors, drug and weapon-related felonies, and other misdemeanors.

Two injury outcomes were created from patient-level hospital information extracted from the SPARCS system for the years 2010 to 2016 using external cause of injury codes (E-codes) pertaining to firearm and stabbing injuries.

Analysis and Results

To assess whether changes in outcomes were significantly different from zero, researchers used t-tests to compare average monthly rates before and after the initial launch of MAP in July 2014. In general, MAP sites outperformed comparison sites (Figure 1). MAP sites experienced stronger results in all three offense indices coded by NYPD:

  • “Seven Major” felony offenses (MAP sites down –7%, comparison sites up +1%);
  • “Non-Seven Major” felonies (MAP sites down –4%, comparison sites up +4%); and
  • Misdemeanors (MAP sites down –21%, comparison sites down –14%).

Decreases in two of the indicators were significant (i.e., significantly different from zero) for MAP sites while only one indicator was significant for the comparison sites. A similar pattern was observed for most of the alternative offense categories created by JohnJayREC (Figure 2). Five of six categories revealed stronger outcomes among the MAP sites.

While there was no difference in person-related felonies (both down –5%), the change in property-related felonies was stronger in MAP communities compared with non-MAP communities (reduction of -6% versus an increase of +8%). The difference in other felonies favored MAP as well (down –11% in MAP sites versus up 7% in the comparison sites).

All misdemeanor indices improved more in MAP sites than in comparison sites. For example, person-related misdemeanors declined –15% in MAP sites versus a decline of –5% in comparison sites.

Changes in four of the six indices in MAP sites were found to be statistically significant from zero (i.e., person-related misdemeanors, property-related misdemeanors, other misdemeanors, and weapon/drug felonies), while just two indices in non-MAP sites were significant (p<.05).

The mere existence of declining rates is not surprising. Crime rates were falling across much of New York City during this period. The fact that twice as many indices in MAP sites showed significant declines, however, is at least promising.

Changes in victimization outcomes also tended to be better in MAP sites (Figure 3). The average change before and after MAP was slightly better for both gunshot injuries and shooting victimizations in MAP sites. The difference in stabbing injuries favored MAP sites as well, although rates increased in both MAP and non-MAP areas (up +1% in MAP sites and up +8% in comparison sites).

The percentage change in homicide rates appeared to be much better in comparison sites (–56%) than in MAP sites (–14%), but homicide data must be considered carefully. The base rate for homicides is very low, exposing percent-change calculations to drastic fluctuations (Figure 4).

Significant results are seen more often in property and misdemeanor offenses. This is not surprising and may not be meaningful. These offenses are far more common than offenses against persons and statistical analyses are more likely to detect significant changes among large volume indicators. Future analyses by the evaluation team will examine all crime categories in greater detail.

Moreover, widely varying base rates among the offense categories may affect the appearance and interpretation of simple comparisons of percentage change. In subsequent analyses, researchers will apply other statistical techniques to detect changes over time. Based on these preliminary analyses, however, MAP sites appear to have stronger results than comparison sites.

Finally, researchers examined long-term trends for key outcomes by plotting monthly rates per 1,000 residents for each indicator and creating a trend line for each offense in MAP sites versus comparison sites (Figure 5). The thin, faded blue and gray lines represent the actual monthly averages of events in MAP developments and comparison developments. The heavier lines represent the best fitted trend line for each outcome using a hexic (6 degrees) polynomial trend line. This type of trend line is used when values change continuously and dramatically. Goodness of fit is determined using the R-squared value of the line which ranges from 0 to 1, with the fit improving as the value approaches 1.

This analysis does not provide final results about the differences between MAP sites and comparison sites. It examines the volatility and general trends of key outcomes. When portrayed in graphic form, the analysis demonstrates the extent to which rates of infrequent events are likely to fluctuate when measured in relatively small communities. More analyses will be needed to identify the degree to which long-term trends differ between MAP sites and comparison sites after controlling for other factors.
(Line graphs for other outcomes are available in the Technical Appendix.)

Conclusion

This report addresses the central question of the evaluation of the New York City Mayor’s Action Plan for Neighborhood Safety. Namely, was the presence of the MAP initiative in some NYCHA developments associated with greater improvements in crime and victimization outcomes compared with the same outcomes in NYCHA developments not involved in MAP? The results presented here do not answer the question in full, but they offer an early look at efforts by the research team to generate more precise answers. Additional analyses are needed to rule out competing explanations and to examine the complex series of relationships among all the study’s variables. Based on the preliminary findings in this report, however, the results of MAP to date may be considered promising.

________________________

Notes
Funding for this report was provided by the New York City Mayor’s Office of Criminal Justice (MOCJ). Points of view or opinions contained within this document are those of the authors and do not necessarily represent the official position or policies of the City University of New York, John Jay College, or their funding partners. The shooting victimization data presented in this report were provided by and belong to the New York City Police Department. Points of view or opinions contained within this document are those of the authors and do not necessarily represent the official position or policies of the New York City Police Department. Any further use of these data must be approved by the New York City Police Department. Data about gun and stabbing injuries were obtained from the New York State Department of Health and may not be released without permission.

Acknowledgments
The authors are grateful to the staff and leadership of the Mayor’s Office of Criminal Justice for their guidance and support during the development of the project. The authors are also grateful for the assistance received from current and former colleagues who contributed to the project: Wogod Alawlaqi, Nicole Alexander, Rebecca Balletto, Justice Banks, Patricia Cobar, Arlana Henry, Rhoda Ramdeen, Jason Szkola, Kathy Tomberg, Anthony Vega, and Kevin Wolff.

Recommended Citation
Delgado, Sheyla A., Richard Espinobarros, Gina Moreno, and Jeffrey A. Butts (2019). Public Safety Trends in MAP Communities and Matched Comparison Areas. MAP Evaluation Update 3. New York, NY: Research and Evaluation Center, John Jay College of Criminal Justice, City University of New York.

Copyright
Research and Evaluation Center at John Jay College of Criminal Justice
City University of New York (CUNY)
524 59th Street, Suite 605BMW
New York, NY 10019
http://www.JohnJayREC.nyc