Findings from Household Surveys in MAP Communities and non-MAP Communities
UPDATE 4
by Sheyla Delgado, Jeffrey A. Butts and Gina Moreno
INTRODUCTION
As part of an evaluation of the New York City Mayor’s Action Plan for Neighborhood Safety (MAP), researchers from John Jay College of Criminal Justice collaborated with survey specialists from NORC at the University Chicago to collect data from two probability samples of residents in public housing developments in New York City. One sample of residents came from communities involved in the MAP initiative. A second sample was from statistically matched housing developments not involved in MAP (See Evaluation Update 1).
This report describes results from the first iteration of surveys conducted in early 2019. After a second iteration is completed in early 2020, the evaluation team will analyze the data to detect changes in resident perceptions and to identify any changes that may be related to the effects of MAP.
EVALUATING THE MAP INITIATIVE
New York City launched the Mayor’s Action Plan for Neighborhood Safety in 2014, describing it as a “targeted and comprehensive approach” to public safety in housing developments operated by the New York City Housing Authority (NYCHA). The goal of MAP is to improve the public safety of NYCHA communities by supporting the general well-being of residents, facilitating community empowerment, strengthening community connections, and increasing the presence of active community space in and around NYCHA developments. According to the NYC Mayor’s Office of Criminal Justice (MOCJ), housing developments involved in MAP accounted for one-fifth of all violent crimes in NYCHA communities during several years preceding MAP.
In 2017, John Jay College’s Research and Evaluation Center (JohnJayREC) began the evaluation of MAP and immediately enlisted the partnership of researchers from NORC at the University of Chicago. Most of the evaluation relies on administrative data from police, social services, and other partners, but adding NORC allowed the study to collect survey data directly from residents.
Together, the research teams from John Jay and NORC designed the household survey to measure perceptions, attitudes, and opinions of people living in public housing. Experts from NORC administered the surveys to large samples of residents from more than 30 public housing developments.
Researchers designed the survey to measure perceptions of community safety, the availability of services and social supports for residents, and various other indicators of community well-being. Drawing on this information, the research team will analyze differences between MAP communities and the matched set of non-MAP communities in 2019 and 2020. Key outcomes measured by the “NYCHA Resident Survey” include social cohesion (trust in one’s community), belief in government legitimacy, perceptions of safety, and the extent to which residents are willing to engage with government in the interests of their community. The survey measured collective efficacy (neighbors solving problems together) using two different forms of this important question (categorical and dichotomous) to test their comparative utility.
To create a useful and theoretically salient set of survey questions and scales (i.e. groups of questions measuring the same concept), the research team first reviewed more than 40 previous studies (Figure 1). Whenever possible, the team preserved the original wording of questions from those studies. Often, however, it was necessary to adapt questions to make them appropriate for a study of New York City public housing residents (Figure 2). Some questions used in previous research referred to topics and activities that would be relevant only in smaller cities and suburban areas (e.g., lawn care).
Before launching data collection, researchers from NORC and JohnJayREC presented the questionnaire to officials from MOCJ and NYCHA for their review and approval. The NYCHA Resident Survey was then pilot tested with a small group of residents. Feedback from the pilot group helped to ensure the suitability of language used in the questionnaire and to confirm the accuracy and accessibility of instructions provided for survey respondents.
SAMPLING AND RECRUITMENT
Residents were sampled from 34 public housing developments with similar population sizes and demographic characteristics, including 17 developments involved in the MAP initiative (treatment group) and 17 developments not involved in MAP (comparison group).1 To begin the sampling process, NYCHA provided NORC analysts with a de-identified list of more than 80,000 adult residents (ages 18 and older) across all 34 study sites. NORC randomly selected 17,630 of those residents as the initial study sample (Figure 3).
1. The MAP initiative is often described as an intervention focused on 15 housing developments, but NYCHA considers three of those developments (Red Hook, Queensbridge, and Van Dyke) as comprising two distinct communities each. Thus, MAP could be defined as an effort involving 18 sites. One of those sites, however, is exclusively for older residents (Van Dyke II). It was excluded from the study. Thus, the John Jay College evaluation conceptualizes MAP as an initiative affecting 17 NYCHA communities.
Soon thereafter, each sampled participant received an envelope via U.S. Mail with a letter explaining the survey, its purpose, and its sponsorship. Respondents were assured that—while NYCHA endorsed the survey—the housing authority was not conducting the survey and would not see the answers of any residents, nor would any resident’s participation or lack of participation in the survey affect their housing status.
Every invitation envelope contained a $2 bill and the letter described additional incentives for respondents who completed the survey. Respondents could answer the survey by phone or by using a website accessible with a desktop or mobile device after entering their unique log-in credentials. Everyone completing the survey received a $15 gift card as well as a $10 bonus if they completed it within two weeks of receiving the invitation letter. The survey was available in four languages: English; Spanish; Cantonese; and, Mandarin.
To ensure effective understanding of the survey and to clarify the independence of the project, the research team from NORC and JohnJayREC met with leaders of NYCHA resident associations before beginning data collection. Two information sessions were held at the MOCJ offices with Resident Association Leaders (RAL) and other important stakeholders. During the first meeting, residents recommended promotional materials (e.g., posters) to familiarize residents with the survey and to encourage those receiving invitation letters to take the survey seriously. Before data collection commenced, the research team delivered posters in English, Spanish, and Cantonese to each RAL. Resident leaders placed the posters in heavily trafficked areas of NYCHA buildings, including elevators and stairways.
After invitation letters were mailed and responses began to appear on the survey website, NORC researchers monitored the response rate and conducted ongoing analyses to ensure data quality. Based on their estimation of the sample sizes required for adequate statistical power, the research team hoped to receive at least 40 valid and complete responses from each of the 34 sampled developments. The actual response rate was much better than expected (N=50+).
NORC researchers expected to use phone follow-ups with non-respondents to ensure sufficient response rates. Phone interviews were scheduled to begin during the fifth week of data collection, but they began and ended two weeks early due to the study’s unexpectedly high response rate. In the end, fewer than 80 telephone interviews were required in just 9 of 34 developments. All other developments provided 50 or more completed surveys prior to the start of phone follow-ups. Data collection began on February 5th, 2019 and concluded five weeks later.
RESULTS: SAMPLE CHARACTERISTICS
The final respondent pool from this first iteration of the NYCHA Resident Survey was just under 2,000 (N=1,942), half from MAP communities and half from comparison communities, with few significant differences between respondents in MAP and non-MAP sites. Only small differences were observed in age, gender, ethnicity, education level, and employment status (Table 1).
Respondents from MAP and non-MAP communities were very similar in age. More than half of all respondents in both groups were between ages 25 and 69. Most respondents were female (MAP 72%; Comparison 67%), which is representative of NYCHA residents overall according to city data.
Some differences were observed in self-reported ethnicity, with more Black or African American respondents in the MAP group (47% versus 32%) and somewhat more Asian respondents in the comparison group (13% versus 5%).
About eight in ten respondents reported they had earned at least a high school diploma, and at least four in ten reported some college experience (MAP 43%; Comparison 45%). More than a third of all respondents reported being employed either part-time or full-time (MAP 36%; Comparison 37%).
Two-thirds of all respondents reported that they had been living in their NYCHA developments more than ten years (Table 2). Very few respondents were newcomers. Among respondents in MAP sites, just 10 percent had been residing in the development fewer than 3 years while the same was true for just 8 percent of respondents from comparison sites.
Most respondents reported that their households included at least three people (MAP 54%; Comparison 46%). Nearly one in six respondents reported households of five or more, and this was due to the presence of children. When household size was reported for adults only (age 18 or older), fewer than five percent of survey respondents reported more than 4 people in their households (MAP 2%; Comparison 3%).
The analysis of respondent characteristics in treatment and comparison sites suggests the two samples were very similar. The results provide support for the comparability of treatment and comparison sites as intended by the research team.
RESULTS: OUTCOME MEASURES
Researchers employed factor analysis to identify sets of survey items that were sufficiently correlated with one another to qualify as multi-item scales or factors. Of the 70 potential outcome items measured in the survey, 61 were used to create 10 different scales. Nine items were not used in the analysis because they failed to combine into scales with sufficient reliability. (Responses to all items and scales are provided in the Appendix.)
The NYCHA Resident Survey was intended to measure key outcomes of the MAP initiative by comparing changes in MAP and non-MAP communities between iterations of the survey.
Only preliminary conclusions may be drawn from this first iteration of the survey. After the second iteration, the evaluation team should be able to assess changes in resident responses to estimate the potential effects of MAP. The team’s ability to conduct such an analysis depends on the reliability of the outcome measures tracked by the study.
Researchers assessed each scale using Cronbach’s alpha coefficient, a traditional statistic used to judge reliability. Alpha coefficients range from zero to one. As the value approaches 1.0, the internal consistency among items is stronger. Values above 0.9 are considered excellent, while those above 0.8 are good. Values above 0.7 are considered acceptable, but values lower than 0.7 are considered not useful. All scales reported in this analysis were above the acceptable threshold (Figure 1).
After the second iteration of the survey, researchers will test for differences between treatment and comparison sites. Even in this first iteration, however, two scales already show significant differences (Table 3). Perceptions of respondents from MAP sites were slightly more positive than those from comparison sites: “awareness of social support services” (MAP 4.67; Comparison 4.34); and “collective efficacy: categorical” (MAP 12.23; Comparison 11.85).
Assessing the significance of differences in the survey scales depends on the particular method used and the study’s assumptions about the distribution of scores for each scale. Based on the scales as constructed, the research team tried two different methods to compare differences between MAP and non-MAP sites: independent samples t-tests and Mann-Whitney U tests (Table 4). The Mann-Whitney U test was added because the distributions of responses to many survey scales were skewed (responses tended to cluster at one end or the other of a scale rather than being evenly distributed across all values).
After the second iteration of the survey, the evaluation team will address the potential effects of non-response when analyzing changes in MAP outcomes. Approximately 40 of the 88 survey items had missing values above an acceptable threshold of 10 percent (combining “I don’t know” and “prefer not to answer” responses). Such “nonattitude” responses could accurately reflect the absence of an opinion, or they may be random choices by respondents who feign engagement in a survey while randomly completing items to reach the end and secure the financial incentive. After the second round of data collection, the research team will apply missing data techniques such as imputation or mean substitution to correct for potential bias.
CONCLUSION
Preliminary results suggest the NYCHA Resident Survey was administered as intended by the evaluation team. Respondent samples from the MAP and non-MAP communities were demographically similar and should allow fair comparisons of changes in attitudes and experiences once the second iteration of surveys is completed. The primary focus of this Evaluation Update was to examine the first iteration of data collected using the survey instrument and to test baseline differences between respondents in MAP and non-MAP sites. Significant differences in two survey scales (awareness of social supports and collective efficacy) could be due to the presence of MAP, and it is too soon to draw strong causal inferences. Still, these results may be interpreted as promising.
REFERENCES
Crist, Janice D., Cathleen Michaels, Donald E. Gelfand and Linda R. Phillips (2007). Defining and measuring service awareness among elders and caregivers of Mexican descent. Research and Theory for Nursing Practice 21(2): 119-134.
Earls, Felton J., Jeanne Brooks-Gunn, Stephen W. Raudenbush and Robert J. Sampson (2007). Project on human development in Chicago neighborhoods: Community Survey, 1994-1995. Ann Arbor, MI: Inter-university Consortium for Political and Social Research.
Elo, Irma T., Laryssa Mykyta, Rachel Margolis and Jennifer F. Culhane (2009). Perceptions of neighborhood disorder: The role of individual and neighborhood characteristics. Social Science Quarterly 90(5): 1298-1320.
Fox, Claire L., David Gadd and Julius Sim (2015). Development of the Attitudes to Domestic Violence Questionnaire for Children and Adolescents. Journal of Interpersonal Violence 30(14): 2506-2525.
Kim, Eric S., Nansook Park and Christopher Peterson (2013). Perceived neighborhood social cohesion and stroke. Social Science & Medicine 97(2013): 49-55.
Rosenbaum, Dennis P., Daniel S. Lawrence, Susan M. Hartnett, Jack McDevitt and Chad Posick (2015). Measuring procedural justice and legitimacy at the local level: The police-community interaction survey. Journal of Experimental Criminology 11(3): 335-366.
Tyler, Tom R., Kenneth A. Rasinski and Kathleen M. McGraw (1985). The influence of perceived injustice on the endorsement of political leaders. Journal of Applied Social Psychology 15(8): 700-725.
________________________
Notes
Funding for this report was provided by the New York City Mayor’s Office of Criminal Justice (MOCJ). Points of view or opinions contained within this document are those of the authors and do not necessarily represent the official position or policies of the City University of New York, John Jay College, or their funding partners.
Acknowledgments
The authors are grateful to the staff and leadership of the Mayor’s Office of Criminal Justice for their guidance and support during the development of the project. The authors are also grateful for the assistance received from current and former colleagues who contributed to the project: Wogod Alawlaqi, Nicole Alexander, Rebecca Balletto, Justice Banks, Patricia Cobar, Arlana Henry, Rhoda Ramdeen, Jason Szkola, Kathy Tomberg, Anthony Vega, and Kevin Wolff.
Recommended Citation
Delgado, Sheyla A., Jeffrey A. Butts and Gina Moreno (2019). Opinions and Perceptions of Residents in New York City Public Housing — Findings from Household Surveys in MAP Communities and non-MAP Communities. MAP Evaluation Update 4. New York, NY: Research and Evaluation Center, John Jay College of Criminal Justice, City University of New York.
Copyright
Research and Evaluation Center at John Jay College of Criminal Justice
City University of New York (CUNY)
524 59th Street, Suite 605BMW
New York, NY 10019
http://www.JohnJayREC.nyc
