Public Safety Initiatives Managed by the New York City Department of Youth and Community Development
by Kwan-Lamar Blount-Hill, Patricia Cobar, Fidel L. Osorio, Sheyla A. Delgado, and Jeffrey A. Butts
November 2023
Executive Summary
New York City relies heavily, but not exclusively, on law enforcement to prevent crime and violence. Other interventions are essential to prevent crime and to avoid undue reliance on policing. City officials recently expanded three such programs: the Crisis Management System (CMS) which includes core components of the Cure Violence approach, the Mayor’s Action Plan for Neighborhood Safety (MAP) focused on the safety of public housing, and the Precision Employment Initiative (PEI) that supports the work skills and job readiness of city residents.
New York’s Department of Youth and Community Development asked the Research and Evaluation Center at John Jay College of Criminal Justice (JohnJayREC) to review the three programs and consider their suitability for evaluation research. The three initiatives were designed according to officials’ interpretation of the best available scientific evidence, but they have not yet been subject to rigorous research. More evidence is needed to establish existing program effects with enough reliability to ensure that any new efforts to expand each program would support communities and significantly reduce violence.
Each of the three programs launched by New York City presents methodological challenges for evaluation. Previous research at John Jay College examined two programs, CMS and MAP. Those studies offer some guidance for future measurement and analysis strategies. The third program, PEI, has not yet been involved in a detailed evaluation. However, the research literature related to its substance and focus (employment support) provides considerable background material.
Rigorous evaluations of the three programs would have to:
- Measure each program’s intentions and activities, not only apparent outcomes.
- Collect data at relatively small geographic levels (streets, neighborhoods, etc.).
- Collect data over an extended period and account for the influence of elapsed time and other factors that occurred over the same period.
- Measure the content and frequency of a program’s interactions with other social programs, city officials, and agency staff.
- Measure the perceptions and opinions of agency staff and community residents regarding the actions and effects of programs.
- Collect data about the well-being and job performance of staff and volunteers.
- Measure other social and economic factors not directly related to program activities that may be correlated with its intended outcomes.
Staff members from DYCD asked JohnJayREC to review the three programs and provide information that could shape future efforts to improve their effectiveness in preventing and reducing community violence. The following report incorporates the research team’s review of available program documentation, each program’s key components and strategies, and relevant opinions and perceptions of program staff and community residents.
Introduction
New Yorkers are generally much safer today than they were in the 1990s. Like most of the world, however, violent crime in the city surged from 2020 to 2022. According to the New York City Police Department, violent crimes (murder, rape, robbery, and felony assault) increased more than 20 percent between 2019 and 2022. Shooting incidents grew more than 75 percent, from 967 in 2019 to 1,706 in 2022. Subsequent crime trends were promising, but neighborhoods experienced unequal degrees of improvement. Of the six New York City police precincts with the highest rates of gun violence historically, only five had fewer shootings in the first six months of 2023 than in the first half of 2022. Gun violence remains a serious challenge in New York City and other communities across the United States.
Local governments draw on many resources to reduce violence. Policing is a core component, but effective prevention requires a diverse set of tools. In 2023, New York City adopted an ambitious violence prevention strategy involving multiple community-centered approaches. In “A Blueprint for Community Safety,” the city’s Gun Violence Prevention Task Force acknowledged the role of enforcement (especially in neighborhoods most affected by gun violence) but pointed to other resources known to improve the health and well-being of residents in distressed and vulnerable communities.
New York’s violence prevention approach included three programs in particular: the Crisis Management System (CMS), the Mayor’s Action Plan for Neighborhood Safety (MAP), and the Precision Employment Initiative (PEI). The initiatives are managed by New York City’s Department of Youth and Community Development (DYCD). Staff from DYCD asked researchers from John Jay College of Criminal Justice and its Research and Evaluation Center (JohnJayREC) to review the three programs and analyze their readiness for rigorous evaluation.
Building Evidence
Researchers must do more than compare major outcomes before and after launching a new program or policy. Rigorous evaluations measure the apparent effects of an intervention by establishing a statistical connection between the results of the intervention and the actions and processes used to achieve those results. Studies do this, in part, by estimating what would have happened if a program or policy had never been implemented — what researchers call measuring the counterfactual. Evaluation studies can use various methods and designs and will not always involve experimental or “randomized controlled trials” (RCT). Experimental studies are not always practical or ethical when evaluating social policy and community interventions.
What if lawmakers see crime fall in their state after a law enforcement strategy takes effect? This would please policing advocates, but researchers must ask more questions. How did the change in police operations make crime go down? How did it work exactly? Was it the behavior of street patrols, changes in arrest practices, staffing levels, or something about the local legal culture? Did criminal violations drop more in this community than in any others after the new procedures took effect? Was the decline more noticeable in some areas than others? If the pattern was not universal, did changes in crime align with the launch and intensity of the new effort?
Rigorous evaluations use data to connect process and outcomes. Process measures document how programs conduct their day-to-day work. Researchers use them to assess an intervention’s design and delivery. Outcome measures establish whether results occur as planned. Statistical relationships between process and outcome indicate whether an intervention worked as intended.
BASICS |
| Evaluation frameworks provide structured approaches to program planning, monitoring, and evaluation. They enhance program design, clarify goals and objectives, and promote accountability and learning. Ultimately, accurate and useful frameworks contribute to the effective implementation and improvement of interventions. They can serve as the foundation for evaluation research by providing roadmaps for data collection and allowing evaluators to assess whether a program is achieving its intended outcomes and impacts. |
Based on a program’s theory of change or conceptual framework, researchers must show its effects are credible (i.e., that outcomes are consistent with the actions assumed to create them). Process measures include data about program activities, their timing, location, duration, and the extent to which individuals or neighborhoods participate. Process measures are also helpful in establishing whether relevant data can be collected reliably and consistently and whether identical data elements are available for suitable comparison groups. Without effective process measures, an outcome evaluation may not be able to generate conclusive findings. Statistical techniques cannot make up for an analysis that fails to account for both process and outcomes. A program might remain a black box of undifferentiated causes with no real connection to observed effects, even if those effects are welcome and impressive.
Program leaders and staff must work with researchers to develop effective evaluation frameworks with several distinct elements.
Resources and Activities:
The resources used and steps involved in implementing program activities or policy changes. What does a program or policy actually do? What resources are involved?
Outputs and Intermediate Outcomes:
The direct and immediate products, capacities, and deliverables that result from program activity or policy movement. What changes in people or community conditions result from a program’s activities or what actions are produced by a policy?
Outcomes and Impacts:
The durable changes that occur in people or conditions because of these activities and their effects. What long-term outcomes are attributable to program efforts or a policy’s actions? How do they follow from the intermediate outcomes?
Evaluation frameworks identify the key components of a program or intervention and propose hypothetical relationships between them. The John Jay research team developed preliminary evaluation frameworks for the three programs reviewed here.
Community Violence Intervention
The three initiatives reviewed by JohnJayREC are the type of interventions the U.S. Department of Justice (DOJ) calls Community Violence Interventions (CVI) because they use “evidence-informed strategies to reduce violence through tailored community-centered initiatives.” According to DOJ, community-centered initiatives rely on the “active, meaningful involvement of a wide range of community members in a community’s governing structure and other organizations that influence community decisionmaking” (sic). The DOJ definition denotes strategies “backed by evidence generated by multiple disciplines [of science] and a variety of methods,” including findings from evaluation research but also “case studies, expert opinions, or documented lessons learned from the field.” In short, CVI initiatives combine expertise from researchers, policymakers, practitioners, and community members.
Federal support for CVI is consistent with DYCD’s mission to “invest in a network of community-based organizations and programs to alleviate the effects of poverty and to provide opportunities for New Yorkers and communities to flourish.” The DYCD mission aligns more closely with a class of CVIs that JohnJayREC designates as CVI-R (CVI at the Roots) or programs that operate at the grassroots level by relying on the talents and power of communities and their residents. Such interventions help to remedy the structural forces leading to individual harm while supporting those already harmed. They invest in communities to alleviate the effects of poverty and engage residents in varying strategies to reduce violence and overcome the historical remnants and present-day systems of bigotry and neglect.
The complexity of these strategies raises an urgent need for evaluation research. The CVI-R report (pages 1-2) noted some of the challenges involved in building strong evidence for community-centered and community-resourced violence interventions:
Researchers are just starting to investigate these issues. Studies demonstrate that some [CVI-R] strategies offer promising results, but even the most celebrated programs do not yet qualify as “evidence-based.” In other words, it is not yet possible to use the findings of research to identify and implement the most effective strategies while rejecting or reforming ineffective approaches. The nascent quality of research evidence is at least partly due to the unique challenges CVI strategies present for evaluation research. Building strong research evidence for the CVI approach is difficult because many different strategies and outcomes are included. Evaluating CVI is also politically complicated. Unlike the consistent support provided to law enforcement, elected officials must be convinced that non-policing approaches to violence prevention are effective. Common-sense appeals or political rhetoric are not enough. Advocates for CVI must be willing to answer tough questions about their methods and results. Communities must invest in rigorous research to identify CVI effects if the strategies are to become key elements of public safety.
Origins and Foundations
Every program designed to reduce and prevent violence expresses a theory of crime causation, whether explicitly or implicitly. Among many theoretical explanations for why, when, and where violent crime occurs, several schools of thought point to various factors leading to crime, including social learning, social strain, social bonds, and labeling theory. Criminological theories attribute the origins of crime and violence to varying combinations of individual behavior and social conditions. Scholars note that neighborhoods with high crime rates are typically afflicted with other social and structural harms, including economic distress, negative health outcomes, substandard housing, and poor quality of life.
The three CVI programs managed by DYCD could be described as drawing to varying degrees on these established theories of crime causation. The Crisis Management System (CMS) coordinates services and supports for at-risk residents and implements gun violence prevention strategies inspired by the Cure Violence approach. Cure Violence programs assume that violence spreads from person to person, much like a virus. To reduce the harm of violence, communities must treat those already affected while delivering prevention for those not yet affected. “Outreach workers” connect residents with preventive services and collaborate with them to resolve various risks such as unemployment and inadequate housing. “Credible messengers” and “violence interrupters” — respected individuals with local “street” credibility — form relationships with at-risk youth and young adults to serve as role models and impart anti-violence norms and values, all to stop routine conflicts from escalating into violent confrontations.
CRIME THEORY |
|
| Social Learning | People learn social norms (i.e., accepted values and behavior) from significant others, relatives, friends, neighbors, and authority figures. Social norms may support or hinder violence. |
| Social Strain | People are more likely to commit crime when their social environment adds significant stress and makes achieving life goals, economic security, and social well-being difficult. |
| Social Bonds | People who are positively connected to their communities, friends, and families are less likely to commit crime. Connections may include relationships, commitment to goals supported by the community, time spent in activities to reach those goals, and shared values. |
| Social Labeling | Assigning a label of “criminal” to individuals, socially or legally, makes it difficult for them to belong to the community and increases their likelihood of committing more crime or other social harms in the future. |
Similarly, the Mayor’s Action Plan (MAP) is a multi-component initiative that relies on social services and community engagement to enhance safety and living conditions in public housing developments operated by the New York City Housing Authority (NYCHA). The initiative creates opportunities for public housing residents to engage in government decision-making and coordinate with public and private organizations to support and respond to resident needs. In essence, the model presumes that empowering neighborhood residents to address the needs of their communities is the most effective way to manage and reduce community-level strain.
The Precision Employment Initiative (PEI) is a workforce development program designed to assist residents of violence-prone neighborhoods working to overcome the barriers that typically impede their success in “traditional” jobs and training programs. The effort does not simply provide individuals with financial support. The initiators of PEI designed the program to increase participants’ commitment to legitimate work and career pathways, to provide them with skills to build attachments necessary for professional success, and to get them to internalize the values of the modern workplace. Beyond alleviating economic strain, PEI offers career opportunities that connect people to civic society.
All three initiatives were designed to reduce individuals’ contact with the criminal justice system’s punitive elements and to reduce the harmful effects on those already contending with the “criminal” label. Together, the programs comprise a strategy to reduce crime and violence in communities most affected by crime using short- and long-term community development strategies involving individual- and neighborhood-level interventions.
Program Reviews
The goals of John Jay’s program reviews were to 1) identify the underlying frameworks used to organize and manage programs, 2) explore the strengths and challenges of programs while considering their potential effects on neighborhood safety and resident well-being, and 3) begin to shape a plan of action for pursuing evaluation studies that may establish the effectiveness and value of each program for residents of New York City. Researchers began each review by interviewing agency staff involved in community-based programming. The team then interviewed CMS, MAP, and PEI staff and reviewed available documentation about each program, both internal and external, including contracts and other legal and policy documents, promotional and explanatory materials, media and press releases, official reports, and internal memoranda. Guiding questions were: What is the conceptual framework underlying each program? What appears to work well? What are the most pressing challenges, and how could the program’s success be measured and evaluated?
Researchers began their interviews in the Spring of 2023. DYCD staff helped to identify the initial set of interview candidates. Interviewees themselves provided additional names and suggested that researchers contact other people with relevant information and a willingness to participate. Prospective interviewees were contacted through email and phone. Each interview invitation included a description of project goals and an overview of data collection methods and guidelines. If an initial outreach failed, the research team made additional attempts. When data collection ended eight weeks later, the study’s “snowball” sampling technique yielded twenty individual interviews and one focus group of former program participants.
Program staff must have worked with their programs for at least 12 months to qualify as interviewees. This prerequisite ensured the team understood day-to-day operations. Researchers offered to conduct in-person interviews, but virtual interviews were extended as a courtesy at whatever time and location suited everyone’s needs. Most respondents opted for virtual interviews conducted via video (i.e., the Zoom application). All interviewees were informed that their involvement was voluntary and that responses would remain confidential. Researchers emphasized that any information obtained from the interviews would only be revealed in an aggregated and de-identified format. Interviews were recorded and transcribed verbatim.
To analyze interview transcripts, the research team employed template analysis. An interview guide and prior knowledge of the programs provided an initial set of thematic categories (e.g., challenges, successes, key principles), which were then adjusted and modified as interviews were coded, eventually adding to (and, in some cases, replacing) original themes with modified themes and codes based on response data. Interviews were conducted in a semi-structured fashion. Each interviewee was asked a standard set of questions, but all were encouraged to add whatever thoughts and observations they deemed relevant. The research team deviated from standard questions as necessary if interviewees were thought to have more information and insight about areas for which previous interviews provided insufficient information. While the research team obtained responses covering a wide array of content for each program, not every interviewee provided detailed input for every topic.
COMMUNITY VOICE |
| So when it comes to the community and the good thing about the staff we have, we pretty much all have community ties. [T]hat makes it a little easier, right? … I’m from here, I’m from right up the street. And this is where we work. … I get a lot of respect and love from the community. [W]hen they see my face, and my name is on something, the community is pretty much receptive. … When it comes to the staff, a lot of people we work with were friends prior to being in this space. Knowing them, it’s kind of personal. It’s personal, and that’s something that’s fulfilling, knowing that we are in a position to do something positive, do something great. Save lives. Change lives.
— Excerpt from project interviews |
The final sample was diverse, encompassing stakeholders, staff, administrators, organization leaders, former program participants, and individuals from partner organizations, including the private sector. As with all qualitative research, the interview and focus group sample was not intended to be statistically representative of all those involved in the programs but rather to give a detailed and informed view of key concepts and a range of opinions about how each program was operating. The following descriptions of the three programs derive from the researchers’ syntheses of the results of the template analyses, document review, and logic analysis, which is a method of assessing the facial validity (i.e., rational “sense” in light of scientific knowledge) of each program and how it was described by staff and administrators.
Crisis Management System (CMS)
The Crisis Management System (CMS) grew from a pilot program to curb gun violence launched in New York City a decade ago. Two successive mayoral administrations embraced it. At its core, CMS employs the Cure Violence model to combat gun violence. Developed in Chicago, Cure Violence has a well-established theory of change rooted in public health concepts, and it is deployed across the United States and internationally. Cure Violence staff members approach gun violence as a public health problem, a harmful population-level contagion that spreads from person to person. According to local staff and administrators, the CMS theory of change assumes that individuals engage in gun violence because they learn and adopt norms that permit or prescribe violence as a response to conflict or some other challenge.
According to those implementing CMS in New York, the first step is to stop the contagion. Violence interrupters (or VIs) work in the neighborhood to monitor social dynamics and leverage their relationships with residents to learn about and thwart any brewing conflicts that may lead to violence. The VIs are able to engage with residents most at risk for gun violence because of their “street credibility” derived from local knowledge, experience, and even their own previous participation in crime and violence. Their credibility is location specific. Credible messengers are most effective when they work in neighborhoods where they have spent significant time and where they are known to others.
Violence interrupters use their local credibility to form relationships with the (mostly) young residents at greatest risk for gun violence. Workers first try to persuade residents that violence is not a good solution to whatever specific problem is causing conflict. The next step is to continue talking with residents to persuade them that violence is not a valid solution to any problem. This process may take months or years. Violence interruption is an ongoing investment in neighborhood safety. It is not an emergency response protocol.
Violence interrupters are supported by outreach workers (or OWs), who are ideally also credible messengers. Outreach workers connect participants with services and resources that help ease the pressures causing them to turn to violence in the first place. Services often include mental health interventions, employment readiness support, access to housing, and legal assistance. In their daily tasks, outreach workers resemble counselors or social workers.
Finally, in addition to the individual-level efforts of VIs and OWs, CMS staff members work to change social norms at the neighborhood level through messaging campaigns, public events, and other demonstrations of community solidarity against violence.
New York City’s network of CMS organizations is distinguished by its size, the scale of wraparound supports, and the depth of coordination with other human services systems. In addition to direct violence interruption and conventional street outreach, CMS workers liaise with New York City schools. Other staff are assigned to work with hospitals whenever emergency patients present with injuries due to gun violence. New York City’s CMS programs are distinct and separate from the criminal justice system. Staff members work to retain the program’s legitimacy and the support of criminal justice agencies and other public officials. Funding for CMS central management is baselined as a routine part of the city budget, although CMS provider organizations depend on annual or multiannual contracts.
Program Logic :
|
Successes and Challenges
The research team interviewed eight CMS practitioners and a focus group of current and former program participants. Researchers asked respondents about the program’s successes and challenges. Practitioners typically credited its success to the qualities and skills of staff. Credible messengers and other staff are usually from the same neighborhoods as participants, providing them with unique knowledge and expertise. Staff could demonstrate their understanding of gun violence and its consequences as they empathized with participants and their families. They were more capable of building strong relationships with residents because they were familiar with events and conditions in program catchment areas.
Current and former participants generally agreed with the practitioners, telling researchers that CMS staff were effective mentors who cared about each participant’s success, providing constructive advice to help them shift their behavior away from violence. CMS staff are predominantly Black and Latinx, much like their communities. Staff members benefit from their backgrounds, which help them understand and identify with residents’ experiences and shape effective interventions and services. Interviewees noted that CMS’s school conflict mediation component was especially valuable for helping staff meet potential participants and connect with them directly.
Interviewees also voiced confidence in the CMS suite of supportive services. Staff believed the availability of social support was crucial for shifting participant attitudes and behavior. The wraparound services offered by CMS exposed participants to new opportunities and helped them to understand that engaging in gun violence was extremely risky to their health and freedom. Interviewees witnessed positive effects when participants were taken to cultural and recreational attractions outside their neighborhoods. Others mentioned their appreciation for CMS workers who encouraged participants to become more physically active and to engage in skill-based training for jobs and careers.
Interviews also surfaced some challenges affecting CMS operations in New York City. Agency leaders and program participants praised staff members, but they believed even staff would continue to be marginalized because of their backgrounds. One interviewee discussed how some city officials questioned why individuals with street credibility and previous involvement in the justice system were needed to do the work. Interviewees shared how the Department of Education required experienced CMS staff with previous justice involvement to be cleared by security each time they visited city schools, delaying their work significantly.
Credible messengers in CMS programs occupy a specialized role that others cannot fill easily. Interviewees expressed how challenging it can be for staff to maintain relationships with participants. Youth from high-risk neighborhoods can be hard to engage. Program staff must have the credibility to gain participants’ trust and, in the words of one interviewee, the skill required to “break down walls” with participants and build the relationships needed to carry out the work. At times, program staff work to form supportive relationships with family members of participants, as families may object to youth becoming involved with any government-funded program.
Per capita rates of violence in the United States are sometimes higher in rural and suburban communities, but programs like the Crisis Management System are usually implemented in urban neighborhoods. Furthermore, urban communities are often affected by historical patterns of disinvestment and economic isolation with primarily racially and ethnically minoritized residents. Program staff must be competent to work in these communities and to form equitable and supportive relationships with residents.
Interviewees argued that adding more trained staff would decrease caseloads and possibly enhance the program’s effectiveness. CMS staff receive training from the Cure Violence Global organization, but the training does not occur regularly. The inconsistent routine makes it difficult for staff to start work. Some wait months to receive training, delaying their full engagement in the work. The urgent need for training highlights the specialized skills needed to carry out the work of violence interruption. Violence interruption work can also be dangerous. Programs implement safety strategies for staff, but interviewees recounted instances when they were in potentially violent situations and their expertise was needed, but just showing up meant risking their safety.
Despite their efforts, interviewees mentioned that some residents complain that CMS programs must not be doing enough to prevent gun violence because shootings still occur. For their part, interviewees believed the number of violent incidents and shootings would likely be higher if they were not already working with participants, and CMS cannot be expected to stop every incident of gun violence given limits of staffing and resources. The programs may need more capacity to engage adequately with participants while also working to shift broader social and behavioral norms.
Other interviewees pointed to a school conflict mediation program that faced critical challenges in connecting with participants because programming was limited to school lunch periods, placing it in competition with students’ need to eat and their natural desire to socialize. One suggestion was to make this component an elective course to give participants a further incentive to join and commit. The logistics of school conflict mediation should also be determinative when assigning schools. CMS staff need to be positioned to get to schools quickly in cases of emergency and should not need to support neighboring catchment areas instead of their own. One interviewee noted that the officials who assign school services should communicate more often with school administrators to explain CMS goals and operations. Some school staff may believe that CMS’s presence reflects poorly on their ability to deal with conflict. More careful and effective communication from City leaders could help clarify CMS staff intentions to support schools’ efforts.
In addition to maneuvering to avoid “stepping on the toes” of school officials, interviewees noted that better relations between staff and police would improve their effectiveness. Police officers do not always take the efforts of CMS seriously. Some have been observed openly expressing skepticism or disdain for the program. Interviewees also noted that dealing with City bureaucracy can be challenging. For example, CMS’s recent transition to DYCD may have affected training protocols, database systems, and standardization of implementation. Interviewees told researchers that adjusting to these changes consumed valuable time as they tried to navigate new processes and comply with new requirements. Interviewees expressed dismay at being repeatedly asked to defend their strategies and expertise to agency administrators. Program leaders and staff alike acknowledged that tracking data about program efforts was necessary, but several interviewees complained about the current and past databases, describing them as deficient and not user-friendly.
Finally, CMS interviewees advised that staff experience burnout due to daily stresses in their line of work. Staff members remain invested in their relationships with participants, and they share how much it affects them when they lose someone to gun violence. The leaders and managers of CMS offer retreats and other activities to help staff decompress, but interviewees told researchers that even more support was needed, particularly routine therapeutic services, mandatory days off, and quarterly staff retreats.
Evaluation Agenda
The Crisis Management System is a community organization network forming supportive relationships and delivering services to the community members most at risk of gun violence. At its core, CMS deploys a team of credible messengers to mediate street conflicts and connect high-risk individuals to services that may dissuade them from engaging in firearm violence. The logic of the Crisis Management System assumes that gun violence can be reduced through two causal pathways. The first pathway provides individuals with services designed to minimize their exposure to gun violence, recognizing that individuals facing socioeconomic challenges are often more vulnerable to becoming victims or perpetrators of gun violence. Program staff members help create alternative futures for individuals who may otherwise be trapped in environments conducive to violence.
A second pathway targets the community at large. To shift societal norms and perceptions surrounding gun violence, CMS staff organize public events and educational campaigns to denormalize violence and serve as a platform for the community to unite against gun violence, mourn victims, and stand together in solidarity. CMS leverages public health strategies to advance this mission. Public education campaigns are pivotal in disseminating a strong anti-gun violence message throughout the community. The program aims to create a cultural shift that makes gun violence unacceptable and denormalized within the community.
New York City faces considerable obstacles in designing and managing an evaluation plan for CMS programs. The program model itself introduces many complications because it combines interventions at individual and neighborhood levels. An evaluation design that measures outcomes among individual participants alone would disregard the Cure Violence model and fail to capture all possible effects. On the other hand, measuring effects at the neighborhood level limits possible sample sizes and introduces substantial risks of unmeasured correlates and variations in program design and implementation. Community-level analyses also face “contamination problems,” in which innovations being tested in a group of treatment neighborhoods are admired and adopted by neighborhoods supposedly serving as comparison areas. It can be difficult for researchers to control or even identify and track numerous factors likely to be involved in neighborhood-level evaluations.
Evaluating the CMS model in New York City would be different and possibly more complex than research on the original Cure Violence approach. City officials designed CMS to include many of the basic concepts developed by Cure Violence, but the model involves a broader array of complementary supportive services. While the original Cure Violence approach mentioned supportive services and individual assistance, the model focused on community-level interventions and social norm change.
When JohnJayREC began investigating the effects of Cure Violence, researchers proposed an evaluation framework to guide their data-collection efforts. The 2014 framework divided the program’s focus into two causal pathways: behavior change and norm change. The behavior change path included outreach and mediation with individuals and intermediate outcomes related to education, employment, relationships, skills, and values—the only other activities leading to those outcomes involved hospital interventions and “meetings” with high-risk youth. A new framework would be needed to account for Cure Violence’s subsequent expansion and enhancement in the city’s Crisis Management System.
The revised framework may be more appropriate for guiding evaluation efforts for the 2023 version of CMS. First, it reflects public health framing for the major causal pathways following program activities. Rather than “Norm Change” and “Behavior Change,” the new model divides CMS efforts into those designed to “Mobilize” the community against violence and “Interrupt, Identify, and Treat” individuals most at risk from violence. The activities and intermediate outcomes along each pathway in the model portray the goals and methods of CMS programs in greater detail. Efforts focused on individuals begin with mediation and wrap-around services, including hospital-based referrals, mental health supports, and employment assistance, and the locus of activity shifts to outreach efforts that connect participants to services.
Second-level intermediate outcomes are operationalized as increasing the skills and knowledge of individuals learning to avoid violence and focus on their social development. At the same time, communities gain increased motivation and capacity to reduce violence. Together, these are hypothesized to produce reduced violence and a social environment where violence is no longer seen as normal.
Mayors Action Plan for Neighborhood Safety (MAP)
The Mayor’s Action Plan for Neighborhood Safety (MAP) was developed in response to the disproportionate rate of crime and violence occurring in and around residential developments administered by the New York City Housing Authority (NYCHA), the city’s public housing agency. Beginning in 2014, MAP introduced greater opportunities for NYCHA residents to influence and even determine the sort of services and public support provided for public housing communities. Soon, MAP was operating in more than two dozen NYCHA developments.
Originally, MAP comprised three investments. First, it invested in people by providing services through various city agencies and programs, including the Human Resources Administration (HRA) and the Summer Youth Employment Program (SYEP). Second, it invested in places through various physical improvements and community beautification efforts. And third, it invested in communities and social networks by creating processes for residents to influence government decision-making and shape public housing policies.
According to available documentation and staff interviews conducted for this assessment, MAP’s original theory of change derived from insights related to human behavior and the influence of physical space on the social environment. MAP staff members addressed historical impediments to community cohesion and collective efficacy. They worked to build a sense of individual and collective hope by providing opportunities for resident empowerment and group decision-making. Stakeholder teams established at each MAP development involved 15 residents recruited for their diversity and previous experience with civic leadership. Up to ten additional members were recruited from city agencies and nonprofit partners. Each stakeholder team worked with MAP representatives to address problems and propose solutions for their community.
Program Logic :
|
City government and nonprofit partners provided additional staff. MAP Engagement Coordinators (MECs) collaborated with resident stakeholder teams to connect NYCHA residents with community organizations and city agencies. MECs were usually staff members at nonprofit partners, such as the Center for Justice Innovation, Jacob A. Riis Settlement, and Los Sures. They served as advocates for community concerns, shared information, and acted as thought partners in developing solutions and implementation plans.
Finally, the engagement coordinators and stakeholder teams collaborated to organize NeighborhoodStat (NStat) meetings, where MAP leaders, residents, and city agencies discussed community challenges and “co-produced” solutions. NStat meetings encouraged members of MAP stakeholder teams to voice their opinions on issues affecting community safety. The meetings provided a space to develop collaborative solutions and opportunities to connect residents with existing services, such as those provided by the Human Resource Administration (HRA), and to develop new services and programs, such as cultural programming and youth mentoring. NStat involved two levels of meetings. Original NStat was designed as a series of cross-community meetings for participating NYCHA developments in each of the five New York City boroughs (Bronx, Brooklyn, Queens, Manhattan, and Staten Island). Local NStat was launched to provide more focused meetings in one NYCHA development at a time, allowing residents to discuss challenges facing their specific communities.
COMMUNITY VOICE |
| We build good relationships with residents, with community partners, and with city agencies. We just do a good job of bringing the stakeholders together on a regular basis. And so, at this point, it doesn’t feel transactional. It’s not just us asking agencies, [“H]ey, can you fix the street?” I think they truly see us as partners. It has taken a lot of time building those relationships.
— Excerpt from project interviews |
In addition to the worthy goal of strengthening resident engagement and collaboration, MAP addressed the historical lack of investment in NYCHA communities, both social and economic. Staff worked across NYCHA communities to provide various social services, including after-school programming, access to community centers, and summer youth employment opportunities. Staff and nonprofit partners also worked to remedy the physical dilapidation of NYCHA buildings and grounds. Capital projects and construction investments spurred by MAP led to revitalized common spaces and began to lessen opportunities for crime in recognized “hot spots.” Projects included the installation of closed-circuit television (CCTV) cameras, layered access doors, and improved lighting.
Successes and Challenges
Researchers interviewed practitioners working in MAP’s community engagement and program support efforts to assess the initiative’s success and challenges. Interviewees believed notable successes of MAP included the “centering of community voice” for the co-production of public safety, stakeholder team investments, changes in local government culture, and the coordination of resources and services available to residents. They were especially proud of the MAP initiative’s successes in leveraging residents’ knowledge and experience in addressing challenges to public safety and quality of life. Local NStat meetings were described as a critical aspect of the initiative because they provided accessible social spaces where residents could work together to achieve tangible “wins” for their communities. Interviewees indicated that stakeholder teams, MAP staff, and partners worked together through all phases of the effort, from resident-driven project design to implementation, including government policy changes. These experiences underscored residents’ ability to affect city government and community decisions, processes, and actions to ensure community safety and well-being.
Interviewees praised MAP’s efforts to engage NYCHA residents. Key strategies included local events and meetings. Both were effective opportunities for resident stakeholders to hone their leadership and service skills. Public activities united residents and kept them involved with their neighbors and communities. The interviewees were particularly adamant about prioritizing residents’ self-determination and perspectives. Residents deserve the power of self-determination and the ability to set the direction for change in their neighborhoods. They are most familiar with their communities, the nature of pressing challenges, and, often, the most practical source for feasible and effective resolutions. Residents also respond when they see their input being valued, further strengthening their collective voice and power. With this ethos, MAP staff created strong relationships with residents to carry out the Initiative’s efforts.
COMMUNITY VOICE |
| I think we are changing the way government operates. We are — at least our unit — we’re flexible. We’re involved. And we’re not just funders at a distance. We are actively designing and implementing the work with community members. That’s a good thing — like, government, obviously, should have rules and processes, but we should also be responsive and proactive.
— Excerpt from project interviews |
Interviewees offered examples to demonstrate that they had witnessed a shift in the cultures of their government colleagues and other agencies concerning the centrality and priority of resident voices. Furthermore, they noted headway in getting agencies not traditionally associated with public safety to accept their role in shaping resident perceptions and experiences. Increasingly, according to interviewees, government agencies placed greater value on resident experience as a form of expertise. Some considered changing procedural and operational norms to better account for this experience. Interviewees noted the extensive investment in time and relationship management required to move the initiative’s work forward but also expressed satisfaction that they had been able to do so. In many instances, this included bridging chasms between community residents as much as between residents and government bodies.
Despite the successes of MAP, interviewees noted key challenges. Among them, interviewees mentioned initiative branding, program management, resident association relationships, and capacity building. Not surprisingly, MAP residents do not always associate MAP itself with the initiative’s accomplishments. Interviewees reported that MAP efforts are easily mislabeled. Staff must remind residents that they are city employees and that their efforts are representative of the city’s commitment to their well-being. MAP is still not a recognized brand in many communities. Interviewees acknowledge that, like many city initiatives, MAP depends on its visibility to engage residents. With consistent messaging and branding, residents would be more likely to see connections between MAP-related projects. They would learn to recognize the initiative as a comprehensive and effective response to community needs and desires.
Managing an initiative the size of MAP presents many challenges. Interviewees expressed concern that local government policies and practices are not always managed “smoothly,” which can lead to disruption in community services and support. For example, the change in MAP’s official agency “home” at DYCD introduced different protocols and procedures. Interviewees also expressed concern about losing team members due to changes in fiscal management and administrative complications. The slowness and rigidity of government bureaucracy were frequent themes in comments from interviewees.
While interviewees saw government-community collaboration as a strength of MAP, they described it as a challenge. Interviewees mentioned the challenge of interactions with resident associations, which required staff to connect one-on-one with RA leaders, support RA-sponsored activities, and publicly affirm RA leadership. Limits on MAP’s financial and organizational capacity complicated relationships with residents. The wide-ranging nature of MAP programming and the dedication of its staff sometimes gave residents the impression that MAP was a vehicle for resolving any problem, including issues with NYCHA. Residents often expressed their desire for greater support in dealing with NYCHA policies or various personal matters MAP was not equipped to address. For example, staff members had to acknowledge that MAP could not address all NYCHA property issues, including those affecting individual apartments.
MAP’s resident stakeholders received an $85 monthly stipend for their participation, but interviewees favored additional compensation. More than financial investments to individual stakeholders, interviewees hoped that MAP could provide residents with the skills to continue mobilizing their neighborhoods long after funding for the initiative ended. Interviewees expressed concern that even residents working with the initiative for years may not be fully prepared to sustain the effort without the technical assistance and coordination expertise that MAP’s city employees could bring. Many residents once participated in MAP-related work, such as participatory budgeting, but the resident stakeholder teams now shoulder more of the workload with less support from residents in general.
Evaluation Agenda
Evaluating the Mayor’s Action Plan is at least as challenging as the Crisis Management System. The program model combines interventions at the level of individuals and communities, but MAP addresses an even wider array of outcomes. As with CMS, it is difficult for researchers to control and track the many factors hypothesized as MAP-relevant outcomes. John Jay College was engaged to evaluate MAP in 2017 when the Mayor’s Office of Criminal Justice (MOCJ) supervised the initiative. While simultaneously assisting city officials with the management of the initiative, researchers devised a quasi-experimental evaluation to measure outcomes in NYCHA communities participating in MAP and compare them with a matched set of NYCHA communities not participating in MAP.
The evaluation team began collecting data in 2017, a year before many of MAP’s core components were operational and before MOCJ leaders began referring to the fully realized initiative as “MAP 2.0.” The JohnJayREC team asked NORC at the University of Chicago to create a survey to measure residents’ experiences, perceptions, and opinions in the 17 MAP developments and the 17 comparison sites. The addition of resident surveys resulted in an evaluation with three key components: 1) administrative data from police and other partner agencies, 2) interviews and observations with MAP leaders and resident participants, and 3) surveys of NYCHA residents in MAP sites and matched comparison sites.
The research team fashioned a beginning evaluation framework to guide the effort based on their initial conversations with city agency staff and MAP team members. The model was relatively ambitious on the outcome side but virtually unspecified on the left side of inputs and outputs. The only input specified in the model was MAP itself, followed by a series of outputs reflecting the goals and objectives of MAP (government competence and efficiency, collaboration, defensible space, etc.). Every output identified in the model flowed through the same two pathways toward the outcomes and final impacts. Even if an evaluation found all outcomes and impacts were significantly affected by MAP in just the way portrayed in the MAP framework, what policy-relevant inferences would be possible? MAP is effective, but which component? Which outputs were important, and how did the initiative produce those outputs? An evaluation that cannot measure the full chain of events involved in the relationships among each input, output, and outcome does not generate actionable conclusions for policy and practice.
Researchers developed a more detailed framework for evaluating the MAP initiative after reviewing available reports and documents and interviewing staff from city agencies and nonprofit partners during this assessment. Importantly, the revised model incorporated shifts in how program staff evolved in their conceptualization of MAP. For instance, the updated model prominently features co-production of safety as a joint government-community endeavor and emphasizes the community’s role in advocating government responsiveness. The previous model captured MAP’s concern with generating community trust in government but may have implied deference to government processes (i.e., legitimacy). The updated model focuses on generating trust through staff and government collaboration.
While the updated model is more detailed than the previous model, the complexity of its many causal pathways would overwhelm data collection plans. Researchers would be required to edit the model to disregard some constructs, or they could measure the relationships between activities and outcomes in sequence during several studies. Simply estimating the effects of NStat, for example, would contribute important information about the effectiveness of MAP. Researchers could measure variations in NStat meetings and activities between NYCHA developments and then track the association of those differences across the expected outcomes of resident engagement, community cohesion, and subjective reports of resident satisfaction with MAP, NStat, and NYCHA generally.
Distinguishing outcomes from inputs and intermediate outcomes is important for understanding the evolution of MAP. It would provide a method of testing whether differences in implementation lead to expected differences in end outcomes (i.e., crime reduction and public safety). The MAP initiative involves a very wide range of services and supports. These services are provided by public and private agencies and by residents themselves. To include an estimate of MAP implementation in the evaluation, researchers would need to compile an assortment of indicators to represent the activities undertaken in NYCHA developments and measure them over time as the initiative affects several outcomes.
The staff and partners of MAP collaborate to 1) ensure the provision of social support services for NYCHA residents, 2) coordinate government action, both formally and through informal channels, 3) advocate for the interests of residents, both formally and informally, 4) directly engage with residents, 5) create public events and public programming, and 6) make capital investments in repairing, beautifying, and maintaining MAP investments. Some of these activities are facilitated through two processes, NeighborhoodStat (NStat) and participatory budgeting, which typically takes place during NStat meetings. Activities are expected to foster a culture of resident engagement with greater deference to resident choice within city government, enhanced trust between the community and government officials, resident commitment to the aims of the Initiative, greater engagement between residents as a social group, and increased use of public spaces. These should lead to enhanced government responsiveness, more opportunities for community-city co-production, community cohesion, and collective efficacy.
Ultimately, the MAP program works when residents have greater access to social support and greater capacity to petition the government, mobilize resident actions, implement solutions to development issues, and create a shared set of priorities. The lasting impact of these outcomes is hypothesized to be greater quality of life, reductions in the fear of crime, an increased sense of safety, and objective reductions in crime and community violence. The challenge for future evaluations is to establish these causal assertions with defensible evidence.
Precision Employment Initiative (PEI)
New York implemented the Precision Employment Initiative (PEI) in response to increases in gun violence and unemployment during the social and economic upheaval of the COVID-19 pandemic. The program is described as a combination of 1) job training and placement, 2) violence and crime reduction, and 3) economic investment in “green jobs.” The United States Bureau of Labor Statistics defines green jobs as work that may either “benefit the environment or conserve natural resources” or make “production processes more environmentally friendly” and less resource intensive.
The intended outcomes of PEI include reduced criminal activity and violence among participants, reduced crime and increased socioeconomic development at the level of the community, and meaningful contributions to improving the environment. As of 2023, the initiative served neighborhoods in Brownsville (Brooklyn), Mott Haven (Bronx), and Jamaica (Queens). Overseen by DYCD’s Workforce Connect team, PEI operated as the Civilian Climate Corps by BlocPower, a private-sector climate technology company focused on analyzing, financing, and upgrading homes and buildings to reduce environmental hazards or resource inefficiencies. ARA Emotional Wellness & Mental Hygiene provides therapeutic support to PEI participants, while job training is by private contractors partnering with BlocPower primarily on a contract-by-contract basis.
Participants begin the program with needs assessments to gauge their math skills, reading comprehension, and familiarity with workplace culture. BlocPower provides approximately one month of job readiness and “soft skills” training, including resume-building, interview preparation, digital and financial literacy, workplace etiquette, interpersonal skills, and conflict management. Resources are added as necessary to address weaknesses in these or other areas. Other assessments weigh the need for social support like childcare or housing. ARA conducts assessments of participants’ psychological and emotional well-being and may offer services and training on stress management and conflict coping skills.
PEI’s core curriculum consists of a stepwise progression from basic technical or trade skills acquisition or certification (e.g., OSHA 40-hour training) to specialized training (e.g., energy auditing) and on-the-job training. Such training would usually be obtained at participants’ expense, but PEI provides hourly wages for trainees. At the end of on-the-job training and wraparound services, participants should have sufficient professional socialization, interpersonal and technical skills, certifications, work experience, and references to be employable in the green sector.
When it launched, PEI aimed to serve approximately 1,500 participants with a $37 million budget. In 2022, funding was increased to $54 million, allowing the program to serve up to 3,000 participants. While the program initially recruited participants through social media and publicity campaigns, subsequent recruitment efforts often came by word of mouth, primarily from current or former participants.
Program Logic :
|
Successes and Challenges
Researchers interviewed a sample of individuals either recently working as part of PEI or who assisted in its development. Among the most important successes and points of distinction emphasized by interviewees was that participants were reliably paid for every hour spent in training. This was perceived as an essential component of PEI’s success. PEI participants earn $20 an hour for training while in the program. Compensation is deposited directly into their bank accounts, which resembles the mode and regularity of a paycheck in professional workplaces. This practice is expected to reinforce financial management skills. Program staff members report that training compensation attracts a wider range of individuals to PEI.
The program provides a selection of green career paths, flexible training hours, and a liberal and inclusive set of eligibility criteria. While recruitment tends to focus on individuals with lower chances of job success without support, its participant population has a variety of risk profiles related to prior “street involvement.” Training is provided in locations throughout the neighborhood to promote accessibility, and PEI’s lack of an age requirement has allowed it to accommodate the rising average age of individuals involved in violence. Interviewees pointed to the cohort-style induction of PEI participants and peer-group training as a mechanism of community-building, program completion, and engagement. Furthermore, participants also have opportunities to engage with previous participants, further fostering a cross-cohort sense of community. These features provide participants with a safe learning environment close to home and free from distractions or potential violent altercations.
Some interviewees emphasized that program administrators could be more selective about contracted service providers than is usually allowable in City programming. The wide variety of wraparound and job development services offered to participants through PEI comes from the program’s strong relationship with partner agencies. Elite Learners, also a CMS provider, offers PEI participants services ranging from housing services to entrepreneurship training. Blocpower’s community connections contribute to the abundance of training options offered to participants with employers who understand the dynamics of working with participants. Its agency partners offer various social and mental health services, emotional wellness supports, and green job development opportunities that PEI participants can access. PEI is currently supported with tax levy funds, which provides flexibility for program management, makes the program sustainable at the municipal level, and demonstrates the city’s commitment to economically neglected communities.
Interviewees also noted challenges, especially the need for data to guide decision-making, which was an early focus of DYCD staff as the agency began to accept responsibility for the program. Aside from internal and interagency data collection and sharing, interviewees stressed consistent data collection and sharing from program service providers. Program leaders required more – and more precise – knowledge about program outcomes, participant experience, and participant outcomes. For instance, what might impact an individual’s participation and force them to end their involvement with the program? Staff described the need to track participant outcomes during the program and after graduation, specifically whether participants gain full and non-temporary employment post-completion. Interviewees were initially unable to estimate the number of individuals with current or prior involvement in the criminal legal system. In early cases, program data revealed some anomalies that went unnoticed. For example, the database of participant addresses contained some outside city limits and outside of the program’s enrollment criteria.
COMMUNITY VOICE |
| [We] pay throughout training. As soon as people enter the program, we compensate them for the hours they put in. It makes a real difference. For some specialization opportunities, there are certifications people can get when they exit, certifications that [other people] have to pay for. We cover that as well. So that’s kind of removing some barriers to entry.
— Excerpt from project interviews |
Underscoring data collection issues was the general sense that PEI launched without well-defined outcomes to determine whether its efforts were successful. As the program began to serve hundreds of individuals, definitions of success were still being developed. If PEI were understood primarily as a jobs program, then unsubsidized employment should determine success. If PEI was fundamentally intended to reduce violence, then decreases in the number of violent incidents within its service areas would be more relevant.
Interviewees also highlighted variability in the process for participants. Sometimes, this variance occurred due to differences in the level of support participants required and the time it would take to bring them to similar levels of job readiness. In other cases, variation may result from the program being tailored to meet the unique aspirations of participants. The time investments necessary to produce a construction trades expert, electric vehicle technician, or green entrepreneur likely differ. Some participants may reach job readiness without needing to complete all aspects of programming. Overall, greater curriculum clarity would allow the program to perform better. Additionally, clear definitions would inform a more consistent reporting mechanism to understand other program outcomes. Finally, interviewees suggested similar programs with which they were familiar operate with more staff and administrative and technological resources than those currently allocated to PEI.
Evaluation Agenda
Unlike CMS and MAP, there are no previous evaluations of PEI from which an established theory and logic for its activities can be refined. Nonetheless, the research team developed an understanding of the program’s activities and outcomes, and the relationships between them, that could inform future evaluation. PEI’s cohort design provides participants with important opportunities for social interaction and relationships, which encourage commitment to the program. The program works with participants long enough to provide them with job referrals and recommendations, a valuable form of social capital. Recruitment is conducted mainly through networking, providing program graduates and current participants with an additional layer of social connection beyond the formal aspects of the program. The various forms of social support are encouraged and fortified by the program and the resources derived from it.
The PEI program (1) provides social and short-term economic support and (2) job training and professional development for green sector employment. The main components of PEI are income subsidies and case management, which provide the foundation for participants to become ready for subsequent activities. Other core activities include technical training, specialized skills, job readiness, and on-the-job training. The immediate outcomes of job readiness training are the participants’ familiarity with workplace norms and expectations (i.e., professional socialization) and newly acquired professional interpersonal skills (e.g., conflict resolution strategies).
Completion of technical skills training leads to certification credentialing. On-the-job training supports individuals by providing social capital (i.e., contacts and references) and work experience. Combining all intermediate outcomes should enhance participants’ employability, eventually leading to a career in the growing green sector.
As noted in the discussion of MAP, an evaluation framework where all program activities and outputs flow to outcomes through one or two pathways may reveal an overly simplified view of a program. In the case of PEI, all program activities are designed to improve just one outcome – employability. This one outcome hypothetically results in several long-term outcomes. According to the PEI theory of change, program participants are expected to be economically stable, less prone to crime and violence, and to express intentions for career development and asset growth. As fewer individuals engage in violence and crime, aggregate levels of violence and crime are expected to drop, and aggregate economic indicators should improve.
A Path Forward for Practice and Evaluation
New York City’s Department of Youth and Community Development implemented three potentially effective programs to prevent violence, reduce crime, and support resident well-being. Each program was designed to leverage the community’s resources to avoid relying exclusively on law enforcement.
City officials and agency staff designed each initiative after consulting with experts and examining the results of similar efforts in other jurisdictions. In this way, each program could satisfy the federal government’s definition of evidence-informed policy and practice. They are not yet, however, evidence-based as defined by the field of evaluation research. Programs must undergo rigorous evaluations to merit that label.
Rigorous evaluations are designed to identify the sources of program effects. Such studies are not necessarily experimental, random-controlled trials. Actionable and policy-relevant research, however, must do more than simply catalog a program’s intentions and then measure whatever outcomes follow implementation. Rigorous evaluations measure outcomes but also test their empirical relationships to the activities and processes of the programs and policies that were intended to produce them.
Effective evaluations rely on detailed frameworks that guide data collection, data analysis, and the interpretation of results. To design reliable evaluations of complex programs, researchers collaborate with policymakers, agency leaders, program staff, and community residents to create detailed evaluation frameworks based on the logic of each program or policy. The frameworks are used to design every step of an evaluation, revising as necessary to account for changes in policy and practice. The evaluation frameworks presented here are draft versions offered as starting points for additional efforts to employ evidence-based public safety strategies in New York City.
__________________________________
Funding
The New York City Department of Youth and Community Development (DYCD) provided funding support for this analysis, but all conclusions are those of the authors. Funders and partners of the Research and Evaluation Center are not responsible for any findings presented in Center publications.
Acknowledgments
The authors appreciate the contributions of other staff members at the Research and Evaluation Center who organized this project and reviewed drafts of this report. In particular, we appreciate the assistance of Kathleen Tomberg, Rebecca Balletto, and Rhoda Ramdeen. The authors are also indebted to the many agency staff and community members who met with the research team to share their knowledge and experience. Their contributions helped to shape and inform the final product.
Recommended Citation
Blount-Hill, Kwan-Lamar, Patricia Cobar, Fidel L. Osorio, Sheyla A. Delgado, and Jeffrey A. Butts (2023). Designing Safety — Public Safety Initiatives Managed by the New York City Department of Youth and Community Development. New York, NY: Research and Evaluation Center, John Jay College of Criminal Justice, City University of New York.