Martha Wade Steketee, Consultant, New York, NY
Jeffrey A. Butts, Director, Research & Evaluation Center
October 5, 2015
Preparation of this report was supported with funding from the Manhattan Theatre Club (MTC) in New York City. The authors acknowledge the support and guidance of David Shookhoff, MTC Director of Education, and Evan Elkin, director of Reclaiming Futures in Portland, Oregon and formerly of the Vera Institute of Justice in New York City. The authors were engaged by MTC to attend the rehearsals and performances of Stargate Theatre Company during its summer 2015 season, to comment on its evaluability, and to prepare evaluation-ready materials. Martha Wade Steketee led the work and conducted interviews with David Shookhoff, Paul Gutkowski, Judy Tate, Stephen DiMenna and Evan Elkin. Ms. Steketee attended four rehearsals and both authors joined the Stargate company at a performance of the Broadway musical Hamilton, and attended public performances of Stargate’s Deeper Than Skin.
Copyright is jointly held by the Manhattan Theatre Club and the Research & Evaluation Center, John Jay College of Criminal Justice, City University of New York.
PREFACE: The Stargate Experience
Stargate Theatre Company began in 2013 as a theatre-making, workforce readiness, and literacy project for justice-involved youth. For seven weeks each summer, a small group of young men meets at least four days per week to write, rehearse and perform a collaboratively crafted play in an Off-Broadway venue in New York City.
Stargate members are recruited from youth service agencies around the city, vetted and then hired as employees of the Manhattan Theatre Club. An in-depth application process includes interviews and a workshop experience that serves as an introduction to Stargate. The application process clarifies the expectations of the program and facilitates the selection process for staff.
Two senior teaching artists have been with Stargate since the beginning: Judy Tate and Stephen DiMenna. Paul Gutkowski became company manager in 2015 and brought previous experience in theatre as well as in working with justice-involved youth.
The first three weeks of Stargate focus on writing exercises to generate the raw material that eventually becomes performable text. By the fourth week, the company is focused on performance and staging, with movements designed by a professional choreographer. The company’s final performances serve as the dramatic and often emotional conclusion to the Stargate experience, but the primary goal of the program is to help company members develop crucial life skills. By conceiving, creating, and performing their own stories, youth members learn skills that are important for theatre as well as for all their future endeavors: discipline, resilience, teamwork, and creative problem solving.
About three-quarters of the members recruited each summer complete the entire seven-week program. Even those who are unable to meet the stringent time commitment, however, are still credited in the final program for their contributions to the written material and to the final production.
The staff and supporters of Stargate Theatre Company hope that every member of the company finds his unique voice by writing and performing, and that through theatre he may discover a path to personal success.
Stargate operations are consistent with existing research about adolescent development and the role of social assets and skill acquisition in preventing anti-social behavior. The theatre company’s next task is to document the actual effectiveness of their approach by conducting a rigorous evaluation that measures youth outcomes.
The Stargate Theatre Company, embedded in the resources, expertise and artistry of the Manhattan Theatre Club Education Program, has been described in rich and evocative detail over the three years of its existence in annual reports, white papers, funding applications and reports, and print and broadcast media coverage. In 2015 the Company Manager described this seven-week program of theatre games, script development and ensemble building for justice-involved young men as a means to “develop the skills and capacities to be successful and to develop empathy as a tool to disengage from the criminal justice system” and to “figure out who you are and what your values are and what you want in a place where you won’t feel judged or you can feel vulnerable through theatre.” As one Stargate member said after the final 2015 performance: “There’s nothing like being paid to write your own story.” There’s power in this program, and the Stargate model is now ready for review.
Stargate members are justice-involved young men who craft a new theatre piece over the course of seven summer weeks and perform it to an invited audience. The program has described itself as providing transitional employment, work-readiness training and literacy education through enforced standards and writing expectations in a supportive and rigorous environment – that happens to be theatre. The young writer-performers work for pay, guided by professional theatre artists and educators to compose, rehearse, and ultimately to perform their own words about their own life experiences, hopes and fears. Three original pieces of theatrical collage have been staged in the first three program development years: Behind My Eyes performed by seven collaborators in 2013, Weathering the Storm crafted by nine collaborators in 2014, and Deeper Than Skin performed in 2015.
The writers were assisted by a large ensemble of Stargate staff and consultants that included: co-Artistic Directors; Company Manager; professional choreographer; stage manager; MTC Project Director, Project Manager, and Assistant Education Director; project assistant; and MTC Education Program intern.
Stargate is a programmatic sibling of youth theatre projects developed over the past several decades in different parts of the world to develop the artistic voice of young people, some of which are described in this report. Stargate was informed by these other programs and findings and born out of connections among Board members and staff of MTC and the Vera Institute of Justice. The first version of the program was outlined in a 2012 white paper to provide theatre programming as non-traditional work for young men transitioning out of Rikers Island. The goals outlined in 2012 were to prepare the young men for “successful reentry into their communities with work readiness and literacy skills.”
The 2012 paper continued:
“With a primary goal of ensuring successful reentry and reducing recidivism, the main components of this initiative include: therapeutic theatre intervention, reentry support services, transitional employment and work readiness, and literacy education.”
The first year of operation was 2013, and the program experienced continual improvement and refinement in each of its three years. The recruitment process always included an orientation workshop for potential applicants, with an evening of warm-up activities, writing, and performance. In 2015 the company added a one-on-one interview with each youth in which staff discussed the artistic goals for the season and explored each youth’s social and health history. Together, these steps generated a cohort willing and able to engage in teamwork, group creation and performance, with individual responsibility and less attrition than in prior years. The program schedule was expanded in 2015 to seven weeks to allow more development and rehearsal time, and the stage manager was brought in earlier in the rehearsal process to monitor staging and to establish a leadership role with the group. Daily staff debriefings of potential issues during rehearsal weeks addressed problems early.
What Stargate may uniquely contribute to the literature on youth development arts programming is its yeasty and exciting amalgam of creation, ensemble building, arts exposure, and public performance. Senior Project Advisor and founding collaborator Evan Elkin described the basic program elements in an essay included as part of the Stargate Theatre Company Guide (in development).
Stargate is, at its core, an experiential theatre-making program for court-involved youth. But it’s also a program which has intentionally sought to integrate the core principles of positive youth development in an authentic way: Stargate is at once a paid job for our young ensemble members, a work readiness training program, a literacy program, and a transformative experience which allows young people to work through some of the same emotional and behavioral issues that might be targeted in a traditional therapeutic intervention. — Evan Elkin
The program is in part pure theatre–holding to theatre rehearsal room and performance standards with rehearsal and performance reports and the stage manager, and meeting in rehearsal spaces of Manhattan Theatre Club in the middle of Manhattan’s theatre district. The program also addresses social justice and racial equity for Evan Elkin, for whom the program allows the justice-involved youth to create and make mistakes just like other young men their age.
The following descriptions and reflections are gleaned from observations and conversations during the 2015 Stargate Theatre Company season. Interviews with the founding and current staff and consultants provide clear statements of mission and vision for the project and reflections on program changes. Observations of rehearsals scattered throughout the 2015 season and the public performances provide themes, quotations, and a palpable sense of enthusiasm among participants and observers alike about Stargate’s possibilities.
Reports from staff members and a review of past program descriptions show a staff constantly calibrating program elements (intake, processing, review of youth progress, program termination), and drawing from decades of combined expertise in theatre education, youth development, and theatre creation with young people. Staff openness to change, constant surveillance of their own practice, integration of research knowledge from the first moments of the program’s existence matched with the program’s stability make it ready for evaluation. An effective outcome evaluation of Stargate would collect data that correlate activities at the level of individual youth with outcomes also measured at the youth level.
RECRUITMENT & MAINTENANCE: Managing the Company
Stargate revamped recruitment strategies and methods during the third season so that writer-performers arrived after extensive one-on-one interviews with program staff. “We need to know who’s on medication, who’s got any kind of mental illness,” recalls co-Artistic Director Stephen DiMenna, who credits Project Director David Shookhoff with great flexibility in making program refinements. “We felt on day one where we were last year at the end of week one and just getting to know who these kids were. We felt like we knew who they were on day one,” co-Artistic Director Judy Tate reflects. “We knew we needed a Kid Whisperer so to speak, a Company Manager. We knew we needed somebody else so that we could focus on the art and somebody else could be the issue person.”
Paul Gutkowski, the Company Manager hired in 2015, knows juvenile justice populations and theatre, and describes his skill set as “combining theatre and trying to find truth through characters and through stories and learning about who you are and how you interact with the world through the safe space of like creating a character to deal with it.” He worked with Stargate’s co-Artistic Directors to turn initial interviews into bonding experiences. “By the end of it, the person stands up and shakes your hand and smiles, and at the end they’re more fired up about being here than when they sat down.” In the rehearsal room, all staff work to maintain consistent support. Gutkowski underscores safety and consistency as key elements:“The level of support for the guys, the safe space for the guys is obviously connected. And I think being really consistent, treating them like professionals until they have in some way demonstrated that they need different kind of support than that, is kind of a cool experience for them.”
All of these efforts build the context in which the art is made, through writing and acting exercises, building a script, and putting it on stage.
COLLAGE CREATION: In Notebooks, On the the Walls, On the Page
Judy Tate brings all her skills as a professional actor, playwright, and writer for television soap operas to the table once the Stargate participants are selected. She leads the writing development activities in the rehearsal room for the first several weeks of the program – in notebooks and on sheets posted on the walls – until the script is frozen and all efforts turn to staging under DiMenna. The participants are creators while she is the playwright who selects, assembles, and has the final say on the script which she crafts afterhours out of the work the young men bring into the rehearsal room. She describes her work as “Studs Terkel meets The Me Nobody Knows” and as puzzle assembly. “It’s not unlike writing on a team on a soap on television. Writing a soap is a combination of puzzle solving and art. You get to create but you’re creating within a puzzle.”
There is a lesson in what she leaves in and what she leaves out every year, but editing and selection will not come as a surprise to the writers, Tate reports. Editing is now embedded in the workshops where writers are asked to review, rewrite, and edit their own work while the final script focus is on the group collage. Writers are told that out of scores of pages of writing a single line might appear in the final play, but the draft work and portfolio belong to them.
Each year’s play has a theme informed by what’s on the stages of MTC at the time. Tate mines what the youth will see for themes and ideas (in 2015 they see MTC’s production Of Good Stock and a preview performance of Hamilton), in what she describes as a dance. “I have ideas that inform the writing, and then what I get from the writers informs the next questions I ask. It’s a dance that goes on and a feedback loop that goes through the entire process.”
Perhaps most important, she works to instill in the young men a belief in themselves as artists. Encouraging them to be open to inspiration around them, during writing exercises she notes, “Artists are awake to the world, awake to themselves” and to “approach everything with a beginner’s mind.”
REHEARSAL: Ensemble and Bodies in Motion
More than 30 rehearsals over the seven-week project period are managed and monitored by a number of staff members, including a project intern who crafts rehearsal reports until the stage manager arrives in the fourth week. Tate leads writing exercises in the first few weeks and DiMenna leads theatre games throughout the process and directs the final production once the script is set and staged rehearsals begin.
Professional choreographer Elise Hernandez conducts dance rehearsals during selected four-hour program work sessions. The young men learn to think like a team in motion and see their gestures as a group. “It’s a different gesture when you have a lot of people doing it.” Hernandez notes in one rehearsal. “Every gesture I gave you has to pop, and then it goes where it goes.”
Tate appreciates DiMenna’s focus on movement and words in the transition from words on the page to movement on stage. “He is respectful and mindful of the script. And unlike a lot of people that do this kind of work with kids, he cares that it is understandable to the audience.” DiMenna keeps the actors focused on text in rehearsal by reminding them that “Each sentence is a new thought and each period is a pause.”
DiMenna in rehearsal is a director conveying life lessons to an emerging ensemble through theatre games, getting it into the performers’ bones. “We know we’re an ensemble if there is an unbroken rhythm,” he says at one point. And in games with balls he repeats, “Think of the ball as the play. We have to keep the ball moving.”
As in camp cohorts and school teams and indeed families, each Stargate ensemble develops its own language and short hand for their shared experiences. In 2015, one of those family words, a code word, was shared in the post-show conversation after the second public performance at New York City Center Stage II on August 22, 2015. “Ens-amily,” the Company member told the audience, means “ensemble plus family.”
That evening, 13 young men performed their collage production Deeper Than Skin, built from their specific stories and exercises into the script by Tate that is staged by DiMenna. Some stories in the final script come from exercises conducted on the first full rehearsal day, while some sections are added in the final program weeks. All of stories come from the lives of the young performers.
From writing to staging, rehearsal to performance, programs to real life, the rehearsal words of co-artistic director DiMenna ring. “Transitions are some of the most tricky things in life. I’m trying to ease your transition from writing to rehearsal.” Stargate aims to provide artistic and pragmatic experience with transitions of many types.
Stargate Theatre Company anticipated that program model development and refinement would occur throughout the first three program years. Nuanced adjustments to staffing (including hiring a Company Manager with multiple relevant areas of expertise and co-Artistic Directors with distinct and compatible skills who model collaboration and problem solving) and procedure modifications have set Stargate on a steady and replicable course. All is prepared for the next stage of research effort: assessing short-term program effects (e.g. standing taller and empathizing) and long-term program effects (e.g. staying in school, being employed, and avoiding the justice system). The 2015 post-performance enthusiasm of the writer who said “There’s nothing like being paid to write your own story” earns a smile and applause. What remains to be proven are the additional effects of this theatrical job readiness program.
DOCUMENTING THE EFFECTS OF STARGATE THEATRE
Stargate Theatre Company is a strong program that engages justice-involved young people in pro-social development and personal expression through the arts. The company’s approach is consistent with the best research on adolescent development and effective strategies for reducing problematic behaviors through social bonds and skill acquisition (Butts, Bazemore and Saa Meroe 2010).
While Stargate is based on sound principles, the next step for the company is to conduct evaluation research that can document its effectiveness. The staff of MTC could retain the services of researchers to design, implement, and report the findings of a high-quality outcome evaluation. The study would have to measure individual outcomes among youth involved with Stargate and compare them with outcomes among another sample of similar youth who are not involved with Stargate.
Such an evaluation would likely occur in three phases. The first phase would begin with the selection of the research consultant and end with the finalization of a research design and data collection plan. The second phase would begin with the initiation of data collection activities and end with the completion of data analysis. The third phase would involve the completion of research products, including a comprehensive final report, an executive summary or project brief, and a video presentation. Depending on the particular design selected for the evaluation, the entire project could be completed in 12 to 18 months.
The central goal of the Stargate Evaluation would be to gather data on the effectiveness of arts programming for youth already involved, or at risk of involvement in the justice system. Arts programming for youth in the justice system involves a wide range of activities, often including music, theatre, poetry, dance, and fiction or non-fiction writing. The extent of a youth’s involvement in art-related activities is theorized to have both short-term and long-term effects on youth outcomes. In the short term, involvement in the arts is hypothesized to increase youth engagement and cooperation with justice and social services programs, promoting rule compliance and facilitating successful reintegration with families and communities.
In the long term, arts programming is thought to affect youth self-awareness, self-expression, social aspirations, and attachment to school, family, and other pro-social institutions. The key research questions to be answered by an evaluation of Stargate
Theatre Company would include:
- Does a positive experience with the arts affect a youth’s compliance with program rules and successful adaptation to other groups and organizational systems?
- Does a positive experience with the arts affect a young person’s school/job attendance and/or academic/vocational performance?
- Does a positive experience with the arts have positive effects on a youth’s attitudes, beliefs, and attachment to family, positive peers, school, and other conventional institutions?
- Does a positive experience with the arts affect a youth’s participation in risky behaviors and the likelihood of additional involvement with police, courts and other official systems?
- Are the effects of a youth’s experience in arts programming related to the diversity, consistency, and duration of his or her participation in art-related activities?
An evaluation of Stargate Theatre Company would begin with a comprehensive review of the research literature on the effects of arts programming with at-risk youth, but the bulk of the evaluation would focus on measuring the process and outcomes of the program itself.
A process evaluation would need to identify the program qualities and attributes that encourage youth participation and support the efficacy of arts programming with at-risk and disconnected youth.
An outcome evaluation would assess the effectiveness of arts programming by tracking individual-level outcomes among a group of Stargate youth and comparing with similar youth not in Stargate. The measurement of youth outcomes would have to focus on behaviors as well as attitudes, and it would need to control for each youth’s background and the extent and duration of their involvement in arts participation.
By conducting the evaluation with a qualified research partner, MTC could ensure that the study is responsive to the needs of the youth involved in Stargate as well as their parents and families, and it could avoid negative consequences for the non-Stargate youth involved in the project as members of a comparison or control group. A professional research partner would help to keep the study methodologically sound, on schedule, and within budget. The research team should submit all reports and deliverables to MTC prior to any external release, and youth involved with Stargate should be provided with copies of all study protocols and products and be invited to review and comment on the study prior to the external release of any findings. In addition, a youth advisory panel could be convened during the study design phase to provide ideas and suggestions to the study team throughout the implementation and dissemination period.
Active and positive involvement in the arts is an effective tool for engaging youth in pro-social activities and learning, encouraging them to grow as individuals and providing them with opportunities for self-expression. Particularly for disadvantaged youth and young people at risk of involvement with the justice system, an effective arts program may be their first opportunity to participate in self-expressive activities, such as music, dance and movement, theatre, poetry and creative writing,
A growing body of research literature on the effects of arts programming with young people suggests that they may have positive effects on youth self image, ability to empathize with others, and other socially valued interaction outcomes. The arts, in fact, may be especially effective during adolescence:
“The creative arts provide access to thoughts and feelings unavailable through words alone, because they take an adolescent one step away from real life into pretense, where the adolescent can experience power and powerlessness, weakness and self-direction, joy and sorrow, without fear of reprisal. Creative self-expression, as both reality and pretense, gives troubles adolescents opportunities to try out a range of previously inaccessible options and roles, and experience the emotions and outcomes that can help them make choices or the future” (Emerson and Shelton 2001: 190).
Researchers have tested the role that arts programming may have in preventing and/or reducing risking behavior and promoting positive life outcomes for program participants. Of course, more research is needed, including rigorous outcomes studies that can estimate the effects of arts programming on youth at-risk of justice involvement, particularly youth from disadvantaged communities where organized arts activities are not always accessible.
Catterall et al. (1999) analyzed data from the National Educational Longitudinal Survey to test the relationship between involvement in the arts and academic success. The results suggested that students involved in school-based arts programs showed increased creativity and greater social skills, and were less likely to drop out of school. Moreover, the positive effects of arts programming were sometimes greater for youth from low-income families than for youth from privileged backgrounds.
The effects of arts programming have sometimes been evaluated in studies that focus on after-school programs. A Canadian study, for example, explored youth outcomes associated with participation in an art, sports, and life-skills program (Strategic Policy and Youth Research 2003). The study found strong effects on criminal and delinquent behavior among participants in an arts program, although the analysis didn’t reveal significant effects on school performance or behavior at home.
A Department of Justice report from 2001 described the results of the YouthARTS Development Project (Clawson and Coolbaugh 2001), an arts programs that worked with high-risk youths (youths with juvenile records) in Atlanta, Portland (OR), and San Antonio. Each program site involved youth in an array of arts activities as well as job training. Official justice records were sampled up to 22 months after participation. While the findings were mixed, the project seemed to be associated with a reduction in delinquent behavior among the participants. Some of the factors that appeared to promote successful program implementation included collaboration with skilled and qualified artists, onsite caseworkers and probation officers, a comprehensive training program for all staff, transportation for participant, and the use of a range of arts services and activities.
Researchers examined outcomes of the National Arts and Youth Demonstration Project (NAYDP) in Canada, collecting data about the effects of community-based, after-school arts activities with youth from low-income neighborhoods and disadvantaged families (Wright et al. 2006). The program had three objectives: 1) evaluate whether community-based groups can effectively recruit and engage youth in a nine-month arts program focusing on theatre and video production; 2) assess the youth’s actual progress in the development of artistic and social skills; and 3) measure the association between involvement in the arts and subsequent psychosocial functioning, behavioral and emotional problems, and self-esteem (Wright et al. 2006:189). The findings were mixed, but the study found that, compared with youth in a matched comparison sample, youth involved in NAYDP exhibited fewer emotional problems at follow-up.
Some studies have evaluated the impact of arts-based programs on youth at risk of delinquency and a few focused on youth already involved in the justice system. As part of a broader youth development strategy, however, arts programming has been shown to have great promise for adolescents. More research is needed to determine that role that arts programming should play in working with youthful offenders.
To evaluate an arts program for at-risk and delinquent youth, researchers must clearly define program goals and intended youth outcomes, monitor and document program implementation and service-delivery processes, and collect relevant outcome and follow-up data from youth involved in the arts program as well as an appropriate comparison or control group of youth not involved in the program.
A program evaluation must answer two basic questions: 1) “What happened to the program participants?” and 2) “What would have happened had they not participated in the program?”
Of course, it is impossible to know the answer to the second question, but researchers can approximate an answer by creating a counterfactual, or a hypothetical scenario in which a program or intervention never happened. Evaluation researchers often create this counterfactual by collecting data from a comparison group or a control group made up of people that could have received the intervention, but did not.
Researchers use the term “comparison group” to denote a counterfactual derived from a “quasi-experimental” evaluation design (Shadish, Cook and Campbell 2002). Subjects receiving a treatment or intervention are compared with another group or population that exists naturally, and researchers do not control the processes or conditions that lead subjects to be in the comparison group versus the treatment group. Youth in a school-based, violence prevention program, for example, could be compared with youth from a neighboring school, with a similar student population, but without a violence prevention program.
The term “control group” implies that a counterfactual is created with an “experimental” evaluation design. An experiment, or randomized controlled trial (RCT) is a study in which researchers exert total control over the process that determines whether each subject belongs to the treatment group or to the control group.
A randomized process (e.g., a coin flip or random number generator) is used to assign each member of a larger population to one of two equivalent groups. One group is the “treatment” or “experimental” group. A second group serves as the “control” – its members do not receive the treatment being evaluated. As long as the assignment process is truly random, the treatment and control groups should be statistically equivalent and any difference between them at a later time can be assumed to result from the treatment itself.
Researchers try to confirm the success of randomization by comparing treatment and control subjects on a wide range of factors to ensure that there are no systematic differences between them, other than the fact that one group received the treatment while the other group did not.
An evaluation of Stargate Theatre Company could rely on either one of these methods to establish a counterfactual. The final choice between a quasi-experimental or experimental design would be made during the first few months of the project, after researchers and staff from MTC were able to review all the design options for the study.
Design Option 1: Quasi-Experimental
The first method to be considered for the Stargate evaluation would be a quasi-experimental design in which the counterfactual would be created with data from a matched comparison group (Figure 1). Ideally, the comparison group would be identified from another disadvantaged or at-risk youth population, such as youth from the local schools or youth under the supervision of the New York City Department of Probation.
The comparison group youth would have to be as similar as possible to youth involved in Stargate but not including those invited to participate in Stargate but who chose not to participate or withdrew early. Youth who declined to participate in or complete the Stargate program would likely be systematically different from Stargate completers in various ways. They may be less socially developed, less cooperative, more resistant, or more delinquent. Relying on non-participants as a comparison group would create a misleading statistical bias, one that researchers call “selection bias.” The outcomes of the treatment group may be more positive than those of the comparison group, not because youth in the treatment group were affected in some way by Stargate but because they started out more positive than youth in the comparison group.
A suitable comparison group may be found by contacting authorities at the public schools or the Department of Probation. The research team and MTC staff would need to meet with the organization’s leaders and explain the goals and methods of the study and clarify the limited contact that youth would have with the study (two questionnaires). Researchers would also have to confirm that the youth in the comparison group were not likely to be exposed to arts programming, but that they were similar to Stargate youth in all other ways, including gender, age, ethnicity, educational status, and prior delinquency record.
If a comparison population was identified, and the administrators of the agency agreed to cooperate, the study team would have to select the actual comparison group members with a probability-based matching process. Individual subjects from the comparison population would be selected one at a time to match each youth in the treatment (Stargate) population.
In other words, for every Stargate youth who was male, Latino, 18 years old, with two prior criminal charges and a previous probation experience, researchers would need to identify an 18-year-old, Latino probationer with two prior charges in the comparison population. The study would need to obtain his consent to participate in the study and collect data from him just as they would with the Stargate participant. Whenever more than one suitable matching youth was available, researchers would choose from among all possible comparison subjects using a random (or probability-based) process.
Using this design, the conclusions of the evaluation would be highly reliable as they would be based upon a statistical comparison of a specific number of youth (e.g. N=50) who participated in Stargate with another 50 youth who were very similar to Stargate youth but who did not participate in the program. The project would need to work closely with MTC and the administrators of the comparison group organization to select each research subject and to design standard data collection protocols that were implemented consistently for both groups.
Even if the research design portrayed in Figure 1 were implemented exactly as planned, however, there would be two critical problems facing the study.
First, while the probability-based matching process may work well enough to control or compensate for most possible sources of difference between the treatment group and the comparison group, it would never be perfect. Evaluations based on non-random comparison groups are always open to criticism. Skeptics would be likely to argue that comparison groups were somehow different from treatment groups, and these differences (even if unconfirmed) might be responsible for whatever effects seem to be associated with participation in Stargate.
Second, even if the matching process worked perfectly, and even if there were no serious questions about whether the resulting comparison group was representative, using a different population to identify and recruit comparison subjects increases the cost and complexity of evaluations. Researchers have to work with at least two separate agencies, organizational structures, and administrative systems to implement data collection protocols. Researchers may have to hire and train additional research assistants to obtain informed consent agreements from each youth and to administer subject questionnaires at multiple points in time. It might turn out to be more cost-effective and statistically defensible to use an experimental design.
Design Option 2: Staggered-Start Experiment
The second evaluation approach to be considered for the Stargate evaluation would rely on a randomized, staggered-start experimental design (Figure 2). The study would assess the effectiveness of arts programming by working with MTC to modify the process used to recruit and enroll youth in Stargate. Half of all youth that agreed to participate in would be randomly selected to begin Stargate immediately; the other half would start after a waiting period (most likely six months). Outcomes would be measured for all study youth from the moment they agreed to participate in the study until either the time they left the program or 12 months following random assignment.
The staggered-start design would provide strong statistical results, as it would allow the study to construct a rigorous test of the impact of Stargate participation by measuring the differences between treatment-group youth (immediate starters) and control-group youth (delayed starters) during the period of the delay and for the remainder of the follow-up period. It would also allow the evaluation to avoid many of the ethical complications traditionally involved in experimental designs. The evaluation would not require any youth to be assigned to a “no treatment” control group, which is a common obstacle to experimental evaluation designs. All study youth would be invited to participate in Stargate, but the timing of their experiences would vary according to the experimental condition to which they were assigned. (All youth would have to be informed of this possibility and consent to participate in advance of assignment.)
Design Option 3: No-Performance Control Group
A third option for the Stargate evaluation would be to create a parallel program operated by the MTC staff and for youth similar to those in Stargate, but without the key ingredients of the Stargate experience — writing, rehearsal and performance (Figure 3). For purposes of discussion, imagine that MTC designed a discussion group for at-risk and disadvantaged youth. The participants in the comparison group would convene as often as the Stargate group, and they would address the same topics that come up during rehearsals, but youth in the comparison group would only discuss these topics with MTC staff. They would not participate in any guided theatre experiences. They would not write or rehearse any dialogue, and they would not be involved in performances of any kind.
This third option would offer a very strong statistical design for the Stargate evaluation. It would create a control condition that was very similar to Stargate Theatre Company, involving the same staff, location, and schedule, but without the theatre experience itself. Of course, the creation and management of the control condition (i.e. the discussion group) would entail costs to MTC and to the research team. The staff time devoted to Stargate overall would increase, and the discussion group would have to be well designed and managed. It would be unethical to involve control-group youth in a program that was not genuinely intended to be a positive experience.
Either Option 2 or Option 3 would be complicated. The principal challenge in any experimental design is to create an accurate randomization process and then to devise administrative routines to ensure the process is implemented correctly throughout the study period. To secure the continued cooperation of program staff, researchers sometimes have to compromise the design. In some studies, for example, researchers have to allow for “over-rides,” or cases in which the study design may call for a youth to be assigned to the control group (waiting period), but staff prefer to see the youth in the treatment group. Over-rides are acceptable within limits, but all youth affected have to be excluded from the study sample. The number of over-rides would have to be kept to a minimum to protect the statistical validity of the design and to prevent the use of over-rides from excluding so many cases from the study sample that data-collection period would have to be extended, thereby delaying the availability of results.
The Stargate evaluation would need to consider the feasibility of all of these research designs and then select the final design based upon ethical concerns (e.g., the confidentiality and privacy rights of youth and families), administrative factors (e.g., complexity and costs of data collection), and research concerns (e.g., statistical power, sample size, and time to completion). Regardless which design were selected, the evaluation would have to merge individual-level data extracted from justice agencies with detailed questionnaires about each youth’s attitudes and perceptions of their families, schools, and other pro-social institutions, as well as measures of self-reported delinquency, substance abuse, and other risky behaviors.
Sample size will be a challenge regardless which design is selected. The Stargate program is relatively small, with only 10 to 15 youth completing the program each year. Conducting an evaluation with such a small program could take several years. Researchers would have to wait for several programmatic cycles just to recruit 50 cases, and obtaining complete data for a sample of 50 youth would take even longer, as some youth would inevitably refuse to participate, and others would fail to complete the program or the follow-up measures (i.e., subject attrition).
Project Design and Work Plan
The Stargate evaluation would be organized in three phases: 1) design, 2) implementation, and 3) dissemination. The design phase involves several critical goals. The first goal would be to establish productive relationships and good communication between the researchers, MTC and any other partners. The study will be successful only if all partners work closely and cooperatively. Establishing teamwork would require a number of pre-evaluation, face-to-face meetings, supplemented by frequent and ongoing communication during the study itself.
The first tasks of the design phase would be to finish the study’s literature review and to select the final design for the study. The next task would create and authorize the project’s procedures and instruments for collecting data. As soon as the evaluation design was finalized, the project would submit all of its data-collection protocols for review and comment by MTC and other partners.
In the implementation phase, the project would begin a comprehensive process evaluation. The research team would observe and assess how Stargate ensures that it creates an effective experience for each youth. Process evaluations include on-site observations during program activities as well as interviews with staff members and managers. Evaluators would assess how youth are recruited and engaged in Stargate and how frequently they participate in various activities.
The most labor-intensive component of the implementation phase would likely be the data collection tasks required to evaluate youth outcomes. Regardless which design option is selected, the study would rely on a combination of interviews, self-administered questionnaires, and administrative data to measure the characteristics, attitudes, and behavior of youth at three points in time: T1) baseline; T2) six months after referral; and T3) 12 months following referral. Program impact would be estimated by comparing outcomes for Stargate youth with outcomes for non-Stargate youth. In the project’s final phase, the evaluation team would disseminate the study findings through a final report, executive summary or brief, and videos and slide show presentations. The entire project would need to be organized carefully to meet its key goals and objectives. (See an example task plan).
Methods and Measurements
Before any data are collected, the research team would have one face-to-face interview with each youth in order to obtain written, informed consent to participate in the study. Questionnaire data would be collected three times: (T1) at baseline or shortly after each youth is recruited into the study; (T2) three to four months after recruitment; and (T3) twelve months after recruitment.
The evaluator would exercise primary responsibility for all instrumentation and data collection, but MTC and the study partners would consult in identifying key constructs and creating items to measure them. Whenever possible, the research team would adapt instruments and items from existing studies to create the study questionnaires.
As data collection for the project would involve direct contact with youth (a vulnerable population) and sensitive data (self-reported delinquent behavior and substance abuse), the external evaluator should ensure that the project is reviewed by an accredited IRB (Institutional Review Board) and that human subject protections are followed. Informed consent would have to be obtained before the completion of any questionnaires. Written, informed consent would be obtained at baseline (T1), and prior to each follow-up survey, the study team would need to review the youth’s rights as a respondent.
During the first data-collection contact, the study team would invite each youth to complete the T1 questionnaire. If a paper form were used, respondents would be asked to insert the questionnaire into an envelope, seal the envelope and put it into a locked box. Approximately six months after completing a T1 questionnaire, the study would ask each youth to complete the T2 questionnaire. Approximately one year after T1, the study would again contact the youth for the completion of a T3 questionnaire.
Working in concert with MTC, the study would organize all available information about each youth’s participation in Stargate (and the non-Stargate discussion group if applicable). In addition, the study team would work with local law enforcement agencies to compile data about justice system contacts among all study youth. All data files would be protected using appropriate security protocols. Only de-identified data files would be retained for analysis.
Formal constructs to be measured by the research team could include: youth maturity of judgment, family communication and functioning, educational achievement and attachment, perceptions of authority, substance abuse, and delinquency. Measures for these constructs are available from the research literature. The study team would work with MTC to develop final measurements. Whenever possible, published scales and items should be used for the evaluation to ensure that all measurements have demonstrated reliability and validity characteristics and that the study draws upon commonly available measurements as appropriate. The latter criterion is important as it may allow normative comparisons of results from this evaluation with results of other studies involving adolescent offenders and arts programming.
Baseline Measures: The evaluation could collect data from youth at baseline on a variety of factors that could interact with Stargate’s effects. Factors would likely include individual characteristics (e.g., age and ethnicity), education status (e.g. enrolled vs. drop-out), and employment (e.g., part-time, full-time, unemployed). Other items could measure environmental conditions through self-report scales. Stressful life circumstances could be assessed with the Urban Adolescent Life Experiences Scale (Allison et al. 1999), which asks respondents to report on the frequency of 15 major events that can have negative impacts on urban adolescents.
Family climate may be assessed with questions from the supervision/monitoring, autonomy-granting, and family conflict subscales of the Family Climate Inventory (Kurdek, Fine, & Sinclair 1995). Another important variable at baseline could be an adolescent’s ability to exercise mature judgment (Cauffman and Steinberg 2000). Items from known instruments will be adapted by the evaluation and administered to youth. The Psychosocial Maturity Inventory scale (alpha = .87) uses simple statements of opinion about the role of luck versus hard work in determining life events. The Consideration of Future Consequences scale (alpha = .75) measures the ability to understand short-term versus long-term consequences. It consists of 12 items coded as 5-point scales in which subjects are asked whether various perspectives of time are similar to their own.
Educational Performance and Attachment: To build self-reported measures of educational performance and attachment, the study could explore a combination of questions from National Center for Education Statistics surveys, including the National Educational Longitudinal Survey of 1988 Eighth Graders and the Education Longitudinal Survey of 2002. Items from these data series measure youth impressions of school, attendance, preparation, and self-reported grades. Other questions ask about disciplinary problems and include a series on the student’s interest in school and the perceived importance of education. Questions are designed to be used in a self-administered format.
Perceptions of Authority: A number of measures used in previous research with adolescents are available for measuring perceptions of authority. The General Attitude toward Institutional Authority Scale examines respondents’ approval or disapproval of the police, the army, the law, and the teaching professions (Rigby 1982). Gouveia-Pereira et al. (2003) describe their measures of adolescent perceptions of authority (Tyler 1997) as well as relational, procedural and distributive justice (Lind and Tyler 1988). Items ask whether judges treat people with respect and consideration, give people a chance to state their point of view, are honest in the decisions they make, and give sentences appropriate for what the person has done. Reliabilities are acceptable with coefficients of .78 and .75 for perceptions of judges and general authorities, respectively.
Self-Reported Delinquency: The study could draw upon items from the National Youth Survey (NYS) to measure self-reported delinquency. The NYS questionnaire has over 30 measures of delinquent behavior. The evaluation could draw upon a subset of questions. The delinquency questions on the NYS run from minor offenses, such as “stole something worth $5 or less,” to serious offenses, such as “attacked someone with the idea of seriously hurting or killing them.” Two items in the NYS refer to selling drugs. Reliability and validity of measures from the NYS items have been evaluated by Huizinga and Elliot (1986) and have been determined to be acceptable. The questions are easily adaptable for self-administered formats.
Drug Use: The study could also measure other risky behaviors that may affect outcomes for Stargate youth. Self-reported substance abuse, for example, could be measured with items from Monitoring the Future (MTF). Conducted by the University of Michigan, the MTF study uses self-administered questionnaires to ask about the prevalence and frequency of substance use, from cigarettes to alcohol and illegal drugs. Reliability and validity have been assessed by O’Malley, Bachman and Johnston (1983).
Observations of Stargate Theatre Company reveal many strengths: staff openness to change, constant review of practice, and an ongoing integration of research knowledge. Stargate staff integrated research knowledge from the very first vision of the company formulated with consultation from Evan Elkin, then at the Vera Institute of Justice.
Stargate Theatre Company has operated successfully for three seasons and demonstrated its ability to inspire justice-involved youth to create and perform in their own dramatic works. Stargate’s approach is consistent with the extensive research literature on the role of arts programming for at-risk and disconnected youth, but the company has not yet participated in an evaluation of its effectiveness. This report provides a possible starting point for evaluation efforts.
A number of implementation issues and measurement challenges will likely confront evaluators. The program will have to identify a small set of hypothesized key components of the program in order to fashion effective approaches for data collection. Research suggests that arts involvement for youth could have positive effects on behavior and skill development, but previous studies have not specified the exact mechanisms that produce these outcomes.
Evaluators will have to collect primary data from youth members of Stargate in addition to relying on administrative data from local agencies and justice officials. This would include interviews and surveys administered directly to youth. The study would also have to collect interview and survey data from staff and other individuals with knowledge of each youth’s behavior (e.g. family members). Each of these components involves labor and costs.
Finally, much of the hypothesized effectiveness of Stargate occurs during group interactions and rehearsals. An evaluation would need to observe a range of program activities, which would require the presence of researchers for extended periods of time.
Despite many complexities, the success of Stargate Theatre Company and its obvious value from a youth development perspective suggests that evaluation would be a sound investment.
Steketee, Martha Wade and Jeffrey A. Butts (2015). Stargate Theatre Company: Engaging Justice-Involved Youth in Creativity and Growth. New York, NY: Manhattan Theatre Club.
Copyright by the Manhattan Theatre Club/Stargate Theatre Company and the Research & Evaluation Center, John Jay College of Criminal Justice, City University of New York (CUNY).
524 59th Street
New York, NY 10019
Published online October 2015
Any views expressed here are those of the authors and do not necessarily reflect the policies of Manhattan Theatre Club, John Jay College, the City University of New York, or their funders.
Allison, K. W., L. Burton, S. Marshall, A. Perez-Febles, J. Yarrington, L.B. Kirsh and C. Merriwether-DeVries. (1999). Life experiences among urban adolescents: Examining the role of context. Child Development 70(4), 1017-1029.
Butts, Jeffrey A., Gordon Bazemore, and Aundra Saa Meroe (2010). Positive Youth Justice: Framing Justice Interventions Using the Concepts of Positive Youth Development. Washington, DC: Coalition for Juvenile Justice.
Catterall, J.S., R. Chapleau and Iwanaga, J. (1999). Involvement in the Arts and Human Development: General Involvement and Intensive Involvement In Music and Theatre Arts (briefing paper). Los Angeles: The Imagination Project, Graduate School of Education & Information Studies, UCLA,
Cauffman, Elizabeth and Lawrence Steinberg (2000). (Im)maturity of judgment in adolescence: Why adolescents may be less culpable than adults. Behavioral Sciences & the Law 18(6): 741-760.
Clawson, Heather J. and Kathleen Coolbaugh (2001). The YouthARTS Development Project. OJJDP Juvenile Justice Bulletin. Washington, DC: Office of Juvenile Justice and Delinquency Prevention.
Cohen, Mary L. and Meade Palidofsky (2013). Changing Lives: Incarcerated Female Youth Create and Perform with the Storycatchers Theatre and Chicago Symphony Orchestra. American Music, 31(2), 163-182.
de Roeper, Julia and Harry J. Savelsberg (2009). Challenging the youth policy imperative: engaging young people through the arts. Journal of Youth Studies, 12(2), 209-225.
Dutton, Susanne E. (2001). Urban Youth Development- Broadway Style: Using Theatre and Group Work as Vehicles for Positive Youth Development. Social Work with Groups, 23(4): 39-58.
Emerson, Elissa and Deborah Shelton (2001). Using creative arts to build coping skills to reduce domestic violence in the lives of female juvenile offenders. Issues in Mental Health Nursing 22(2): 181 – 195.
Ezell, Mark and Michelle Levy (2003). An Evaluation of An Arts program for Incarcerated Juvenile Offenders. Journal of Correctional Education 54(3), 108-114.
Getting Out Staying Out http://www.gosonyc.org/HomePage.php [accessed 9/9/15].
Gouveia-Pereira, Maria, Jorge Vala, Augusto Palmonari and Monica Rubini (2003). School experience, relational justice and legitimation of institutional justice. European Journal of Psychology of Education 18(3), 309-325.
Hughes, Jenny, Andrew Miles and Angus McLewin (2005). Doing the Arts Justice: A Review of Research Literature, Practice and Theory. The Unit for the Arts and Offenders, Centre for Applied Theatre Research, London.
Hughes, Jenny and Karen Wilson (2004). Playing a part: the impact of youth theatre on young people’s personal and social development. Research in Drama Education 9(1): 57 – 72.
Huizinga, David and Delbert S. Elliot (1986). Reassessing the reliability and validity of self-report delinquency measures. Journal of Quantitative Criminology 2(4), 293-327.
International Theatre & Literacy Project http://www.itlp.org/about/history.php [accessed 9/8/15]
Kurdek, Lawrence A., Mark A. Fine and Ronald J. Sinclair (1995). School adjustment in sixth graders: Parenting transitions, family climate, and peer norm effects. Child Development 66, 430-445.
Lazzari, M.M., K.A. Amundson and Jackson, R.L. (2005). “We Are More Than Jailbirds”: An Arts Program for Incarcerated Young Women. Affilia 20(2): 169-185.
Lind, E. Allan and Tom R. Tyler (1988). The Social Psychology of Procedural Justice. New York: Plenum.
Mitchell, Martin (2004). Rhythm of the Machine: Theater, Prison Community, and Social Change. Reflections 4(1): 136-146.
Mosaic Youth Theatre of Detroit. Excellence on Stage and in Life: The Mosaic Model. Accessed on line 9/8/2015 (http://mosaicdetroit.org/who-we-are/).
Opening Act http://openingact.org/ [accessed 9/9/15]
O’Farrell, Larry and Margaret Meban (2003). Arts Education and Instrumental Outcomes: An Introduction to Research, Methods, and Indicators Methods. Paper Commissioned by UNESCO under contract with the Faculty of Education at Queen’s University, Kingston, Canada (50 pages).
O’Malley, Patrick, Jerald G. Bachman and Lloyd D. Johnston (1983). Reliability and consistency in self-reports of drug use. The International Journal of the Addictions 18(6) 805-824.
Rapp-Paglicci, Lisa, Chris Stewart and William Rowe (2012). Improving Outcomes for at-risk Youth: Findings from the Prodigy Cultural Arts Program. Journal of Evidence-Based Social Work, 9(5), 512-523.
Rigby, Ken (1982). A concise scale for the assessment of attitudes towards institutional authority. Australian Journal of Psychology 34: 195-204.
Shadish, William R., Thomas D. Cook and Donald T. Campbell (2002). Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin Co.
Stargate Theatre Company, WLIW Arts Beat, Aired: 01/12/2015 http://watch.wliw.org/video/2365401095/ [accessed 9/9/15]
Stella Adler Studio of Acting at Rikers Island http://www.stellaadler.com/outreach/rikers-island/ [accessed 9/9/15]
Strategic Policy and Youth Branch, Saskatchewan Culture, Youth and Recreation (2003). The Impact of Culture/Arts, Sport and Recreation on Antisocial Behavior in Youth: Evidence and Analysis. Unpublished report.
Tyler, Tom R. (1997). The psychology of legitimacy: A relational perspective on voluntary deference to authorities. Personality and Social Psychology Review 1(4), 323-345.
Tyson, Edgar H. (2002). Hip Hop Therapy: An Exploratory Study of a Rap Music Intervention with At-Risk and Delinquent Youth. Journal of Poetry Therapy 15(3), 131-144.
Wallis, David (2014). The Broadway Theater Company Giving Troubled Teens a Second Act. http://nationswell.com/broadway-theater-company-giving-troubled-teens-second-act/ [accessed 9/8/15].
Wright, Robin, Lindsay John, Stephen Ellenbogen, David R. Offord, Eric K. Duku and William Rowe (2006). Effect of a Structured Arts Program on the Psychosocial Functioning of Youth From Low-Income Communities: Findings From a Canadian Longitudinal Study. Journal of Early Adolescence, 26(2):186-205.
Write on the Edge (WROTE), Manhattan Theatre Club Education Department, http://www.manhattantheatreclub.com/education/programs/teachers-and-school-administrators/ [accessed 9/9/15]