Every SS/HS grantee is required to conduct a local evaluation. The purpose of the local evaluation, in the words of the program announcement, is to “provide timely information for creating strategic plans, measuring progress, and keeping the project focused on the overall objective of the SS/HS initiative—promoting healthy childhood development and preventing violence and substance abuse.” SS/HS grantees were required to include an evaluation design in their applications. This chapter will provide guidance in implementing your local evaluation as well as complying with other Federal evaluation requirements – notably the cross-site National Evaluation and the requirements of the Government Performance Results Act (GPRA). Additional tools, information, and resources on the topics discussed in this chapter can be found in the National Center’s Evaluation Toolkit.
Why Evaluate Your SS/HS Initiative? Evaluating your SS/HS initiative provides timely information for managing both the day-to-day and long-term aspects of your project, measuring progress toward goals and objectives, and understanding if the initiative is, in fact, promoting healthy childhood development and preventing violence and substance abuse. Positive evaluation results can also gain the support needed to sustain components of the initiative after Federal funding ends. Evaluation is important for a number of reasons:
- Evaluation demonstrates to your local partners, stakeholders, and the community that your initiative (and the SS/HS initiative as a whole) is effective and worth sustaining. The Substance Abuse and Mental Health Services Administration, Department of Education, and Office of Juvenile Justice and Delinquency Prevention also use evaluation information to justify requests to Congress for monies to support prevention and intervention programs.
- Evaluation helps a project respond to changes that may impact its effectiveness by documenting implementation and how implementation interacts with the institutional and social environment in which the project takes place. Knowing this can help you decide where best to channel your resources and what parts of the initiative to institutionalize and sustain.
- Evaluation can improve program efficiency by revealing opportunities to streamline program delivery or enhance coordination between program components. Increased efficiency can help reduce cost or allow you to provide more services to a larger audience at the same cost.
Participatory Evaluation
In the participatory approach to evaluation, the evaluator works closely with the program, advising (rather than mandating) decisions related to the evaluation. Many SS/HS projects find that participatory evaluation is especially effective, as it reflects the inherent collaborative nature of the SS/HS initiative. The participatory approach relies on an Evaluation Team composed of one or more individuals trained in evaluation, program staff, and representatives of other stakeholders (such as students who are receiving the program or their parents). The members of the evaluation team work together to plan and implement all evaluation tasks. If you would like to undertake a participatory evaluation of your program, make sure to discuss this approach with candidates during the screening process.
Participatory evaluation is frequently recommended for a number of reasons, including the following:
- When an evaluation is planned and developed collaboratively between program staff and an evaluator, there is less potential for misunderstanding and an increased likelihood that program staff and other stakeholders will understand the benefits of evaluation. In being active participants in the process, program staff are also more likely to accept and use the findings from the evaluation.
- Since front-line staff are often more knowledgeable than an evaluator about the unique needs, culture, and circumstances of the target population, their input can make the evaluation more relevant, appropriate, and sensitive – this creates a more accurate estimate of why and how the program affected these people.
The participatory approach has also some potential disadvantages. It requires more time on the part of both the program staff and the evaluator. This includes meeting time, consulting time, and time to share and resolve differences of opinions. It also increases the potential for bias, since evaluators working closely with program staff may want to show project success as it reflects work done by their colleagues.
The project director is responsible for managing the evaluator regardless of whether a participatory approach is used. Guidance on this relationship can be found in Managing An Evaluator, in the Evaluation Toolkit. The participatory approach entails additional responsibilities for the project director, including the following:
- Working with the evaluator to make certain the evaluation design responds to the project’s needs, is consistent with available resources and the initiative’s timeline, and fulfills all federal requirements
- Identifying key individuals to sit on an Evaluation Team (usually consisting of the project director, evaluators, and other stakeholders interested in the evaluation)
- Monitoring the evaluation to ensure that it is being carried out efficiently and on time
- Providing support to those carrying out the evaluation (including the Evaluation Team and those in the field who are engaged in activities such as administering surveys or abstracting and submitting data)
- Using information from the process evaluation to monitor and manage the initiative and make modifications to programming as needed
- Keeping partners and stakeholders informed about the evaluation and seeking their input and support
- Using evaluation data to fulfill Federal reporting requirements.
Working with an Evaluator
Most SS/HS initiatives contract with a professional evaluator (e.g., university affiliate or evaluation firm) to plan and carry out the evaluation. In many cases, evaluators will assist with conceptualizing and writing the application, writing the evaluation section of the application, and developing the SS/HS evaluation plan that is required at six-months post-award. If you still need to hire an evaluator (if, for example, the evaluator with whom you worked during the application process is no longer available or you created the evaluation plan without the assistance of an evaluator), Hiring an Evaluator (which can be found in the National Center Evaluation Toolkit), will provide some guidance on this process. Some school districts require that evaluators be hired through a “request for proposals” (RFP) process. You should consult your district’s legal or contracting staff to find out if your district requires such a process.
Questions to Guide Your Thinking
|
Evaluation Design
Evaluations usually include both a process evaluation and an outcome evaluation:
- Process evaluation documents whether project activities are implemented as planned. They measure, for example, the number of teachers trained to deliver a curriculum, the number of classrooms receiving a bullying prevention program, the number of sessions delivered per classroom, and measures of program fidelity (i.e., faithfulness to the original model). These are described in the “process measures” column of your SS/HS logic model. Process data can help you understand whether or not the activities were a good fit, and whether the quality of implementation was sufficient to permit a good test of the program.
- Outcome evaluation is used to help determine if the project produced the desired outcomes—this might include changes in knowledge, attitudes, skills, or behaviors. It might also include things like policy and/or systems change. These are described in the “goals” and “objectives” columns of your SS/HS logic model (i.e., how different you hope something will look as a result of your project activities).
Examples of the types of information that can be gleaned from process and outcome evaluations can be found in the following documents:
- Logic Model Sample (05-06 cohort)
- Logic Model Sample (07 cohort)
- Evaluation Plan Sample (05-06 cohort)
- Evaluation Plan Sample (07 cohort)
Performance Indicators
SS/HS projects are required to identify and provide annual data for at least one measurable and attainable performance indicator for each of the elements of the SS/HS Initiative, (and at least one performance indicator for the SS/HS Partnership for 2005 and 2006 grantees), and GPRA measures (discussed below).
Government Performance and Results Act
SS/HS grantees are required to collect and report information every six months on the following Government Performance and Results Act (GPRA) indicators:
For 2005 and 2006 grantees:
- Decrease in the number of violent incidents at schools during the three-year grant period
- Decrease in substance abuse during the three-year grant period
- Improved school attendance during the three-year grant period
- Increase in mental health services to students and families during the three-year grant period
For 2007 Grantees:
1. Student Victimization/Perception of School Safety
- Percentage of grantees that experience a decrease in students who did not go to school on 1 or more days during the past 30 days because they felt unsafe at school or on their way to and from school.
- Percentage of grantees that experience a decrease in students who have been in a physical fight on school property in the 12 months prior to the survey.
2. Student Substance Use and Abuse
- Percentage of grantees that experience a decrease in students who did not go to school on 1 or more days during the past 30 days because they felt unsafe at school or on their way to and from school.
- Percentage of grantees that report a decrease in students who report current (30-day) marijuana use.
- Percentage of grantees that report a decrease in students who report current (30-day) alcohol use.
3. Mental Health Services Provided
- Percentage of grantees that report an increase in the number of students receiving schoolbased mental health services.
- Percentage of grantees that report an increase in the percentage of mental health referrals for students that result in mental health services being provided in the community.
Your performance indicators may or may not also satisfy GPRA requirements. For sample measures that can be used to satisfy the SS/HS GPRA indicators and other information on GPRA, see Government Performance and Results Act Section in the Evaluation Toolkit.
The National Evaluation
While the GPRA indicators provide information about the overarching goals of the SS/HS initiative, they are not exhaustive. Thus, the Federal Government has created a cross-site national evaluation designed to collect further information on the GPRA indicators for the SS/HS initiative, and to examine additional issues relevant to the initiative. These findings will be used to demonstrate the success of the SS/HS initiative and provide support for the continuation of funding. The National Evaluation Team (NET) that is conducting this cross-site evaluation consists of representatives from MANILA Consulting Group, Inc., and its partners RMC Research Corporation and Battelle Centers for Public Health Research and Evaluation.
The National Evaluation will integrate process data (program activities implemented, services received) with outcome data (effects on students and schools, and qualitative effects on community conditions) collected through a school-level and a project-level survey. Data from NET surveys can also be used at the local level to assess program fidelity, make site-specific improvements, measure the degrees of program utilization and coalition coordination, and meet other site-specific evaluation needs. Project Directors and project evaluators are encouraged to work with the NET liaison to discuss how they can access and use data from the national evaluation. Using this data can help avoid redundant data collection, which is both inefficient and burdensome.
Each SS/HS site is assigned a NET liaison as a link between the local site and the national evaluation team. SS/HS sites should hear from their NET liaison shortly after they receive their formal grant award.
For more information on this topic, see the National Evaluation Section in the Evaluation Toolkit.
Logic Model
Developing and/or revisiting your logic model is an important step in conceptualizing your evaluation. It should be done prior to sitting down to draft the Evaluation Plan (described below). SS/HS projects are required to submit a logic model with their first six month performance report.
Your logic model serves as a blueprint for the evaluation by demonstrating the perceived linkages between local needs, goals, objectives, activities, and process and outcome measures. Your logic model can assist in the identification of evaluation procedures that can help you determine whether your activities are working in the intended way as described in your logic model. Having a strong logic model has been a hallmark of successful SS/HS sites.
For a Logic Model Worksheet and other resources on logic models, see the Logic Model Section of the Evaluation Toolkit. The Logic Model Worksheet and Evaluation Plan Worksheet (also included in the Evaluation Toolkit) were designed to be used together to create an evaluation plan that is congruent with your logic model.
Evaluation Plan
An evaluation plan provides an overview of all evaluation activities and their relationships to one another and to other activities. A good evaluation plan can help form the basis of a work plan for your evaluator and can assist you in managing the evaluation process. Your evaluation plan should be based on the evaluation section of your original application, your SS/HS logic model, and the federal reporting requirements previously described. The evaluation plan describes the following:
- The evaluation design: What evaluation method will be used; what data will be collected; how these data will be collected; and how these data will be analyzed.
- How the evaluation will be implemented: Who will collect data; when; what data will be reported to you as project director and other stakeholders, at what interval, and in what form.
For an Evaluation Plan Worksheet and other information on this topic, see the Evaluation Plan Section of the Evaluation Toolkit.
Questions to Guide Your Thinking
|
Your evaluation plan should be reviewed by your FPO and disseminated to your partnership members. Your evaluation plan and data collection instruments may also have to be approved by an Institutional Review Board to satisfy Federal Protection of Human Subjects requirements and comply with the requirements of both the Protection of Pupil Rights Amendment and Family Educational Rights and Privacy Act. Information on all of these issues can be found in the Evaluation Toolkit.
Implementing, Supporting, and Monitoring the Evaluation
Once the evaluation plan has been created and approved, you need to determine whether the project has the capacity to carry out the plan or whether additional capacity must be created. For example, you may need to purchase analytical software, secure storage for data forms, and train the people who will collect the necessary data.
In most cases, the evaluator is primarily responsible for supporting the implementation of the evaluation in the field. The evaluator can train or coach interviewers, provide survey or interview materials, and solve problems that arise while collecting data, such as assisting the schools in scheduling surveys.
Although monitoring the evaluation is the responsibility of the evaluator, you, as the project director, need to stay fully informed about the progress of the evaluation, since you are responsible for ensuring that all aspects of the initiative work together smoothly. For example, a delay in administering pretests could produce a situation in which there is not enough time to teach an evidence-based program before the end of the academic year, which could affect its fidelity.
For more information, please see the Working with an Evaluator Section of the Evaluation Toolkit.
Questions to Guide Your Thinking
|
Reporting and Using Evaluation Results
Evaluation is an essential tool for project management. It is a powerful means to demonstrate the potential value of an initiative and gain the support needed to sustain effective activities and desired outcomes. The ability of partners and stakeholders to understand whether or not an initiative is making a difference in the community—and to articulate this understanding to people who would be involved in sustaining the initiative’s efforts—is critical.
All too often, evaluations are conducted and the results sit stagnant on hard drives, in file drawers, or on bookshelves. This pitfall can be avoided when project staff and partners take on an active role in the evaluation and work with their evaluator to ensure that the right questions are being asked and that they are not collecting data just for the sake of collecting data. To maximize the utility of evaluation data, it is important to think through the end use of each piece of information being collected. This can be facilitated by working with staff and partners to identify the priorities of the initiative along with the level of information desired for each strategy or activity.
Questions to Guide Your Thinking
|
Once the data have been received, it is important to present them to people in ways that are understandable and easily communicated to others. This might include the use of charts and graphs to visually display important information, and the use of anecdotes to help humanize the data. Positive evaluation data can build institutional, political, and community support for sustaining your initiative’s components. People are more likely to support the dedication of resources to continue a program if they can see that it is making a real difference in their community.
Questions to Guide Your Thinking
|
VIGNETTE: EVALUATION
Dr. Gopal’s advice for project directors is as follows:
|
