Hiring an Evaluator

Hiring an evaluator can save your program considerable time and effort. It can also yield information that is more accurate, useful and credible than would be generated by an evaluation designed and implemented by staff without evaluation expertise. This publication offers some advice on how to locate and hire an evaluator appropriate to your program’s needs. Although this publication refers to “hiring” an evaluator, this information holds true if your project is going to contract for evaluation services, rather than actually put that person on staff. The choice of hiring an “internal” evaluator or contracting with an “external” evaluator depends on several factors, including cost, the time frame in which the evaluation will take place, and in some cases, the evaluator’s desire to remain independent.

Reasons to Involve an Evaluator Earlier Rather than Later*

It is never too soon to bring an evaluator into a project. Early involvement in the prevention planning process provides time and opportunity for evaluators to:

• Gain a thorough understanding of the program. Evaluators who are involved in program planning have a better understanding of the program's design and intent, the workings of each component, and the relationships among the program components.
• Conduct an evaluability assessment An evaluability assessment can determine whether a program is mature enough to evaluate and it is functioning as intended. The assessment should also determine whether program outcomes or impact can be measured. Such an assessment can reveal potential problems and prevent premature evaluations that waste time and resources.
• Design the evaluation, select measures, and develop instruments. Once the evaluation design and measures have been chosen, the measures need to be organized into a questionnaire or other data collection instruments and pilot-tested. Additional time is required if these instruments need to be translated into other languages. Ideally, all of this work should be completed before your program begins – especially if the evaluation includes a pretest.
• Obtain an Institutional Review Board (IRB) approval. In some cases, you will not be able to collect data until your procedures for protecting the participants have been reviewed and approved by an IRB. The IRB review process cannot begin until all instruments and participant protection procedures have been developed.
• Develop a report with program staff. Program staff should see the evaluation as an important activity and not a distraction.

Waiting to hire an evaluator until after the program has been fully implemented can be costly. It usually takes three to six months of evaluation planning and preparation before data collection can begin. If you wait too long, your evaluator may not have sufficient time to help you collect the data you need to understand whether your program is having an impact.

Locating an Evaluator

Not all evaluators will possess all the skills your project needs. The more thorough your search, the more likely it is that you will find a good match for your project. There are many methods you can use to locate qualified candidates, including the following:

• Ask your Technical Assistance Specialist at the National Center for Mental Health Promotion and Youth Violence Prevention. Help in identifying an evaluator is an appropriate technical assistance request.
• Contact programs similar to your own. Other agencies that have implemented and evaluated similar prevention activities may be able to suggest evaluators who will be a good fit for your program. Be sure to ask whether there is anyone with whom they were dissatisfied.
• Ask funders. Funders see many evaluation reports and may know some good candidates in your area. Using an evaluator whom your funder knows and respects can go a long way in ensuring the funder will be satisfied with the evaluation report.
• Inquire at local colleges and universities. Faculty in departments of sociology, social work, education, community psychology, and public health, and in university-based research centers often have training and experience in program evaluation. Some professors do work outside their institutions and might be willing to work on your evaluation. Or they may be able to recommend qualified graduate students.
• Work with professional associations. Associations such as the American Evaluation Association (http://www.eval.org/) and the Society for Prevention Research (http://www.preventionresearch.org/) may be able to provide names of their members in your area.
• Consult with state or local agencies. Ask representatives from state agencies responsible for administering Federal mental health, substance abuse, and violence prevention grants. Most state or local government agencies, such as departments of public health and education, have planning and evaluation departments. These people may be able to help you or direct you to local organizations or individuals who can provide evaluation expertise.
• Ask research institutes and consulting firms. Private research organizations often employ evaluators.
• Read the evaluation literature. Find evaluation studies on programs like your own. If the authors are local, they may be a good fit for your program. If the authors are not local, they may know of qualified evaluators in your area. By reading these articles, you will also learn a lot about how evaluation studies are conducted.
• Review conference presentations. Look through agendas of conferences that focus on mental health and violence prevention. Contact local researchers to discuss your program or ask researchers in other areas for local contacts.

Deciding What You Want – and What You Can Afford

Before scheduling any interviews, sit down with your planning team and talk through what you are looking for in an evaluator. You need some sense of how complicated the evaluation will be – and thus the level of expertise and expense required. You should identify any skills that are particularly important to your project. For example:

• The ability to speak and design survey instruments in the language of your target audience is essential.
• The ability of an evaluator to understand and respect differing views – even controversial ones – while retaining his or her scientific objectivity may be key to your project.
• A great deal of tact and the ability to translate scientific evaluation concepts into language that can be understood by lay people are desirable traits for an evaluator working on a project that has high visibility or one that requires collaboration, especially with organizations such as local government or funding agencies.

Screening and Assessing Candidates

Take the time to recruit candidates who meet your criteria, then carefully assess their qualifications. Screen applicants carefully so you do not waste time interviewing candidates who are not appropriate for the needs of your project.

Applicants will send you materials, which your hiring committee will have to read and rank. There are two approaches to scoring or rank-ordering applications: formal and informal.

A formal approach involves assigning points to each criterion, such as evaluation philosophy, education, cost, living in the target community, particular kinds or amounts of personal or work experience. Candidates accrue a certain number of points for each criterion they meet: the better their qualifications, the more points. You can also score cover letters according to how well they are written, how much care was taken with them, and how well they are presented. Once applicants are scored, you can interview candidates with the highest scores, or use the scores as a starting point for group discussion.

An informal approach looks at the same criterion but uses discussion within the hiring committee or project staff, rather than a scoring guide, to compare and contrast candidates. The approach you select will depend largely on how your hiring committee works together and likes to process information.

Preparing for the Interview

Be prepared with your key discussion points before the interview. Have your questions ready. Key items that you should make a point of discussing with candidates include the following:

• Evaluation philosophy. Consider the evaluation philosophy or approach that you find most comfortable and appealing. If your project is going to use a participatory approach to evaluation, it is important that your evaluator will be comfortable with it. Ask candidates about their experiences with the approach they have chosen.
• Education and experience. If you cannot identify – or afford – someone with formal training in program evaluation, look for individuals with graduate-level training in social science research methods. They should also have professional experience in evaluation design, data collection, and statistical analysis. An ideal candidate will have experience that is relevant to your specific program. Ask candidates whether or not they have evaluated similar programs with similar target populations. If they have, they may have knowledge and resources that will save time and money.
• Ability to respond to outcome needs creatively. For example, by using proxy data.
• Oral communication skills. Evaluators must be able to communicate effectively with a broad range of people including parents, program staff, other evaluators, and the media. They should be able to avoid or translate scientific jargon when necessary. Someone who cannot clearly explain evaluation concepts to a lay audience is not a good candidate. An evaluator needs to be able to connect comfortably with program staff and participants if he or she is going to successfully gather information from these people. Have candidates bring
• writing samples, including evaluation reports, articles, and the script or PowerPoint slides for presentations that they have developed to share findings.
• Writing skills. An evaluator must have strong writing skills. Having to rewrite evaluation reports takes time. The scientific integrity of evaluation results can be threatened if the report has to be rewritten by someone other than the evaluator.
• Cultural sensitivity. An evaluator needs to respect the cultures of the communities with which he or she works. Mutual respect and some understanding and acceptance of how others see the world are crucial. Genuine sensitivity to the culture and community will increase the comfort level of program staff, participants, and other stakeholders. It will also ensure that data collection tools are appropriate and relevant, thus increasing the accuracy of the findings.
• Budget and cost. Ideally, you should also ask candidates to prepare a written proposal for your evaluation, including a budget. You might want to reserve this request for your pool of finalists. To get a good proposal, provide candidates with clear information about your program’s goals, activities, and audience.
• Time and access. Make sure candidates have the time to complete the necessary work. Ask them what their other professional commitments are and how much time they will be able to devote to your project. Compare their responses to your estimate of the time needed to do the work. Make sure to consider the necessity of site visits and regular meetings. The more contact your evaluator has with your program, the better he or she will understand how it works and the more opportunities he or she will have to monitor data collection activities. Regular meetings also let you monitor the evaluator’s performance.
• Commitment to your agenda. Researchers, particularly those attached to universities, may have their own reasons for embarking on an evaluation. It may fit into a doctoral dissertation, a book that the researcher is writing, or another long-term research interest. Researchers may have strong prejudices about the research methods they want to use or what they expect to find. You may want to discuss these issues up front. An evaluator’s agenda needs to be congruent with that of the program.
• Data ownership and control. Insist that the evaluator obtain your prior approval for all public dissemination of results. If you don’t, your evaluator may write articles about your program or make conference presentations on the data without your knowledge.
• Chain of command. It is important to establish that the evaluator will be working for the project, not the funder. It is essential that project staff review, in advance, all evaluation reports and presentations before they are released to the funder or other audiences.
• References. Ask for references and check them.

Managing Interview Logistics

There are a number of things you can do before, during, and after the interview to ensure that the interview process runs smoothly and yields the information you need to make an informed hiring decision. One of these is to talk with staff and key partners to develop an interview process. Before you begin scheduling interviews, consider these questions:

• Who will sit on the hiring committee? It is usually a good idea to create a hiring committee that includes a diverse group of program staff and other stakeholders such as members of your community advisory board, if your program has one.
• What kinds of materials should candidates bring to the interview? Ask candidates to provide resumes and writing samples, especially copies of evaluation reports.
• What kinds of questions will you ask? Generate a list of four to seven questions that you will ask everyone.
• Will there be any other aspect to the interview besides conversation? For example, will you take candidates on a tour of the school or community? Will they meet with people other than the interview committee? Will you ask them to demonstrate a competency, such as explaining evaluation concepts?

Finally, keep in mind that an important part of an evaluator’s job is to assist in building the skills, knowledge, and abilities of other staff and stakeholders. It is critical that all parties can work well together. Make sure to invite finalists to meet project staff and others with whom they will be working to see who best fits with individual styles and your organizational culture. If the fit is good, your evaluation is off to a great start!

Additional Resources

Additional evaluation resources can be found in the Evaluation Toolkit on the National Center for Mental Health Promotion and Youth Violence Prevention Web site.

* Adapted from How to Find and Work with an Evaluator. Washington, DC: Center for Substance Abuse Prevention (Evaluation Basics Prevention DSS 3.0.)