What To Look For In An Evaluator

FUNDING TIPS

Finding A Consultant to Evaluate Your Program: It's a Two-Way Street

When hiring an evaluation consultant, first realize that to get a good assessment you will have to help. Effective evaluation is a "2-way street," though many nonprofits don't always understand this, expert James Fremming tells a Washington Support Center workshop. Consequently, don't expect the consultant to know everything. For example, consultants often use a standard approach that rarely fits an organization's needs perfectly.

You want to be an "active customer," Fremming advises. Know beforehand the types of data you seek, then assist the evaluator to obtain it. This sort of "participative evaluation" will yield better information and more useful results, notes Fremming, who has 15 years experience assessing private nonprofit and federal programs. Your program manager is the key player in participative evaluations. He has primary responsibility in managing the contract and designing the study.

Therefore, have your program manager coordinate the process. Have him assign data-collecting duties to staff and other stakeholders in the evaluation. Also, let him determine what activities will occur first, at which points in the process results will be reviewed and what types of decisions will be made and why.

In addition, the program manager should make sure staff people feedback to the evaluator. For instance, the consultant needs to know if staff fully understand the evaluation design plan and if data collection forms are working as intended. In addition, the consultant will be interested in the types of data staff find most useful and if there are any information collection problems. Further, tell him how evaluation reports will be used.

When hiring a evaluator, first check with key staff and stakeholders to get "multiple perspectives" on how the study should be done, Fremming says. Then, interview at least three consultants before selecting one. The consultant must have good credentials and references. But it's just as important that he "fits your personality and organization," Fremming says. "When looking for an evaluator, shop the same way as you would for a house, not a blender."

Top

A FEW TIPS:

Here are some tips from the Administration on Children and Families on holding a competition for an evaluation contract:

• Provide clear and accurate descriptions of the technical requirements for the materials, services and products to be procured by the evaluator.
• Avoid restrictive specifications or evaluation criteria that could limit competition.
• Solicit offers from as many responsible, qualified sources as possible.
• Advertise the contract as widely as feasible and allow adequate time to receive and review offers.
• Do price and cost analysis to determine if offers are reasonable.

Top

A CHECKLIST OF ISSUES:

Following is an ACF list of issues to consider when selecting a consultant:

• Is there a language barrier? You need somebody who uses layman terms to explain how to do the evaluations, so all staff will understand. Ask the consultant to explain terms with which you're not familiar.
• Does he want to know you? A good consultant asks questions about your organization and what it's trying to accomplish, particularly before starting to suggest how to evaluate it.
• Is he local? Somebody who has only studied state or regional programs may not be useful evaluating local ones. The best candidate will have already assessed organizations like yours.
• Does he know your clients? There may be critical factors about the types of people you serve that must be taken into account by the evaluation. Unless the consultant has experience with your target population, he might overlook these.
• Is he a numbers person? An evaluator must know how to generate and analyze statistics if he's going to spot subtle changes in your program. Or, he should have access to a qualified statistician.
• Does he have pet theories? You expect a consultant, particularly an experienced one, to have his own ideas about how to approach the evaluation and what should be tested. But if he is reluctant to abandon them in favor of your concerns -- or at least compromise -- he probably won't fit with your organization.
• Does he plan on sticking around? The consultant must be on site enough to observe staff and program operations. This is the only way he can adjust the evaluation design as the program changes.

Top

INTERVIEWING A CONSULTANT:

Questions when interviewing evaluation consultant candidates include:

• What is your motivation for evaluating our program? Asking this helps weed out people wanting to use your project to test hypotheses of personal interest to them.
• Your don't want the consultant adding aspects to the study design that has little to do with your needs.
• Have you ever developed data collection forms for a local program evaluation? If the consultant plans to use measuring instruments from past jobs, ask to see them to make sure they will fit your program, are neat and easy to complete and that you understand them.
• Are you willing to amend the evaluation design after the program is fully implemented? A good consultant will understand the need to change the study plan in midcourse. Consequently, be wary of candidates who insist that changing the assessment will weaken it.
• How experienced are you in using statistical methods? You don't need a statistician, but the consultant should be used to using quantitative data and know a broad array of statistical methods. Also, the evaluator should be able to explain statistical methods to you.
• Will you write the evaluation report? This may seem too basic to ask, but you want to be sure the consultant will write the report and not farm it out to a colleague or assistant. Also, request samples of past reports (not journal articles).
• Are you willing to produce periodic reports throughout the study? Preliminary reports can be valuable to program management and let you know if you need to make modifications. A good consultant will also see the value of interim reports and not mind doing them.

Finally, let the evaluator into your organization. Allow the evaluator to attend meetings to know what's happening. Similarly, have the evaluator spend time with key staff, at least initially. It's also important for him to attend staff brainstorming and policy making sessions. All this breeds familiarity with your organization and can produce more useful assessments. The consultant can't help you unless a trusting and practical relationship develops.

Top

For more information, call the Support Center of Washington, DC at 202/833-0300.The following article appeared in the Families in Crisis Funding Report, July 18, 1994. Prepared by the staff of CD Publications, the article offers some pointers to agencies looking for an evaluation consultant.
Copyright 1994, CD Publications, 8204 Fenton Street, Sliver Spring, Maryland 20910. Used by permission.