A deep understanding of the problem and the people it affects
Knowledge of the causes of the problem and its context
A description of what success looks like
A survey of stakeholders and other influences
Your chosen approach to solving the problem
A theory of change for your solution
An understanding of the behaviors that are needed for your theory of change to work
Recognition of how your strategy might go wrong
A set of indicators and targets to assess your progress
The ultimate outcome of your program
Intro
Success means that your ultimate outcome met or exceeded your targets. Or does it?
Earlier you learned about the importance of collecting feedback and assessing progress as you implement your theory of change. Evaluation goes a step further to ask: If the ultimate outcome of your theory of change occurred, did it happen because of your work or unrelated factors?
In this step, you will discover why evaluating impact is critically important for your social enterprise, your beneficiaries, and other stakeholders. You’ll also learn some fundamental concepts—along with basic approaches to evaluation—that will help you hire and communicate with an expert evaluator.
Until now, everything you’ve learned in this module has prepared you to personally engage in developing a strategy for social change from start to finish. In contrast, program evaluation is a highly technical skill that requires considerable statistical knowledge and field experience to do well—which is why you need an outside expert.
Evaluating Your Impact
A pack of wolves howls at the moon to try to make it disappear so that the sun can rise. The moon goes down and the sun comes up, but one wolf asks the key question:
If the howling wolves only measured progress, they would conclude that they achieved their desired outcome of making the sun rise. But the skeptical wolf suggests that their activity had no impact on that outcome. This illustrates the difference between measuring outcomes and assessing impact—which is the task of program evaluation.
Measuring the Impact of a Social Enterprise
Are you wondering how the skeptical wolf’s question applies to a more human-centered program like yours?
Consider a social enterprise that prepares people released from prison for stable jobs. Without the program, an average of 30% of ex-offenders are employed one year after they are released. The organization’s outcome target is that 45% of its clients will be employed.
A year after the program is launched, 45% of its clients are indeed employed. Success? Suppose that severe immigration restrictions were imposed during the year. The businesses where the ex-offenders work were suddenly in desperate need of employees, and would have hired the ex-offenders even without the social enterprise’s job training.
For your program to have impact means that it is causing, or contributing to, an outcome that would not occur—or not occur to as great an extent—without your activities. To use a technical term, the “counterfactual” is what would have happened without your intervention. Impact means that your program has, in effect, beaten the counterfactual.
Causation or Correlation?
Social outcomes inevitably have multiple causes. So your program may be a cause of an outcome, but not necessarily the sole cause. For most programs, it’s more accurate to say that they contributed to some extent to the outcome.
For one event to cause another necessarily means that the events are correlated (or connected). But the fact that events are correlated does not mean that one caused the other.
Imagine that it’s a hot sunny day and you’re at the beach. People are getting sunburns and are eating ice cream—sometimes the same people. Perhaps getting sunburnt causes people to eat ice cream to cool off. Or perhaps eating ice cream causes people to get sunburns—they feel so cool they hang out in the sun. Of course you know that something else is at work. While sunburns and ice cream eating are correlated, both are caused by another variable: the weather.
In this example correlation and causation are easy to distinguish. But in more complex situations—such as the complicated system surrounding the social problem you’re trying to solve—it’s not so easy to determine causation. That’s the goal of program evaluation.
Why You Should Measure Impact
Program evaluation takes time, money, and a lot of work. So why bother? Because your program is consuming human and financial resources—and you, your stakeholders, and your funders will want to know whether it’s actually making a difference in improving people’s lives.
Evaluations of your program will be valuable for future social entrepreneurs considering whether to launch similar programs. In designing your own program, evaluations indicating that similar programs have or have not worked can be an important guide to developing your theory of change. Sound evaluations are the core of what people mean by strategies informed by evidence.
Who Should Measure Impact and When
Determining your program’s impact—whether the program accomplished better results than would have happened without it—is a complicated task. While monitoring the progress of a strategy can be done by an organization’s own staff, impact evaluation typically requires an outside expert. Most program staff don’t have the necessary expertise, and the findings are more credible to stakeholders (including funders) when someone outside your organization evaluates the program.
Measuring impact is an activity you do after your program is operational. The timing of evaluation is different for every program. As a general rule, you can dive into the process when you’ve operated the program long enough at scale to collect adequate data on program outcomes. But it’s a mistake to wait until after you develop and implement a program before planning for evaluation. In most cases, it’s impossible to assemble the necessary data retroactively.
A professional evaluator will use some of the same feedback that you collect to monitor your progress along the theory of change. That’s why it’s so critical to have a good set of monitoring indicators on day one, combined with a reliable system for collecting the data. Feedback is invaluable for the evaluation process as well as for informing your ongoing decision-making.
Basic Evaluation Techniques
Now let’s explore four basic techniques that are used to evaluate a program’s impact.
1
2
3
4
Jordan's Journey
Let’s check-in with Jordan and see what she finds when assessing the impact of her company’s diabetes prevention program.
Statistically Significant or a Matter of Chance?
Even if there is a noticeable difference between the average outcomes of the groups compared in an evaluation, it is possible that the difference is just a matter of chance rather than the result of the program’s activities. Underlying every evaluation technique is the concept of statistical significance. Essentially, statistical significance is a measure of whether it’s appropriate to attribute the difference to the program—or just to chance.
To oversimplify, statistical significance is a function of the size of the sample groups and the size of the difference in outcomes between the two groups. The larger the sample size, the more likely that any observed differences between groups is statistically significant. Similarly, if the size of the difference in outcomes is large, then the difference is more likely to be significant—even if the sample size is not very large.
Assessing your impact is a complicated job, which is why you need an outside expert to help.
Even though you won’t be the one evaluating your program, you still have an important role in the process. Position yourself for a successful evaluation before you launch. Lay the groundwork for a robust plan to collect feedback and monitor your progress. And when the time is right, devote the resources necessary to hire a professional evaluator.
You want to change the world. At some point, you’ll need to learn whether your program’s impact is significant enough that others can build on your experience and funders will want to continue to support your work.