As we’ve explored in our last two posts, Pay for Success (PFS), an innovative model to scale social programs, requires that project investors investigate the “4Ps”—Partnerships, Program, Policy and Process—to evaluate the likelihood that the project will succeed, meet social impact targets and, in turn, payback investors. Today we explore the core outcome of a PFS transaction, whether or not a “Program” can produce and measure impact. Since the repayment of a PFS investment is ultimately dependent on the program’s ability to deliver successful outcomes, the evaluation of the program is probably the most critical aspect of a PFS project.
We examine the three components of the Program—the intervention, the implementation and evaluation plan:
The intervention refers to the set of supportive services and/or improved social outcomes that the target population will receive through the PFS investment, such as higher levels of employment, fewer days of incarceration, or improved graduation rates. These services and outcomes determine the ultimate repayment (or not) of the investment. Using research, evaluation studies, or hiring a third-party program expert, can help an investor understand the intervention’s ability to produce the results needed to get repaid. Some questions that helped guide us: What does evidence-based research say about the success of the intervention in the past? Has the intervention achieved the desired outcomes in a similar target population before? How long did it take to achieve these effects? Are there differences between the evaluated intervention and the proposed intervention in the Pay for Success investment? What are these differences and how do they make the intervention more or less likely to achieve the targeted social outcomes?
Once the intervention is evaluated, the next step is to understand how it will be implemented. A well designed intervention cannot produce social impact if the provider does a poor job of running the program or does not anticipate implementation issues. In our work, we looked closely at who would be implementing the services. We considered whether there was a single provider or multiple. And if it was a team effort, had any of the providers worked together before? We looked at the providers’ proven track record in delivering the specific intervention or similar programs. To further evaluate each provider we investigated: Has the provider(s) been evaluated by a third party? Does the provider(s) have a data driven culture and use data to change their service provision to focus on what works? If there are multiple providers, have they shared data? Will the intervention require significant scaling up of staff or other resources? Does the organization have experience scaling up programs successfully? In the PFS contract, are the incentives of the provider(s) aligned with producing the targeted outcomes needed to repay investors? Digging deeply into the capacity and incentive structure of the PFS service provider(s) is critical to evaluating the credit worthiness of the transaction.
3. Evaluation Plan
In PFS transactions, investments will be repaid if, and when social impacts are achieved and verified. Without an evaluation plan that is methodologically sound and robust enough to survive the various twists and turns of a project, investors and government may not be able to agree on the extent to which impacts occurred as a direct result of the intervention and the implementation rather than other factors like national growth in employment. Basically, sound program evaluation is how the government knows whether or not it is obligated to repay the investors. A few key questions we explored in our work: What is the evaluation plan to measure the outcomes? Are there adequate data systems in place to measure and track the outcomes? Will the outcomes of the treatment group be compared to a control group, to history, or to something else? Can we distinguish effects of the program from other environmental factors? Will environmental factors out of the control of the service provider(s) impact the outcome payments? Do we have a sufficient sample size for the chosen evaluation method to detect real effects and provide sufficient statistical power? What might prevent meeting the sample size or other evaluation requirements? Is there a back-up evaluation plan proposed or a process agreed-upon to develop an alternative evaluation? Will the evaluation plan be fully financed at the closing of the Pay for Success transaction? Ideally, at closing, a PFS transaction should have the evaluation plan set and fully funded with, at minimum, pre-agreed process to develop a backup plan if there are issues collecting enough program participants, data, or other requirements of the evaluation plan.
The ability of the program to produce the outcomes needed for repayment is probably the most critical aspect of any PFS transaction. Foundations are in an ideal position to bring program expertise of multiple issue areas and financial expertise together for the underwriting of PFS transactions. Many other investors may need to “buy” that program expertise. And, if an investor does need to buy the expertise, it is money well spent and may lead to valuable partnerships that will help the PFS project’s performance.
Next week we’ll dig into Policy. How do investors begin to evaluate policy risk? Stay tuned and sign-up to learn more.
Don’t miss the first two blogs introducing the 4 Ps of PFS and P for Partnership. We encourage you to keep the conversation running in the comments section. Thanks to all that have commented to open up the dialogue and share your insights.