Evaluation can help answer the question: How do you change a system?

This Throwback Thursday, as we think about how our approach to evaluation and systems change has evolved in the last year, we’re looking back at three critical questions that can be helpful for any organization to consider before taking the leap into developmental evaluation. This post originally appeared on October 17, 2013.

As I write about a lot on this blog, Living Cities believes that we need to transform systems that currently produce opportunity for some into systems that produce opportunity for all. Which begs the question: how do you change a system?

I wish I could say that we know the answer. But we don’t. Because systems are complex - they are made up of the interactions between an array of actors and institutions, policies and practices that produce a specific result. But, through all our work, we are trying to figure it out.

Around the same time we began to focus in on systems change, we also began using developmental evaluation, an approach focused on collecting and using data strategically to assist in the development of the work as it is emerging. More than two years into this work, we have gleaned many lessons. And with the American Evaluation Association’s Annual Conference taking place this week in Washington, DC, we thought it was a good time to share three questions for organizations to consider before taking the leap into developmental evaluation.

Different ways to approach and understand evaluation.

  1. Is developmental evaluation the right approach for the work you are doing? Early on, we were trying to apply developmental evaluation to all our work at Living Cities, but we quickly learned that a lot of our work was better-suited to formative or traditional evaluation approaches. But, developmental evaluation was the right fit for the emergent work we were doing, the work where we could envision where we wanted to get to, but didn’t know how to get there.

For instance, the conventional wisdom in community development finance is that the biggest challenge in making investments that will benefit low-income people is the limited supply of capital. Accepting this wisdom, we created a supply of flexible debt to be issued to the five sites participating in The Integration Initiative. But, closing these investments proved more difficult than we anticipated, and revealed other big issues around the ability of communities to accept and deploy capital, and the financial products available in the marketplace. This experience gave birth to a new and fast-evolving body of work called capital absorption which is trying to figure out what it takes for communities to be able to make effective use of different forms of capital to provide needed goods and services to low-income residents. Developmental evaluation was embedded in the process from the get go, helping the team identify their hypotheses and reflect on what they were learning from the feedback on each research report, workshop, presentation and blog post to evolve their thinking and the strategy.

  1. Is your organization prepared to make the most of developmental evaluation? Traditionally, evaluation has been viewed as a tool for accountability, with a focus on identifying and communicating inputs and outputs and outcomes, often so that funders and board members know where their investment is going. However, in developmental evaluation, the measure of progress is learning. For a long time, we at Living Cities struggled with trying to reconcile the concepts of “accountability” and “learning” in our evaluation work. And as a result, we weren’t taking full advantage of all we were learning through developmental evaluation.

When I asked our CEO, Ben Hecht about this, he shared this great insight into how our view of this has changed and why learning and accountability aren’t a dichotomy, “As a philanthropic investor, generally speaking, you want to know that your money was well spent. There are three forms of accountability. One is financial accountability. Did you spend the money the way you said you were going to spend it. The second one is learning accountability. Did you use the resources to learn, share with the field, and can you show that you actually did have influence in the field? And then the third accountability is did the lives of people change?”

  1. Is your staff prepared to incorporate developmental evaluation? The status quo is that practitioners plan their effort and implement it, and evaluation comes into play afterwards to measure if it was effective or not. As a result, staff members often have limited understanding of evaluation and may even feel like it is a tool for judgment.
    To use developmental evaluation, this relationship between staff and evaluation has to change dramatically. As Living Cities’ former evaluation director, Kathy Brennan, describes it, the developmental approach embeds “evaluation fundamentals into the day-to-day work of the organization so that real-time learning and rapid course corrections can be incorporated as strategy is developed.”

We know from our own experience at Living Cities that this shift is a huge one and requires staff to be taught and supported to 1) articulate the goals of the work, and their hypotheses about what it will take to achieve them; 2) design their approaches in a way that they can use developmental evaluation data to improve their strategies as they are implementing them, and 3) build in time to reflect on what they are learning by doing their work.

I’m excited to learn from others about your experiences preparing your organization for and using developmental evaluation. And if you happen to be in Washington, DC for the American Evaluation Association Conference, join Kathy Brennan and me for the session Making it Real: Moving Developmental Evaluation From Theory to Practice on Friday, October 18, 2013 from 4:30-6 pm.

To learn more about what distinguishes developmental evaluation from more traditional evaluation, check out Evaluating Social Innovation.

Image Source: Collective Impact: Embracing Emergence Presented by John Kania, Blair Taylor, & Mark Cabaj on May 01, 2013.