In 2012 Living Cities began to employ a developmental approach to evaluate our work. In the last three years, the work has grown and become pivotal to our organization. This Throwback Thursday, we’re looking back at where we started. Over the course of the next two weeks, we’ll be sharing reflections on our evaluation journey and details about our current thinking.
“Do you thrive in ambiguity?”
The interview up until that moment had been going well, but this question gave me pause. I sat back in my chair, and thought quickly. My initial reaction to the question was that it was an odd one, maybe even antithetical, given that Living Cities was searching for an evaluator to oversee its organizational evaluation and learning efforts.
My second reaction was quite the opposite. When an organization strives for something as complex as Living Cities does—to harness the financial and creative power of cross-sector collaborators to transform systems that are meant to spur economic opportunity, so that they work for all people in an urban area–this is uncharted territory; territory that is by nature, ambiguous.
Each time I have evaluated systems change initiatives, I walk into ambiguity and work within it.
So I responded. “Each time I have evaluated systems change initiatives, I walk into ambiguity and work within it. As an evaluator, however, my role is to bring a new level of clarity and systematic inquiry so that organizations like yours can learn and grow your impact.”
Then I shot back. “Do you think Living Cities is ready for evaluation?”
“Absolutely,” they responded.
Now, four months after that conversation, the Living Cities’ organizational evaluation is underway. I am overseeing this work and we have brought on Fourth Quadrant Partners, LLC, a firm versed in emergent learning principles, to help design and implement the evaluation and to make sure we are embedding evaluative inquiry and practice throughout the organization. We are taking a developmental approach—a relatively new, nontraditional approach to evaluation, but arguably the one best suited for complex systems-change work and a rapid-paced, innovative, organization like ours. Frankly, I am very excited.
“Developmental evaluation“ is a term coined by Michael Quinn Patton, author of, ”Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use,“ the seminal work on this subject. The developmental approach differs from traditional summative and formative approaches in that it is designed for emergent strategies and models rather than ones that are fully “baked”. It is useful for innovative organizations tackling complex social change issues where the paths to success are unclear, nonlinear, and dynamic and where trying to map them from the outset would lead to premature guesswork, inaccuracy, and frustration.
Too often a "one size fits all” approach is pushed by stakeholders who define rigorous approaches very narrowly.
I am excited that the Living Cities Board and Members had the foresight to match the evaluation approach to the complex and dynamic work of Living Cities. In my experience, this does not happen nearly as often as it should. Too often a “one size fits all” approach is pushed by stakeholders who define rigorous approaches very narrowly. This rigid definition often leads to evaluation work that is neither useful nor relevant to strengthening the work of organizations. (For more on the rigor debate, see Lisbeth B. Schorr’s article: “Expanding the Evidence Universe: Doing Better by Knowing More.)”
Developmental evaluation still maintains the fundamentals of strong evaluation—developing evaluation questions, testing hypotheses, collecting data systematically and making judgments from that data. It works, however through embedding these evaluation fundamentals into the day-to-day work of the organization so that real-time learning and rapid course corrections can be incorporated as strategy is developed — hence the name “developmental evaluation”. So as I stated in the title, we have a unique opportunity. I think we have chosen the right evaluation approach, and I believe we are organizationally ready for evaluation. Now the learning begins.
In this evaluation and learning process we WILL make missteps along the way and there WILL be strategies that we find are not working as well as others in our work. But as Bob Giloth and Colin Austin say in their book, “Mistakes to Success: Learning and Adapting when Things go Wrong,” discussing, analyzing and learning from mistakes should be common practice and is often more powerful than discussing successes. We recognize that sharing what we are learning is a large part of what will move our work, and the work of other social innovators, forward.
So, I will regularly be writing and talking openly about what we are learning throughout this process - our successes and failures - both in terms of lessons gleaned about developmental evaluation in practice in a complex organization and about how the results are having implications for our work. This is my first step. What aspects of our evaluation effort would you like to learn more about? I welcome your feedback and comments.
This blog was originally posted on February 28, 2012.