Halfway There: Reflections on the TII Midterm Outcomes Report and Evaluating Systems Change

Posted by Beth Siegel on
User Blog Posts Image 89

Early in the formation of The Integration Initiative (TII), Living Cities engaged my company, Mt. Auburn Associates, along with Mathematica Policy Research to help design a national evaluation of TII across all five sites – Baltimore, Cleveland, Detroit, Newark, and the Twin Cities. In designing the evaluation, we started by focusing on the underlying assumption of TII: that it is critical to take a “systems” approach to addressing the challenges low-income people face, such as poor schools, barriers to employment, and poor transportation access to name a few. Living Cities believes that efforts to address such challenges have had limited success due to a tendency to focus on programs and projects, rather than on changing systems. Programs, they believe, should be a means to identifying challenges and solutions that can drive systems change rather than an end in itself.

Operating within complex systems means there is an ever changing set of actors, a large number of overlapping efforts, and other collaborative activities occurring in each site. As a result, the national evaluation team worked with local evaluators to develop a common approach to assessing the contribution to enduring system outcomes rather than the attribution (see figure). In other words, the focus of the evaluation has been on the degree to which the investments of TII contributed to outcomes rather than assessing whether those investments were the cause.

Given the ambitious system change goals of TII, this first outcomes report seeks to answer two overriding questions: 1) Has TII led to system changes and improved outcomes that have broader impacts on low-income individuals; and 2) What can we learn about the design and implementation of TII that influences the type and level of system change and outcomes?

Through baseline analysis, interviews, a Network Survey, site visits, document review, and local evaluator responses to common questions, the report looks at the initial outputs and outcomes associated with the work of the five sites during the first half (or 18 months) of the Initiative (you can access the midterm outcomes report here).

What we’ve learned so far is that Living Cities and the TII sites are on the right track. In fact, in a lot of areas, the work has surpassed where we thought the sites would be according to the Theory of Change. For instance, there has been a high level of network building and strengthening in the first year. In Cleveland, one of the outcomes that has already been institutionalized is a new local procurement policy at University Hospitals, to ensure that UH works with businesses in the area to meet their operational needs, increasing local economic investment in the process. There has also been increased integration across discipline areas and stakeholder groups in all sites, with one of the biggest changes being increased relationships with CDFIs. Specifically, in both Baltimore and Detroit, CDFI capacity has been strengthened by the introduction of The Reinvestment Fund (TRF) and NCB Capital Impact respectively as lenders.

Some of the key success factors producing the positive results come down to leadership. While strengthened networks and governance structures are helping to change the status quo, staffing these tables seems to make a big difference to their success. In addition, it is evident that the project directors in each site are playing increasingly important roles as connectors between the multiple leadership tables and initiatives going on in each site. Living Cities staff has also played an important role in coaching project directors and shaping the governances tables, helping to align work and accelerate progress.

While many gains have been made, there are some areas where the work is not as far along as expected. Deploying financing in particular is the one area where sites are not on track, with some sites closing deals at a slower pace than anticipated. In addition, the complexity of the systems framework has constrained progress and made communication of work difficult.

Despite the challenges to this work, the midterm outcomes report reveals that the five sites have made considerable progress in building a strong civic infrastructure, and making connections across disciplines and stakeholders. Still, it is important to note that this is really just the beginning. While TII was planned as a 3 year initiative, systems change is a much longer process. Given the financial terms alone, the initiative could really be thought of as a 10-year effort at minimum. As the work continues to progress over this longer timeframe, the sites will face many changes – current participants will exit the effort while new ones will come on board. However the work evolves, the national evaluation will continue to be a learning process for Living Cities and the broader field, and can serve as a demonstration on how to begin changing urban systems to benefit low-income people.

Beth Siegel is President of Mt. Auburn Associates