Developing richer evaluation methods for community development initiatives
In the mid 1990s, a number of community development initiatives in the USA started to develop and use a new approach to evaluation, born out of a need to understand with more clarity not only whether complex development initiatives were leading to improvements in the communities they targeted, but if they were, how and why they were effective. This tool, championed by the Aspen Institute, is Theory of Change. It adopts and builds on several well-known approaches to evaluation and project management including realist evaluation and implementation frameworks.
Complex health interventions: why Theory of Change may offer greater learning
Public health interventions, like community development initiatives, are characterised by their complexity. Effectiveness trials increasingly adopt approaches integrating multiple interacting components, delivered by often diverse work forces. In mental health, this complexity is increased by recognised challenges, such as desperately inadequate provision of human resources to meet needs, high levels of stigma at both local and institutional levels, and low levels of awareness and support to mental health care. Interventions designed to address multiple challenges are being developed and tested, and theory of change offers an approach to evaluation which may allow deeper understanding of how positive change in mental health care service provision and ultimately, outcomes for people with mental health conditions, their families and their communities, happen.
A roadmap for evaluation and dissemination
Further, Theory of Change offers a roadmap by which to assess and to communicate what works in clear and direct terms to new audiences, throughout the lifetime of the project. It can provide an explanation of a proposed novel intervention approach, the justification for that approach, and the assumptions to be tested. As such, it helps to guide formative research, and allows the project theory to be revised based on the outcomes of this research. The revised theory is then further tested through implementation. A Theory of Change framework determines indicators to measure every outcome expected to be achieved within the lifetime of an intervention. Because of this, project teams can use the Theory of Change to determine which components of the intervention are successful and where the weaknesses lie with the given approach. This 'story' is communicated through the framework itself, as it is redrawn on completion of the intervention to show what worked, how, and to what extent.
Finding a champion
What's important, is finding a way to champion this approach throughout the intervention development, implementation and evaluation. This requires buy in to the Theory of Change approach to evaluation from key stakeholders from an early stage. Also necessary is careful management of the theory of change throughout the project, adapting the framework based on the outcomes of formative research, using it to determine the final approach to implementation, undertaking the evaluation based on this framework, and telling the final story in a compelling way to new audiences.
An individual champion within a project team is a useful start. MHIN is developing tools to support the process, too. These include guidelines on how to achieve stakeholder buy in for the process and how to develop a Theory of Change, templates for reporting the development of the Theory of Change and evaluating progress using Theory of Change, and an animated, narrated guide to developing and testing a Theory of Change.
Theory of Change: moving forward
As with any evaluation approach, Theory of Change isn't perfect. Developing a Theory of Change can be a complicated process, which must be revisited throughout the lifespan of an intervention as new insights are generated through research and practice. Further, more should be done to ensure that the nuances of the local context are captured; testing assumptions may not go far enough. However, given the complexity of mental health interventions and growing efforts to build the evidence base for these interventions, we as a community of innovators can benefit from the standardised, replicable approach that Theory of Change provides us to help structure our formative research, implementation, evaluation and dissemination. We need champions to ensure not only that Theory of Change is taken up, but that it is taken up in the right way - as a flexible tool to be enhanced through further discussion and debate as we uncover its weaknesses and strengths through critical application in the field.
Images courtesy of Valentina Iemmi. Copyright © 2014 Valentina Iemmi. All rights reserved