Process Evalution
Contextual factors shape the theory of change and affect the implementation, causal mechanisms, and outcomes of an intervention. Process evaluators should capture how context is affected by an intervention, as well as how contextual factors can change an intervention. The theory of change explains how an intervention is intended to produce the desired effect. It entails making hypothesis about the causal mechanisms by which the components and activities of an intervention will lead to outcomes. A different publication by Durlak et al. examines 500 studies to assess the degree to which the level of implementation affects the outcomes obtained in promotion and prevention programs, as well as identify factors that affect the implementation process.
By their nature, public health problems and the systems in which they are created and shaped are complex [40], and as a result, we might expect to see a more explicit attempt to use complexity concepts to generate evidence on public health interventions. Complexity science introduces a number of additional concepts that may be of value to researchers who seek to evaluate the mechanisms by which public health interventions have impacts in real-world environments. These concepts are used to describe, analyze, measure, and estimate attributes of change. The change first occurs within and across the system elements, and these collective changes result in emergent system change. In 2008 the MRC updated their guidance on the development and evaluation of complex interventions.
Why was this study done?
The purpose of a process evaluation is to explain how an intervention generates outcomes or effects. In health research, process evaluations may be undertaken in conjunction with a randomised controlled trial of a complex intervention. Process evaluations, like other theory-based approaches to evaluation (such as realist evaluation and ‘theory of change’ approaches) investigate the underpinning theory of the intervention. Data collection and analyses in process evaluations are usually structured around logic models which represent this theory and which illustrate the causal pathways thought to be operating in the intervention. The case studies were from low- and middle-income countries and Indigenous communities in Canada. Each of the process evaluations generated findings on whether interventions were implemented with fidelity, the extent of capacity building, contextual factors and the extent to which relationships between researchers and community impacted on intervention implementation.
For example, delivering information on the internet for younger populations but using a leaflet for older participants (fidelity of function). Conducting a meaningful process evaluation requires careful design and study management. The most important considerations during the planning phase of your project evaluation are prioritizing short and long-term goals, identifying your target audience(s), determining methods for collecting data, systematic testing and assessing the feasibility of each for your target audience(s). Lessons learned from an adolescent sexual and reproductive health programme in West AfricaVan Belle, S., Marchal, B., Dubourg, D., Kegels, G. Lessons learned from an adolescent sexual and reproductive health programme in West Africa, BMC Public Health, 10, 741. An outcome evaluation measures a program’s results and determines whether intended outcomes were achieved.
Evidence Matters
This can help to highlight a broad range of system requirements such as new policies and capacity building to support implementation. Process evaluation is crucial in understanding contextual factors that may impact intervention implementation which is important in considering whether or not the intervention can be translated to other contexts. In November 2010 an MRC PHSRN-funded workshop discussed the need for guidance on process evaluation in complex public health intervention studies.
The most important learning was that although process evaluation is time consuming, it enhances understanding of factors affecting implementation of complex interventions. The research highlighted the need to initiate process evaluations early on in the project, to help guide design of the intervention; and the importance of effective communication between researchers responsible for trial implementation, process evaluation and outcome evaluation. We conducted a systematic search to identify examples of public health evaluations that apply a complex systems perspective to process evaluations involving qualitative methods.
The findings of process evaluations are crucial to understand the pathways between intervention and impact, so as to optimize implementation, impact and inform scalability of the interventions. This requires planning from the project outset, and engagement with implementers and decision makers throughout program implementation [1]. This is contrary to the classical set-up of most trials in which deviation from the study protocol is not acceptable because it interferes with the evaluation of effectiveness. The choice about whether to involve implementers in the process evaluation depends on the design of the primary study.
We are currently testing out this approach in an evaluation of how a system and its elements adapt and co-evolve in response to a local alcohol intervention that raises additional revenue to police and manage the night-time economy. We intend that this 2-phase framework can be of use, and be further refined, by public health practitioners and researchers who seek to produce evidence to improve health in complex social settings. Following the workshop, the MRC PHSRN is supporting development of guidance for process evaluation of complex public health interventions, focusing predominantly upon process evaluations within trials or other outcomes evaluations. It will focus upon priorities for process evaluation identified within the MRC framework for complex interventions, in terms of understanding implementation, causal mechanisms and the relationship of interventions with their contexts.
- A common theme that emerged was that while mixed methods approaches can be time consuming and generate a vast amount of data, they significantly enhance understanding of the implementation of complex interventions as well as generate a wealth of learning to inform future projects.
- Qualitative methods can be used to explore processes in more depth, such as participant perceptions of the intervention and how the intervention and its context interact with each other.
- A process evaluation can also explain why an intervention failed and indicate how you might need to redesign it.
- However, relationships between intervention aspects are not necessarily linear and may include feedback loops.
- The first step in the process is to define exactly what you want to find out from the evaluation.
Although these methods may begin with some qualitative work, such as participatory workshops to map a system of interest, their aim is to generate quantitative estimates of future or hypothetical impacts [31]. Compared with quantitative methods, there is little consensus, and less has been written on how to explicitly draw on a complex systems approach for process evaluations that use qualitative methods. The analysis began with an in-depth reading of, and familiarization with, the included studies, with specific attention paid to the ways in which they drew on systems thinking and/or complexity science and the methods utilized to achieve their evaluative aims.
Complex systems have been framed as a kind of antidote to reductionist approaches to health research [5]. Finding ways to bring a complex systems perspective to public health evaluation could, it is hoped, shed new light on how to address public health challenges in a complex world. A complex systems perspective can be applied to many different types of research design and methodology. In this paper, we focus on how such a perspective has been applied to process evaluations that utilize qualitative methods. The remainder of this section elaborates on what is meant by complex systems and process evaluations and discusses why qualitative methods are a particular area of interest for public health evaluators interested in complex systems. ‘Process evaluations’ employ a variety of methods used in the social sciences, normally including both qualitative and quantitative methods.
